Method of encoding a video signal based on inter prediction processing and data transmission method

Abstract

A method and a data transmission method for processing and encoding video signals based on inter-frame prediction. Specifically, the method for processing video signals based on inter-frame prediction includes: configuring a merge list based on neighboring blocks of the current block; if the number of merge candidates included in the merge list is less than a preset first predetermined number, adding historical merge candidates from a historical merge candidate list to the merge list, where historical merge candidates represent motion information of blocks previously encoded for the current block; obtaining a merge index indicating merge candidates from the merge list to be used for inter-frame prediction of the current block; and generating a prediction block for the current block based on the motion information of the merge candidates indicated by the merge index, wherein adding historical merge candidates to the merge list may include: checking whether a preset second predetermined number of historical merge candidates in the historical merge candidate list have motion information overlapping with the motion information of merge candidates included in the merge list.

Description

[0001] This application is a divisional application of the original invention patent application No. 201980016043.X (International Application No.: PCT / KR2019 / 017243, Application Date: December 6, 2019, Invention Title: Method and Apparatus for Processing Video Signals Based on Inter-Frame Prediction). Technical Field

[0002] Embodiments of this disclosure relate to methods and apparatus for processing video signals based on inter-frame prediction, and more specifically, to methods and apparatus for performing inter-frame prediction using historical motion vector prediction. Background Technology

[0003] Compression coding refers to a series of signal processing techniques used to transmit digital information over communication lines or to store information in a form suitable for storage media. Media including pictures, images, and audio can be targets of compression coding, and specifically, the techniques used to perform compression coding on pictures are called video image compression.

[0004] Next-generation video content should feature high spatial resolution, high frame rates, and large-scale scene rendering. Processing this content will necessitate a dramatic increase in memory storage, memory access speeds, and processing power.

[0005] Therefore, there is a need to design encoding tools for efficiently processing next-generation video content. Summary of the Invention

[0006] Technical issues

[0007] The purpose of this disclosure is to provide constraints on redundancy checks performed in order to add HMVP candidates to the merge list (or AMVP list).

[0008] The technical objectives to be achieved in the embodiments of this disclosure are not limited to those mentioned above, and other technical objectives not described above can be clearly understood by those skilled in the art to which this disclosure pertains, based on the following description.

[0009] Technical solution

[0010] In one aspect of the embodiments of this disclosure, a method for processing a video signal based on inter-frame prediction may include the following steps: configuring a merge list based on neighboring blocks of the current block; adding historically based merge candidates included in a historically based merge candidate list to the merge list when the number of merge candidates included in the merge list is less than a first predetermined number; obtaining a merge index indicating merge candidates in the merge list for inter-frame prediction of the current block; and generating a prediction block of the current block based on motion information of the merge candidates indicated by the merge index, wherein the step of adding the historically based merge candidates to the merge list includes the following steps: checking whether a second predetermined number of historically based merge candidates in the historically based merge candidate list have the same motion information as the merge candidates included in the merge list.

[0011] In an implementation, when the second predetermined number of history-based merge candidates have motion information different from that of the predefined merge candidates included in the merge list, the second predetermined number of history-based merge candidates can be added to the merge list.

[0012] In an implementation, when the second predetermined number of history-based merge candidates have different motion information than the third predetermined number of predefined merge candidates included in the merge list, the second predetermined number of history-based merge candidates can be added to the merge list.

[0013] In an implementation, when the second predetermined number of history-based merging candidates have motion information different from the predefined spatial merging candidates included in the merging list, the second predetermined number of history-based merging candidates can be added to the merging list.

[0014] In an implementation, the first predetermined number may be defined as the value obtained by subtracting 1 from the maximum number of merged candidates.

[0015] In an implementation, the step of adding a history-based merge candidate may include the following steps: when the merge list includes three current merge candidates, check whether two history-based merge candidates have the same motion information as the merge candidates included in the merge list.

[0016] In another aspect of the implementation, an apparatus for processing video signals based on inter-frame prediction may include: a memory configured to store video signals; and a processor incorporated into the memory, wherein the processor is configured to: configure a merge list based on neighboring blocks of the current block; add historically based merge candidates included in a historically based merge candidate list to the merge list when the number of merge candidates included in the merge list is less than a first predetermined number; obtain a merge index indicating merge candidates in the merge list for inter-frame prediction of the current block; and generate a prediction block for the current block based on motion information of the merge candidates indicated by the merge index, wherein the processor is further configured to check whether a second predetermined number of historically based merge candidates in the historically based merge candidate list have the same motion information as the merge candidates included in the merge list.

[0017] Technical effect

[0018] According to embodiments of this disclosure, by limiting the redundancy checks performed for addition to the merge list (or AMVP list), the complexity of redundancy checks can be improved and efficiency can be increased.

[0019] The effects achievable in this disclosure are not limited to those mentioned above, and other technical effects not described above will be clearly understood by those skilled in the art to which this disclosure pertains, based on the following description. Attached Figure Description

[0020] The accompanying drawings, which are included as part of the detailed description in order to aid in understanding this disclosure, provide embodiments of the disclosure and together with the detailed description, describe the technical features of the disclosure.

[0021] Figure 1 An example of the functional configuration of an encoder, which is an example of a video signal processing device according to an embodiment of the present disclosure, is illustrated.

[0022] Figure 2 This is a schematic block diagram of an embodiment of the present disclosure, and of a decoding device in which video / image signals are decoded.

[0023] Figure 3 This is an implementation of the present disclosure and is a view showing examples of multiple types of tree structures.

[0024] Figure 4 This is an illustration of a signaling mechanism for segmenting information of a quadtree structure with nested multi-type trees, which can be implemented using the present disclosure.

[0025] Figure 5The present disclosure provides an embodiment that can be applied, and is illustrated in the diagrams showing a method for dividing a CTU into multiple CUs based on a quadtree and nested multi-type tree structures.

[0026] Figure 6 The embodiments of this disclosure can be applied, and the diagram illustrates a method for limiting ternary tree segmentation.

[0027] Figure 7 The embodiments of this disclosure can be applied, and the diagram illustrates redundant segmentation patterns that may occur in binary tree segmentation and ternary tree segmentation.

[0028] Figure 8 and Figure 9 These are illustrations of a video / image coding method based on inter-frame prediction according to embodiments of the present disclosure and an inter-frame prediction unit in a coding apparatus according to embodiments of the present disclosure.

[0029] Figure 10 and Figure 11 These are, respectively, the video / image decoding process based on inter-frame prediction according to embodiments of the present disclosure and the inter-frame predictor within the decoding device.

[0030] Figure 12 This is an implementation of the present disclosure and is a view used to illustrate adjacent blocks used in merge mode or jump mode.

[0031] Figure 13 This is a flowchart illustrating a method for configuring and merging candidate lists according to an embodiment of the present disclosure.

[0032] Figure 14 This is a flowchart illustrating a method for constructing a merged candidate list according to an embodiment of the present disclosure.

[0033] Figure 15 An example of a motion model according to an embodiment of the present disclosure is illustrated.

[0034] Figure 16 An example of a control point motion vector for affine motion prediction according to an embodiment of the present disclosure is illustrated.

[0035] Figure 17 An example is shown where the motion vectors of each sub-block of a block have been applied according to an embodiment of the present disclosure.

[0036] Figure 18 An example of a neighboring block for affine motion prediction in affine merging mode according to an embodiment of the present disclosure is illustrated.

[0037] Figure 19An example is illustrated of a block to which affine motion prediction has been performed using neighboring blocks that have already been applied with affine motion prediction according to an embodiment of this disclosure.

[0038] Figure 20 This is a diagram illustrating a method for generating a merge candidate list using neighboring affine coding blocks according to embodiments of the present disclosure.

[0039] Figure 21 and Figure 22 This is a diagram illustrating a method for constructing an affine merging candidate list using neighboring blocks encoded by affine prediction, according to embodiments of the present disclosure.

[0040] Figure 23 An example of a neighboring block for affine motion prediction in affine inter-frame mode according to an embodiment of the present disclosure is illustrated.

[0041] Figure 24 An example of a neighboring block for affine motion prediction in affine inter-frame mode according to an embodiment of the present disclosure is illustrated.

[0042] Figure 25 and Figure 26 This is an illustration of a method for deriving motion vector candidates using motion information of neighboring blocks in an affine inter-frame mode according to an embodiment of the present disclosure.

[0043] Figure 27 An example of an affine motion vector field of a sub-block unit according to an embodiment of the present disclosure is illustrated.

[0044] Figure 28 An exemplary method and motion vector for generating prediction blocks under inter-frame prediction with an affine motion model applied are shown, according to embodiments of the present disclosure.

[0045] Figure 29 This is a diagram illustrating a method for performing motion compensation based on motion vectors of control points according to an embodiment of the present disclosure.

[0046] Figure 30 This is a diagram illustrating a method for performing motion compensation based on the motion vector of control points in a non-square block according to an embodiment of the present disclosure.

[0047] Figure 31 This is a diagram illustrating a method for performing motion compensation based on the motion vector of control points in a non-square block according to an embodiment of the present disclosure.

[0048] Figures 32 to 38 This is a diagram illustrating a method for performing motion compensation based on the motion vector of control points in a non-square block according to an embodiment of the present disclosure.

[0049] Figure 39 This is a flowchart describing a method for storing an HMVP according to an embodiment of the present disclosure.

[0050] Figure 40 This is a diagram illustrating an HMVP table operating in an unconstrained FIFO mode according to an embodiment of the present disclosure.

[0051] Figure 41 This is a diagram illustrating an HMVP table operating in constrained FIFO mode according to an embodiment of the present disclosure.

[0052] Figure 42 This is a diagram illustrating an HMVP LUT and a long-term HMVP LUT according to an embodiment of the present disclosure.

[0053] Figure 43 This is a diagram illustrating an example of a method for updating an HMVP LUT according to an embodiment of the present disclosure.

[0054] Figure 44 This is a diagram illustrating a method for limiting the number of HMVP candidates (i.e., the target of pruning checks) according to an embodiment of this disclosure.

[0055] Figure 45 This is a flowchart illustrating an example of a method for performing a trimming check according to an embodiment of the present disclosure.

[0056] Figure 46 This is a diagram illustrating a method for generating H-STMVP candidates using motion vectors from different reference images according to embodiments of this disclosure.

[0057] Figure 47 This is a diagram illustrating the location of a block for deriving an inherited affine HMVP candidate according to an embodiment of the present disclosure.

[0058] Figure 48 This is a diagram illustrating an affine merge list or affine AMVP list according to an embodiment of the present disclosure.

[0059] Figure 49 This is a flowchart illustrating a method for processing video signals based on inter-frame prediction according to an embodiment of the present disclosure.

[0060] Figure 50 An example block diagram of an apparatus for processing video signals according to an embodiment of the present disclosure is shown.

[0061] Figure 51 An example of a video coding system using this disclosure is provided.

[0062] Figure 52A structural diagram of a content streaming system is shown.

[0063] Figure 53 This is a schematic diagram illustrating an example of a service system that includes digital devices.

[0064] Figure 54 This is a block diagram illustrating a digital device according to an embodiment.

[0065] Figure 55 This is a configuration block diagram illustrating another implementation of a digital device.

[0066] Figure 56 This is a block diagram illustrating a digital device according to another embodiment.

[0067] Figure 57 This is an example Figures 54 to 56 A block diagram showing the detailed configuration of the control unit.

[0068] Figure 58 This is an illustration of an example in which the screen of a digital device simultaneously displays a main image and a secondary image according to an embodiment. Detailed Implementation

[0069] Preferred embodiments of the present disclosure will be described below with reference to the accompanying drawings. The description below, using the drawings, is intended to illustrate exemplary embodiments of the present disclosure and is not intended to describe the only embodiments that can implement the present disclosure. Specific details are included in the following description to provide a perfect understanding of the present disclosure. However, those skilled in the art will understand that the present disclosure can be practiced without these specific details. In some cases, to prevent ambiguity regarding the technical concept of the present disclosure, well-known structures or devices may be omitted, or these structures or devices may be depicted as block diagrams focusing on the core functions of the structures or devices.

[0070] In some cases, to prevent the technical concept of this disclosure from being unclear, well-known structures or devices may be omitted, or these structures or devices may be depicted as block diagrams focusing on the core functions of the structures or devices.

[0071] Furthermore, while widely used general terms are selected as much as possible as possible for use in this disclosure, terms chosen arbitrarily by the applicant are used in specific cases. Since the meaning of such terms will be clearly described in the corresponding section of the specification in such cases, it should be understood that this disclosure is not to be interpreted solely by the terms used only in the specification of this disclosure, but rather that the meaning of the terms should be clarified.

[0072] Specific terminology may be provided for use in the description below to aid in understanding this disclosure. Furthermore, these specific terminology may be modified to other forms within the scope of the technical concept of this disclosure. For example, signals, data, samples, pictures, slices, tiles, frames, blocks, etc., may be appropriately substituted and interpreted in each encoding process.

[0073] This document relates to video / image coding. For example, the methods / implementations disclosed in this document can be applied to methods disclosed in the Universal Video Coding (VVC) standard or the next-generation video / image coding standard.

[0074] In this document, an image generally refers to a unit representing an image within a specific time period, and a slice / tile is a unit that constitutes part of an image in the coding process. A slice / tile may include one or more coding tree units (CTUs). An image can be composed of multiple slices / tiles.

[0075] A pixel, or image unit, can refer to the smallest unit that makes up a picture (or image). A "sample" can also be used as the term corresponding to a pixel. A sample can typically represent a pixel or pixel value, and can represent only the pixel / pixel value of the luminance component, or only the pixel / pixel value of the chrominance component.

[0076] A unit can represent a basic unit of image processing. A unit may include a specific region of an image and at least one of the information associated with that region. In some cases, the term "unit" may be used interchangeably with terms such as "block" or "region." In common cases, an M×N block may represent a group (or array) of samples or transform coefficients consisting of M columns and N rows.

[0077] In this document, " / " and "," are interpreted as "and / or". For example, "A / B" is interpreted as "A and / or B", and "A, B" is interpreted as "A and / or B". Additionally, "A / B / C" means "at least one of A, B, and / or C". Furthermore, "A, B, and C" means "at least one of A, B, and / or C". (In this document, the terms " / " and "," should be interpreted as indicating "and / or". For example, the expression "A / B" can mean "A and / or B". Additionally, "A, B" can mean "A and / or B". Additionally, "A / B / C" can mean "at least one of A, B, and / or C". Additionally, "A, B, C" can mean "at least one of A, B, and / or C".)

[0078] Furthermore, the word "or" in this document will be interpreted as "and / or". For example, "A or B" can mean 1) only "A", 2) only "B", or 3) "A and B". In other words, "or" in this document can mean "additionally or alternatively". (Additionally, in this document, the term "or" should be interpreted as indicating "and / or". For example, the expression "A or B" can include 1) only A, 2) only B, and / or 3) both A and B. In other words, the term "or" in this document should be interpreted as indicating "additionally or alternatively".)

[0079] In the following description, "processing unit" means a unit that performs encoding / decoding processes such as prediction, transformation, and / or quantization. For ease of description, a processing unit may be referred to as a "processing block" or "block".

[0080] A processing unit can be interpreted as having the meaning of having units for the luminance component and units for the chrominance component. For example, a processing unit can correspond to a coding tree unit (CTU), a coding unit (CU), a prediction unit (PU), or a transform unit (TU).

[0081] Furthermore, the processing unit can be interpreted as a unit for the luma component or a unit for the chroma component. For example, the processing unit may correspond to a coding tree block (CTB), coding block (CB), prediction block (PB), or transform block (TB) for the luma component. Alternatively, the processing unit may correspond to a coding tree block (CTB), coding block (CB), prediction block (PB), or transform block (TB) for the chroma component. Moreover, this disclosure is not limited thereto, and the processing unit can be interpreted to include the meaning of both units for the luma component and units for the chroma component.

[0082] Furthermore, the processing unit is not inherently limited to square blocks and can be constructed in the form of a polygon with three or more vertices.

[0083] Furthermore, in the following text, pixels, etc., are generally referred to as samples. Additionally, using a sample can mean using pixel values.

[0084] Figure 1 An example of the functional configuration of an encoder, which is an example of a video signal processing device according to an embodiment of the present disclosure, is illustrated.

[0085] Reference Figure 1The encoding device 100 can be configured to include an image divider 110, a subtractor 115, a transformer 120, a quantizer 130, an inverse quantizer 140, an inverse transformer 150, an adder 155, a filter 160, a memory 170, an inter-frame predictor 180, an intra-frame predictor 185, and an entropy encoder 190. The inter-frame predictor 180 and the intra-frame predictor 185 are often referred to as predictors. In other words, a predictor can include an inter-frame predictor 180 and an intra-frame predictor 185. The transformer 120, quantizer 130, inverse quantizer 140, and inverse transformer 150 can be included in a residual processor. The residual processor can also include a subtractor 115. In one implementation, the image divider 110, subtractor 115, transformer 120, quantizer 130, dequantizer 140, inverse transformer 150, adder 155, filter 160, inter-frame predictor 180, intra-frame predictor 185, and entropy encoder 190 can be configured as a single hardware component (e.g., an encoder or processor). Furthermore, memory 170 may include a decoded picture buffer (DPB) and may be configured with a digital storage medium.

[0086] Image segmenter 110 divides the input image (or picture or frame) input to encoding device 100 into one or more processing units. For example, a processing unit may be referred to as a coding unit (CU). In this case, coding units can be recursively segmented from coding tree units (CTUs) or maximum coding units (LCUs) based on a quadtree-binary tree (QTBT) structure. For example, a coding unit can be segmented into multiple deeper coding units based on a quadtree structure and / or a binary tree structure. In this case, for example, a quadtree structure can be applied first, followed by a binary tree structure. Alternatively, a binary tree structure can be applied first. The encoding process according to this disclosure can be performed based on the final coding units that are no longer segmented. In this case, the maximum coding unit can be directly used as the final coding unit based on encoding efficiency according to image characteristics, or, if necessary, the coding unit can be recursively segmented into deeper coding units. Thus, a coding unit with an optimal size can be used as the final coding unit. In this case, the encoding process may include processes such as prediction, transformation, or reconstruction, which will be described subsequently. In another example, the processing unit may also include a prediction unit (PU) or a transform unit (TU). In this case, each of the prediction unit and the transform unit can be partitioned or segmented from each final coding unit. The prediction unit may be a unit that predicts samples, and the transform unit may be a unit that derives transform coefficients and / or derives the residual signal from the transform coefficients.

[0087] Units can be used interchangeably with blocks or regions, depending on the situation. In common cases, an M×N block can indicate a set of samples or a set of transform coefficients comprising M columns and N rows. Typically, a sample can indicate a pixel or pixel value, and can indicate only the pixel / pixel value of the luminance component or only the pixel / pixel value of the chrominance component. Within a sample, a picture (or image) can be used as the term corresponding to a pixel or graphic element.

[0088] Encoding device 100 can generate a residual signal (residual block or residual sample array) by subtracting the prediction signal (prediction block or prediction sample array) output by inter-frame predictor 180 or intra-frame predictor 185 from the input image signal (original block or original sample array). The generated residual signal is sent to converter 120. In this case, as illustrated, the unit within the encoder that subtracts the prediction signal (prediction block or prediction sample array) from the input image signal (original block or original sample array) can be referred to as subtractor 115. The predictor can perform prediction on the processing target block (hereinafter referred to as the current block) and can generate a prediction block including prediction samples for the current block. The predictor can determine whether to apply intra-frame prediction or inter-frame prediction on the basis of the current block or CU. The predictor can generate various information about the prediction, such as prediction mode information, as described below for each prediction mode, and can send said information to entropy encoder 190. The information about the prediction can be encoded in entropy encoder 190 and can be output in the form of a bitstream.

[0089] The intra-frame predictor 185 can refer to samples within the current image to predict the current block. The referenced samples can be located as neighbors of the current block, or spaced apart from the current block depending on the prediction mode. In intra-frame prediction, the prediction mode can include multiple non-angular modes and multiple angular modes. For example, non-angular modes can include DC mode and planar mode. For example, depending on the fineness of the prediction direction, angular modes can include 33 or 65 angular prediction modes. In this case, for example, depending on the configuration, more or fewer of these 33 or 65 angular prediction modes can be used. The intra-frame predictor 185 can use the prediction modes applied to neighboring blocks to determine the prediction mode applied to the current block.

[0090] Inter-frame predictor 180 can derive the predicted block of the current block based on a reference block (reference sample array) specified by motion vectors on a reference image. In this case, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted on a block, sub-block, or sample basis based on the correlation between motion information between neighboring blocks and the current block. Motion information may include motion vectors and reference image indices. Motion information may also include inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction) information. In the case of inter-frame prediction, neighboring blocks may include spatially neighboring blocks within the current image and temporally neighboring blocks within the reference image. The reference image including the reference block and the reference image including the temporally neighboring block may be the same or different. The temporally neighboring block may be represented by a name called a co-localization reference block or co-localization CU (colCU). The reference image including the temporally neighboring block may be called a co-localization image (colPic). For example, inter-frame predictor 180 can construct a motion information candidate list based on neighboring blocks and can generate information indicating which candidate is used to derive the motion vector and / or reference image index of the current block. Inter-frame prediction can be performed based on various prediction modes. For example, in jump mode and merge mode, the inter-frame predictor 180 can use motion information from neighboring blocks as motion information for the current block. In jump mode, unlike merge mode, residual signals cannot be sent. In Motion Information Prediction (MVP) mode, motion vectors from neighboring blocks can be used as motion vector prediction terms. The motion vector of the current block can be indicated by signaling the motion vector difference.

[0091] The predicted signal generated by the inter-frame predictor 180 or the intra-frame predictor 185 can be used to generate the reconstructed signal or the residual signal.

[0092] Transformer 120 can generate transform coefficients by applying a transform scheme to the residual signal. For example, the transform scheme may include at least one of Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), Karhunen-Loève Transform (KLT), Graphical Based Transform (GBT), or Conditional Nonlinear Transform (CNT). In this case, GBT refers to a transform obtained from a graph when representing the relationship information between pixels as a graph. CNT refers to a transform obtained based on a prediction signal generated using all previously reconstructed pixels. Furthermore, the transform process can be applied to square blocks of pixels of the same size, or it can be applied to blocks of variable size that are not square.

[0093] Quantizer 130 quantizes the transform coefficients and transmits them to entropy encoder 190. Entropy encoder 190 encodes the quantized signal (information about the quantized transform coefficients) and outputs it as a bitstream. This information about the quantized transform coefficients can be referred to as residual information. Quantizer 130 can rearrange the block-form quantized transform coefficients into a one-dimensional vector based on the coefficient scan order, and can generate information about the quantized transform coefficients based on this one-dimensional vector form. Entropy encoder 190 can perform various encoding methods such as exponential Golomb, context-adaptive variable-length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). In addition to the quantized transform coefficients, entropy encoder 190 can also encode information required for video / image reconstruction (e.g., values ​​of syntax elements) together or separately. The encoded information (e.g., encoded video / image information) can be transmitted or stored as a bitstream at network abstraction layers (NALs). The bitstream can be transmitted over a network or stored in a digital storage medium. In this configuration, the network may include a broadcast network and / or a communication network. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. The transmitter (not shown) that sends the signal output by the entropy encoder 190 and / or the storage device (not shown) that stores the signal may be configured as an internal / external element of the encoding device 100, or the transmitter may be an element of the entropy encoder 190.

[0094] The quantized transform coefficients output by quantizer 130 can be used to generate a prediction signal. For example, the residual signal can be reconstructed by applying inverse quantization and inverse transform to the quantized transform coefficients using inverse quantizer 140 and inverse transform 150 within the loop. Adder 155 can add the reconstructed residual signal to the prediction signal output by inter-frame predictor 180 or intra-frame predictor 185, thus generating a reconstructed signal (reconstructed image, reconstructed block, or reconstructed sample array). If there is no residual for processing the target block, as in the case where a jump mode has been applied, the prediction block can be used as a reconstructed block. Adder 155 can be referred to as a reconstructor or reconstructed block generator. The generated reconstructed signal can be used for intra-frame prediction of the next processing target block within the current image, and can be used for inter-frame prediction of the next image by filtering, as will be described later.

[0095] Filter 160 can improve the quality of subjective / objective images by applying filtering to the reconstructed signal. For example, filter 160 can generate a modified reconstructed image by applying various filtering methods to the reconstructed image. The modified reconstructed image can be stored in memory 170, specifically in the DPB of memory 170. Various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filtering, and bilateral filtering. Filter 160 can generate various information for filtering as described below in the description of each filtering method, and can send them to entropy encoder 190. The filtered information can be encoded by entropy encoder 190 and output as a bitstream.

[0096] The modified reconstructed image sent to memory 170 can be used as a reference image in inter-frame predictor 180. If inter-frame prediction is applied, the encoding device can avoid prediction mismatch between encoding device 100 and decoding device and improve encoding efficiency.

[0097] The DPB of memory 170 can store modified reconstructed images so that they can be used as reference images in inter-frame predictor 180. Memory 170 can store motion information of blocks from which motion information within the current image has been derived (or encoded) and / or motion information of blocks within reconstructed images. The stored motion information can be sent to inter-frame predictor 180 to be used as motion information for spatially or temporally neighboring blocks. Memory 170 can store reconstructed samples of reconstructed blocks within the current image and can send them to intra-frame predictor 185.

[0098] Figure 2 This is a schematic block diagram of an embodiment of the present disclosure, and of a decoding device in which video / image signals are decoded.

[0099] Reference Figure 2The decoding device 200 can be configured to include an entropy decoder 210, an inverse quantizer 220, an inverse transformer 230, an adder 235, a filter 240, a memory 250, an inter-frame predictor 261, and an intra-frame predictor 262. The inter-frame predictor 261 and the intra-frame predictor 262 can be collectively referred to as predictor 260. That is, the predictor can include inter-frame predictor 180 and intra-frame predictor 185. The inverse quantizer 220 and the inverse transformer 230 can be collectively referred to as a residual processor. That is, the residual processor can include inverse quantizer 220 and inverse transformer 230. The entropy decoder 210, inverse quantizer 220, inverse transformer 230, adder 235, filter 240, inter-frame predictor 261, and intra-frame predictor 262 can be configured as a single hardware component (e.g., a decoder or a processor) according to the implementation. Furthermore, the memory 170 can include a decoded picture buffer (DPB) and can be configured with a digital storage medium.

[0100] When the input includes a bitstream containing video / image information, the decoding device 200 can proceed according to... Figure 2 The process of processing video / image information in an encoding device reconstructs the image. For example, the decoding device 200 can perform decoding using a processing unit applied in the encoding device. Therefore, the processing unit used for decoding can be, for example, an encoding unit. Depending on the quadtree structure and / or binary tree structure, encoding units can be segmented from encoding tree units or maximal encoding units. Furthermore, the reconstructed image signal decoded and output by the decoding device 200 can be played back by a playback device.

[0101] Decoding device 200 can receive from Figure 1The signal is a bitstream output by the encoding device. The received signal can be decoded by the entropy decoder 210. For example, the entropy decoder 210 can derive information for image reconstruction (or picture reconstruction) (e.g., video / image information) by parsing the bitstream. For example, the entropy decoder 210 can decode the information within the bitstream based on encoding methods such as exponential Golomb coding, CAVLC, or CABAC, and can output the values ​​of syntax elements for image reconstruction or quantized values ​​of transform coefficients with respect to the residuals. More specifically, in the CABAC entropy decoding method, bins corresponding to each syntax element can be received from the bitstream. A context model can be determined by decoding the target syntax element information and decoding information of neighboring and decoded target blocks or information of symbols / bins decoded in previous steps. The probability of bin occurrence can be predicted based on the determined context model, and symbols corresponding to the values ​​of each syntax element can be generated by performing arithmetic decoding on the bins. In this case, in the CABAC entropy decoding method, after determining the context model, the context model can be updated using information of symbols / bins decoded for the context model of the next symbol / bin. Information about prediction from the information decoded in entropy decoder 2110 can be provided to the predictors (inter-frame predictor 261 and intra-frame predictor 262). Parameter information (i.e., quantized transform coefficients) related to the residual values ​​that have undergone entropy decoding in entropy decoder 210 can be input to dequantizer 220. Furthermore, information about filtering from the information decoded in entropy decoder 210 can be provided to filter 240. Additionally, a receiver (not shown) that receives the signal output from the encoding device can be further configured as an internal / external element of decoding device 200, or the receiver can be a component of entropy decoder 210.

[0102] The dequantizer 220 can dequantize the quantized transform coefficients and output the transform coefficients. The dequantizer 220 can rearrange the quantized transform coefficients into two-dimensional blocks. In this case, the rearrangement can be performed based on the coefficient scan order executed in the encoding device. The dequantizer 220 can perform dequantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain the transform coefficients.

[0103] The inverse converter 230 can output the residual signal (residual block or residual sample array) by applying an inverse transform to the transform coefficients.

[0104] The predictor can perform predictions on the current block and generate a prediction block that includes prediction samples for the current block. The predictor can determine whether to apply intra-frame prediction or inter-frame prediction to the current block based on the prediction information output by the entropy decoder 210, and can determine the detailed intra-frame / inter-frame prediction mode.

[0105] Intra-predictor 262 can refer to samples within the current image to predict the current block. The referenced samples can be located as neighbors of the current block, or spaced apart from the current block depending on the prediction mode. In intra-prediction, the prediction mode can include multiple non-angular modes and multiple angular modes. Intra-predictor 262 can use the prediction modes applied to neighboring blocks to determine the prediction mode applied to the current block.

[0106] Inter-frame predictor 261 can derive the prediction block of the current block based on a reference block (reference sample array) specified by a motion vector on a reference image. In this case, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted on a block, sub-block, or sample basis based on the motion information correlation between neighboring blocks and the current block. Motion information may include motion vectors and reference image indices. Motion information may also include inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction) information. In the case of inter-frame prediction, neighboring blocks may include spatially neighboring blocks within the current image and temporally neighboring blocks within the reference image. For example, inter-frame predictor 261 can configure a motion information candidate list based on neighboring blocks and can derive the motion vector and / or reference image index of the current block based on received candidate selection information. Inter-frame prediction can be performed based on various prediction modes. Information about the prediction may include information indicating the mode of inter-frame prediction for the current block.

[0107] Adder 235 can generate a reconstruction signal (reconstructed image, reconstruction block, or reconstruction sample array) by adding the obtained residual signal to the prediction signal (prediction block or prediction sample array) output by inter-frame predictor 261 or intra-frame predictor 262. If there is no residual for processing the target block, as in the case where a jump mode has been applied, the prediction block can be used as the reconstruction block.

[0108] Adder 235 may be referred to as a reconstructor or reconstruction block generator. As will be described later, the generated reconstructed signal can be used for intra-frame prediction of the next processing target block within the current image, and can be used for inter-frame prediction of the next image through filtering.

[0109] Filter 240 can improve the quality of subjective / objective images by applying filtering to the reconstructed signal. For example, filter 240 can generate a modified reconstructed image by applying various filtering methods to the reconstructed image, and the modified reconstructed image can be sent to memory 250, specifically to DPB 250 in memory. Various filtering methods can include, for example, deblocking filtering, sample adaptive offset (SAO), adaptive loop filter (ALF), and bilateral filter.

[0110] The reconstructed image sent (modified) in the DPB of memory 250 can be used as a reference image in inter-frame predictor 261. Memory 250 can store motion information of blocks from which motion information within the current image has been derived (or decoded) and / or motion information of blocks within the reconstructed image. The stored motion information can be sent to inter-frame predictor 261 to be used as motion information for spatially or temporally neighboring blocks. Memory 170 can store reconstructed samples of reconstructed blocks within the current image and can send them to intra-frame predictor 262.

[0111] In this disclosure, the embodiments described in the filter 160, inter-frame predictor 180 and intra-frame predictor 185 of the encoding device 100 can be applied to the filter 240, inter-frame predictor 261 and intra-frame predictor 262 of the decoding device 200 in the same or corresponding manner.

[0112] Block segmentation

[0113] The video / image coding method according to this document can be performed based on various detailed techniques. These detailed techniques are illustrated below. It will be apparent to those skilled in the art that these techniques can be associated with related processes in the video / image coding / decoding process already described and / or subsequently described, such as prediction, residual processing (inverse transform, inverse quantization), syntax element encoding, filtering, and segmentation / partitioning.

[0114] The block segmentation process according to this document can be performed in the image segmenter 110 of the encoding device. The segmentation-related information can be (encoded) in the entropy encoder 190 and sent to the decoding device in the form of a bitstream. The entropy decoder 210 of the decoding device can derive the block segmentation structure of the current image based on the segmentation-related information obtained from the bitstream, and can perform a series of processes (e.g., prediction, residual processing, block reconstruction, and in-loop filtering) based on the block segmentation structure.

[0115] Segment the image into CTUs

[0116] An image can be segmented into a series of coding tree units (CTUs). Each CTU may correspond to a coding tree block (CTB). Alternatively, a CTU may include a coding tree block for luma samples and two corresponding coding tree blocks for chroma samples. In other words, relative to an image comprising three sample arrays, a CTU may include an N×N block of luma samples and two corresponding blocks of chroma samples.

[0117] The maximum allowed size of the CTU used for encoding and prediction can differ from the maximum allowed size of the CTU used for transformation. For example, the maximum allowed size of the luma block within a CTU can be 128×128.

[0118] CTU is segmented using a tree structure.

[0119] CTUs can be segmented based on a quadtree (QT) structure. A quadtree structure can be simply referred to as a quadtree structure. This is to incorporate various local features. Furthermore, in the literature, CTUs can be segmented based on multiple tree structure partitions, including binary trees (BT) and ternary trees (TT), in addition to quadtrees. Below, a QTBT structure can include segmentation structures based on quadtrees and binary trees. QTBTTT can include segmentation structures based on quadtrees, binary trees, and ternary trees. Alternatively, a QTBT structure can include segmentation structures based on quadtrees, binary trees, and ternary trees. In the coding tree structure, the CU can have a square or rectangular shape. The CTU can be segmented into a quadtree structure. Subsequently, the leaf nodes of the quadtree structure can be additionally segmented using multiple tree structure types.

[0120] Figure 3 This is a view that applies an embodiment of the present disclosure and illustrates examples of multiple types of tree structures.

[0121] In one embodiment of this disclosure, the multi-type tree structure may include four partitioning types, such as Figure 3 As shown in the diagram, these four segmentation types can include vertical binary segmentation SPLIT_BT_VER, horizontal binary segmentation SPLIT_BT_HOR, vertical ternary segmentation SPLIT_TT_VER, and horizontal ternary segmentation SPLIT_TT_HOR. Leaf nodes in the multi-type tree structure can be referred to as CUs. These CUs can be used in the prediction and transformation processes. In this document, typically, CUs, PUs, or TUs can have the same block size. In this case, if the maximum supported transformation length is less than the width or height of the color component of the CU, then the CU and TU can have different block sizes.

[0122] Figure 4 This is an illustration of a signaling mechanism for segmenting information of a quadtree structure with nested multi-type trees, which can be implemented using the present disclosure.

[0123] In this scenario, the CTU is treated as the root of a quadtree and is initially split into a quadtree structure. Each quadtree leaf node can be further split into multi-type tree structures. In the multi-type tree structure, a first flag (e.g., `mtt_split_cu_flag`) is signaled to indicate whether the corresponding node should be further split. If the corresponding node is further split, a second flag (e.g., `mtt_split_cu_vertical_flag`) is signaled to indicate the split direction. Subsequently, a third flag (e.g., `mtt_split_cu_binary_flag`) is signaled to indicate whether the split type is a binary or ternary split. For example, the multi-type tree splitting mode (`MttSplitMode`) of the CU can be derived based on `mtt_split_cu_vertical_flag` and `mtt_split_cu_binary_flag` as shown in Table 1.

[0124] [Table 1]

[0125] MttSplitMode mtt_split_cu_vertical_flag mtt_split_cu_binary_flag SPLIT_TT_HOR 0 0 SPLIT_BT_HOR 0 1 SPLIT_TT_VER 1 0 SPLIT_BT_VER 1 1

[0126] Figure 5 The embodiments of this disclosure can be applied, and the diagram illustrates a method for dividing a CTU into multiple CUs based on a quadtree and nested multi-type tree structures.

[0127] In this configuration, the bolded block edges indicate quadtree partitioning, while the remaining edges indicate multi-type tree partitioning. Quadtree partitioning with multi-type trees can provide a content-adaptive coding tree structure. A CTU can correspond to a coding block (CB). Alternatively, a CU can include a coding block for luma samples and two coding blocks for corresponding chroma samples. Using luma samples as units, the size of the CU can be larger than the CTU or smaller than 4×4. For example, in a 4:2:0 color format (or chroma format), the maximum chroma CB size can be 64×64, and the minimum chroma CB size can be 2×2.

[0128] In this document, for example, the maximum permissible luminance TB size can be 64×64, and the maximum permissible chrominance TB size can be 32×32. If the width or height of the CB segmented based on the tree structure is greater than the maximum transform width or height, the corresponding CB can be segmented until the TB size constraints in the horizontal and vertical directions are automatically (or implicitly) satisfied.

[0129] Furthermore, for quadtree coding schemes with multiple tree types, the following parameters can be qualified and identified as SPS syntax elements.

[0130] -CTU size: Size of the root node of the quadtree

[0131] -MinQTSize: Minimum allowed size of quadtree leaf nodes

[0132] -MaxBtSize: Maximum allowed size of the root node of the binary tree

[0133] -MaxTtSize: Maximum allowed size of the root node of the ternary tree

[0134] -MaxMttDepth: The maximum allowed depth of hierarchical division of a multi-type tree from quadtree leaves.

[0135] -MinBtSize: Minimum allowed size of binary leaf nodes

[0136] -MinTtSize: Minimum allowed size of a ternary leaf node

[0137] For example, in a quadtree coding tree structure with multiple tree types, the CTU size can be set to 128×128 luma samples and two corresponding chroma samples in a 64×64 block (in 4:2:0 chroma format). In this case, MinOTSize can be set to 16×16, MaxBtSize can be set to 128×128, MaxTtSize can be set to 64×64, and MinBtSize and MinTtSize (for both width and height) can be set to 4×4, and MaxMttDepth can be set to 4. Quadtree partitioning can be applied to the CTU to generate quadtree leaf nodes. Quadtree leaf nodes can be called leaf QT nodes. Quadtree leaf nodes can have a size from 16×16 (i.e., MinOTSize) to 128×128 (i.e., CTU size). If the leaf QT node is 128×128, it can be additionally partitioned into a binary / ternary tree without further partitioning. The reason for this is that although the leaf QT node is split, it exceeds MaxBtsize and MaxTtsize (i.e., 64×64). In other cases, the leaf QT node can be additionally split into a multi-type tree. Therefore, the leaf QT node is the root node for the multi-type tree, and the multi-type tree depth (mttDepth) value of the leaf QT node can be 0. If the multi-type tree depth reaches MaxMttdepth (e.g., 4), additional splits are no longer considered. If the width of the multi-type tree node is equal to MinBtSize and less than or equal to 2×MinTtSize, additional horizontal splits are no longer considered. If the height of the multi-type tree node is equal to MinBtSize and less than or equal to 2×MinTtSize, additional vertical splits are no longer considered.

[0138] Figure 6The embodiments of this disclosure can be applied, and the diagram illustrates a method for limiting ternary tree segmentation.

[0139] Reference Figure 6 To enable a 64×64 luma block and a 32×32 chroma pipeline design in the hardware decoder, TT segmentation can be constrained under certain conditions. For example, if the width or height of the luma encoding block is greater than a preset specific value (e.g., 32, 64), then... Figure 6 As shown, TT segmentation can be constrained.

[0140] In this disclosure, the coding tree scheme supports separate block tree structures for luma and chroma blocks. With respect to P-slices and B-slices, the luma CTB and chroma CTB within a CTU can be constrained to have the same coding tree structure. However, relative to I-slices, luma and chroma blocks can have separate block tree structures. If a separate block tree mode is applied, the luma CTB can be segmented into CUs based on a specific coding tree structure. The chroma CTB can be segmented into chroma CUs based on different coding tree structures. This may mean that a CU within an I-slice can be configured with a coding block containing luma components or two chroma components, and a CU within a P- or B-slice can be configured with a block containing three color components.

[0141] In the above-mentioned "Splitting the CTU using a tree structure," a quadtree coding tree structure with multiple tree types has been described, but the structure for splitting the CU is not limited to this. For example, the BT structure and TT structure can be interpreted as concepts included in the Multi-Split Tree (MPT) structure, and the CU can be interpreted as being split by the QT structure and the MPT structure. In one example of splitting the CU by the QT structure and the MPT structure, the splitting structure can be determined by signaling a syntax element that includes how many blocks the leaf nodes of the QT structure are split into (e.g., MPT_split_type) and a syntax element that includes information indicating which direction (vertical or horizontal) the leaf nodes of the QT structure are split (e.g., MPT_split_mode).

[0142] In another example, a different approach than the QT, BT, or TT structures can be used to segment the CU. That is, instead of segmenting a deeper CU into 1 / 4 the size of a shallower CU based on a QT structure, or into 1 / 2 the size of a shallower CU based on a BT structure, or into 1 / 4 or 1 / 2 the size of a shallower CU based on a TT structure, a deeper CU can be segmented into 1 / 5, 1 / 3, 3 / 8, 3 / 5, 2 / 3, or 5 / 8 the size of a shallower CU. The methods for segmenting the CU are not limited to these.

[0143] If a portion of a tree node block extends beyond the bottom or right image boundary, the corresponding tree node block can be constrained so that all samples of all encoded CUs lie within the image boundary. In this case, for example, the following partitioning rule can be applied.

[0144] -If a part of a tree node block extends beyond both the bottom and right image boundaries.

[0145] - If the block is a QT node and the block size is greater than the minimum QT size, then the block is forced to be split in QT splitting mode.

[0146] Otherwise, force the blocks to be split in the SPLIT_BT_HOR mode.

[0147] Otherwise, if part of a tree node block extends beyond the bottom image boundary,

[0148] - If the block is a QT node and the block size is greater than the minimum QT size and the block size is greater than the maximum BT size, then the block is forced to be split in QT splitting mode.

[0149] Otherwise, if the block is a QT node and the block size is greater than the minimum QT size and the block size is less than or equal to the maximum BT size, then the block is forced to be split in either QT split mode or SPLIT_BT_HOR mode.

[0150] Otherwise (if the block is a BTT node or the block size is less than or equal to the minimum QT size), the block will be forced to be split in the SPLIT_BT_HOR mode.

[0151] Otherwise, if part of the tree node block extends beyond the right image boundary,

[0152] - If the block is a QT node and the block size is greater than the minimum QT size and the block size is greater than the maximum BT size, then the block is forced to be split in QT splitting mode.

[0153] Otherwise, if the block is a QT node and the block size is greater than the minimum QT size and the block size is less than or equal to the maximum BT size, then the block is forced to be split in either QT split mode or SPLIT_BT_VER mode.

[0154] Otherwise (if the block is a BTT node or the block size is less than or equal to the minimum QT size), the block will be forced to be split in SPLIT_BT_VER mode.

[0155] Furthermore, the aforementioned quadtree-based coding block structure with multiple tree types provides a highly flexible block partitioning structure. Due to the partitioning types supported by the multiple tree types, the same coding block structure result may potentially be obtained depending on different partitioning patterns. By constraining the occurrence of these redundant partitioning patterns, the amount of data for partitioning information can be reduced. (See also...) Figure 7 This will be described.

[0156] Figure 7 The embodiments of this disclosure can be applied, and the diagram illustrates redundant segmentation patterns that may occur in binary tree segmentation and ternary tree segmentation.

[0157] like Figure 7 As illustrated, two consecutive binary segmentations in one direction have the same coded block structure as the binary segmentation used for center segmentation after ternary segmentation. In this case, the binary segmentation used for center segmentation of ternary segmentation (in a given direction) can be constrained. This constraint can be applied to the CUs of all images. If such a particular segmentation is constrained, the signaling of the corresponding syntax element can be modified by incorporating this constraint case. Therefore, the number of bits used for signaling segmentation can be reduced. For example, as... Figure 7 As in the example shown, if the binary tree split used for the center split of the CU is constrained, the mtt_split_cu_binary_flag syntax element indicating whether the split is a binary or ternary split is not signaled, and the corresponding value can be interpreted as 0 by the decoder.

[0158] predict

[0159] To recover the currently processed unit that was being decoded, the current image or the decoded portion of another image that includes the current processed unit can be used.

[0160] A picture (slice) that uses only the current image for reconstruction (i.e., performs only intra-picture prediction) can be called an intra-picture or I-picture (slice). A picture (slice) that uses at least one motion vector and a reference index to predict the corresponding unit can be called a prediction picture or P-picture (slice). A picture (slice) that uses up to two motion vectors and two reference indices can be called a double-prediction picture or B-picture (slice).

[0161] Intra-frame prediction refers to a method of deriving the prediction of the current processing block from data elements (e.g., sample values) of the same decoded image (or slice). In other words, it means predicting the pixel values ​​of the current processing block by referring to the reconstructed region in the current image.

[0162] Inter-frame prediction will be described in more detail below.

[0163] Inter-frame prediction (or inter-image prediction)

[0164] Inter-frame prediction refers to a prediction method that derives the current processing block based on data elements (e.g., sample values ​​or motion vectors) from images other than the current image. In other words, it means predicting the pixel values ​​of the current processing block by referring to a reconstructed region in another reconstructed image besides the current image.

[0165] Inter-frame prediction (or inter-image prediction) is a technique for removing redundancy between images, and it is primarily performed through motion estimation and motion compensation.

[0166] This disclosure describes the above references Figure 1 and Figure 2 The detailed techniques of the inter-frame prediction method are described. In the case of the decoder, the following descriptions can be used. Figure 10 Video / image decoding methods based on inter-frame prediction and Figure 11 The inter-frame prediction unit in the decoding device. Furthermore, relative to the encoder, it can be described subsequently. Figure 8 Video / image coding methods based on inter-frame prediction and Figure 9 The inter-frame prediction unit in the coding device. Additionally, through... Figure 8 and Figure 9 Encoded data can be stored as a bit stream.

[0167] The prediction unit of an encoding / decoding device can derive prediction samples by performing inter-frame prediction based on blocks. Inter-frame prediction can be represented as a prediction derived in a way that depends on data elements (e.g., sample values, motion information, etc.) of images other than the current image. When applying inter-frame prediction to the current block, the prediction block (prediction sample array) of the current block can be derived based on the reference block (reference sample array) specified by the motion vector on the reference image indicated by the reference image index.

[0168] In this case, to reduce the amount of motion information transmitted in inter-frame prediction mode, the motion information of the current block can be predicted on a block, sub-block, or sample basis based on the correlation between motion information between neighboring blocks and the current block. Motion information may include motion vectors and reference image indices. Motion information may also include inter-frame prediction type information (L0 prediction, L1 prediction, Bi prediction, etc.).

[0169] When applying inter-frame prediction, neighboring blocks can include spatially neighboring blocks present in the current image and temporally neighboring blocks present in a reference image. The reference image including the reference block can be the same as or different from the reference image including the temporally neighboring block. Temporally neighboring blocks can be represented by names such as co-localization reference blocks or colCU, and the reference image including the temporally neighboring block can be called the co-localization image colPic. For example, a candidate list of motion information can be constructed based on the neighboring blocks of the current block, and a signal can be sent indicating which candidate to select (use) to derive the motion vector of the current block and / or the reference image index, along with flags or index information.

[0170] Inter-frame prediction can be performed based on various prediction modes. For example, in jump mode and merge mode, the motion information of the current block can be the same as the motion information of the selected neighboring blocks. In jump mode, unlike merge mode, residual signals cannot be sent. In Motion Vector Prediction (MVP) mode, the motion vectors of the selected neighboring blocks are used as motion vector prediction terms, and the motion vector difference can be signaled. In this case, the motion vector of the current block can be derived using the sum of the motion vector prediction terms and the motion vector difference.

[0171] Figure 8 and Figure 9 These are illustrations of a video / image coding method based on inter-frame prediction according to embodiments of the present disclosure and an inter-frame prediction unit in a coding apparatus according to embodiments of the present disclosure.

[0172] Reference Figure 8 and Figure 9 S801 can be executed by the inter-frame predictor 180 of the encoding device, and S802 can be executed by the residual processing unit of the encoding device. Specifically, S802 can be executed by the subtractor 115 of the encoding device. In S803, the inter-frame predictor 180 derives prediction information, and the entropy encoder 190 can encode the prediction information. In S803, the residual processing unit derives residual information, and the entropy encoder 190 can encode the residual information. The residual information is information about the residual samples. The residual information may include information about the transform coefficients after quantization of the residual samples.

[0173] As described above, the residual samples can be derived into transform coefficients by the transformer 120 of the encoding device, and the transform coefficients can be derived into quantized coefficients quantized by the quantizer 130. Information about the quantized transform coefficients can be encoded in the entropy encoder 190 through a residual encoding process.

[0174] Encoding device 100 performs inter-frame prediction on the current block (S801). Encoding device 100 can derive the inter-frame prediction mode and motion information of the current block, and can generate prediction samples for the current block. In this case, the inter-frame prediction mode determination, motion information derivation, and prediction sample generation processes can be performed simultaneously, or any one process can be performed before the other. For example, the inter-frame predictor 180 of encoding device 100 may include a prediction mode determination unit 181, a motion information derivation unit 182, and a prediction sample derivation unit 183. The prediction mode determination unit 181 can determine the prediction mode of the current block. The motion information derivation unit 182 can derive the motion information of the current block. The prediction sample derivation unit 183 can derive the prediction samples of the current block.

[0175] For example, the inter-frame predictor 180 of the encoding device 100 can search for a given region (search region) of a reference image for blocks similar to the current block through motion estimation, and can derive a reference block with the smallest difference relative to the current block or a given reference or a smaller difference. The inter-frame predictor 180 can derive a reference image index indicating the reference image in which the reference block is located based on the reference block, and can derive motion vectors based on the positional difference between the reference block and the current block. The encoding device 100 can determine the mode applicable to the current block among various prediction modes. The encoding device can compare the RD costs of various prediction modes and determine the optimal prediction mode for the current block.

[0176] For example, when applying a transition mode or a merge mode to the current block, the encoding device 100 can configure a merge candidate list, which will be described later, and can export a reference block from the reference blocks indicated by the merge candidates included in the merge candidate list that has the smallest difference relative to the current block, or a difference of a given reference or smaller. In this case, a merge candidate associated with the exported reference block can be selected. Merge index information indicating the selected merge candidate can be generated and signaled to the decoding device 200. Motion information of the current block can be exported using the motion information of the selected merge candidate.

[0177] For another example, if the (A)MVP mode is applied to the current block, the encoding device can configure an (A)MVP candidate list, which will be described later, and can use the motion vector of the selected MVP candidate from the motion vector prediction term (MVP) candidates included in the (A)MVP candidate list as the MVP of the current block. In this case, for example, the motion vector of the reference block derived by motion estimation can be used as the motion vector of the current block. The MVP candidate that includes the motion vector with the smallest difference relative to the motion vector of the current block can become the selected MVP candidate. The motion vector difference (MVD) can be derived as the difference obtained by subtracting the MVP from the motion vector of the current block. In this case, information about the MVD can be signaled to the decoding device 200. Furthermore, if the (A)MVP mode is applied, the value of the reference picture index can be configured as reference picture index information and can be signaled to the decoding device separately.

[0178] The encoding device 100 can derive residual samples based on the predicted samples (S802). The encoding device 100 can derive residual samples by comparing the original samples of the current block with the predicted samples.

[0179] Encoding device 100 encodes image information including prediction information and residual information (S803). The encoding device can output the encoded image information in bitstream form. The prediction information may include information about prediction mode information (e.g., transition flag, merge flag, or mode index) and motion-related information as information related to the prediction process. The motion-related information may include candidate selection information (e.g., merge index, MVP flag, or MVP index) as information for deriving motion vectors. Furthermore, the motion-related information may include information about MVD and / or reference image index information.

[0180] In addition, motion-related information may include indications of whether L0, L1, or bidirectional prediction is applied. Residual information is about the residual samples. Residual information may include information about the transformation coefficients after quantization of the residual samples.

[0181] The output bitstream can be stored in (digital) storage media and transmitted to the decoding device, or it can be transmitted to the decoding device via a network.

[0182] Furthermore, as described above, the encoding device can generate a reconstructed image (including reconstructed samples and reconstructed blocks) based on reference samples and residual samples. This is to derive the same prediction result in the encoding device 100 as the prediction result performed in the decoding device 200. Therefore, encoding efficiency can be improved. Thus, the encoding device 100 can store the reconstructed image (or reconstructed samples and reconstructed blocks) in memory and use it as a reference image for inter-frame prediction. As described above, an in-loop filtering process can also be applied to the reconstructed image.

[0183] Figure 10 and Figure 11 These are, respectively, the video / image decoding process based on inter-frame prediction according to embodiments of the present disclosure and the inter-frame predictor within the decoding device.

[0184] Reference Figure 10 and Figure 11 The decoding device 200 can perform operations corresponding to those performed in the encoding device 100. The decoding device 200 can perform predictions on the current block based on the received prediction information and can derive prediction samples.

[0185] S1001 to S1003 can be executed by the inter-frame predictor 261 of the decoding device, and the residual information of S1004 can be obtained from the bitstream by the entropy decoder 210 of the decoding device. The residual processing unit of the decoding device can derive the residual samples of the current block based on the residual information. Specifically, the dequantizer 220 of the residual processing unit derives the transform coefficients by performing dequantization based on the quantized transform coefficients derived from the residual information. The inverse transformer 230 of the residual processing unit can perform an inverse transform on the transform coefficients to derive the residual samples of the current block. S1005 can be executed by the adder 235 or the recovery unit of the decoding device.

[0186] Specifically, the decoding device 200 can determine the prediction mode for the current block based on the received prediction information (S1001). The decoding device 200 can determine which inter-frame prediction mode to apply to the current block based on the prediction mode information within the prediction information.

[0187] For example, decoding device 200 may determine whether to apply a merge mode or (A)MVP mode to the current block based on a merge flag. Alternatively, decoding device 200 may select one of a variety of inter-frame prediction mode candidates based on a mode index. Inter-frame prediction mode candidates may include jump modes, merge modes, and / or (A)MVP modes, or may include a variety of inter-frame prediction modes described below.

[0188] Decoding device 200 derives motion information for the current block based on the determined inter-frame prediction mode (S1002). For example, if a jump mode or merge mode is applied to the current block, decoding device 200 can configure a merge candidate list (described later) and select one of the merge candidates included in the merge candidate list. The selection can be performed based on the selection information (merge index) described above. Motion information for the current block can be derived based on the motion information of the selected merge candidate. The motion information of the selected merge candidate can be used as the motion information for the current block.

[0189] For another example, if the (A)MVP mode is applied to the current block, the decoding device 200 can configure an (A)MVP candidate list, which will be described later, and can use the motion vector of the selected MVP candidate from the motion vector prediction (MVP) candidates included in the (A)MVP candidate list as the MVP of the current block. Selection can be performed based on selection information (MVP flag or MVP index). In this case, the decoding device 200 can derive the MVD of the current block based on information about the MVD. The decoding device can derive the motion vector of the current block based on the MVP and MVD of the current block. Furthermore, the decoding device can derive the reference image index of the current block based on reference image index information. The image indicated by the reference image index in the reference image list for the current block can be derived as the reference image referenced for inter-frame prediction of the current block.

[0190] Furthermore, as will be described later, motion information for the current block can be exported without a candidate list configuration. In this case, the motion information for the current block can be exported according to the process disclosed in the prediction mode, which will be described later. In this case, candidate list configurations such as those described above can be omitted.

[0191] The decoding device 200 can generate prediction samples for the current block based on the motion information of the current block (S1003). In this case, the decoding device 200 can derive a reference image based on the reference image index of the current block, and can derive prediction samples of the current block indicated on the reference image by the motion vector of the current block. In this case, as will be described later, a prediction sample filtering process can be further performed on some or all of the prediction samples of the current block, depending on the situation.

[0192] For example, the inter-frame predictor 260 of the decoding device 200 may include a prediction mode determination unit 261, a motion information derivation unit 262, and a prediction sample derivation unit 263. The decoding device 200 may determine the prediction mode of the current block based on the prediction mode information received from the prediction mode determination unit 261, and may derive the motion information (motion vector and / or reference image index) of the current block based on information related to the motion information received from the motion information derivation unit 262. The prediction sample derivation unit 263 may derive prediction samples for the current block.

[0193] The decoding device 200 generates residual samples for the current block based on the received residual information (S1004). The decoding device 200 can generate reconstructed samples for the current block based on the predicted samples and the residual samples, and can generate a reconstructed image based on the reconstructed samples (S1005). Thereafter, as described above, an in-loop filtering process can also be applied to the reconstructed image.

[0194] As described above, the inter-frame prediction process may include an inter-frame prediction mode determination step, a motion information derivation step based on the determined prediction mode, and a prediction execution (prediction sample generation) step based on the derived motion information.

[0195] Determine inter-frame prediction mode

[0196] Various inter-frame prediction modes can be used to predict the current block in an image. For example, modes such as merge mode, jump mode, MVP mode, and affine mode can be used. Decoder-side motion vector correction (DMVR) mode, adaptive motion vector resolution (AMVR) mode, etc., can be further used as additional modes. Affine mode can be referred to as affine motion prediction mode. MVP mode can be referred to as advanced motion vector prediction (AMVP) mode.

[0197] The encoding device can signal the decoding device with prediction mode information indicating the inter-frame prediction mode of the current block. This prediction mode information can be included in the bitstream and received by the decoding device. The prediction mode information may include index information indicating one of several candidate modes. Alternatively, the inter-frame prediction mode can be indicated by hierarchical signaling of flag information.

[0198] In this context, the prediction pattern information may include one or more flags. For example, flags may be further signaled to indicate whether a jump pattern is applied by signaling a jump flag, whether a merge pattern is applied when a jump pattern is not applied by signaling a merge flag, and whether an MVP pattern is applied when a merge pattern is not applied or for additional identification purposes. Affine patterns may be signaled as independent patterns or as patterns dependent on either the merge or MVP pattern. For example, an affine pattern may be configured as one of a merge candidate list or an MVP candidate list, as described below.

[0199] Motion information is derived from the inter-frame prediction mode.

[0200] Encoding device 100 or decoding device 200 can perform inter-frame prediction using motion information of the current block. Encoding device 100 can derive the optimal motion information for the current block based on the motion estimation process. For example, encoding device 100 can search for reference blocks with similar relevance using original blocks in the original image of the current block within a search range defined within a reference image, on a fractional pixel basis. Therefore, the encoding device can derive motion information. Block similarity can be derived based on the difference between phase-based sample values. For example, block similarity can be calculated based on the SAD (sum of absolute differences) between the current block (or its template) and a reference block (or its template). In this case, motion information can be derived based on the reference block with the minimum SAD within the search area. The derived motion information can be signaled to the decoding device using various methods based on the inter-frame prediction mode.

[0201] Merge mode and jump mode

[0202] Figure 12 This is an implementation of the present disclosure and is a view used to illustrate adjacent blocks used in merge mode or jump mode.

[0203] If the merge mode is applied, the motion information of the current prediction block is not sent directly, but is derived using the motion information of neighboring prediction blocks. Therefore, the encoding device 100 can indicate the motion information of the current prediction block by sending a flag indicating that the merge mode has been used and a merge index indicating which neighboring prediction block has been used.

[0204] Encoding device 100 can search for candidate blocks to merge, used to derive motion information of the current predicted block, in order to execute a merging mode. For example, up to five candidate blocks can be used, but this disclosure is not limited thereto. Furthermore, a maximum number of candidate blocks can be transmitted in the slice header or tile group header, and this disclosure is not limited thereto. After searching for candidate blocks, encoding device 100 can generate a list of candidate blocks to merge, and can select the candidate block with the lowest cost from among them as the final candidate block to merge.

[0205] The embodiments of this disclosure provide various implementations of constructing a merge candidate block for a merge candidate list.

[0206] For example, the merge candidate list can use 5 merge candidate blocks. Alternatively, it can use 4 spatial merge candidates and 1 temporal merge candidate.

[0207] For a detailed example, in the case of spatial merging candidates, Figure 12 The blocks illustrated in the example can be used as space merge candidates.

[0208] Figure 13 This is a flowchart illustrating a method for configuring and merging candidate lists according to an embodiment of the present disclosure.

[0209] Reference Figure 13 The encoding device (encoding device 100 or decoding device 200) searches for spatial neighbor blocks of the current block and inserts the derived spatial merging candidates into the merging candidate list (S1301). For example, spatial neighbor blocks may include the lower-left neighbor block, left neighbor block, upper-right neighbor block, upper neighbor block, and upper-left neighbor block of the current block. This is just an example, and in addition to spatial neighbor blocks, additional neighbor blocks such as right neighbor blocks, lower neighbor blocks, and lower-right neighbor blocks can be further used as spatial neighbor blocks. The encoding device can detect available blocks by searching spatial neighbor blocks based on priority, and can derive the motion information of the detected blocks as spatial merging candidates. For example, the encoding device 100 or decoding device 200 can search in the order of A1, B1, B0, A0, and B2. Figure 11 The example shows five blocks, and the list of merged candidates can be configured by indexing the available candidates sequentially.

[0210] The encoding device searches for temporally neighboring blocks of the current block and inserts the exported temporal merge candidates into the merge candidate list (S1302). Temporally neighboring blocks can be located on a reference picture, i.e., on a picture different from the current picture where the current block is located. The reference picture where the temporally neighboring block is located can be called a co-location picture or a col picture. Temporally neighboring blocks can be searched in the sequence of the lower-right corner neighboring block and the lower-right center block of the co-location block of the current block on the col picture.

[0211] Furthermore, if motion data compression is applied, specific motion information may be stored in a col picture as representative motion information for each given storage unit. In this case, it is not necessary to store the motion information of all blocks within a given storage unit, so that a motion data compression effect can be obtained. In this case, for example, the given storage unit may be predetermined as a 16×16 sample unit or an 8×8 sample unit, or size information for the given storage unit may be signaled from the encoding apparatus 100 to the decoding apparatus 200. If motion data compression is applied, the motion information of a temporally neighboring block may be replaced with the representative motion information of the given storage unit in which the temporally neighboring block is located.

[0212] That is, in this case, in terms of implementation, after performing an arithmetic right shift by a given value based on the coordinates (top-left sample position) of a temporally neighboring block of a prediction block, rather than the coordinates of the temporally neighboring block, a temporal merge candidate may be derived based on the motion information of the prediction block covering the position after the arithmetic left shift. For example, if the given storage unit is a 2n×2n sample unit, assuming that the coordinates of the temporally neighboring block are (xTnb, yTnb), the motion information of the prediction block located at ((xTnb>>n)<<n), (yTnb>>n)<<n)) (i.e., the modified position) may be used for the temporal merge candidate.

[0213] Specifically, for example, if the given storage unit is a 16×16 sample unit, assuming that the coordinates of the temporally neighboring block are (xTnb, yTnb), the motion information of the prediction block located at ((xTnb>>4)<<4), (yTnb>>4)<<4)) (i.e., the modified position) may be used for the temporal merge candidate. Alternatively, for example, if the given storage unit is an 8×8 sample unit, assuming that the coordinates of the temporally neighboring block are (xTnb, yTnb), the motion information of the prediction block located at ((xTnb>>3)<<3), (yTnb>>3)<<3)) (i.e., the modified position) may be used for the temporal merge candidate.

[0214] The encoding apparatus may check whether the current number of merge candidates is less than the maximum number of merge candidates (S1303). The maximum number of merge candidates may be predefined, or may be signaled from the encoding apparatus 100 to the decoding apparatus 200. For example, the encoding apparatus 100 may generate information on the maximum number of merge candidates, may encode the information, and may transmit the information in the form of a bitstream to the decoding apparatus 200. If the maximum number of merge candidates is reached, the candidate adding procedure may not be performed.

[0215] If, as a result of the check, the current number of merge candidates is less than the maximum number of merge candidates, the encoding device inserts the added merge candidate into the merge candidate list (S1240). For example, the added merge candidate may include ATMVP (Adaptive Temporal Motion Vector Prediction), combined bidirectional prediction merge candidates (if the current slice type is B), and / or zero vector merge candidates.

[0216] If, as a result of the check, the current number of merge candidates is not less than the maximum number of merge candidates, the encoding device can terminate the configuration of the merge candidate list. In this case, the encoder can select the best merge candidate from the merge candidates used to construct the merge candidate list based on the rate distortion (RD) cost, and can signal the decoder to indicate the selection information (e.g., merge index) of the selected merge candidate. The decoder can select the best merge candidate based on the merge candidate list and the selection information.

[0217] The motion information of the selected merging candidate can be used as the motion information of the current block. As described above, a prediction sample for the current block can be derived based on the motion information of the current block. The encoder can derive a residual sample for the current block based on the prediction sample and can signal the decoder with residual information associated with the residual sample. The decoder can generate a reconstructed sample based on the residual sample derived from the residual information and the prediction sample. As described above, the decoder can generate a reconstructed image based on the reconstructed sample.

[0218] If the jump mode is applied, the motion information of the current block can be derived using the same method as when the merge mode is applied. In this case, the residual signal of the corresponding block is omitted when the jump mode is applied. Therefore, the predicted samples can be directly used as reconstructed samples.

[0219] MVP mode

[0220] Figure 14 This is a flowchart illustrating a method for constructing a merged candidate list according to an embodiment of the present disclosure.

[0221] If the motion vector prediction (MVP) model is applied, it can be based on the reconstructed spatial neighbor blocks (e.g., Figure 12 The motion vectors of the spatially neighboring blocks (as described in the diagram) and / or the motion vectors corresponding to the temporally neighboring blocks (or Col blocks) are used to generate a candidate list of motion vector prediction terms (MVPs). That is, the motion vectors of the reconstructed spatially neighboring blocks and / or the motion vectors of the temporally neighboring blocks can be used as candidates for motion vector prediction terms.

[0222] Information about the prediction can include selection information (e.g., MVP flag or MVP index) of the best motion vector prediction candidate selected from the motion vector prediction candidate candidates included in the indicator list. In this case, the predictor can use the selection information to select the motion vector prediction term for the current block from the motion vector prediction candidate candidates included in the motion vector candidate list. The predictor of the encoding device 100 can calculate the motion vector difference (MVD) between the motion vector of the current block and the motion vector prediction term, encode the MVD, and output the encoded MVD in bitstream form. That is, the MVD can be calculated as the value obtained by subtracting the motion vector prediction term from the motion vector of the current block. In this case, the predictor of the decoding device can obtain the motion vector difference included in the information about the prediction and can derive the motion vector of the current block by adding the motion vector difference to the motion vector prediction term. The predictor of the decoding device can obtain or derive the reference image index of the indicator reference image based on the information about the prediction. For example, it can be as follows: Figure 14 The example demonstrates the configuration of the candidate list for motion vector prediction terms.

[0223] Affine motion prediction

[0224] Figure 15 An example of a motion model according to an embodiment of the present disclosure is illustrated.

[0225] In conventional image compression techniques (e.g., High Efficiency Video Coding (HEVC)), a single motion vector is used to represent the motion of a coded block. While a single motion vector can be used for each block to represent the optimal motion of a block unit, it may not be the actual optimal motion for every picture element. Therefore, determining the optimal motion vector at the pixel element level improves coding efficiency. Thus, embodiments of this disclosure describe a motion prediction method for encoding or decoding video signals using a multi-motion model. Specifically, motion vectors can be represented at 2 to 4 control points in each picture element unit of a block or sub-block. A prediction scheme using motion vectors at multiple control points can be referred to as affine motion prediction or affine forecasting.

[0226] The affine motion model according to embodiments of this disclosure can represent, for example, Figure 14 The four motion models illustrated herein are as follows. An affine motion model representing three of the motions (translation, scaling, and rotation) that can be represented by an affine motion model is called a similar (or simplified) affine motion model. In describing embodiments of this disclosure, for ease of description, a similar (or simplified) affine motion model is substantially described, but this disclosure is not limited thereto.

[0227] Figure 16An example of a control point motion vector for affine motion prediction according to an embodiment of the present disclosure is illustrated.

[0228] like Figure 16 As shown, in affine motion prediction, the motion vectors of the image elements (or sub-blocks) included in a block can be determined using two control point motion vectors (CPMV) pairs v_0 and v_1. In this case, the set of motion vectors can be called the affine motion vector field (MVF). In this case, Equation 1 can be used to determine the affine motion vector field.

[0229] [Formula 1]

[0230]

[0231] In Equation 1, v_0 (v_0 = {v_0x, v_0y}) indicates the motion vector CPMV0 at the first control point at the upper left position of the current block 1300. v_1 (v_1 = {v_1x, v_1y}) indicates the motion vector CPMV1 at the second control point at the upper right position of the current block 1300. Furthermore, w indicates the width of the current block 1300. v (v = {v_x, v_y}) indicates the motion vector at the position {x, y}. Equation 1 can be used to derive the motion vectors of sub-blocks (or image elements). In one implementation, the motion vector precision can be rounded to 1 / 16 precision.

[0232] Figure 17 An example is shown where the motion vectors of each sub-block of a block have been applied according to an embodiment of the present disclosure.

[0233] Reference Figure 17 During encoding or decoding, the affine motion vector field (MVF) can be determined using image elements or blocks as units. That is, in affine motion prediction, the motion vector of the current block can be derived using image elements or sub-blocks as units.

[0234] If the affine motion vector field is determined using image elements as units, the motion vector can be obtained based on the value of each image element. In the case of block units, the motion vector of the corresponding block can be obtained based on the image element value of the center of the block. In this paper, it is assumed that... Figure 17 Similarly, the affine motion vector field (MVF) is determined using 4×4 blocks as units. In this case, this is for ease of description, and the embodiments of this disclosure are not limited thereto. Figure 17 An example is shown where the coding block is configured with 16×16 samples and the affine motion vector field (MVF) is determined in 4×4 blocks.

[0235] Affine motion prediction can include affine merging mode (or AF_MERGE) and affine inter-frame mode (or AF_INTER). The AF_INTER mode can include AF_4_INTER mode using a motion model based on 4 parameters and AF_6_INTER mode using a motion model based on 6 parameters.

[0236] Affine Merge Pattern

[0237] AF_MERGE determines the control point motion vector (CPMV) based on the affine motion model of neighboring blocks encoded as affine motion predictions. Affine-encoded neighboring blocks in the search sequence can be used for AF_MERGE. When one or more neighboring blocks are encoded as affine motion predictions, the current block can be encoded as AF_MERGE.

[0238] That is, if the affine merge mode is applied, the CPMV of the current block can be derived using the CPMV of the neighboring blocks. In this case, the CPMV of the neighboring blocks can be used as the CPMV of the current block without any changes, and the CPMV of the neighboring blocks can be modified based on the size of the neighboring blocks and the size of the current block, and can be used as the CPMV of the current block.

[0239] Figure 18 An example of a neighboring block for affine motion prediction in affine merging mode according to an embodiment of the present disclosure is illustrated.

[0240] In affine merging (AF_MERGE) mode, the encoder can perform encoding as follows.

[0241] Step 1: Scan the neighboring blocks A to E 1810, 1820, 1830, 1840, and 1850 of the current coded block 1800 in alphabetical order, and determine the block that is first encoded according to the affine prediction mode based on the scan sequence as a candidate block for affine merging (AF_MERGE).

[0242] Step 2: Use the control point motion vectors (CPMV) of the determined candidate blocks to determine the affine motion model.

[0243] Step 3: Determine the control point motion vector (CPMV) of the current block 1800 based on the affine motion model of the candidate blocks, and determine the MVF of the current block 1800.

[0244] Figure 19 An example is illustrated of a block to which affine motion prediction has been performed using neighboring blocks that have already been applied with affine motion prediction according to an embodiment of this disclosure.

[0245] For example, such as Figure 19Similarly, if block A 1920 has already been encoded according to the affine pattern, then after block A 1920 is determined as a candidate block, the affine motion model can be derived using the control point motion vectors (CPMVs) (e.g., v2 and v3) of block A 1920, and the control point motion vectors (CPMVs) v0 and v1 of the current block 1900 can be determined. The affine motion vector field (MVF) of the current block 1900 can be determined based on the control point motion vectors (CPMVs) of the current block 1900, and encoding can be performed.

[0246] Figure 20 This is a diagram illustrating a method for generating a merge candidate list using neighboring affine coding blocks according to embodiments of the present disclosure.

[0247] Reference Figure 20 If affine merge candidates are used to determine CPMV pairs, then methods such as Figure 20 The candidates are exemplified in the example. Figure 20 In this context, it is assumed that the scan sequence of the candidate list has been set to A, B, C, D, and E. However, this disclosure is not limited to this, and the various sequences can be preset.

[0248] In an implementation, if the number of candidates (hereinafter referred to as affine candidates) encoded according to the affine patterns (or affine predictions) available in neighboring blocks (i.e., A, B, C, D, E) is 0, the affine merging pattern of the current block can be skipped. If the number of available affine candidates is 1 (e.g., A), the motion model of the corresponding candidate can be used to derive the control point motion vectors (CPMV_0 and CPMV_1) of the current block. In this case, it is not necessary to indicate (or encode) the index of the corresponding candidate. If the number of available affine candidates is 2 or more, two candidates on the scan sequence can be configured as a candidate list for AF_MERGE. In this case, candidate selection information, such as indicating the index of the candidate selected within the candidate list, can be signaled. The selection information can be a flag or index information and can be referred to as AF_MERGE_flag or AF_merge_idx.

[0249] In embodiments of this disclosure, motion compensation can be performed on the current block based on the size of its sub-blocks. In this case, the sub-block sizes of the affine block (i.e., the current block) are derived. If each of the width and height of the sub-block is greater than 4 luminance samples, the motion vector for each sub-block can be derived, and DCT-IF-based motion compensation (1 / 16 pel for luminance and 1 / 32 pel for chrominance) can be performed on the sub-blocks. Otherwise, motion compensation based on an enhanced bilinear interpolation filter can be performed on all affine blocks.

[0250] In embodiments of this disclosure, if the merge / jump flag is true and both the width and height of the CU are greater than or equal to 8, the affine flag is signaled at the CU level by indicating whether to use the affine merge mode bitstream. When the CU is encoded as AF_MERGE, a merge candidate index with a maximum value of "5" is signaled to specify the use of motion information candidates for the CUs in the affine merge candidate list.

[0251] Figure 21 and Figure 22 This is a diagram illustrating a method for constructing an affine merging candidate list using neighboring blocks encoded by affine prediction, according to embodiments of the present disclosure.

[0252] Reference Figure 21 Configure the affine merge candidate list using the following steps.

[0253] 1) Inserting model-based affine candidates

[0254] Model-based affine candidate means that the candidate is derived from the effective neighboring reconstruction block encoded according to the affine pattern. For example... Figure 21 As illustrated, the scanning order of the candidate blocks is from left A, top B, top right C, and bottom left D to top left E.

[0255] If the neighboring lower-left block A is encoded using a 6-parameter affine model, the motion vectors (v_4, v_5, v_6) of the upper-left, upper-right, and lower-left corners of the CU including block A are obtained. Based on the 6-parameter affine model, the motion vector (v_0, v_1, v_2) of the upper-left corner of the current block is calculated based on the motion vectors (v_4, v_5, and v_6).

[0256] If the adjacent lower-left block A is encoded using a 4-parameter affine model, the motion vectors (v_4, v_5) of the upper-left and upper-right corners of the CU including block A are obtained. Based on the 4-parameter affine model, the motion vector (v_0, v_1) of the upper-left corner of the current block is calculated based on the motion vectors (v_4, v_5).

[0257] 2) Insert affine candidates based on control points

[0258] Reference Figure 21 Candidates based on control points refer to the configuration of candidates by combining the neighboring motion information of control points.

[0259] First, from Figure 21 The specified spatial and temporal neighbor blocks, as illustrated in the example, derive the motion information of the control points. Cp_k (k = 1, 2, 3, 4) represents the k-th control point. Furthermore, A, B, C, D, E, F, and G are used to predict the spatial location of CP_k (k = 1, 2, 3), and H is used to predict the temporal location of CP4.

[0260] The coordinates of CP_1, CP_2, CP_3, and CP_4 are (0,0), (W,0), (H,0), and (W,H), respectively. In this case, W and H are the width and height of the current block.

[0261] Motion information for each control point is obtained based on the following priorities.

[0262] Relative to CP_1, the check priority is A→B→C, and if A is available, then A is used. Otherwise, if B is available, then B is used. If neither A nor B is available, then C is used. If none of these three candidates are available, then motion information for CP_1 cannot be obtained.

[0263] Compared to CP_2, the inspection priority is E→D.

[0264] Compared to CP_3, the inspection priority is G→F.

[0265] Use H instead of CP_4.

[0266] Secondly, a combination of control points is used to configure the motion model.

[0267] To compute the transformation parameters in a 4-parameter affine model, motion vectors for two control points must be used. These two control points can be selected from one of the following six combinations: {CP_1,CP_4}, {CP_2,CP_3}, {CP_1,CP_2}, {CP_2,CP_4}, {CP_1,CP_3}, and {CP_3,CP_4}. For example, using control points CP_1 and CP_2 when constructing a 4-parameter affine motion model is labeled as "affine (CP_1,CP_2)".

[0268] To compute the transformation parameters in a 6-parameter affine model, motion vectors for three control points must be used. These three control points can be selected from one of the following four combinations: {CP_1,CP_2,CP_4}, {CP_1,CP_2,CP_3}, {CP_2,CP_3,CP_4}, and {CP_1,CP_3,CP_4}. For example, using control points CP_1, CP_2, and CP_3 in constructing a 6-parameter affine motion model is labeled as "affine (CP_1,CP_2,CP_3)".

[0269] Furthermore, in embodiments of this disclosure, if an affine merge candidate exists in the affine merge mode, this can always be considered a 6-parameter affine mode.

[0270] Affine inter-frame mode

[0271] Figure 23An example of a neighboring block for affine motion prediction in affine inter-frame mode according to an embodiment of the present disclosure is illustrated.

[0272] Reference Figure 23 Affine motion prediction can include affine merging mode (or AF_MERGE) and affine inter-frame mode (or AF_INTER). In affine inter-frame mode (AF_INTER), after determining the two-point control vector prediction (CPMVP) and CPMV, the control vector difference (CPMVD) corresponding to the difference can be sent from the encoder to the decoder. The detailed encoding process for affine inter-frame mode (AF_INTER) can be the same as the detailed encoding process described below.

[0273] Step 1: Identify two CPMVP candidates

[0274] Step 1.1: Identify a maximum of 12 CPMVP candidate combinations (refer to Equation 2)

[0275] [Equation 2]

[0276] {(v0, v1, v2)|v0={v A v B v C}, v1={V D V E}, v2={v F v G}}

[0277] In Equation 2, v_0 indicates the motion vector CPMV0 at the upper left control point 2310 of the current block 2300. v_1 indicates the motion vector CPMV1 at the upper right control point 2311 of the current block 2300. v_2 indicates the motion vector CPMV2 at the lower left control point 2312 of the current block 2300. v_A indicates the motion vector of the neighboring block A2320, which is adjacent to the upper left of the upper left control point 2310 of the current block 2300. v_B indicates the motion vector of the neighboring block B2322, which is adjacent to the upper left of the upper left control point 2310 of the current block 2300. v_C indicates the motion vector of the neighboring block C2324, which is adjacent to the left of the upper left control point 2310 of the current block 2300. v_D is the motion vector of the neighboring block D2326, which is adjacent to the upper right control point 2311 of the current block 2300. v_E indicates the motion vector of the neighboring block E 2328, which is adjacent to the upper right of the upper right control point 2311 of the current block 2300. v_F indicates the motion vector of the neighboring block F 2330, which is adjacent to the left of the lower left control point 2312 of the current block 2300. v_G indicates the motion vector of the neighboring block G 2332, which is adjacent to the lower left of the lower left control point 2312 of the current block 2300.

[0278] Step 1.2: Sort the CPMVP candidate combinations according to the smaller difference (DV) value, and use the top two candidates (see Equation 3 below).

[0279] [Formula 3]

[0280] DV = |(v 1x -v 0x )*h-(v 2y -v 0y )*w|+|(v 1y -v 0y )*h+(v 2x -v 0x )*w|

[0281] v_0x indicates the x-axis element of the motion vector (V0 or CPMV0) at the top-left control point 2310 of the current block 2300. v_1x indicates the x-axis element of the motion vector (V_1 or CPMV_1) at the top-right control point 2311 of the current block 2300. v_2x indicates the x-axis element of the motion vector (V_2 or CPMV_2) at the bottom-left control point 2312 of the current block 2300. v_0y indicates the y-axis element of the motion vector (V_0 or CPMV_0) at the top-left control point 2310 of the current block 2300. v_1y indicates the y-axis element of the motion vector (V_1 or CPMV_1) at the top-right control point 2311 of the current block 2300. v_2y indicates the y-axis element of the motion vector (V_2 or CPMV_2) at the bottom-left control point 2312 of the current block 2300. w indicates the width of the current block 2300. h indicates the height of the current block 2300.

[0282] Step 2: When the number of candidates for the control point motion vector prediction term (CPMVP) is less than 2, use the AMVP candidate list.

[0283] Step 3: Determine the Control Point Motion Vector Prediction Term (CPMV) for each of the two candidates, and optimally select the candidate with the smaller value by comparing the RD cost with the CPMV.

[0284] Step 4: Send the index and control point motion vector difference (CPMVD) corresponding to the best candidate.

[0285] In embodiments of this disclosure, a process for constructing CPMVP candidates under AF_INTER is provided. Similar to AMVP, the number of candidates is 2, and an index indicating the position of the candidate list is signaled.

[0286] The process of constructing the CPMVP candidate list is as follows.

[0287] 1) Check whether neighboring blocks are encoded as affine motion predictions by scanning neighboring blocks. If the scanned block is encoded as an affine prediction, derive the motion vector pair of the current block based on the affine motion model of the scanned neighboring blocks until the number of candidates becomes 2.

[0288] 2) If the number of candidates is less than 2, the candidate configuration process is executed. Furthermore, in embodiments of this disclosure, a 4-parameter (2 control points) affine inter-frame pattern is used to predict the motion model and content for zooming in / out and rotation. For example... Figure 16 The example illustrates the affine motion field of a block, described by motion vectors from two control points.

[0289] Equation 1 above describes the motion vector field (MVF) of the block.

[0290] In conventional techniques, Advanced Motion Vector Prediction (AMVP) mode is necessary for scanning the MVP index and motion vector difference (MVD). When applying the AMVP mode to this disclosure, an affine flag (affine_flag) is signaled to indicate whether affine prediction is used. If affine prediction is applied, the syntax of inter_dir, ref_idx, mvp_index, and the two MVDs (mvd_x and mvd_y) is signaled. A list of affine MVP pair candidates, including two affine MVP pairs, is generated. One of the two affine MVP pairs is selected using the signaled mvp_index. Affine MVP pairs are generated from two types of affine MVP candidates: spatially inherited affine candidates and angle-derived affine candidates. Spatially inherited affine candidates can be generated if neighboring CUs are encoded in affine mode. The motion vectors of the 2-control-point MVP pairs are generated using the affine motion model of the neighboring affine coding blocks. The MV of the 2-control-point MVP pairs of the spatially inherited affine candidates is derived using the following formula.

[0291] [Formula 4]

[0292] V 0x =V B0x +(V B2_x -V B0x )*(posCurCU_Y-posRefCU_Y) / RefCU_height+(V B1x -V B0x )*

[0293] (posCurCU_X-posRefCU_X) / RefCU_width

[0294] [Formula 5]

[0295] V 0y =VB0y +(V B2_y -V B0y )*(posCurCU_Y-posRefCU_Y) / RefCU_height+(V B1y -V B0y )*

[0296] (posCurCU_X-posRefCU_X) / RefCU_width

[0297] If V_B0, V_B1, and V_B2 can be replaced by the top-left MV, top-right MV, and bottom-left MV of a given reference / neighboring CU, then (posCurCU_X, posCurCU_Y) is the position of the top-left sample of the current CU for the top-left sample of the frame. (posRefCU_X, posRefCU_Y) is the position of the top-left sample of the reference / neighboring CU for the top-left sample of the frame.

[0298] [Formula 6]

[0299] V 1x =V B0x +(V B1x -V B0x )*CU_width / RefCU_width

[0300] [Formula 7]

[0301] V 1y =V B0y +(V B1y -V B0y )*CU_width / RefCU_width

[0302] Figure 24 An example of a neighboring block for affine motion prediction in affine inter-frame mode according to an embodiment of the present disclosure is illustrated.

[0303] Reference Figure 24 When the number of MVP pairs is less than 2, use the affine candidate derived from the angle. For example... Figure 24 The example illustrates how adjacent motion vectors are used to derive affine MVP pairs. The first MVP pair is configured using the first available MV in set A (A0, A1, and A2) and the first available MV in set B (B0 and B1) relative to the affine candidates derived from the first angle. The MV of the upper right control point is computed using the first available MV in set A and the first available MV in set C (C0 and C1) relative to the affine candidates derived from the second angle. The first available MV in set A and the computed upper right control point MV constitute the second MVP pair.

[0304] In embodiments of this disclosure, two candidate sets comprising two (three) candidates {mv_0,mv_1}({mv_0,mv_1,mv_2}) are used to predict two (three) control points of an affine motion model. The given motion vector difference (mvd_0, mvd_1, mvd_2) and the control points are calculated using the following formula.

[0305] [Formula 8]

[0306]

[0307]

[0308]

[0309] Figure 25 and Figure 26 This is an illustration of a method for deriving motion vector candidates using motion information of neighboring blocks in an affine inter-frame mode according to an embodiment of the present disclosure.

[0310] In the affine candidate list, affine motion is extended from spatially neighboring blocks (extrapolated affine candidates), and the affine candidate list is accompanied by combinations of motion vectors from spatially neighboring blocks (virtual affine candidates). The candidate set is set as follows:

[0311] 1. Derive at most two distinct affine MV prediction itemsets from the affine motion of neighboring blocks. For example... Figure 25 The neighboring blocks A0, A1, B0, B1, and B2 are examined in the example. If the neighboring blocks are encoded using an affine motion model and the corresponding reference frame is the same as the reference frame of the current block, then two control points (for a 4-parameter affine model) or three control points (for a 6-parameter affine model) of the current block are derived from the affine model of the neighboring blocks.

[0312] 2. Figure 26 The example illustrates neighboring blocks used to generate a virtual affine candidate set. The neighboring MVs are divided into three groups: S_0 = {mv_A, mv_B, mv_C}, S_1 = {mv_D, mv_E}, and S_2 = {mv_F, mv_G}. mv_0 is the first MV in S0 that shares the same reference picture as the current block's reference picture. mv_2 is the first MV in S1 that shares the same reference picture as the current block's reference picture.

[0313] Given mv_0 and mv_1, mv_2 can be derived using Equation 9 below.

[0314] [Formula 9]

[0315]

[0316] In Equation 9, the current block size is W×H.

[0317] If only mv_0 and mv_2 are given, mv_1 can be derived using Equation 10 below.

[0318] [Formula 10]

[0319]

[0320] In embodiments of this disclosure, affine inter-frame prediction can be performed in the following order.

[0321] Input: Affine motion parameters, reference image sample

[0322] Output: CU prediction block

[0323] processor

[0324] Derive the size of sub-blocks of an affine block

[0325] -If both the width and height of the sub-block are greater than 4 brightness samples,

[0326] --Relative to each sub-block

[0327] - Derive the motion vectors of the sub-blocks

[0328] - Perform motion compensation on the (called) sub-block based on DCT-IF (1 / 16 pel for luma and 1 / 32 pel for chroma).

[0329] - Otherwise, perform (call) compensation based on the enhanced bilinear interpolation filter on all affine blocks.

[0330] Furthermore, in embodiments of this disclosure, if the merge / jump flag is false and the width and height of the CU are greater than or equal to 8, the affine flag is signaled to indicate whether an affine inter-frame mode will be used at the CU level. If the CU is encoded in affine inter-frame mode, the model flag is signaled to indicate whether a 4-parameter affine model or a 6-parameter affine model is applied to the CU. If the model flag is true, the AF_6_INTER mode (6-parameter affine model) is applied and the MVD is parsed. Otherwise, the AF_4_INTER mode (4-parameter affine model) is applied and both MVDs are parsed.

[0331] In AF_4_INTER mode, similar to affine merging mode, motion vector pairs extrapolated from neighboring blocks encoded by affine mode are generated and preferentially inserted into the candidate list.

[0332] Subsequently, if the size of the candidate list is less than 4, neighboring blocks are used to generate candidates with motion vector pairs {(v_0,v_1)|v0={v_A,v_B,v_c},v_1={v_D,v_E}}. Figure 26 As illustrated, v_0 is selected from the motion vectors of blocks A, B, and C. The motion vectors from neighboring blocks are scaled based on the reference list, the POC used to reference neighboring blocks, the POC used to reference the current CU, and the relationship between the current CU and the current CU. Furthermore, the proximity method for selecting v_1 from neighboring blocks D and E is similar. When the candidate list is greater than 4, the candidates are first sorted based on the consistency of neighboring motion vectors (similar to the consistency of two motion vectors in a candidate pair), and the top 4 candidates are retained.

[0333] If the number of candidates in the candidate list is less than 4, the list is populated by motion vector pairs by copying the AMVP candidates.

[0334] In AF_6_INTER mode, similar to affine merging mode, motion vector triples extrapolated from neighboring blocks encoded in affine merging mode (affine motion vector triples) are generated and preferentially inserted into the candidate list.

[0335] Subsequently, when the size of the candidate list is less than 4, neighboring blocks are used to generate candidates including motion vector pairs {(v_0,v_1,v_2)|v0={v_A,v_B,v_c},v1={v_D,v_E},v2={v_G,v_H}}. Figure 26 As illustrated, v_0 is selected from the motion vectors of blocks A, B, or C. The motion vectors from neighboring blocks are scaled based on the reference list, the POC used to reference neighboring blocks, the POC used to reference the current CU, and the relationship between the current CU and the current CU. Furthermore, the method for selecting v_1 from neighboring blocks D and E is similar to selecting v_2 from F and G. When the candidate list is greater than four, the candidates are sorted based on the consistency of neighboring motion vectors (similar to two motion candidates out of three motion vectors), and the top four candidates are retained.

[0336] When the number of candidates in the candidate list is less than 4, the list is populated by copying the motion vector triples configured for the corresponding AMVP candidates.

[0337] After deriving the CPMV of the current CU, the MVF of the current CU is generated according to Equation 11 for a 4-parameter affine model, and the MVF of the current CU is generated according to Equation 12 for a 6-parameter affine model, based on the number of affine parameters.

[0338] [Equation 11]

[0339]

[0340] [Equation 12]

[0341]

[0342] In this case, the sub-block size M×N is derived in Equation 13, and MvPre is the precision of the motion vector portion (1 / 16).

[0343] [Equation 13]

[0344]

[0345] After deriving from Equation 12, M and N need to be down-adjusted if necessary so that they become factors of w and h. When M or N is less than 8, WIF is applied. Otherwise, sub-block-based affine motion compensation is applied.

[0346] Figure 27 An example of an affine motion vector field of a sub-block unit according to an embodiment of the present disclosure is illustrated.

[0347] Reference Figure 27 To derive the motion vector for each M×N sub-block, calculations are performed according to Equation 11 or Equation 12, such as... Figure 27 The motion vectors illustrated are the motion vectors of the center samples for each sub-block. The SHVC upsampling interpolation filter is used to generate predictions for each sub-block using the derived motion vectors.

[0348] An SHVC upsampling interpolation filter with the same filter length and normalization factor as the HEVC motion-compensated interpolation filter can be used as the motion-compensated interpolation filter for the additional fractional pixel locations. The chroma component motion vector accuracy is 1 / 32 sample. The additional interpolation filter for the 1 / 32 pixel locations is derived by averaging the filters from two neighboring 1 / 16 pixel locations.

[0349] The AF_MERGE mode can be selected on the encoder side using the same method as selecting a common merging mode. A candidate list is generated first, and the candidate with the lowest RD cost is selected for comparison with the RD costs of other inter-frame modes. The result of the comparison determines whether to apply AF_MERGE.

[0350] For the AF_4_INTER mode, an RD cost check is used to determine whether to select motion vector pairs as candidates for the current CU's control point motion vector prediction (CPMVP). After determining the CPMVP of the current affine CU, affine motion estimation is applied, and the control point motion vector (CPMV) is obtained. Therefore, the difference between CPMV and CPMVP is determined.

[0351] On the encoder side, the AF_6_INTER mode is only recognized if the AF_MERGE or AF_4_INTER mode was determined as the optimal mode during the previous mode selection phase.

[0352] In embodiments of this disclosure, the affine inter-frame (affine AMVP) mode can be performed as follows:

[0353] 1) AFFINE_MERGE_IMPROVE: As an alternative to searching for the first neighboring block in affine mode, the improvement is to search for the neighboring block with the largest coding unit size as an affine merging candidate.

[0354] 2) AFFINE_AMVL_IMPROVE: Neighboring blocks in affine mode are added to the affine AMVP candidate list, similar to the common AMVP process.

[0355] The detailed process of generating the affine AMVP candidate list is as follows.

[0356] First, identify if the bottom-left neighboring block uses an affine motion model and has the same reference index as the current reference index. If no neighboring block exists, use the same method to identify the left neighboring block. If no neighboring block exists, identify if the bottom-left neighboring block uses an affine motion model and has a different reference index. If a neighboring block exists, the scaled affine motion vector is added to the reference image list. If no neighboring block exists, use the same method to identify the left neighboring block.

[0357] Secondly, the same method is used to identify the top-right neighboring block, the top neighboring block, and the top-left neighboring block.

[0358] After these processes, if two candidates are found, the process of generating the affine AMVP candidate list terminates. If no two candidates are found, the original operations within the JEM software are executed to generate the affine AMVP candidate list.

[0359] 3) AFFINE_SIX_PARAM: In addition to the 4-parameter affine motion model, a 6-parameter affine motion model is added as an additional model.

[0360] A 6-parameter affine motion model is derived using Equation 14.

[0361] [Formula 14]

[0362]

[0363] Because there are 6 parameters in the motion model, determining the model requires three motion vectors located at the top left (MV_0), top right (MV_1), and bottom left (MV_2). These three motion vectors can be determined using a method similar to that used for the two motion vectors in a 4-parameter affine motion model. The affine model merging is always set to a 6-parameter affine motion model.

[0364] 4) AFFINE_CLIP_REMOVE: Removes motion vector constraints for all affine motion vectors. Performs motion compensation to control the motion vector constraints themselves.

[0365] Affine motion model

[0366] As mentioned above, various affine motion models can be used or considered in affine inter-frame prediction. For example, an affine motion model can represent four motions, such as... Figure 15 The same applies. An affine motion model that can represent the three types of motion (translation, scaling, and rotation) that can be represented by an affine motion model can be called a similar (or simplified) affine motion model. The number of CPMVs and / or the method of deriving the sample / sub-block unit MVs of the current block can vary depending on which affine motion model is used.

[0367] In embodiments of this disclosure, adaptive four- and six-parameter motion models are used. In AF_INTER, in addition to the four-parameter motion model present in JEM, a six-parameter motion model is also proposed. The six-parameter affine motion model is described as in Equation 15.

[0368] [Formula 15]

[0369] x′=a*x+b*y+c

[0370] y′=d*x+e*y+f

[0371] In this case, the coefficients a, b, c, d, e, and f are affine motion parameters. (x, y) and (x', y') are the coordinates of the pixel positions before and after the affine motion model transformation. In video coding, in order to use the affine motion model, if CPMV0, CPMV1, and CPMV2 are the MVs of CP0 (top left), CP1 (top right), and CP2 (bottom left), then Equation 16 can be described as follows.

[0372] [Formula 16]

[0373]

[0374] In this case, CPMV_0 = {v_0x, v_0y}, CPMV_1 = {v_1x, v_1y}, CPMV_2 = {v_2x, v_2y}, and w and h are the width and height of each coded block. Equation 16 is the motion vector field (MVF) of this block.

[0375] The flag is parsed at the CU level to indicate whether a 4-parameter affine motion model or a 6-parameter affine motion model is used when neighboring blocks are encoded as affine predictions. If no neighboring blocks are encoded as affine predictions, the flag is omitted and the 4-parameter model is used for affine prediction. In other words, the 6-parameter model is considered when one or more neighboring blocks are encoded in the affine motion model. Regarding the number of CPMDs, each of the two or three CPMDs is signaled relative to the 4-parameter and 6-parameter affine motion models.

[0376] Furthermore, in embodiments of this disclosure, pattern matching motion vector correction can be used. In JEM's Pattern Matching Motion Vector Derivation (PMMVD, hereinafter referred to as PMVD in the JEM encoder description), the decoder needs to evaluate some motion vectors (MVs) to determine initial MV candidates for CU-level search. In sub-CU-level search, some MV candidates are added in addition to the best CU-level MV. The decoder needs to evaluate these MV candidates to search for the best MV. This requires a large storage bandwidth. In the proposed Pattern Matching Motion Vector Correction (PMVR), the concepts of template matching and bilateral matching from PMVD are adopted in JEM. A PMVR_flag is signaled when a jump mode or merge mode is selected to indicate whether PMVR is available. To meaningfully reduce storage bandwidth requirements compared to PMVD, a list of MV candidates is generated. If PMVR is applied, the initial MV candidate index is explicitly signaled.

[0377] The candidate list is generated using a merged candidate list generation process, but sub-CU merged candidates, such as affine and ATMVP candidates, are excluded. For bilateral matching, only single-predicted MV candidates are included. Double-predicted MV candidates are split into two unit-predicted MV candidates. Furthermore, similar MV candidates (those with an MV difference less than a predefined threshold) are removed. For CU-level search, a diamond search MV correction is performed starting from the signaled MV candidate.

[0378] Sub-CU level search is only available in bilateral matching and merging mode. The search window for sub-CU level search is the same as the search window for CU level search for all sub-CUs. Therefore, no additional bandwidth is required in sub-CU level search.

[0379] Template matching is also used to refine the MVP in a certain mode. In AMVP mode, the HEVC MVP generation process is used to generate two MVPs, and an MVP index is signaled to select one of the two MVPs. Template matching in PMVR is used to further refine the selected MVP. If Adaptive Motion Vector Resolution (AMVR) is applied, the MVP is rounded to the corresponding precision before template matching refinement. This refinement process is called Pattern Matching Motion Vector Prediction Term Refinement (PMVPR). In the remainder of this document, unless otherwise defined, PMVR includes Template Matching PMVR, Bidirectional Matching PMVR, and PMVPR.

[0380] To reduce storage bandwidth requirements, PMVR is not applicable to 4×4, 4×8, and 8×4 CUs. To further reduce the amount of additional storage bandwidth required, the search range for {template matching, bidirectional matching} for CU regions of 64 or more can be reduced to {±2, ±4}. For CU regions greater than 64, the search range for {template matching, bidirectional matching} can be reduced to {±6, ±8}. Compared to the worst-case scenario in HEVC, using all the methods described in the PMVR section of this document, the required storage bandwidth is reduced from 45.9x in JEM-7.0 PMVD to 3.1x in PMVR.

[0381] Affine: A technique used when using affines in non-QT blocks.

[0382] Figure 28 An exemplary method and motion vector for generating prediction blocks under inter-frame prediction with an affine motion model applied are shown, according to embodiments of the present disclosure.

[0383] Reference Figure 28 As can be seen, the equation for deriving motion vectors when applying the affine motion model can be used. Motion vectors can be derived based on Equation 17 below.

[0384] [Equation 17]

[0385]

[0386] In this context, v_x indicates the x-component of the sample unit motion vector of the (x, y) coordinate sample within the current block. v_y indicates the y-component of the sample unit motion vector of the (x, y) coordinate sample within the current block. That is, (v_x, v_y) is the sample unit motion vector of the (x, y) coordinate sample. In this case, a, b, c, d, e, and f indicate parameters of the equation used to derive the sample unit motion vector (motion information) of the (x, y) coordinates from the control point (CP) of the current block. The CP can be represented as a control picture element. These parameters can be derived from the motion information of the CP of each PU sent as a unit. An equation for deriving the sample unit motion vector derived from the motion information of the CP can be applied to each sample of the block, and this equation can be derived as the sample position within a reference image depending on the relative position between the x-axis and y-axis of each sample. The sample unit motion vector can be derived differently depending on the block size, based on the QTBT(TT) block segmentation structure, asymmetric or symmetric type, or block position. This will be described later. Figures 29 to 38 A detailed implementation method is illustrated.

[0387] Figure 29 This is a diagram illustrating a method for performing motion compensation based on motion vectors of control points according to an embodiment of the present disclosure.

[0388] Reference Figure 29 Let's assume the current block is a 2N×2N block for description. For example, the motion vector of the top-left sample within the current block can be described as v_0. Furthermore, using samples from neighboring blocks adjacent to the current block as CPs, the motion vectors of CPs can be v_1 and v_2. That is, if the width and height of the current block are S and the coordinates of the top-left sample position in the current block are (xp, yp), then the coordinates of CP0 of CP can be (xp, yp), the coordinates of CP1 of CP can be (xp+S, yp), and the coordinates of CP2 of CP can be (xp, yp+S). The motion vector of CP0 can be v_0, the motion vector of CP1 can be v_1, and the motion vector of CP2 can be v_2. The motion vectors of the sample cells can be derived using the motion vectors of the CPs. The motion vectors of the sample cells can be derived based on Equation 18 below.

[0389] [Formula 18]

[0390]

[0391] In this context, v_x and v_y indicate the x and y components of the motion vector of the sample with (x, y) coordinates within each current block. v_x0 and v_y0 indicate the x and y components of the motion vector v_0 of CP0, respectively. v_x1 and v_y1 indicate the x and y components of the motion vector v_1 of CP1, respectively. v_x2 and v_y2 indicate the x and y components of the motion vector v_2 of CP2, respectively. The motion vector of each sample within the current block can be derived based on its relative position within the current block using equations such as Equation 1.3.4.4-2 for deriving the motion vector of the sample unit.

[0392] Figure 30 This is a diagram illustrating a method for performing motion compensation based on the motion vector of control points in a non-square block according to an embodiment of the present disclosure.

[0393] Figure 30 An example of a CP divided into N×2N blocks is shown. The equation for deriving the motion vectors of sample cells within the current block can be derived using the same method as for the 2N×2N segmentation type. During the derivation of this equation, a width value suitable for the shape of the current block can be used. To derive the sample cell motion vectors, three CPs can be derived, and their positions can be adjusted as in Equation 1.3.4.4-3. That is, if the width and height of the current block are S / 2 and S respectively, and the coordinates of the upper left sample position of the current block are (xp, yp), then the coordinates of CP0 can be (xp, yp), the coordinates of CP1 can be (xp+S / 2, yp), and the coordinates of CP2 can be (xp, yp+S). The sample cell motion vectors can be derived based on Equation 19 below.

[0394] [Formula 19]

[0395]

[0396] In this case, vx and vy indicate the x and y components of the motion vector of the sample at the (x, y) coordinates within the current block, respectively. v_x0 and v_y0 indicate the x and y components of the motion vector v_0 of CP0, respectively. v_x1 and v_y1 indicate the x and y components of the motion vector v_1 of CP1, respectively. v_x2 and v_y2 indicate the x and y components of the motion vector v_2 of CP2, respectively. Equation 19 illustrates the equation for deriving the motion vector of the sample unit by considering the width of the current block as S / 2. The motion vector of the sample unit can be derived based on the relative position within the current block using equations such as Equations 1.3.4.4.-3, based on the segmentation type N×2N for each sample within the current block segmented from the CU.

[0397] Figure 31This is a diagram illustrating a method for performing motion compensation based on the motion vector of control points in a non-square block according to an embodiment of the present disclosure.

[0398] Figure 31 An example is shown where a block is segmented based on a 2N×N segmentation type. To derive the motion vector of the sample unit, three CPs can be derived. This can be done as follows: Figure 31 The position of the CP can be adjusted in the same way, and it can be based on Figure 31 The shape of the current block illustrated in the example is adjusted to a height of S / 2. That is, if the width and height of the current block are S and S / 2 respectively, and the coordinates of the top-left sample position of the current block are (xp, yp), then the coordinates of CP0 of CP can be (xp, yp), the coordinates of CP1 of CP can be (xp+S, yp), and the coordinates of CP2 of CP can be (xp, yp+S / 2). The sample cell motion vector can be derived as shown in Equation 20 below.

[0399] [Formula 20]

[0400]

[0401] In this case, v_x and v_y indicate the x and y components of the motion vector of the sample at the (x, y) coordinates within the current block, respectively. v_x0 and v_y0 indicate the x and y components of the motion vector v_0 of CP0, respectively. v_x1 and v_y1 indicate the x and y components of the motion vector v_1 of CP1, respectively. v_x2 and v_y2 indicate the x and y components of the motion vector v_2 of CP2, respectively. Equation 20 illustrates the equation for deriving the motion vector of the sample unit by considering the height of the current block as S / 2. The motion vector of the sample unit can be derived based on the relative position within the current block using equations such as Equations 1.3.4.4.-4, based on the segmentation type N×2N for each sample within the current block segmented from the CU.

[0402] Figures 32 to 38 This is a diagram illustrating a method for performing motion compensation based on the motion vector of control points in a non-square block according to an embodiment of the present disclosure.

[0403] Figure 32 This example illustrates the CP of the current asymmetric block. For example... Figure 32 As illustrated, the width and height of the asymmetric current block can be denoted as W and H, respectively. To derive the sample cell motion vector, three CPs can be derived for each current block, and the coordinates of the CPs can be adjusted based on the width and height according to the shape of the current block, as shown in... Figure 32The same applies. That is, if the width and height of the current block are W and H, and the coordinates of the top-left sample position of each current block are (xp, yp), then the coordinates of CP0 of CP can be set to (xp, yp), the coordinates of CP1 of CP can be set to (xp+W, yp), and the coordinates of CP2 of CP can be set to (xp, yp+H). In this case, the motion vector of the sample unit within the current block can be derived based on Equation 21 below.

[0404] [Equation 21]

[0405]

[0406] In this case, v_x and v_y indicate the x-component and y-component of the motion vector of the sample at the (x, y) coordinates within the current block, respectively. v_x0 and v_y0 indicate the x-component and y-component of the motion vector v_0 of CP0, respectively. v_x1 and v_y1 indicate the x-component and y-component of the motion vector v_1 of CP1, respectively. v_x2 and v_y2 indicate the x-component and y-component of the motion vector v_2 of CP2, respectively. Equation 21 illustrates the equation used to derive the motion vector of the sample unit by considering the width and height of the asymmetric current block.

[0407] Furthermore, according to this disclosure, in order to reduce the amount of motion information for a CP indicated by blocks, motion information prediction candidates for at least one CP can be selected based on the motion information of the current block or neighboring samples of the current block. Motion information prediction candidates can be referred to as affine motion information candidates or affine motion vector candidates. Affine motion information candidates can include, for example, in... Figures 33 to 38 The content disclosed in the document.

[0408] ATMVP Simplification

[0409] Advanced Temporal Motion Vector Prediction (ATMVP) is an inter-frame prediction using temporal motion information candidates, but it can also indicate the use of motion information candidates proposed to improve existing TMVPs. In embodiments of this disclosure, ATMVP is not limited to its name, and ATMVP can be referred to as a sub-block temporal merging candidate, a sub-block-based temporal merging candidate, a sub-block temporal motion vector prediction term, a sub-block-based temporal motion vector prediction term, etc.

[0410] Specifically, using the motion vector of colPB located at the bottom right block of the current block or the center of the current block as a candidate for temporal motion information can prevent intra-frame motion from being included in the TMVP. Conversely, if ATMVP is applied, the motion vector of colPB located at the position indicated by the motion vector of a neighboring block can be used as an ATMVP candidate.

[0411] In the example of applying ATMVP, after first searching for the motion vectors of neighboring blocks in the available space while sequentially checking the merge candidate configurations, the position indicated by the time vector in the reference image can be derived as col-PB (ATMVP candidate). Furthermore, the motion vector of the corresponding block can be used in each sub-block unit using the time vector. In this case, if no MV exists in a particular sub-block, the MV of the block located at the center of the corresponding block is used as the MV of the unavailable sub-block and stored as the representative MV.

[0412] In addition, various ATMVP simplification schemes can be proposed that use ATMVP but reduce the amount of storage used.

[0413] In this implementation, memory usage is reduced because the reference images used to find the corresponding block are constrained to be colocalized images (specified in the slice header). In the example, up to four reference images can be used. To find the corresponding block, a temporal vector can be derived from one of the spatial candidates in scan order. If the current candidate has the same reference image as the colocalized image, the search processor can terminate.

[0414] As an example, the ATMVP and STMVP sub-block merging patterns can be modified.

[0415] 1. Add signaling at the image / slice level to enable various sub-block sizes to be used for motion export based on ATMVP / STMVP.

[0416] 2. Simplify motion export based on ATMVP by enabling the use of a common localization image.

[0417] In the implementation, a granular image / slice-level adaptation supporting motion fields derived from ATMVP and STMVP patterns is proposed. Specifically, a default sub-block size for the motion parameters of ATMVP and STMVP in slices referenced to the Sequence Parameter Set (SPS) is signaled in the slice header. Additionally, a flag is signaled in the slice header. If the flag is equal to 0, it indicates that the default sub-block size, as specified in the SPS, is used for ATMVP / STMVP-based motion derivation in the current slice. Otherwise (i.e., the flag is set to 1), another syntax element is signaled in the slice header to specify the ATMVP / STMVP sub-block size used for the slice.

[0418] In this implementation, a simple encoding method is used to determine the ATMVP / STMVP sub-block size of the current image. One of two ATMVP / STMVP sub-block sizes (4 and 8) is selected based on the average size of the ATMVP / STMVP blocks in the last encoded image at the same time level. It is assumed that the image just encoded is the i-th image at the k-th time level and contains N CUs encoded by ATMVP and STMVP patterns. Furthermore, it is assumed that the sizes of these CUs are S0, S1, ..., S. N-1 The average size of the ATMVP / STMVP CU was calculated as follows: Then, when encoding the (i+1)th image in the same k-th time layer, the size of the corresponding ATMVP / STMVP sub-block is determined according to Equation 22 below.

[0419] [Equation 22]

[0420]

[0421] In the implementation, the ATMVP / STMVP sub-block size can always be set to 4 for the first image in each time layer.

[0422] In this implementation, the following two steps can be used to export ATMVP.

[0423] (1) If a neighboring CU is available and its MV is different from those MVs in the existing candidate list, the encoder / decoder can add motion vectors from the spatially neighboring CUs in the order A1, B1, B0, and A0, such as Figure 12 As shown in the diagram, the number of available and unique spatial candidates can be represented as N0;

[0424] (2) The first MV candidate from N0 spatial candidates can be used to determine the colocalization picture and the position of motion of each sub-block from it (by adding displacement according to MV). If N0 equals 0, the colocalization picture signaled in the slice header and the colocalization position with zero motion are used to obtain the motion of each sub-block.

[0425] If multiple reference images are used, the colocalization images for different CUs in ATMVP may not always be the same. Having different colocalization images for ATMVP export for different CUs in the current image may mean needing to acquire the motion field of multiple reference images, which is undesirable due to increased storage bandwidth. Therefore, in this implementation, a simplified design is proposed to use the same colocalization image as that signaled at the slice header in HEVC as the colocalization image for ATMVP export. At the block level, if the reference image of a neighboring block A is different from this colocalization image, the MV of block A can be scaled using the HEVC temporal MV scaling method, and the scaled MV of block A can be used in ATMVP.

[0426] The motion vectors used to obtain the motion field in the co-localized image Rcol can be represented as MVcol. To minimize the impact of MV scaling, in this response, the MVs in the spatial candidate list for deriving MVcol are selected as follows: If the reference image of a candidate MV is a co-localized image, that MV is selected and used as MVcol without any scaling. Otherwise, the MV with the reference image closest to the co-localized image is selected to derive MVcol with scaling.

[0427] Common historical motion vector prediction (HMVP)

[0428] Typically, image compression techniques employ two main approaches to address spatial and temporal redundancy. For example, High Efficiency Video Coding (HEVC) and VVC both use two motion compression schemes based on inter-frame coding: one is motion merging, and the other is Advanced Motion Vector Prediction (AMVP). Various modifications are being discussed to improve these two prediction modes. These modifications include increasing the number of candidates to search for candidates with a more spatially expanded scope and examining temporal candidates at non-traditional locations. Both approaches involve primarily using available candidates to construct the list, minimizing rate distortion (RD) costs, and signaling selected candidates within the bitstream.

[0429] Specifically, in recent image compression techniques, HMVPs (High-Performance Minimum Viable Modules) are discussed, which store motion information of previously encoded blocks and use this stored motion information for motion prediction of subsequently encoded blocks. Such HMVPs can be added to a merge list (or merge candidate list) or an AMVP list (or AMVP candidate list).

[0430] The decoder maintains a lookup table (LUT) operating in the first-in-first-out (FIFO) system (or method) of the HMVP. In this disclosure, the LUT is not limited to its name and may be referred to as a table, HMVP table, HMVP candidate table, buffer, HMVP buffer, HMVP candidate buffer, HMVP list, or HMVP candidate list. Specifically, when a non-affine prediction unit (PU) (or coding unit (CU)) is decoded, the corresponding motion information is stored in the LUT. The decoder performs decoding on the next PU. In this case, the stored motion information may include motion vectors in the x (horizontal) and y (vertical) directions, reference index information, and pattern information.

[0431] The decoder maintains a LUT containing motion information of the progressively decoded non-affine candidates. The size of the LUT can be limited to a predefined S candidates. In one implementation, the LUT can be reset at the beginning of a slice, the beginning of a CTU row, or the beginning of a CTU.

[0432] HMVP can apply both merge and AMVP modes. The merge list can have B candidates, and the AMVP list can have two candidates. In conventional image compression techniques, the merge list is configured with the following candidates: i) spatial candidates, ii) temporal candidates, iii) bidirectional prediction (Bi-Pred) candidates, and iv) zero-motion candidates. Recently, methods have been discussed that additionally consider Advanced Motion Vector Prediction (ATMVP) as a candidate. For example, ATMVP candidates can be inserted into the merge list before temporal candidates. Candidates in the merge list are added until they reach the maximum merge list size. Duplicate candidates may not be added to the merge list. Two candidates can be inserted into the AMVP list. For example, the first candidate can be selected from the available spatial candidates, and the second candidate can be selected from the temporal candidates. If the list is not full, a zero-motion vector candidate can be added.

[0433] HMVP is applied based on the FIFO order, in which candidates leave the LUT in the same order as their input.

[0434] In one implementation, when HMVP is applied to a merge list configuration, an HMVP candidate can be inserted (or added) to the third position of the list as follows:

[0435] 1. Spatial Candidates

[0436] 2. Time Candidates

[0437] 3. LUT has up to S HMVP candidates

[0438] 4. Combined dual prediction candidates

[0439] 5. Zero motion vector candidates

[0440] In one implementation, when HMVP is applied to a merge list configuration, the HMVP candidate can be inserted in the third position after the time candidate as follows:

[0441] 1. Spatial Candidates

[0442] 2. Time Candidates

[0443] 3. Up to K HMVP candidates

[0444] 4. Candidates for zero motion vectors

[0445] Figure 39 This is a flowchart describing a method for storing an HMVP according to an embodiment of the present disclosure.

[0446] Reference Figure 39 The decoder decodes the current PU (or CU) (S3901).

[0447] The decoder checks whether the current PU is a block encoded in non-affine mode (S3902). To facilitate the use of HMVP candidates, if the current PU is a block encoded in affine mode, the decoder does not store the motion information of the current PU in the table.

[0448] If the current PU is a block encoded in non-affine mode, the decoder stores the motion information of the current PU in the table (or updates it) (S3903).

[0449] In embodiments of this disclosure, two methods (i.e., an unconstrained FIFO method and an ii. constrained FIFO method) can be used to update the HMVP table. In the former, redundant motion information may exist, but no pruning process is applied. This helps reduce the complexity of the overall process. This is described with reference to the following figure.

[0450] Figure 40 This is a diagram illustrating an HMVP table operating in an unconstrained FIFO mode according to an embodiment of the present disclosure.

[0451] Reference Figure 40 Candidates added to the table are appended to the end (right side) of the table. Conversely, candidates that leave the table according to the FIFO method are placed at the front of the table (left side, the oldest candidate).

[0452] If the table is not fully populated with the maximum number of predefined candidates at index L-1 (i.e., the end), new candidates are added without removing existing ones. Conversely, if the table is fully populated, i.e., if the maximum number of candidates is met, candidates at the front (i.e., the oldest in the table) are removed and new candidates are added.

[0453] Figure 41 This is a diagram illustrating an HMVP table operating in constrained FIFO mode according to an embodiment of the present disclosure.

[0454] Reference Figure 41 When using a constrained FIFO, pruning is performed if adding a new candidate results in any redundancy. In this implementation, if a candidate with redundant motion information exists in the table, the redundant candidate is removed from the table, and the motion information of the current candidate can be added.

[0455] Implementation Method 1

[0456] Regarding HMVP candidates, in many cases, the latest historical MV may overlap with the motion information of spatial candidates (or spatially neighboring candidates). Therefore, this implementation proposes a method to set the addition order of HMVP candidates differently from the HMVP LUT index order when HMVP candidates are added to the AMVP or merge list.

[0457] According to embodiments of this disclosure, the candidate list can be efficiently configured by adaptively adjusting the HMVP candidates. Therefore, the number of signaling bins used for binarization can be reduced, and coding efficiency can be improved.

[0458] That is, HMVP candidates added to the merge list or AMVP list are not restricted by their index in the HMVP list. As an implementation method, Table 2 below illustrates a method for changing the order in which HMVP candidates are added to the AMVP or merge list.

[0459] [Table 2]

[0460]

[0461] Referring to Table 2, as mentioned above, it is highly likely that the most recently inserted HMVP candidate may have the same motion information as the spatial candidate. Therefore, the order in which HMVP candidates are added can be pre-defined by considering probability without regard to the HMVP index.

[0462] Furthermore, in one implementation, the encoder or decoder can add HMVP candidates to the merge list or AMVP list starting from the nth candidate in the list. Table 3 below illustrates the changed order in which candidates are added to the AMVP or merge list.

[0463] [Table 3]

[0464]

[0465] Referring to Table 3, HMVP candidates can be added to the merge list or AMVP list starting from the second index.

[0466] In one implementation, information about the order in which HMVP candidates are added within the table (LUT) can be signaled from the encoder to the decoder. For example, such order information can be sent via a High-Level Syntax (HLS). The HLS could be, for example, a sequence parameter set, a picture parameter set, a slice header, a coding tree unit, a coding unit, and / or another suitable syntax header.

[0467] Table 4 illustrates high-level syntactic structures for which the methods proposed in this disclosure can be applied.

[0468] [Table 4]

[0469]

[0470] Referring to Table 4, a set_HMVP_order_flag value of 1 indicates that set_HMVP_order_flag exists in the slice header of non-IDR images in CVS. A set_HMVP_order_flag value of 0 indicates that set_HMVP_order_flag does not exist in the slice header and adaptive HMVP is not used in VCS.

[0471] Table 5 illustrates the slice header syntax structures for which the methods proposed in this disclosure can be applied.

[0472] [Table 5]

[0473]

[0474] Referring to Table 5, slice_HMVP_idx refers to the index of the candidate sequence used. For example, slice_HMVP_idx equal to 0 can represent a basic HMVP sequence such as 0, 1, 2, 3. Similarly, the index value 1 can be used to represent an HMVP sequence of 3, 2, 1, 0.

[0475] Implementation Method 2

[0476] In embodiments of this disclosure, in addition to the HMVP LUT, a method for motion prediction using a long-term list is proposed. Therefore, the number of HMVP candidates maintained can be increased. In embodiments, a 2-HMVP table can be considered. In this case, one table can be used to store general HMVP candidates, while the other can be used as a long-term list containing candidates that need to be further maintained.

[0477] The following describes how to reset and construct a long-term list (or a long-term HMVP list).

[0478] - After decoding the first CTU in a CTU line, one or more historical MVs of subsequent CTUs can be added to a long-term HMVP LUT. This long-term HMVP LUT can be left unused or updated until the next CTU line.

[0479] - At the beginning of the next CTU line, a long-term HMVP LUT can be used to reset the public HMVP LUT. The reason for doing this is that the HMVP candidate of the CTU at the beginning of the CTU line may be more relevant than the historical MV at the end of the previous CTU line.

[0480] - The above process can be repeated.

[0481] Figure 42 This is a diagram illustrating an HMVP LUT of a long-term HMVP LUT according to an embodiment of the present disclosure.

[0482] Reference Figure 42 There can be two LUTs. One can be an HMVP LUT (or a general HMVP LUT or a short-term HMVP LUT), and the other can be a long-term HMVP LUT. When an HMVP candidate is added to both the merge list and the AMVP list, it can be added from either the HMVP LUT or the long-term LUT, as shown below. Figure 30 exemplified in .

[0483] The use of long-term LUTs can be indicated through high-level syntax by adopting new syntax elements. For example, syntax elements can exist as sequence parameter sets, image parameter sets, slice headers, encoding tree units, encoding units, and / or another appropriate syntax header.

[0484] Implementation Method 3

[0485] In embodiments of this disclosure, a method is proposed that considers the flexibility for decoding when adding HMVP candidates to an HMVP LUT. The encoder / decoder may consider decisions for one or more characteristics of the PU (or CU).

[0486] In an implementation, the encoder / decoder may consider the following when adding HMVP candidates to the table. The encoder / decoder may add candidates to the table by considering characteristics such as the PU's mode (e.g., merge mode, affine mode, or AMVP mode) and / or the block size, either individually or in combination. In one implementation, other characteristics may be considered in addition to these. For example, the merge type (e.g., spatial candidate or temporal candidate) considering HMVP LUT updates, whether it is a sub-PU, etc., may be considered as criteria for candidate selection. Selection criteria can be determined to reduce redundancy with previous history (or previous HMVPs). For example, if a PU is encoded in a merge mode and the merge type is spatial merge, the decoder may update the HMVP LUT without the corresponding PU's motion information.

[0487] Figure 43 This is a diagram illustrating an example of a method for updating an HMVP LUT according to an embodiment of the present disclosure.

[0488] Reference Figure 43 The encoder / decoder obtains the encoded candidate motion information (S4301).

[0489] The encoder / decoder evaluates whether to update the LUT with candidate motion information based on predefined decision criteria (S4302). The decision criteria may include one or more characteristics related to the candidate pattern (e.g., merge pattern, affine pattern, or AMVP pattern), the candidate block size, and / or the candidate merge type.

[0490] The encoder / decoder updates the LUT (S4303) based on decision criteria. That is, if a candidate satisfies a predefined decision criterion, the encoder / decoder can add the candidate's motion information to the LUT.

[0491] Implementation Method 4

[0492] In embodiments of this disclosure, constraints are proposed on redundancy checks performed to add HMVP candidates to the merge list (or AMVP list). These constraints on redundancy checks can be implemented in various ways.

[0493] In one implementation, the encoder / decoder can limit the number of pruning checks for a first specific number of candidates in the merge list. Alternatively, the encoder / decoder can limit the number of pruning checks on candidates from the first candidate in the merge list to specific candidates. For example, the encoder / decoder can perform a pruning process on candidates from the first candidate in the merge list to specific candidates. Furthermore, the number of HMVP candidates (i.e., the targets of the pruning checks) can be limited to a predetermined number.

[0494] Furthermore, in one implementation, the encoder / decoder can limit the pruning check by performing a pruning check on specific types of merge candidates within the merge list. For example, the encoder / decoder can perform a pruning check only on spatial candidates in the merge list when adding an HMVP candidate. Alternatively, for example, the encoder / decoder can perform a pruning check only on some spatial candidates in the merge list when adding an HMVP candidate. Some of the spatial candidates may be predefined. For example, predefined spatial candidates may be at least one of left neighbor spatial candidates and / or top neighbor spatial candidates. Implementations of this disclosure are not limited thereto, and other types may be combined and limited to be the targets of the pruning check.

[0495] Figure 44 This is a diagram illustrating a method for limiting the number of HMVP candidates (i.e., the target of pruning checks) according to an embodiment of this disclosure.

[0496] Reference Figure 44 In embodiments of this disclosure, the number of HMVP candidates (i.e., the targets of the pruning check) may be limited to M.

[0497] The encoder / decoder can check for redundancy in motion information between the first M candidates in the HMVP LUT and the merge candidates in the merge list when constructing the merge list using HMVP candidates.

[0498] Alternatively, the encoder / decoder can check for redundancy in motion information between the top M candidates in the HMVP LUT and the currently decoded PU when adding motion information of the decoded processing block (e.g., PU) to the HMVP LUT.

[0499] Figure 45 This is a flowchart illustrating an example of a method for performing a trimming check according to an embodiment of the present disclosure.

[0500] Reference Figure 45 The encoder / decoder obtains the decoded candidate motion information and determines (or decodes) the number of trim checks (S4501, S4502).

[0501] The encoder / decoder performs a trim check based on the determined number of trim checks (S4503).

[0502] In one implementation, as shown in Tables 4 and 5, information related to pruning checks can be signaled via high-level syntax. In this case, syntax elements sent from the encoder to the decoder can be included in a special header to indicate the number of pruning checks to be performed. For example, high-level syntax can be included in sequence parameter sets, image parameter sets, slice headers, coding tree units, coding units, and / or other appropriate syntax data headers.

[0503] Implementation Method 5

[0504] In embodiments of this disclosure, an efficient method for selecting HMVP candidates is proposed. When historical motion vector candidates (i.e., HMVP candidates) are inserted into a merge list (or AMVP list), a pruning check can be performed to ensure that the HMVP candidates do not overlap with existing merge lists.

[0505] To perform a total redundancy check between a merge list of size M and a historical LUT of size N, (M-1)×N checks must be performed. Therefore, in embodiments of this disclosure, the number of HMVP candidates can depend on the existing merge candidates. For example, the number of HMVP candidates can depend on the number of spatial candidates present in the merge list. Alternatively, for example, the number of HMVP candidates can depend on the number of both spatial and temporal candidates present in the merge list.

[0506] If another merge candidate exists in the merge list, the number of HMVP candidates for which pruning checks will be performed based on specific criteria (or rules) that are based on the number of merge candidates and / or the number of HVVPs in the merge list can be reduced. Therefore, the number of redundant checks in the worst case can be reduced.

[0507] For example, in the case of a merge list of size (or length) 6, if the merge list is completely filled, it can include up to 5 spaces or other merge candidates. In the worst case, it may be necessary to perform 30 redundancy checks to insert an HMVP candidate into a list of 6 HMVPs.

[0508] In one implementation, examples relating to the constraint on the number of HMVPs to be checked are illustrated in Equation 23 and Table 6.

[0509] [Equation 23]

[0510] if (existing_candidates>=3)

[0511] number_hist_to_check=7-existing_candidates

[0512] [Table 6]

[0513] Existing candidate # Existing candidates to be checked # Historical MVs to be checked # The # of the inspection 1 1 6 6 2 2 6 12 3 3 4 12 4 4 3 12 5 5 2 10

[0514] Referring to Table 6, by limiting the number of HMVPs (i.e., the targets of pruning checks) to 2, the number of redundant checks performed to add HMVPs in the worst case is reduced by 12 times instead of 30 times.

[0515] Implementation Method 6

[0516] In embodiments of this disclosure, a method is proposed for constructing a merge list using historical temporal motion vector prediction (H-STMVP). The H-STMVP indicator is derived as a candidate derived from the average of two historical spatial MVPs and TMVPs. Two spatial HMVPs can be obtained from the HMVP buffer. The TMVP can be obtained from the current merge list. In this case, the spatial candidates can be candidates obtained from the last two encoded MVs in the decoded sequence preceding the current block.

[0517] For example, the last encoded MV (referred to as MV_L in this disclosure), the penultimate MV (referred to as MV_(L-1) in this disclosure), and MV_TMVP can be used to generate H-STMVP candidates that will be inserted into the merge list.

[0518] If all three candidates are available, the MV to be added to the merge list can be calculated using Equation 24 below.

[0519] [Equation 24]

[0520] {MV L +MV L-1 +MV TMVP}*43 / 128

[0521] In one implementation, if only two of the three candidates are available, then only those two candidates can be averaged to generate the H-STMVP. Similarly, if only one candidate is available, then that single candidate can be used. If no candidates are available, the H-STMVP is not used for the merge list configuration.

[0522] In embodiments of this disclosure, a method is proposed to obtain motion vectors for H-STMVP candidates using a method other than Equation 24.

[0523] For example, as an alternative to averaging three or more candidates at once, averaging the candidates in the space first, and then using the result of averaging two candidates, can simplify the computation. An example of this is illustrated in the following formula.

[0524] [Equation 25]

[0525] <[(MV L +MV L-1 )>>1+MV TMVP ]>>1>

[0526] Alternatively, the average value can be obtained as follows.

[0527] [Equation 26]

[0528] <[(MV L +MV TMVP )>>1+MV L-1 ]>>1>

[0529] [Equation 27]

[0530] <[(MV L-1 +MV TMVP )>>1+MV L ]>>1>

[0531] [Equation 28]

[0532] <[2MV L +MV L-1 +MV TMVP ]>>2>

[0533] The encoder / decoder can first average the two candidates as in Equations 25 through 27, and then average the resulting values ​​using a third candidate. Alternatively, the encoder / decoder can assign higher importance / weight to the candidate (i.e., MV_L) by applying two shift operations as in Equation 28. The average value can be derived using Equations 25 through 28 using only shift operations without division.

[0534] Implementation Method 7

[0535] In embodiments of this disclosure, a method is proposed for using a given number of spatial candidates instead of two history-based spatial candidates when deriving H-STMVP. The n candidates do not need to be an inherently continuous decoding sequence. The n candidates can be selected randomly or according to a certain rule.

[0536] Therefore, a more general method (as in Equation 29 below) can be used to represent Equation 24 above.

[0537] [Equation 29]

[0538] {MV1+MV2+...+MV n +MV TMvP )*1 / (n+1)

[0539] In another implementation, assuming the use of 5 spatial candidates, the impact of increasing the number of spatial candidates on the generation of H-STMVP candidates can be minimized, and spatial and temporal candidates can be appropriately incorporated by increasing the weight applied to the temporal candidates.

[0540] Therefore, after averaging the spatial candidates together using Equation 30, the above objective can be achieved by averaging the averaging result over MV_TMVP.

[0541] [Formula 30]

[0542] [{MV1+MV2++MV n}*1 / n+MV TMVP I*1 / 2

[0543] Implementation Method 8

[0544] In embodiments of this disclosure, a method is proposed for adding weights (or weighting factors) to motion vector candidates used to derive H-STMVP. In this case, the weights can be determined empirically, or by considering the time distance up to a fixed reference frame, or by considering their position in a history table. For example, a new candidate may have a heavier weight than a previous candidate.

[0545] That is, in this embodiment, the above-mentioned formula 24 can be represented as formula 31 below.

[0546] [Equation 31]

[0547] {MV=w1·MV L +w2·MV L-1 +w3·MV TMvP}

[0548]

[0549] In this case, the weights can have the same value or unevenly distributed values.

[0550] Implementation Method 9

[0551] In embodiments of this disclosure, a method is proposed for scaling motion vectors used to derive H-STMVP candidates as a single reference image.

[0552] Figure 46 This is a diagram illustrating a method for describing H-STMVP candidates using motion vectors referenced to different reference images according to embodiments of this disclosure.

[0553] Reference Figure 46Assume that the candidate references (or indicators) for MV_L, MV_L-1, and MV_TMVP are different reference images. That is, Figure 46 An example is given that candidates used to generate H-STMVP candidates can have different reference indices and therefore different reference frames.

[0554] The average values ​​of Equations 24 to 31 can be unequal because frames with closely spaced reference frames actually have a greater impact on the motion vectors of H-STMVP. Therefore, a method is proposed to scale all motion vectors to a single reference frame for equal comparison and merging.

[0555] In this scenario, the encoder can determine which single frame, executed as part of the RD optimization, is most suitable as a reference frame. In implementations, the selected reference frame can be signaled in a slice header similar to the TMVP array index present in the slice header. For example, fixed rules can be used to generate the reference frames to be used. Alternatively, for example, the list can be scaled as the first available reference frame in L0, or the list can be scaled based on the current image sequence count.

[0556] In one implementation, to achieve the above objectives, the encoder may send information about a single fixed image to the decoder using a high-level syntax (HLS), which may be a sequence parameter set, a picture parameter set, a slice header, a coding tree unit, and / or a portion of another data header. For example, a high-level syntax structure such as Table 6 and / or Table 7 below may be defined.

[0557] [Table 7]

[0558]

[0559] Referring to Table 7, a value of 1 for `set_HSTMVP_ref_pic_flag` indicates that `set_HSTMVP_idx` exists in the slice header of a non-IDR image in CVS. A value of 0 for `set_HSTMVP_ref_pic_flag` indicates that `set_HSTMVP_idx` does not exist in the slice header.

[0560] [Table 8]

[0561]

[0562] Referring to Table 8, slice_HMVP_idx specifies the reference index. In one implementation, the reference index can be selected relative to list L0.

[0563] Implementation Method 10

[0564] In the embodiments of this disclosure, more detailed embodiments will be described relative to the embodiments described above. Specifically, a method is proposed for indirectly using affine HMVP candidates by using position and size information to calculate or derive the CPMV of the current block. In this disclosure, the derived CPMV may be referred to as an inherited affine HMVP candidate. The inherited affine HMVP candidate according to the embodiments of this disclosure can be used in the above-described affine merge list and / or affine AMVP list generation process.

[0565] Figure 47 This is a diagram illustrating the location of a block for deriving an inherited affine HMVP candidate according to an embodiment of the present disclosure.

[0566] Reference Figure 47 The CPMV of the current block 4701 can be derived based on the position and size of the affine HMVP candidate in a manner similar to that used to derive the common inherited CPMV from neighboring blocks. That is, the encoder / decoder can derive the motion vector of the control point of the current block 4701 based on the position and size (e.g., width and height) information of the reference block 4702, which serves as an affine HMVP candidate.

[0567] As an implementation, the CPMV of the affine HMVP inherited by the current block can be derived by using Equations 32 and 33 below.

[0568] [Equation 32]

[0569] V 0x =V B0x +(V B2_x -V B0x )*(posCurCU_Y-posRefCU_Y) / RefCU_height+(V B1x -V B0x )*(posCurCU_X-posRefCU_X) / RefCU_width

[0570] [Equation 33]

[0571] V 0y =V B0y +(V B2_y -V B0y )*(posCurCU_Y-posRefCU_Y) / RefCU_height+(V B1y -V B0y )*(posCurCU_X-posRefCU_X) / RefCU_width

[0572] In Equations 32 and 33, posCurCU_Y represents the vertical coordinate value of the top-left sample of the current block 4701, and posRefCU_Y represents the vertical coordinate value of the top-left sample of the reference block 4702. posCurCU_X represents the horizontal coordinate value of the top-left sample of the current block 4701, and posRefCU_X represents the horizontal coordinate value of the top-left sample of the reference block 4702. RefCU_height represents the height of the reference block 4702, and RefCU_width represents the width of the reference block 4702.

[0573] Implementation Method 11

[0574] In one embodiment of this disclosure, when adding affine HMVP candidates (direct or inherited HMVPs), constraints can be added to select affine HMVP candidates that can be used to generate affine merge lists or affine AMVP lists.

[0575] As an example, an affine HMVP candidate can only be added to the affine merge list or the affine AMVP list if it is adjacent to the current block.

[0576] As another example, an affine HMVP candidate can only be added to the affine merge list or the affine AMVP list if it is located (or exists) within a specific distance from the current block. For example, the specific distance could be a predefined pixel distance. The encoder / decoder can determine whether an affine HMVP candidate is within the predetermined specific distance to determine if the affine HMVP candidate is available.

[0577] As another example, an affine HMVP candidate can only be added to the affine merge list or the affine AMVP list if it is located (or exists) in a specific position based on the current block. For example, when an affine HMVP candidate exists in a specific position, it is possible that the affine HMVP candidate is the left or top neighbor block of the current block.

[0578] For an affine HMVP LUT with N elements, the above verification process can be performed on all elements or the first M elements until merging, the AMVP list is full, or a predefined number of HMVP candidates are reached.

[0579] Implementation Method 12

[0580] In one embodiment of this disclosure, a method is proposed for using affine HMVP candidates to replace inherited affine candidates that already exist in the affine merge list and / or the affine AMVP list.

[0581] Figure 48This is a diagram illustrating an affine merge list or affine AMVP list according to an embodiment of the present disclosure.

[0582] Reference Figure 48 The encoder / decoder can replace inherited candidates existing in the affine merge list or affine AMVP list with inherited affine HMVP candidates. In other words, when applying a sub-block-based merge pattern to the current block, the encoder / decoder can use inherited affine candidates and already constructed affine candidates to generate a sub-block-based merge candidate list, derive inherited affine HMVP candidates, and replace at least one inherited affine candidate included in the sub-block-based merge candidate list with the inherited affine HMVP candidates.

[0583] Additionally, in one embodiment of this disclosure, the affine HMVP lookup table (LUT) can be initialized at the slice, CTU row, or the beginning of the CTU. This improves the performance of parallel processing.

[0584] In the following embodiments, a method for reducing the number of worst-case pruning checks for HMVP is proposed.

[0585] Implementation Method 13

[0586] In embodiments of this disclosure, when an HMVP candidate is added to the merge list, the number of pruning checks can be determined based on the number of available candidates in the merge list and the number of HMVP candidates that can be added to the merge list. Hereinafter, for ease of description, variables are defined as follows in describing embodiments of this disclosure.

[0587] -NST: Number of available (or existing) candidates in the merge list

[0588] -NHMVP: The number of HMVP candidates in the table (i.e., the size of the HMVP table).

[0589] -NmrgToBeAdded: The number of HMVP candidates added to the merge list.

[0590] -NHMVPChecked: Number of HMVP candidates that have undergone pruning and checking.

[0591] -Nmax_HMVP_prunning: The worst-case number of pruning checks required to add HMVP candidates to the merge list.

[0592] In one embodiment of this disclosure, HMVP candidates can be added to the merge list based on the following criteria.

[0593] - First condition: When the LUT was previously pruned (i.e., there are no identical mv values ​​among the candidates in the HMVP LUT).

[0594] -Second condition: When the size of the HMVP LUT table is 6

[0595] - Third condition: When the maximum number of available (or existing) merge candidates to be added to the merge list is 4. That is, this is when the number of merge candidates in the merge list is less than the value obtained by subtracting 1 from the maximum merge list size (or the maximum number of merge candidates). For example, the maximum merge list size can be 6, and an HMVP candidate can be added (or inserted) if the number of currently available merge candidates is less than 5. In other words, an HMVP candidate can only be added at most to merge list index 5.

[0596] When an HMVP candidate is added to the merge list (i.e., becomes a merge candidate), each HMVP candidate may require a pruning check to remove duplicates between merge candidates. Based on existing image compression techniques, the worst-case number of pruning checks required to add an HMVP to the merge list can be calculated as shown in Table 8 below.

[0597] [Table 9]

[0598] <![CDATA[N ST ]]> <![CDATA[N mrgToBeAdded ]]> <![CDATA[N HMVPChecked ]]> <![CDATA[N max_hmvp_prumning ]]> 0 5 6 0 1 4 6 4 2 3 6 7 3 2 6 9 4 1 6 10

[0599] Referring to Table 9, based on existing image compression techniques, a trimming check can be performed on the six HMVP candidates in the HMVP table (or HMVP list, HMVP candidate list).

[0600] Specifically, 1) When there is one candidate in the merge list, the number of HMVP candidates added to the merge list can be 4. Then, a pruning check can be performed on all six HMVP candidates. In this case, the worst-case number of pruning checks can be 4. 2) When there are two candidates in the merge list, the number of HMVP candidates added to the merge list can be 3. Then, a pruning check can be performed on all six HMVP candidates. In this case, the worst-case number of pruning checks can be 7. 3) When there are three candidates in the merge list, the number of HMVP candidates added to the merge list can be 2. Then, a pruning check can be performed on all six HMVP candidates. In this case, the worst-case number of pruning checks can be 9. 4) When there are four candidates in the merge list, the number of HMVP candidates added to the merge list can be 1. Then, a pruning check can be performed on all six HMVP candidates. In this case, the worst-case number of pruning checks can be 10.

[0601] Implementation Method 14

[0602] In embodiments of this disclosure, a method for reducing the number of worst-case pruning checks is proposed. If there are more merge candidates in the merge list, the encoding performance of HMVP decreases as the number of merge candidates (i.e., non-HMVP candidates) increases, so it may be necessary to reduce the number of HMVP candidates to be pruned. Therefore, in embodiments of this disclosure, the encoder / decoder can set the number of HMVP candidates to be checked (NHMVPChecked) to be equal to the number of available HMVP candidates to be added (NmrgToBeAdded) in order to reduce the number of worst-case pruning checks. In this case, the number of worst-case pruning checks can be calculated as shown in Table 10 below.

[0603] [Table 10]

[0604] <![CDATA[N ST ]]> <![CDATA[N mrgToBeAdded ]]> <![CDATA[N HMVPChecked ]]> <![CDATA[N max_hmvp_prunning ]]> 0 5 5 0 1 4 4 4 2 3 3 6 3 2 2 6 4 1 1 4

[0605] Referring to Table 10, compared with conventional image compression techniques, HMVP can reduce the number of worst-case trim checks from 10 to 6.

[0606] Referring to Table 10, in the implementation: 1) When there is one candidate in the merge list, the number of HMVP candidates added to the merge list can be 4. Then, a pruning check can be performed on all four HMVP candidates. In this case, the worst-case number of pruning checks can be 4. 2) When there are two candidates in the merge list, the number of HMVP candidates added to the merge list can be 3. Then, a pruning check can be performed on all three HMVP candidates. In this case, the worst-case number of pruning checks can be 6. 3) When there are three candidates in the merge list, the number of HMVP candidates added to the merge list can be 2. Then, a pruning check can be performed on both HMVP candidates. In this case, the worst-case number of pruning checks can be 6. 4) When there are four candidates in the merge list, the number of HMVP candidates added to the merge list can be 1. Then, a pruning check can be performed on one HMVP candidate. In this case, the worst-case number of pruning checks can be 4.

[0607] Implementation Method 15

[0608] In embodiments of this disclosure, to reduce the number of worst-case pruning checks, the encoder / decoder can set the number of HMVP candidates to be pruned (NHMVPChecked) to be the same as the sum of the number of available HMVP candidates to be added (NmrgToBeAdded) and k. Here, k represents a predefined constant value. As an example, when K is 1, the number of worst-case pruning checks can be calculated as shown in Table 10.

[0609] [Table 11]

[0610] <![CDATA[N ST ]]> <![CDATA[N mrgToBeAdded ]]> <![CDATA[N HMVPChecked ]]> <![CDATA[N max_hmwp_prunning ]]> 0 5 6 0 1 4 5 4 2 3 4 7 3 2 3 8 4 1 2 7

[0611] Referring to Table 11, in the implementation: 1) When there is one candidate in the merge list, the number of HMVP candidates added to the merge list can be 4. Then, a pruning check can be performed on five HMVP candidates. In this case, the worst-case number of pruning checks can be 4. 2) When there are two candidates in the merge list, the number of HMVP candidates added to the merge list can be 3. Then, a pruning check can be performed on four HMVP candidates. In this case, the worst-case number of pruning checks can be 7. 3) When there are three candidates in the merge list, the number of HMVP candidates added to the merge list can be 2. Then, a pruning check can be performed on three HMVP candidates. In this case, the worst-case number of pruning checks can be 8. 4) When there are four candidates in the merge list, the number of HMVP candidates added to the merge list can be 1. Then, a pruning check can be performed on two HMVP candidates. In this case, the worst-case number of pruning checks can be 7.

[0612] Implementation Method 16

[0613] In embodiments of this disclosure, in order to reduce pruning checks in the worst case, the number of HMVP candidates to be checked (NHMVPChecked) can be limited to the following formula 34.

[0614] [Formula 34]

[0615] N HMVPChecked =min(N) HMVP ,C*N mrgToBeAdded )

[0616] In Equation 34, C represents a predefined constant value. If C is 2, the worst-case pruning check count can be calculated as shown in Table 12.

[0617] [Table 12]

[0618] <![CDATA[N ST ]]> <![CDATA[N mrgToBeAdded ]]> <![CDATA[N HMVPChecked ]]> <![CDATA[N max_hmvp_prunning ]]> 0 5 6 0 1 4 6 4 2 3 6 7 3 2 4 9 4 1 2 7

[0619] Referring to Table 12, in the implementation: 1) When there is one candidate in the merge list, the number of HMVP candidates added to the merge list can be 4. Then, a pruning check can be performed on the six HMVP candidates. In this case, the worst-case number of pruning checks can be 4. 2) When there are two candidates in the merge list, the number of HMVP candidates added to the merge list can be 3. Then, a pruning check can be performed on the six HMVP candidates. In this case, the worst-case number of pruning checks can be 7. 3) When there are three candidates in the merge list, the number of HMVP candidates added to the merge list can be 2. Then, a pruning check can be performed on the four HMVP candidates. In this case, the worst-case number of pruning checks can be 9. 4) When there are four candidates in the merge list, the number of HMVP candidates added to the merge list can be 1. Then, a pruning check can be performed on the two HMVP candidates. In this case, the worst-case number of pruning checks can be 7.

[0620] For ease of description, the above-mentioned embodiments of this disclosure can be divided and described, but this disclosure is not limited thereto. That is, the above-described embodiments 1 to 12 can be performed independently, or they can be combined and performed as one or more embodiments.

[0621] Figure 49 This is a flowchart illustrating a method for processing video signals based on inter-frame prediction according to an embodiment of the present disclosure.

[0622] refer to Figure 49 For ease of description, the decoder has been described substantially, but this disclosure is not limited thereto. The method for processing video signals based on inter-frame prediction according to embodiments of this disclosure can be performed identically in both the encoder and the decoder.

[0623] The decoder configures the merge list or merge candidate list based on the neighboring blocks of the current block (S4901).

[0624] When the number of merge candidates included in the merge list is less than a predetermined first maximum, the decoder adds the history-based merge candidates from the history-based merge candidate list to the merge list (S4902). Here, the history-based merge candidates represent the motion information of blocks encoded before the current block.

[0625] The decoder obtains the merge index (S4903) indicating the merge candidate in the merge list for inter-frame prediction of the current block.

[0626] The decoder generates the prediction block for the current block based on the motion information of the merging candidates indicated by the merging index (S4904).

[0627] As described above, as an implementation, the step of adding history-based merge candidates to the merge list may include the following steps: checking whether a predefined second number of history-based merge candidates in the history-based merge candidate list overlap with merge candidates included in the merge list in their motion information.

[0628] As described above, as an implementation, if a history-based merge candidate has motion information that does not overlap with the motion information of a predefined merge candidate included in the merge list, then a second number of history-based merge candidates can be added to the merge list.

[0629] As described above, as an implementation, if a history-based merge candidate has motion information that does not overlap with the motion information of a predefined third number of merge candidates included in the merge list, then a second number of history-based merge candidates can be added to the merge list.

[0630] As described above, as an implementation, if a historically based merging candidate has motion information that does not overlap with the motion information of a specific spatial merging candidate included in the merging list, then a second number of historically based merging candidates can be added to the merging candidate list.

[0631] As described above, as an implementation method, the first specific number can be limited to a value obtained by subtracting 1 from the largest merge candidate.

[0632] As described above, as an implementation, the step of adding history-based merge candidates to the merge list may include the following steps: when three current merge candidates are included in the merge list, check whether two history-based merge candidates have motion information that overlaps with the merge candidates included in the merge list.

[0633] Figure 50 An example block diagram of an apparatus for processing video signals according to an embodiment of the present disclosure is shown. Figure 50 Video signal processing devices can be combined with Figure 1 Encoding device 100 or Figure 2 The decoding device 200 is compatible.

[0634] The video signal processing apparatus 5000 according to embodiments of the present disclosure may include a memory 5020 for storing video signals and a processor 5010 for processing the video signals in conjunction with the memory 5020.

[0635] The processor 5010 according to embodiments of the present disclosure may be configured with at least one processing circuit for processing video signals, and may process video signals by executing instructions to encode or decode video signals. That is, the processor 5010 may encode the original video signal or decode the encoded video signal by executing the above-described encoding or decoding methods.

[0636] Figure 51 An example of a video coding system using this disclosure is provided.

[0637] A video encoding system may include a source device and a receiving device. The source device may transmit encoded video / image information or data to the receiving device in file or streaming format via storage media or a network.

[0638] The source device may include a video source, an encoding device, and a transmitter. The receiving device may include a receiver, a decoding device, and a renderer. The source device may be referred to as a video / image encoding device, and the receiving device may be referred to as a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display, and the display may be configured as a separate device or an external component.

[0639] A video source can acquire video / image data through a process of capturing, compositing, or generating video / images. A video source may include a video / image capture device and / or a video / image generation device. A video / image capture device may include, for example, one or more cameras, a video / image archive containing previously captured video / images, etc. A video / image generation device may include, for example, a computer, tablet computer, and smartphone, and can electronically generate video / image data. For example, virtual video / image data can be generated by a computer, etc., and in this case, the video / image capture process can be replaced by a process of generating related data.

[0640] Encoding devices can encode input video / images. They can perform a series of processes such as prediction, transformation, and quantization for compression and coding efficiency. The encoded data (encoded video / image information) can be output as a bitstream.

[0641] A transmitter can send encoded video / image information or data, output as a bitstream, to a receiver in a receiving device via digital storage media or a network in file or streaming format. Digital storage media can include various media such as Universal Serial Bus (USB), Secure Digital SD, Optical Disc (CD), Digital Video Disc (DVD), Blu-ray, Hard Disk Drive (HDD), and Solid State Drive (SSD). The transmitter can include elements for generating media files in a predetermined file format and may include elements for transmission over a broadcast / communication network. The receiver can extract the bitstream and send it to a decoding device.

[0642] Decoding devices can decode video / image data by performing a series of processes such as inverse quantization, inverse transform, and prediction, which correspond to the operations of encoding devices.

[0643] The renderer can render decoded video / images. The rendered video / images can then be displayed on a monitor.

[0644] Figure 52 A structural diagram of a content streaming system is shown.

[0645] Reference Figure 52 The content streaming system using this disclosure can mainly include an encoding server, a streaming server, a network server, a media storage device, a user device, and a multimedia input device.

[0646] An encoding server can compress content input from multimedia input devices such as smartphones, cameras, and camcorders into digital data to generate a bitstream, and then transmit it to a streaming server. As another example, when multimedia input devices 460, such as smartphones, cameras, and camcorders, directly generate bitstreams, the encoding server can be omitted.

[0647] Bitstreams can be generated by applying the encoding method or bitstream generation method disclosed herein, and the streaming server can temporarily store bitstreams during the processing of sending or receiving bitstreams.

[0648] A streaming server transmits multimedia data to a user device via a web server based on a user's request, and the web server acts as a medium to inform the user what services are available. When a user requests a desired service through the web server, the web server forwards the request to the streaming server, which then sends the multimedia data to the user. In this case, the content streaming system may include a separate control server, which controls the commands / responses between devices within the content streaming system.

[0649] A streaming server can receive content from media storage devices and / or encoding servers. For example, a streaming server can receive content from an encoding server in real time. In this case, to provide a smooth streaming service, the streaming server can store the bitstream for a predetermined period of time.

[0650] For example, user devices may include mobile phones, smartphones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation terminals, touchscreen PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, head-mounted displays (HMDs), digital TVs, desktop computers, and digital signage).

[0651] In a content streaming system, each server can operate as a distributed server, and in this case, data received from each server can be processed in a distributed manner.

[0652] As described above, the embodiments described in this disclosure can be implemented and executed on a processor, microprocessor, controller, or chip. For example, the functional units illustrated in the figures can be implemented and executed on a computer, processor, microprocessor, controller, or chip.

[0653] Furthermore, the decoders and encoders disclosed herein can be included in multimedia broadcasting transmitting and receiving devices, mobile communication terminals, home theater video devices, digital cinema video devices, surveillance cameras, video chat devices, real-time communication devices such as video communication, mobile streaming devices, storage media, cameras, video-on-demand (VoD) service providers, over-the-top (OTT) video devices, internet streaming service providers, three-dimensional (3D) video devices, video telephony devices, and medical video devices, and can be used to process video signals or data signals. For example, OTT video devices can include game consoles, Blu-ray players, internet access TVs, home theater systems, smartphones, tablet PCs, and digital video recorders (DVRs).

[0654] Furthermore, the processing methods of this disclosure can be generated in the form of a program executed by a computer and can be stored in a computer-readable recording medium. Multimedia data with data structures according to this disclosure can also be stored in a computer-readable recording medium. A computer-readable recording medium includes all kinds of storage devices in which computer-readable data is stored. Computer-readable recording media can include, for example, Blu-ray discs (BD), Universal Serial Bus (USB), ROM, PROM, EPROM, EEPROM, RAM, CD-ROM, magnetic tape, floppy disks, and optical data storage devices. Furthermore, computer-readable recording media include media implemented in the form of a carrier wave (e.g., transmitted via the Internet). Additionally, bitstreams generated using encoding methods can be stored in a computer-readable recording medium or transmitted via wired or wireless communication networks.

[0655] Furthermore, the embodiments of this disclosure can be implemented as a computer program product using program code. The program code can be executed by a computer according to the embodiments of this disclosure. The program code can be stored on a carrier wave that is readable by a computer.

[0656] The decoding and encoding devices using this disclosure can be included in digital devices. The term "digital device" includes all digital devices capable of performing at least one of the following: transmitting, receiving, processing, and outputting data, content, and services. Here, processing data, content, services, etc., by a digital device includes operations of encoding and / or decoding the data, content, services, etc. Digital devices pair or connect (hereinafter referred to as "pairing") with other digital devices, external servers, etc., via wired / wireless networks to transmit and receive data, and convert it when necessary.

[0657] For example, the digital devices include fixed devices (or floor-standing devices) such as Internet TV, HBBTV (Hybrid Broadcast Broadband TV), Smart TV, IPTV (Internet Protocol TV), and PCs, as well as mobile devices (or handheld devices) such as PDAs (Personal Digital Assistants), smartphones, tablet PCs, and portable computers. In the disclosure described below, for convenience, Figure 54 An implementation of digital TV as a digital device is shown and described, and Figure 55 An implementation of the mobile device as a digital device is shown and described.

[0658] Furthermore, the term "wired / wireless network" as described in this disclosure refers to a communication network that supports interconnection and / or data transmission and reception between digital devices or between a digital device and an external server. These wired / wireless networks may include currently and future supported communication networks and the communication protocols for them, and may be formed by communication standards or protocols for wired connections such as USB (Universal Serial Bus), CVBS (Composite Video Blanking Synchronization), Components, S-Video (analog), DVI (Digital Vision Interface), HDMI (High-Definition Multimedia Interface), RGB, D-SUB, etc., and by communication standards for wireless connections such as Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, Digital Living Networks Alliance (DLNA), Wireless LAN (WLAN) (Wi-Fi), Wibro (Wireless Broadband), WiMAX (World Microwave Access Interoperability), HSDPA (High-Speed ​​Downlink Packet Access), LTE (Long Term Evolution), Wi-Fi Direct, etc.

[0659] In the following, where reference is made simply to a digital device in this disclosure, depending on the context, it may mean a fixed device or a mobile device or include both.

[0660] Furthermore, a digital device is, for example, an intelligent device that supports broadcast reception, computer functions, and at least one external input, and can support email, web browsing, banking, games, and applications via wired / wireless networks. Additionally, a digital device may include an interface for supporting at least one input or control device (hereinafter referred to as an input device), such as a manual input device, a touchscreen, or a spatial remote control. Digital devices can use standardized general-purpose operating systems (OS). For example, digital devices can have various applications added, removed, modified, and updated on a general-purpose OS kernel, and can be configured and provide a more user-friendly environment through them.

[0661] Furthermore, the external inputs described in this disclosure include external input devices, that is, all input devices or digital devices that are wired / wireless connected to the aforementioned digital devices and send / receive relevant data thereon. Here, external inputs can include all devices such as High Definition Multimedia Interface (HDMI), gaming devices such as game consoles or Xboxes, smartphones, tablet PCs, printers, or smart TVs.

[0662] Furthermore, the term "server" described in this disclosure includes a client, that is, all digital devices or systems that supply data to the aforementioned digital devices, and is referred to as a processor. Examples of such servers include portal servers that provide web pages or network content, advertising servers that provide advertising data, content servers that provide content, and social media service (SNS) servers, service servers, or manufacturing servers provided by manufacturers, etc.

[0663] Furthermore, the term "channel" as described herein refers to a path, device, etc., used for sending and receiving data, and can be exemplified by a broadcast channel. Here, broadcast channels are represented according to physical channels, virtual channels, and logical channels, depending on the activation of digital broadcasting. A broadcast channel can be referred to as a broadcast network. As mentioned above, a broadcast channel refers to a channel used to provide or access broadcast content provided by a broadcasting station, and the broadcast content is primarily based on live broadcasting, and is also referred to as a live channel. However, recently, the media used for broadcasting has become more diversified, and non-live broadcasting has been activated in addition to live broadcasting. It can also be understood as a term referring to the entire channel. Therefore, a live channel can be understood as a term that in some cases means not only live broadcasting but also the entire broadcast channel including non-live broadcasting.

[0664] In this disclosure, the term "arbitrary channel" is further defined in relation to channels other than the aforementioned broadcast channels. An arbitrary channel may provide service guides such as Electronic Program Guides (EPGs) in conjunction with a broadcast channel. Service guides, GUIs (Graphical User Interfaces), or OSD screens (screen displays) may be configured / provided using only an arbitrary channel.

[0665] Furthermore, unlike broadcast channels with predetermined channel numbers between transceivers, arbitrary channels are channels arbitrarily assigned within the receiver, and essentially no channel numbers overlapping with those used to represent broadcast channels are assigned to arbitrary channels. For example, when tuning to a specific broadcast channel, the receiver receives a broadcast signal that transmits the broadcast content and its signaling information through the tuned channel. Here, the receiver parses channel information from the signaling information and configures the channel browser, EPG, etc., based on the parsed channel information and provides them to the user. When the user submits a channel change request via an input device, the receiver responds accordingly.

[0666] As mentioned above, since the broadcast channel is pre-committed between the sending and receiving terminals, assigning arbitrary channels to broadcast channels in an overlapping manner can cause confusion for users, or may potentially cause confusion. Therefore, as mentioned above, it is effective not to assign duplicates. Furthermore, even if the arbitrary channel number does not overlap with the broadcast channel number as described above, confusion still exists during the user's channel browsing process, and this needs to be considered when assigning arbitrary channel numbers. This is because, according to this disclosure, arbitrary channels can also be implemented to be accessed in the same way as regular broadcast channels in response to a user's request for channel switching. Therefore, arbitrary channel numbers can be defined and displayed as parallel characters such as arbitrary channel-1, arbitrary channel-2, etc., rather than in the numerical form of broadcast channels, to facilitate user access to arbitrary channels and to facilitate differentiation or identification of broadcast channel numbers. In this case, although the arbitrary channel number is displayed as a character string as in arbitrary channel-1, the receiver can internally recognize the arbitrary channel in numerical form and implement it as a broadcast channel number. In addition, arbitrary channel numbers can be provided in the form of numbers such as broadcast channels, and channel numbers can be defined and displayed in various ways that can be distinguished from broadcast channels such as video channel-1, caption 1, and video-1.

[0667] This digital device executes a web browser for network services and provides users with various types of web pages. Here, web pages also include those containing video content, and in this disclosure, video and web pages are processed separately or independently. Furthermore, as described above, the separated video can be assigned to any channel and provided through service guides, etc., and output according to channel switching requests during the viewing of service guides or broadcast channels. In addition to network services, for services such as broadcast content, games, and applications, predetermined content, images, audio, items, etc., are processed separately from the broadcast content, games, and applications themselves, and can be assigned any channel number for their reproduction, processing, etc., and can be implemented as described above.

[0668] Figure 53 This is a schematic diagram illustrating an example of a service system that includes digital devices.

[0669] Examples of service systems that include digital devices include content provider CP 5310, service provider SP 5320, network provider NP 5330, and home network end-user HNED (customer) 5340. Here, HNED 5340 is, for example, client 5300, i.e., digital device. Content provider 5310 produces and provides various types of content. Figure 53As shown, terrestrial broadcasters, cable SOs (System Operators) or MSOs (Multiple SOs), satellite broadcasters, various Internet broadcasters, and private content providers (CPs) can be cited as examples of such content providers 5310. Furthermore, in addition to providing broadcast content, content provider 5310 also provides various applications.

[0670] Service provider 5320 packages the content provided by content provider 5310 into a service package and provides it to HNED 5340. For example, Figure 53 The service provider 5320 packages and provides HNED 5340 with primary terrestrial broadcasting, secondary terrestrial broadcasting, cable MSO, satellite broadcasting, various Internet broadcasting, and applications.

[0671] Service provider 5320 provides services to client 300 via unicast or multicast. Furthermore, service provider 5320 can send data to multiple pre-registered clients 5300 simultaneously, and for this purpose, the Internet Group Management Protocol (IGMP) can be used.

[0672] The content provider 5310 and service provider 5320 mentioned above can be the same entity or a single entity. For example, the content provided by content provider 5310 can be packaged as a service and provided to HNED 5340, so the functions of service provider 5320 can also be performed together, or vice versa.

[0673] Network provider 5330 provides a network for data exchange between content provider 5310 and / or service provider 5320 and client 5300.

[0674] The client 5300 can establish a home network to send and receive data.

[0675] Furthermore, the content provider 5310 and / or service provider 5320 in the service system may use conditional access or content protection devices to protect the transmitted content. In this case, the client 300 may use processing devices such as CableCARD (POD: point of deployment) or DCAS (downloadable CAS) in response to restricted reception or content protection.

[0676] Additionally, client 5300 can also use bidirectional services via a network (or communication network). In this case, client 5300 can still perform the functions of the content provider, and the existing service provider 5320 can receive it and send it back to another client.

[0677] Figure 54 This is a block diagram illustrating a digital device according to an embodiment. Here, for example, Figure 54 It can correspond to Figure 53 The client 5300 refers to the aforementioned digital device.

[0678] Digital device 5400 includes network interface 5401, TCP / IP manager 5402, service delivery manager 5403, SI decoder 5404, demultiplexer 5405, audio decoder 5406, video decoder 5407, display module (monitor A / V and OSD) 5408, service control manager 5409, service discovery manager 5410, SI and metadata DB 5411, metadata manager 5412, service manager 5413, UI manager 5414, etc.

[0679] Network interface 5401 receives or sends IP packets over the network. That is, network interface 5401 receives services, content, etc. from service provider 5320 over the network.

[0680] TCP / IP manager 5402 participates in the source and destination packet transmission of IP packets received and sent by digital device 5400. Furthermore, TCP / IP manager 5402 classifies received packets to correspond to the appropriate protocol and outputs the classified packets to service delivery manager 5403, service discovery manager 5410, service control manager 5409, metadata manager 5412, etc. Service delivery manager 5403 is responsible for controlling the received service data. For example, when controlling real-time streaming data, service delivery manager 5403 can use RTP / RTCP. When sending real-time streaming data using RTP, under the control of service manager 5413, service delivery manager 5403 parses the received data packets according to RTP and sends them to demultiplexer 5405 or stores them in SI and metadata database 5411. Furthermore, service delivery manager 5403 uses RTCP to feed back network reception information to the server providing the service. The demultiplexer 5405 demultiplexes the received packets into audio, video, and system information (SI) data, and sends them to the audio / video decoders 5406 / 5407 and the SI decoder 5404, respectively.

[0681] The SI decoder 5404 decodes service information such as Program Specific Information (PSI), Program and System Information Protocol (PSIP), and Digital Video Broadcasting Service Information (DVB-SI).

[0682] Furthermore, the SI decoder 5404 stores the decoded service information, for example, in the SI and metadata database 5411. The service information stored in this way can be read and used by the appropriate configuration (e.g., upon user request).

[0683] Audio / video decoders 5406 / 5407 decode each audio and video data demultiplexed by demultiplexer 5405. The decoded audio and video data are then provided to the user via display module 5408.

[0684] The application manager may include, for example, a UI manager 5414 and a service manager 5413. The application manager manages the overall state of the digital device 5400, provides the user interface, and manages other managers.

[0685] UI Manager 5414 provides a graphical user interface (GUI) to the user using an OSD (On-Screen Display) or similar means, and receives key input from the user to perform device operations based on that input. For example, when UI Manager 5414 receives key input from the user for channel selection, UI Manager 5414 sends the key input signal to Service Manager 5413.

[0686] Service Manager 5413 controls service-associated managers such as Service Delivery Manager 5403, Service Discovery Manager 5410, Service Control Manager 5409, and Metadata Manager 5412.

[0687] Furthermore, the service manager 5413 creates a channel map and selects channels using the channel map based on the key input received from the user interface manager 5414. The service manager 5413 also receives channel service information from the SI decoder 5404 and sets the audio / video packet identifier (PID) of the selected channel for the demultiplexer 5405. The PID set in this way is used in the demultiplexing process described above. Therefore, the demultiplexer 5405 uses the PID to filter the audio data, video data, and SI data.

[0688] Service discovery manager 5410 provides the information needed to select service providers to offer services. When it receives a signal about channel selection from service manager 5413, service discovery manager 5410 uses this information to locate services.

[0689] Service Control Manager 5409 is responsible for selecting and controlling services. For example, when a user selects a live broadcast service such as a regular broadcaster's license, Service Control Manager 5409 uses IGMP or RTSP, and when a user selects a service such as VOD (Video on Demand), Service Control Manager 5409 uses RTSP to select and control the service. The RTSP protocol can provide special effects modes for real-time streaming. Furthermore, Service Control Manager 5409 can use the IP Multimedia Subsystem (IMS) and Session Initiation Protocol (SIP) through IMS Gateway 5450 to initialize and manage sessions. The protocol can be one implementation, and other protocols can be used depending on the implementation example.

[0690] Metadata manager 5412 manages metadata associated with services and stores the metadata in SI and metadata database 5411.

[0691] The SI and metadata database 5411 stores service information decoded by the SI decoder 5404, metadata managed by the metadata manager 5412, and information required for selecting service providers provided by the service discovery manager 5410. Furthermore, the SI and metadata database 5411 can store system settings data used by the system.

[0692] The SI and metadata database 5411 can be implemented using non-volatile RAM (NVRAM), flash memory, etc.

[0693] In addition, the IMS Gateway 5450 is a gateway that collects the functions required to access IMS-based IPTV services.

[0694] Figure 55 This is a configuration block diagram illustrating another implementation of a digital device. Specifically, Figure 55 A block diagram of a mobile device is shown as another embodiment of a digital device.

[0695] Reference Figure 55 The mobile device 5500 may include a wireless communication unit 5510, an audio / video A / V input unit 5520, a user input unit 5530, a sensing unit 5540, an output unit 5550, a memory 5560, an interface unit 5570, a control unit 5580, and a power supply unit 5590. Figure 55 The components shown are not required, so mobile devices with more or fewer components can be implemented.

[0696] The wireless communication unit 5510 may include one or more modules that enable wireless communication between the mobile device 5500 and the wireless communication system or between the mobile device and the network in which the mobile device resides. For example, the wireless communication unit 5510 may include a broadcast receiving module 5511, a mobile communication module 5512, a wireless internet module 5513, a short-range communication module 5514, and a location information module 5515.

[0697] The broadcast receiving module 5511 receives broadcast signals and / or broadcast-related information from an external broadcast management server via a broadcast channel. Here, the broadcast channel may include a satellite channel and / or a terrestrial channel. The broadcast management server may be a server that generates and transmits broadcast signals and / or broadcast-related information, or a server that receives previously generated broadcast signals and / or broadcast-related information and transmits them to the terminal. The broadcast signal may include TV broadcast signals, radio broadcast signals, and data broadcast signals, and may also include a broadcast signal combining data broadcast signals with TV broadcast signals or radio broadcast signals.

[0698] Broadcast-related information can refer to information related to a broadcast channel, broadcast program, or broadcast service provider. Broadcast-related information can also be provided via a mobile communication network. In this case, it can be received by the mobile communication module 5512.

[0699] Broadcast-related information can exist in various forms, such as Electronic Program Guide (EPG) or Electronic Service Guide (ESG).

[0700] The broadcast receiver module 5511 can receive digital broadcast signals using digital broadcasting systems such as ATSC, DVB-T (Digital Video Broadcasting on Terrestrial), DVB-S (Satellite), Media-Only Forward Link (MediaFLO), handheld DVB-H, and ISDB-T (Integrated Services Digital Broadcasting on Terrestrial) and other digital broadcasting systems. Of course, the broadcast receiver module 5511 can be configured to be used with other broadcasting systems in addition to those described above.

[0701] Broadcast signals and / or broadcast-related information received by broadcast receiving module 5511 can be stored in memory 5560.

[0702] The mobile communication module 5512 transmits and receives radio signals with at least one of a base station, an external terminal, and a server on a mobile communication network. The radio signals may include various types of data transmitted and received based on voice call signals, video call signals, or text / multimedia messages.

[0703] The wireless internet module 5513 includes a module for wireless internet access and can be built inside or outside the mobile device 5500. Wireless internet technologies that can be used include wireless LAN (WLAN) (Wi-Fi), wireless broadband Wibro, global microwave access interoperability WiMAX, and high-speed downlink packet access HSDPA.

[0704] The short-range communication module 5514 refers to a module used for short-range communication. It can use short-range communication technologies such as Bluetooth, RFID (Radio Frequency Identification), infrared communication (IrDA, Infrared Data Association), UWB (Ultra-Wideband), ZigBee, RS-232, and RS-485.

[0705] The location information module 5515 is a module used to obtain the location information of the mobile device 5500, and a Global Positioning System (GPS) module can be used as an example.

[0706] The A / V input unit 5520 is used for audio and / or video signal input and may include a camera 5521, a microphone 5522, etc. The camera 5521 processes image frames, such as still images or moving pictures, acquired by the image sensor in video call mode or shooting mode. The processed image frames can be displayed on the display unit 5551.

[0707] Image frames processed by camera 5521 can be stored in memory 5560 or transmitted externally via wireless communication unit 5510. Two or more cameras 5521 can be configured depending on the usage environment.

[0708] Microphone 5522 receives external sound signals in call mode, recording mode, voice recognition mode, etc., and processes them as electronic voice data. The processed voice data can be converted into a form that can be transmitted to the mobile communication base station via mobile communication module 5512 in call mode and then output. Microphone 5522 can be implemented using various noise reduction algorithms to remove noise generated during the reception of external sound signals.

[0709] The user input unit 5530 generates input data that the user uses to control the operation of the terminal. The user input unit 5530 may be configured with a keyboard, dome switch, touchpad (pressure type / capacitive type), scroll wheel, toggle switch or the like.

[0710] The sensing unit 5540 displays the current state of the mobile device 5500, such as its on / off state, position, presence of user contact, direction of movement, and acceleration / deceleration. It senses and generates sensing signals for controlling the operation of the mobile device 5500. For example, it can sense the position or tilt of the mobile device 5500 when it moves or tilts. It can also sense whether power is being supplied to the power supply unit 5590 or whether an external device is being connected to the interface unit 5570. Furthermore, the sensing unit 5540 may include a proximity sensor 5541 incorporating Near Field Communication (NFC).

[0711] The output unit 5550 is used to generate outputs related to vision, hearing or touch, and may include a display unit 5551, an audio output module 5552, an alarm module 5553 and a tactile module 5554.

[0712] Display unit 5551 displays or outputs information processed by mobile device 5500. For example, when the mobile device is in call mode, it displays a call-related user interface (UI) or graphical user interface (GUI). When the mobile device 5500 is in video call mode or shooting mode, it displays the captured and / or received video, UI, or GUI.

[0713] The display unit 5551 may include a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT-LCD), an organic light-emitting diode (OLED), a flexible display, and a 3D display.

[0714] Some of these displays can be transparent or light-transmitting, allowing the outside to be seen through them. This can be called a transparent display, and a typical example of a transparent OLED is a TOLED. The rear structure of the display unit 5551 can also be configured as a light-transmitting structure. With this structure, the user can see objects located behind the terminal body through the area occupied by the display unit 5551 on the terminal body.

[0715] Depending on the implementation of the mobile device 5500, there may be two or more display units 5551. For example, the multiple display units may be separated from each other or integrally arranged in the mobile device 5500, or they may be disposed on different surfaces.

[0716] When the display unit 5551 and the sensor that detects touch operations (hereinafter referred to as the "touch sensor") form an interlayer structure (hereinafter referred to as the "touch screen"), the display unit 5551 can also be used as an input device in addition to being an output device. The touch sensor may take the form of, for example, a touch film, a touch sheet, or a touchpad.

[0717] The touch sensor can be configured to convert pressure applied to a specific portion of the display unit 5551, or changes in capacitance generated in a specific portion of the display unit 5551, into an electrical input signal. The touch sensor can be configured to detect not only the location and area of ​​the touch, but also the pressure applied during the touch.

[0718] If a touch input is made to the touch sensor, a corresponding signal is sent to the touch controller. The touch controller processes the signal and then sends the corresponding data to the control unit 5580. Therefore, the control unit 5580 can determine which area of ​​the display unit 5551 has been touched.

[0719] The proximity sensor 5541 can be placed in or near the internal area of ​​a mobile device surrounded by a touchscreen. A proximity sensor is a device that detects the presence or absence of an object near a predetermined detection surface using mechanical force or infrared light without mechanical contact. Proximity sensors have a longer lifespan and higher utilization rate than contact sensors.

[0720] Examples of proximity sensors include transmissive photoelectric sensors, direct reflective photoelectric sensors, specular reflective photoelectric sensors, high-frequency oscillating proximity sensors, capacitive proximity sensors, magnetic proximity sensors, and infrared proximity sensors. When a touchscreen is capacitive, it is configured to detect the proximity of an indicator based on changes in the electric field corresponding to the indicator's proximity. In this case, the touchscreen (touch sensor) can be classified as a proximity sensor.

[0721] In the following text, for ease of description, the action of recognizing an indicator as being on the touchscreen rather than touching the touchscreen with the indicator is referred to as a "proximity touch," and the action of actually touching the touchscreen with the indicator is referred to as a "contact touch." The position of a proximity touch on the touchscreen refers to the position of the indicator perpendicular to the touchscreen when it is brought close to the touch.

[0722] The proximity sensor detects proximity touches and proximity touch patterns (e.g., proximity touch distance, proximity touch direction, proximity touch speed, proximity touch time, proximity touch position, proximity touch movement state, etc.). The information corresponding to the sensed proximity touch operation and proximity touch pattern can be output on the touchscreen.

[0723] The audio output module 5552 can output audio data received from the wireless communication unit 5510 or stored in the memory 5560 in call signal reception, call mode or recording mode, voice recognition mode, broadcast reception mode or similar modes. The audio output module 5552 can also output audio signals related to functions performed in the mobile device 5500 (e.g., call signal reception tone, message reception tone, etc.). The audio output module 5552 may include a receiver, a speaker, and a buzzer.

[0724] Alarm module 5553 outputs a signal to notify that an event has occurred on mobile device 5500. Examples of events occurring on the mobile device include call signal reception, message reception, key signal input, and touch input. In addition to video or audio signals, alarm module 5553 may output a signal to notify of the event in other forms, such as vibration.

[0725] Video or audio signals can also be output through display unit 5551 or audio output module 5552, so that display unit 5551 and audio output module 5552 can be classified as part of alarm module 5553.

[0726] The haptic module 5554 produces various tactile effects that the user can perceive. Vibration is a typical example of the tactile effects produced by the haptic module 5554. The intensity and pattern of the vibrations produced by the haptic module 5554 are controllable. For example, different vibrations can be synthesized and output or output sequentially.

[0727] In addition to vibration, the haptic module 5554 can also produce a variety of haptic effects, including stimulation effects such as pin arrangement that moves vertically relative to the contact surface of the skin, jetting or inhaling force of air, abrasion on the skin surface, contact with electrodes, electrostatic force, and the effect of reproducing hot and cold sensations by using elements that can absorb or generate heat.

[0728] The haptic module 5554 can transmit tactile effects not only through direct contact, but also enable the user to feel the tactile effects through the sensation of muscles such as fingers or arms. At least two or more haptic modules 5554 can be provided depending on the configuration of the mobile device 5500.

[0729] The memory 5560 can store programs for operating the control unit 5580, or it can temporarily store input / output data (e.g., phone book, messages, still images, videos, etc.). The memory 5560 can store data related to various vibration modes and output sound when touch input is applied to the touchscreen.

[0730] The memory 5560 may include at least one of the following storage media: flash memory, hard disk memory, multimedia card micro / card-type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disk, and optical disk. The mobile device 5500 may operate in conjunction with a network storage device that performs the storage functions of the memory 5560 over the Internet.

[0731] Interface unit 5570 serves as a channel for communication with all external devices connected to mobile device 5500. Interface unit 5570 receives data from external devices, receives power, and transmits data to each component within mobile device 5500, or enables data within mobile device 5500 to be transmitted to external devices. For example, wired / wireless headset ports, external charger ports, wired / wireless data ports, memory card ports, ports for connecting devices equipped with identification modules, audio input / output ports, video I / O ports, headphone ports, etc., may be included in interface unit 5570.

[0732] The identification module is a chip that stores various information used to authenticate access to the mobile device 5500, and may include a User Authentication Module (UIM), a Subscriber Identification Module (SIM), a Global Subscriber Identification Module (USIM), etc. The device equipped with the identification module (hereinafter referred to as the "identification device") can be manufactured in a smart card format. Therefore, the identification device can be connected to the mobile device 5500 via a port.

[0733] When the mobile device 5500 is connected to an external bracket, the interface unit 5570 can serve as a channel through which power from the bracket is supplied to the mobile terminal 550, or as a channel through which the user inputs various command signals from the bracket. The various command signals or power input from the bracket can function as signals used to identify that the mobile terminal is correctly mounted on the bracket.

[0734] The control unit 5580 typically controls the overall operation of the mobile device. For example, it performs control and processing related to voice calls, data communications, video calls, etc. The control unit 5580 may include a multimedia module 5581 for multimedia playback. The multimedia module 5581 may be implemented within the control unit 5580 or may be implemented separately from the control unit 5580. Specifically, the control unit 5580, and the multimedia module 5581, may include the encoding device 100 and / or decoding device 200 described above.

[0735] The control unit 5580 can perform a pattern recognition process that can recognize handwritten or drawing input performed on the touch screen as characters and images, respectively.

[0736] The power supply unit 5590 receives external and internal power under the control of the control unit 5580 and supplies the power required for the operation of each component.

[0737] The various implementations described herein can be implemented, for example, using software, hardware, or a combination thereof, in a computer- or similar device-readable recording medium.

[0738] Depending on the hardware implementation, the embodiments of the patterns described herein can be implemented using at least one of an Application-Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field-Programmable Gate Array (FPGA), a processor, a controller, a microcontroller, a microprocessor, and electronic units for performing functions. In some cases, the embodiments described herein can be implemented using the control unit 5580 itself.

[0739] Depending on the software implementation method, implementations such as the processes and functions described herein can be implemented as individual software modules. Each software module can perform one or more functions and operations described herein. The software code can be implemented by a software application written in a suitable programming language. Here, the software code is stored in memory 5560 and can be executed by control unit 5580.

[0740] Figure 56 This is a block diagram illustrating a digital device according to another embodiment.

[0741] Another example of digital device 5600 may include a broadcast receiving unit 5605, an external device interface unit 5656, a storage unit 5640, a user input interface unit 5650, a control unit 5670, a display unit 5680, an audio output unit 5685, a power supply unit 5690, and a shooting unit (not shown). Here, the broadcast receiving unit 5605 may include at least one tuner 5610, a demodulation unit 5620, and a network interface unit 5630. However, depending on the circumstances, the broadcast receiving unit 5605 may include the tuner 5610 and the demodulation unit 5620, but may not include the network interface unit 5630, or vice versa. Additionally, although not shown, the broadcast receiving unit 5605 may be equipped with a multiplexer to multiplex the signal tuned by the tuner 5610 and then demodulated by the demodulation unit 5620, as well as the signal received through the network interface unit 5630. Additionally, although not shown, the broadcast receiving unit 5605 may be equipped with a demultiplexer to demultiplex the multiplexed signal, the demodulated signal, or the signal that has already passed through the network interface unit 5630.

[0742] Tuner 5610 receives RF broadcast signals by tuning to a user-selected channel or all pre-stored channels from the RF broadcast signals received via the antenna. Furthermore, tuner 5610 converts the received RF broadcast signals into intermediate frequency (IF) signals or baseband signals.

[0743] For example, if the received RF broadcast signal is a digital broadcast signal, it is converted into a digital IF signal (DIF), and if it is an analog broadcast signal, it is converted into an analog baseband video or audio signal (CVBS / SIF). That is, tuner 5610 can process both digital and analog broadcast signals. The analog baseband video or audio signal (CVBS / SIF) output from tuner 5610 can be directly input to control unit 5670.

[0744] Tuner 5610 can receive single-carrier RF broadcast signals according to the Advanced Television Systems Committee (ATSC) scheme, or receive RF broadcast signals with multiple carriers according to the Digital Video Broadcasting (DVB) scheme.

[0745] In addition, tuner 5610 can sequentially tune and receive RF broadcast signals from all broadcast channels stored by the channel storage function among the RF broadcast signals received by the antenna, and convert them into intermediate frequency signals or baseband signals.

[0746] Demodulation unit 5620 receives and demodulates the digital IF signal DIF converted by tuner 5610. For example, when the digital IF signal output from tuner 5610 is an ATSC system, demodulation unit 5620 performs, for example, 8-vertical-sideband (8-VSB) demodulation. Additionally, demodulation unit 5620 can perform channel decoding. For this purpose, demodulation unit 5620 may include a trellis decoder, a deinterleaver, a Reed-Solomon decoder, etc., and performs trellis decoding, deinterleaverization, and Reed-Solomon decoding.

[0747] For example, when the digital IF signal output from tuner 5610 is in DVB mode, demodulation unit 5620 performs, for example, coded orthogonal frequency division modulation (COFDMA) demodulation. Additionally, demodulation unit 5620 can perform channel decoding. For this purpose, demodulation unit 5620 may include a convolutional decoder, a deinterleaver, and a ahead Solomon decoder, and performs convolutional decoding, deinterleaverating, and Solomon decoding readout.

[0748] The demodulation unit 5620 can output a streaming signal TS after demodulation and channel decoding. In this case, the streaming signal can be a signal in which video signals, audio signals, or data are multiplexed. For example, the streaming signal can be an MPEG-2 transport stream TS in which MPEG-2 standard video signals, Dolby AC-3 standard audio signals, etc., are multiplexed. Specifically, the MPEG-2 TS can include a 4-byte header and a 184-byte payload.

[0749] Furthermore, the demodulation unit 5620 can be provided for both ATSC and DVB methods. That is, the digital device can include both an ATSC demodulator and a DVB demodulator.

[0750] The streaming signal output from demodulation unit 5620 can be input to control unit 5670. Control unit 5670 can control demultiplexing, video / audio signal processing, etc., and control the image output through display unit 5680 and the audio output through audio output unit 5685.

[0751] The external device interface unit 5656 provides an environment for various external devices to interact with the digital device 5600. For this purpose, the external device interface unit 5635 may include an A / V input / output unit (not shown) or a wireless communication unit (not shown).

[0752] The external device interface unit 5656 can be connected wired / wirelessly to external devices such as digital multifunction discs (DVDs), Blu-ray players, gaming devices, cameras, camcorders, computers (portable computers, tablets), smartphones, Bluetooth devices, and the cloud. The external device interface unit 5656 transmits video, audio, or data (including image) signals input from the connected external device to the control unit 5670 of the digital device. The control unit 5670 can control the processed image, audio, or data signals to output them to the connected external device. For this purpose, the external device interface unit 5656 may also include an A / V input / output unit (not shown) or a wireless communication unit (not shown).

[0753] The A / V input / output unit may include a USB terminal, a composite video blanking synchronization CVBS terminal, a component terminal, an S-video terminal (analog) and a DVI (digital visual interface) terminal, an HDMI (high-definition multimedia interface) terminal, an RGB terminal, a D-SUB terminal, etc., to input video and audio signals from external devices into the digital device 5600.

[0754] The wireless communication unit can perform short-range wireless communication with other electronic devices. For example, the digital device 5600 can network with other electronic devices according to communication protocols (e.g., Bluetooth, RFID, IrDA, UWB, ZigBee, and DLNA).

[0755] Furthermore, the external device interface unit 5656 can be connected to at least one of the various set-top boxes and various terminals mentioned above, and can perform input / output operations using the set-top box.

[0756] In addition, the external device interface unit 5656 can receive applications or application lists from adjacent external devices and transmit them to the control unit 5670 or the storage unit 5640.

[0757] Network interface unit 5630 provides an interface for connecting digital device 5600 to wired / wireless networks, including the Internet. Network interface unit 5630 may include, for example, an Ethernet port for connecting to a wired network, and uses communication standards such as, for example, Wireless LAN (WLAN) (Wi-Fi), Wireless Broadband (Wibro), Global Microwave Access Interoperability (WiMAX), and High-Speed ​​Downlink Packet Access (HSDPA) for connecting to wireless networks.

[0758] The network interface unit 5630 can send or receive data with other users or other digital devices through the connected network or another network linked to the connected network. Specifically, some content data stored in the digital device 5600 can be sent to another user pre-registered in the digital device 5600, or to a selected user or a selected digital device among other digital devices.

[0759] Furthermore, the network interface unit 5630 can access predetermined web pages via the connected network or another network linked to the connected network. That is, it can connect to predetermined web pages via the network and send or receive data with the corresponding server. Additionally, it can receive content or data provided by content providers or network operators. Specifically, it can receive content such as movies, advertisements, games, VOD, broadcast signals, and related information provided by content providers or network operators via the network. Furthermore, it can receive firmware update information and update files provided by network operators. The network interface unit 5630 can also send data to the Internet or to content providers or network operators.

[0760] In addition, the network interface unit 5630 can select and receive desired applications from publicly available applications via the network.

[0761] The storage unit 5640 may store a program for processing and controlling each signal in the control unit 5670, or it may store processed image, voice, or data signals.

[0762] Additionally, storage unit 5640 can temporarily store image, audio, or data signals input from external device interface unit 5635 or network interface unit 5630. Storage unit 5640 can also store information related to a predetermined broadcast channel via a channel memory function.

[0763] Storage unit 5640 can store applications or application lists input from external device interface unit 5635 or network interface unit 5630.

[0764] In addition, storage unit 5640 can store various platforms described later.

[0765] Storage unit 5640 may include at least one storage medium selected from, for example, flash memory, hard disk, multimedia card micro-type and card type memory (e.g., SD or XD memory), RAM and ROM (EEPROM, etc.). Digital device 5600 can play content files (video files, still image files, music files, document files, application files, etc.) stored in storage unit 5640 and provide them to the user.

[0766] Figure 56 An embodiment in which the storage unit 5640 and the control unit 5670 are separately arranged is illustrated, but the scope of this disclosure is not limited thereto. That is, the storage unit 5640 may be included in the control unit 5670.

[0767] The user input interface unit 5650 transmits signals input by the user to the control unit 5670 or transmits signals from the control unit 5670 to the user.

[0768] For example, the user input interface unit 5650 can receive and process control signals related to power on / off, channel selection, and screen settings from the remote control device 4200 according to various communication methods such as RF communication and infrared (IR) communication. Alternatively, the user input interface unit 5650 can process control signals from the control unit 5670 so that they can be transmitted to the remote control device 4200.

[0769] Furthermore, the user input interface unit 5650 can transmit control signals input from local keys (not shown), such as the power key, channel key, volume key, and value setting key, to the control unit 5670.

[0770] The user input interface unit 5650 can transmit control signals input from a sensing unit (not shown) that senses user posture to the control unit 5670, or transmit signals from the control unit 5670 to the sensing unit (not shown). Here, the sensing unit (not shown) may include a touch sensor, a voice sensor, a position sensor, and a motion sensor.

[0771] The control unit 5670 can demultiplex the stream input through the tuner 5610, demodulation unit 5620, or external device interface unit 5656, or process the demultiplexed signal to generate and output a video or audio signal. The control unit 5670 may include the aforementioned encoding and / or decoding devices.

[0772] The image signal processed by the control unit 5670 can be input to the display unit 5680 and displayed as an image corresponding to the image signal. Additionally, the image signal processed by the control unit 5670 can be input to an external output device via the external device interface unit 5656.

[0773] The audio signal processed by the control unit 5670 can be the audio signal output to the audio output unit 5685. Alternatively, the audio signal processed by the control unit 5670 can be input to an external output device via the external device interface unit 5656.

[0774] Although not in Figure 56 As illustrated in the example, the control unit 5670 may include a demultiplexer, an image processor, etc.

[0775] The control unit 5670 can control the overall operation of the digital device 5600. For example, the control unit 5670 can control the tuner 5610 to tune to the RF broadcast corresponding to a user-selected channel or a pre-stored channel.

[0776] The control unit 5670 can control the digital device 5600 according to user commands or internal programs input through the user input interface unit 5650. Specifically, it can access a network and download desired applications or application lists to the digital device 5600.

[0777] For example, the control unit 5670 controls the tuner 5610 to input a signal of a channel selected according to a predetermined channel selection command received through the user input interface unit 5650 and to process the video, audio, or data signal of the selected channel. The control unit 5670 enables the channel information selected by the user to be output along with the processed image or audio signal through the display unit 5680 or the audio output unit 5685.

[0778] As another example, based on an external device image playback command received through the user input interface unit 5650, the control unit 5670 can enable audio signals input from an external device (e.g., a camera or camcorder) through the external device interface unit 5635 to be output through the display unit 5680 or the audio output unit 5685.

[0779] Furthermore, the control unit 5670 can control the display unit 5680 to display images. For example, the control unit 5670 can control the display unit 5680 to display broadcast images input through the tuner 5610, externally input images input through the external device interface unit 5656, images input through the network interface unit, or images stored in the storage unit 5640. In this case, the image displayed on the display unit 5680 can be a still image or a video, and can be 2D video or 3D video.

[0780] Additionally, the control unit 5670 can control the playback of content. The content can be stored in the digital device 5600, received broadcast content, or externally input content. The content can be at least one of broadcast images, externally input images, audio files, still images, connected network screens, and document files.

[0781] When an application is entered to view an entry, the control unit 5670 can control the display of a list or application located within the digital device 5600 or downloadable from an external network.

[0782] The control unit 5670 can control the installation and operation of applications downloaded from an external network, along with various user interfaces. Furthermore, the control unit 5670 can control the display unit 5680 to display images related to the application to be executed, based on user selection.

[0783] Although not shown in the figure, it can be further equipped with a channel browsing processing unit to generate thumbnail images corresponding to channel signals or external input signals.

[0784] The channel browsing processing unit can receive the streaming signal TS output from the demodulation unit 5620 or the streaming signal output from the external device interface unit 5656, extract images from the input streaming signal, and generate thumbnail images.

[0785] The generated thumbnail images can be input to the control unit 5670 either as is or encoded. Alternatively, the generated thumbnail images can be encoded in stream form and input to the control unit 5670. The control unit 5670 can use the input thumbnail images to display a thumbnail list containing multiple thumbnail images on the display unit 5680. The thumbnail images in this list can be updated sequentially or simultaneously. Therefore, the user can easily grasp the content of multiple broadcast channels.

[0786] The display unit 5680 converts the image signals, data signals, OSD signals, etc. processed by the control unit 5670, or the image signals and data signals received from the external device interface unit 5656, into R, G, and B signals respectively, and generates drive signals.

[0787] The display unit 5680 can be a PDP, LCD, OLED, flexible display, 3D display, etc.

[0788] In addition to being an output device, the display unit 5680 can be configured as a touch screen and used as an input device.

[0789] The audio output unit 5685 receives signals (e.g., stereo signals, 3.1 channel signals, or 5.1 channel signals) processed by the control unit 5670 and outputs them as audio. The audio output unit 5685 can be implemented as various types of speakers.

[0790] In addition, to sense the user's posture, as described above, a sensing unit (not shown) having at least one of a touch sensor, a voice sensor, a position sensor, and a motion sensor may be provided in the digital device 5600. The signal detected by the sensing unit (not shown) can be transmitted to the control unit 5670 through the user input interface unit 5650.

[0791] A camera unit (not shown) can also be configured to capture images of the user. The image information captured by the camera unit (not shown) can be input to the control unit 5670.

[0792] The control unit 5670 can detect the user's posture independently or in combination with images captured by the camera unit (not shown) or signals sensed by the sensing unit (not shown).

[0793] The power supply unit 5690 can supply power to the entire digital device 5600.

[0794] Specifically, the power supply unit 5690 can supply power to the control unit 5670, the display unit 5680 for image display, and the audio output unit 5685 for audio output, which can be implemented in the form of a system-on-a-chip (SoC).

[0795] For this purpose, the power supply unit 5690 may include a converter (not shown) that converts AC power into DC power. For example, when the display unit 5680 is implemented as a liquid crystal panel with multiple backlights, a PWM-operable inverter (not shown) for changing brightness or dimming drive may also be provided.

[0796] The remote control device 5700 sends user input to the user input interface unit 5650. For this purpose, the remote control device 5700 can use Bluetooth, RF (radio frequency) communication, infrared (IR) communication, UWB (ultra-wideband), ZigBee, or other methods.

[0797] In addition, the remote control device 5700 can receive image, audio or data signals output from the user input interface unit 5650 and display them on the remote control device 5700, or output voice or vibration.

[0798] The aforementioned digital device 5600 can be a digital broadcast receiver capable of processing fixed or mobile ATSC or DVB type digital broadcast signals.

[0799] Furthermore, the digital device according to this disclosure may omit some components or include components not illustrated, as needed. As mentioned above, the digital device may be without a tuner and demodulator, and may also receive and play content via a network interface unit or an external device interface unit.

[0800] Figure 57 This is an example Figures 54 to 56 A block diagram showing the detailed configuration of the control unit.

[0801] Examples of control units may include a demultiplexing unit 5710, an image processing unit 5720, a screen display OSD generation unit 5740, a mixer 5750, a frame rate converter (FRC) 5755, and a formatter 5760. Additionally, although not shown, the control unit may also include a voice processing unit and a data processing unit.

[0802] The demultiplexing unit 5710 can demultiplex the input stream. For example, the demultiplexing unit 5710 can demultiplex the input MPEG-2TS into a video signal, an audio signal, and a data signal. Here, the stream signal input to the demultiplexing unit 5710 can be a stream signal output from a tuner, demodulator, or external device interface.

[0803] The image processing unit 5720 performs image processing on the demultiplexed image signal. For this purpose, the image processing unit 5720 may include a video decoder 5725 and a scaler 5735.

[0804] The video decoder 5725 decodes the demultiplexed video signal, and the scaler 5735 scales the resolution of the decoded video signal for output on the display.

[0805] The video decoder 5725 supports various standards. For example, when the video signal is encoded using the MPEG-2 standard, the video decoder 5725 performs the function of an MPEG-2 decoder, and when the video signal is encoded using the Digital Multimedia Broadcasting (DMB) method or the H.264 standard, the video decoder 5725 performs the function of an H.264 decoder.

[0806] The video signal decoded by the image processing unit 5720 is input to the mixer 5750.

[0807] OSD generation unit 5740 generates OSD data based on user input or its own input. For example, OSD generation unit 5740 generates data for displaying various data in graphical or textual form on the screen of display unit 5680 based on control signals from user input interface unit. The generated OSD data includes various data such as user interface screens of digital devices, various menu screens, widgets, icons, and viewership information.

[0808] The OSD generation unit 5740 can generate data based on EPG to display captions or broadcast information for broadcast images.

[0809] Mixer 5750 mixes the OSD data generated by OSD generation unit 5740 with the image signal processed by image processing unit and provides it to formatter 5760. Because the decoded video signal and OSD data are mixed, the OSD is displayed as an overlay on the broadcast video or externally input video.

[0810] The frame rate converter FRC 5755 converts the frame rate of the input video. For example, depending on the output frequency of the display unit, the frame rate converter 5755 can convert an input 60Hz image frame rate to a frame rate of, for example, 120Hz or 240Hz. As mentioned above, various methods exist for converting frame rates. For example, when the frame rate converter 5755 converts the frame rate from 60Hz to 120Hz, the same first frame is inserted between the first and second frames, or a third frame predicted from the first and second frames is inserted between the first and second frames. As another example, when the frame rate converter 5755 converts the frame rate from 60Hz to 240Hz, three identical frames or predicted frames can be inserted between existing frames. If a separate frame conversion is not performed, the frame rate converter 5755 can be ignored.

[0811] The formatter 5760 modifies the output of the frame converter 5755 to match the output format of the display unit. For example, the formatter 5760 can output R, G, and B data signals, and these R, G, and B data signals can be output as Low Voltage Differential Signaling (LVDS) or Small LVDS. Additionally, when the output of the frame converter 5755 is a 3D video signal, the formatter 5760 can support 3D services through the display unit by configuring the output to a 3D format according to the display unit's output format.

[0812] The audio processing unit (not shown) in the control unit can perform audio processing on the demultiplexed audio signal. The audio processing unit (not shown) can support various audio formats. For example, even when the audio signal is encoded in formats such as MPEG-2, MPEG-4, AAC, HE-AAC, AC-3, BSAC, the audio processing unit can be equipped with a corresponding decoder.

[0813] In addition, the audio processing unit (not shown) in the control unit can handle bass, treble, volume control, etc.

[0814] The data processing unit (not shown) in the control unit can perform data processing on the demultiplexed data signal. For example, the data processing unit can decode the demultiplexed data signal even when it is being encoded. Here, the encoded data signal can be EPG information, including broadcast information such as the start and end times of broadcast programs broadcast on each channel.

[0815] Furthermore, the digital devices described above are examples according to this disclosure, and each component may be integrated, added, or omitted depending on the specifications of the actual digital device. That is, if necessary, two or more components may be combined into one component, or a component may be subdivided into two or more components. Additionally, the functions performed in each block are for describing embodiments of this disclosure, and specific operations or devices do not limit the scope of this disclosure.

[0816] A digital device can be an image signal processing device that performs signal processing on an image stored in the device or an input image. As another example of an image signal processing device, besides… Figure 56 In addition to the display unit 5680 and audio output unit 5685 shown, examples may include set-top boxes (STBs), DVD players, Blu-ray players, game devices, computers, etc.

[0817] Figure 58 This is an illustration of an example in which the screen of a digital device simultaneously displays a main image and a secondary image according to an embodiment.

[0818] According to the embodiments, the digital device can simultaneously display a main image 5810 and an auxiliary image or auxiliary image 5820 on a screen 5800. The main image 5810 may be referred to as the first image, and the auxiliary image 5820 may be referred to as the second image. The main image 5810 and the auxiliary image 5820 may include, but are not limited to, video, still images, electronic program guides (EPGs), graphical user interfaces (GUIs), on-screen operating systems (OSDs), etc. The main image 5810 may refer to an image that is relatively smaller than the screen 5800 of the electronic device when displayed simultaneously on the screen 5800 together with the auxiliary image 5820. Figure 58 In this display, the main image 5810 is displayed in the upper left corner of the screen 5800 of the digital device, but the display position of the main image 5810 is not limited to this, and the main image 5810 can be displayed at any position on the screen 5800 of the digital device.

[0819] The main image 5810 and the auxiliary image 5820 can be directly or indirectly related to each other. As an example, the main image 5810 can be a streamed video, and the auxiliary image 5820 can be a GUI that sequentially displays thumbnails of the video including information similar to that of the streamed video. As another example, the main image 5810 can be a broadcast image, and the auxiliary image 5820 can be an EPG. As yet another example, the main image 5810 can be a broadcast image, and the auxiliary image 5820 can be a GUI. Examples of the main image 5810 and the auxiliary image 5820 are not limited to these.

[0820] In one implementation, the main image 5810 is a broadcast image received via a broadcast channel, and the auxiliary image 5820 may be information related to the broadcast image received via the broadcast channel. Information related to the broadcast image received via the broadcast channel may include, for example, EPG information, which includes, but is not limited to, a comprehensive channel schedule and detailed broadcast program information, as well as broadcast program commentary information.

[0821] In another embodiment, the main image 5810 is a broadcast image received via a broadcast channel, and the auxiliary image 5820 may be an image generated based on information pre-stored in a digital device. The image generated based on the information pre-stored in the digital device may include, for example, a basic user interface (UI) for an EPG, basic channel information, an image resolution manipulation UI, and a bedtime scheduling UI, but is not limited thereto.

[0822] In another embodiment, the main image 5810 is a broadcast image received via a broadcast channel, and the auxiliary image 5820 may be information related to the broadcast image received via a network. The information related to the broadcast image received via a network may be, for example, information obtained through a network-based search engine. More specifically, for example, information related to the characters currently displayed on the main image 5810 may be obtained through a network-based search engine.

[0823] However, the examples are not limited to this, and information related to broadcast images received over a network can be obtained by using, for example, artificial intelligence (AI) systems. More specifically, for example, the estimated location on a map of a place currently displayed on the main image 5810 can be obtained by using network-based deep learning, and a digital device can receive information about the estimated location on a map of a place currently displayed on the main image 5810 over the network.

[0824] The digital device according to the embodiment can receive at least one of image information of a main image 5810 and image information of an auxiliary image 5820 from an external source. The image information of the main image 5810 may include, for example, broadcast signals received via a broadcast channel, source code information of the main image 5810, and IP packet (Internet Protocol packet) information of the main image 5810 received via a network, but is not limited thereto. Similarly, the image information of the auxiliary image 5820 includes, for example, broadcast signals received via a broadcast channel, source code information of the auxiliary image 5820, and IP packet information of the auxiliary image 5820 received via a network, but is not limited thereto. The digital device can decode and use the image information of the main image 5810 or the image information of the auxiliary image 5820 received from an external source. However, in some cases, the digital device may internally store the image information of the main image 5810 or the image information of the auxiliary image 5820.

[0825] The digital device can display the main image 5810 and the auxiliary image 5820 on the screen 5800 of the digital device based on the image information of the main image 5810 and the information related to the auxiliary image 5820.

[0826] In one example, the decoding device 200 of the digital device includes a main image decoding device and an auxiliary image decoding device, which can decode the image information of the main image 5810 and the image information of the auxiliary image 5820, respectively. The renderer includes a main video renderer (first renderer) and an auxiliary video renderer (second renderer). The main image renderer can display the main image 5810 on a first area of ​​the screen 5800 of the digital device based on the information decoded by the main image decoding device, and the auxiliary image renderer can display the auxiliary image 5820 on a second area of ​​the screen 5800 of the digital device based on the information decoded by the auxiliary image decoding device.

[0827] In another example, the decoding device 200 of the digital device can decode the image information of the main image 5810 and the image information of the auxiliary image 5820. Based on the information decoded by the decoding device 200, the renderer can process the main image 5810 and the auxiliary image 5820 together to display them simultaneously on the screen 5800 of the digital device.

[0828] That is, according to this document, a method for processing image services in a digital device can be provided. The image service processing method may include: receiving image information, decoding a (main) image based on the image information, rendering the decoded image or displaying the decoded image in a first area of ​​a display, and rendering an auxiliary image or displaying the auxiliary image in a second area of ​​the display. In this case, the step of decoding the first image can follow the steps described above. Figure 3The decoding process in the decoding device 200. For example, as described above, the steps of decoding the first image may include: deriving prediction samples for the current block based on inter-frame or intra-frame prediction, deriving residual samples for the current block based on received residual information, and generating reconstructed samples based on the prediction samples and / or residual samples. Additionally, the steps of decoding the first image may include performing an in-loop filtering process on the reconstructed image including the reconstructed samples.

[0829] For example, the supplementary image can be an Electronic Program Guide (EPG), an on-screen OSD, or a Graphical User Interface (GUI). For example, image information can be received via a broadcast network, and information about the supplementary image can also be received via a broadcast network. For example, image information can be received via a communication network, and information about the supplementary image can also be received via a communication network. For example, image information can be received via a broadcast network or a communication network, and information about the supplementary image can be stored in a storage medium on a digital device.

[0830] In the embodiments mentioned above, the elements and features of this disclosure have been combined in a specific manner. Each of these elements or features may be considered optional unless otherwise explicitly described. Each of these elements or features may be implemented without combination with other elements or features. Furthermore, some of these elements and / or features may be combined to form embodiments of this disclosure. The order of operations described in the embodiments of this disclosure may be changed. Some elements or features of an embodiment may be included in another embodiment, or may be replaced by corresponding elements or features of another embodiment. It is apparent that embodiments may be constructed by combining claims that are not expressly referenced in the claims, or may be included as new claims by amendments after filing the application.

[0831] The embodiments of this disclosure can be implemented by various means (e.g., hardware, firmware, software, or combinations thereof). In the case of hardware implementation, the embodiments of this disclosure can be implemented using one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, etc.

[0832] When implemented in firmware or software, embodiments of this disclosure can be implemented as modules, programs, or functions that perform the functions or operations mentioned above. The software code can be stored in memory and driven by a processor. The memory can be located internally or externally to the processor and can exchange data with the processor using various known means.

[0833] It will be apparent to those skilled in the art that this disclosure may be implemented in other specific forms without departing from its essential characteristics. Therefore, the detailed description should not be construed as restrictive, but rather as illustrative in all respects. The scope of this disclosure should be determined through a reasonable analysis of the appended claims, and all modifications within the equivalent scope of this disclosure are included within its scope.

[0834] Industrial applicability

[0835] The preferred embodiments mentioned above have been disclosed for illustrative purposes, and those skilled in the art can improve, modify, substitute or add various other embodiments without departing from the technical spirit and scope of the present disclosure as disclosed in the appended claims.

Claims

1. A method for processing a video signal based on inter-frame prediction by a device, the method comprising the following steps: Obtain a flag indicating whether the merge mode is applied to the current block; Based on the merging flag, it is determined whether the merging mode is applied to the current block among multiple inter-frame prediction modes, wherein the multiple inter-frame prediction modes include the merging mode and the motion vector prediction MVP mode; Based on the determination of the merge mode applied to the current block, a merge list is configured based on the neighboring blocks of the current block; The HMVP candidates included in the historical motion vector prediction HMVP candidate list are added to the merge list until the number of merge candidates included in the merge list is equal to the value obtained by subtracting 1 from the maximum number of merge candidates, wherein the HMVP candidate represents the motion information of the block encoded before the current block; Obtain the merge index indicating the merge candidate for inter-frame prediction of the current block within the merge list; The predicted block for the current block is generated based on the motion information of the merging candidates indicated by the merging index; and The current block is reconstructed based on the predicted block and the residual block of the current block. The step of adding the HMVP candidate to the merge list includes the following steps: Check whether a first number of HMVP candidates in the HMVP candidate list have the same motion information as the merge candidates included in the merge list. Specifically, based on the fact that the first number of HMVP candidates have motion information different from the predefined spatial merging candidates included in the merging list as some of the spatial merging candidates, the first number of HMVP candidates are added to the merging list. The predefined spatial merging candidates include at least one of the left-hand or top-hand blocks adjacent to the current block, and The number of predefined spatial merge candidates is less than the number of spatial merge candidates included in the merge list.

2. A method for encoding video signals based on inter-frame prediction, the method comprising the following steps: Determine whether a merging mode is applied to the current block among multiple inter-frame prediction modes, wherein the multiple inter-frame prediction modes include the merging mode and the motion vector prediction MVP mode, and wherein a merging flag indicating whether the merging mode is applied to the current block is generated; Based on the determination of the merge mode applied to the current block, a merge list is generated based on the neighboring blocks of the current block; The HMVP candidates included in the historical motion vector prediction HMVP candidate list are added to the merge list until the number of merge candidates included in the merge list is equal to the value obtained by subtracting 1 from the maximum number of merge candidates, wherein the HMVP candidate represents the motion information of the block encoded before the current block; Generate a merge index that indicates merge candidates for inter-frame prediction of the current block within the merge list; The predicted block for the current block is generated based on the motion information of the merging candidates indicated by the merging index; and Generate a residual block for the current block based on the current block and the predicted block. The step of adding the HMVP candidate to the merge list includes the following steps: Check whether a first number of HMVP candidates in the HMVP candidate list have the same motion information as the merge candidates included in the merge list. Specifically, based on the fact that the first number of HMVP candidates have motion information different from that of the predefined spatial merging candidates included in the merging list, the first number of HMVP candidates are added to the merging list. The predefined spatial merging candidates include at least one of the left-hand or top-hand blocks adjacent to the current block, and The number of predefined spatial merge candidates is less than the number of spatial merge candidates included in the merge list.

3. A method for transmitting a bit stream, the method comprising the following steps: Determine whether a merging mode is applied to the current block among multiple inter-frame prediction modes, wherein the multiple inter-frame prediction modes include the merging mode and the motion vector prediction MVP mode, and wherein a merging flag indicating whether the merging mode is applied to the current block is generated; Based on the determination of the merge mode applied to the current block, a merge list is generated based on the neighboring blocks of the current block; The HMVP candidates included in the historical motion vector prediction HMVP candidate list are added to the merge list until the number of merge candidates included in the merge list is equal to the value obtained by subtracting 1 from the maximum number of merge candidates, wherein the HMVP candidate represents the motion information of the block encoded before the current block; Generate a merge index that indicates merge candidates for inter-frame prediction of the current block within the merge list; The prediction block of the current block is generated based on the motion information of the merging candidate indicated by the merging index; Generate a residual block for the current block based on the current block and the predicted block; Generate the bitstream including the merge index; and Send the generated bit stream. The step of adding the HMVP candidate to the merge list includes the following steps: Check whether a first number of HMVP candidates in the HMVP candidate list have the same motion information as the merge candidates included in the merge list. Specifically, based on the fact that the first number of HMVP candidates have motion information different from the predefined spatial merging candidates included in the merging list as some of the spatial merging candidates, the first number of HMVP candidates are added to the merging list. The predefined spatial merging candidates include at least one of the left-hand or top-hand blocks adjacent to the current block, and The number of predefined spatial merge candidates is less than the number of spatial merge candidates included in the merge list.

Images