Image decoding / coding method and bit stream transmission method

Abstract

The present invention relates to a video encoding method and a video decoding method, and the video decoding method suitable for one embodiment of the present invention may comprise: a step of determining an initial motion vector of a current block by using a motion vector of a reconstruction region; searching a motion vector of the current block on the basis of the initial motion vector; and generating a prediction sample of the current block by using the motion vector, wherein the initial motion vector can comprise a motion vector in a historical direction and a motion vector in a future direction.

Description

[0001] This application is a divisional application of patent application No. 201980041236.0, filed on June 18, 2019, entitled "Image Encoding / Decoding Method and Apparatus". Technical Field

[0002] This invention relates to an image encoding / decoding method and apparatus, and more particularly to a prediction method and apparatus for the current block using multiple prediction blocks generated based on multiple prediction information. Background Technology

[0003] In recent years, the demand for multimedia data such as video on the Internet has been increasing dramatically. However, the current development speed of channel bandwidth is insufficient to fully meet the rapidly increasing volume of multimedia data. To address this, the Video Coding Expert Group (VCEG) of the International Telecommunication Union Telecommunication Standardization Sector (ITU-T) and the Moving Picture Expert Group (MPEG) of the International Organization for Standardization / International Electrotechnical Commission (ISO / IEC) developed the first version of the video compression standard, High Efficiency Video Coding (HEVC), in February 2014.

[0004] High Efficiency Video Coding (HEVC) defines various techniques such as intra-frame prediction, inter-frame prediction, transform, quantization, entropy coding, and loop filtering. Summary of the Invention

[0005] Technical issues The purpose of this invention is to provide a method and apparatus for predicting the current block using multiple prediction blocks generated based on multiple prediction information.

[0006] Furthermore, the main objective of this invention is to provide an encoding and decoding method that can more accurately derive motion information.

[0007] Furthermore, the present invention aims to provide an efficient encoding / decoding method and apparatus that utilizes the encoded information of the reconstructed region.

[0008] Furthermore, the present invention aims to provide an efficient encoding / decoding method and apparatus for motion vector difference information.

[0009] Furthermore, the present invention aims to provide a recording medium for storing a bitstream generated by an image encoding method or apparatus to which the present invention is applicable.

[0010] Furthermore, the present invention aims to provide a recording medium for storing a bitstream decoded by an image decoding method or apparatus to which the present invention is applicable.

[0011] The technical issues to be addressed by this invention are not limited to those mentioned above. Those skilled in the art to which this invention pertains will be able to further understand other technical issues not mentioned through the following description.

[0012] Technical solution An image decoding method applicable to one aspect of the present invention may include: generating a first prediction block by performing inter-frame prediction on a current block; generating a second prediction block by performing intra-frame prediction on the current block; and generating a final prediction block by performing a weighted sum on the first prediction block and the second prediction block.

[0013] In the image decoding method to which this invention is applied, the second prediction block may be a block predicted using only a portion of the available intra-frame prediction modes.

[0014] In the image decoding method to which the present invention is applied, the step of generating the first prediction block may include: determining the initial motion vector of the current block using the motion vector of the reconstructed region; exploring the motion vector of the current block based on the initial motion vector; and generating a prediction sample of the current block using the motion vector; wherein the initial motion vector may include motion vectors in historical directions and motion vectors in future directions.

[0015] In the image decoding method to which this invention is applied, the step of exploring motion vectors described above can be performed by utilizing the difference between the prediction block of the historical direction indicated by the motion vector of the historical direction and the prediction block of the future direction indicated by the motion vector of the future direction.

[0016] In the image decoding method to which this invention is applied, the step of exploring motion vectors described above can be based on the prediction error generated by applying the difference between the motion vectors in the historical direction and the predicted blocks in the future direction to the sum of absolute differences (SAD) calculation method.

[0017] In the image decoding method of the present invention, the above-described step of exploring motion vectors can be performed within a predefined exploration area.

[0018] In the image decoding method to which this invention is applied, the motion vector of the reconstructed region can be candidate motion information of the merging pattern.

[0019] In the image decoding method applicable to the present invention, the candidate motion information of the above-mentioned merging mode may include at least one of the motion information of the spatial candidate block and the motion information of the temporal candidate block of the current block.

[0020] The image decoding method to which this invention is applied may further include: a step of decoding decoder-side motion vector derivation (DMVD) mode motion information; wherein, when the aforementioned decoder-side motion vector derivation (DMVD) mode motion information indicates decoder-side motion vector derivation (DMVD) mode motion, a step of exploring the motion vector of the current block may be performed based on the aforementioned initial motion vector.

[0021] In the image decoding method to which this invention is applied, the motion vectors explored above can be used in the derivation of motion vectors for blocks decoded after the current block.

[0022] An image encoding method applicable to another aspect of the present invention may include: generating a first prediction block by performing inter-frame prediction on the current block; generating a second prediction block by performing intra-frame prediction on the current block; and generating a final prediction block by performing a weighted sum on the first prediction block and the second prediction block.

[0023] In the image coding method to which this invention is applied, the second prediction block may be a block predicted using only a portion of the available intra-frame prediction modes.

[0024] In the image coding method applicable to the present invention, the step of generating the first prediction block may include: determining the initial motion vector of the current block using the motion vector of the reconstructed region; exploring the optimal motion vector of the current block based on the initial motion vector; and decoding the motion information of the decoder-side motion vector derivation (DMVD) mode by determining whether the optimal motion vector is used; wherein the initial motion vector may include motion vectors in the historical direction and motion vectors in the future direction.

[0025] In the image coding method to which this invention is applied, the step of exploring the optimal motion vector described above can be carried out by utilizing the difference between the prediction block of the historical direction indicated by the motion vector of the historical direction and the prediction block of the future direction indicated by the motion vector of the future direction to explore the optimal motion vector.

[0026] In the image coding method to which this invention is applied, the step of exploring the optimal motion vector described above can be based on the prediction error generated by applying the difference between the motion vector in the historical direction and the predicted block in the future direction to the sum of absolute differences (SAD) calculation method.

[0027] In the image coding method to which this invention is applied, the step of exploring the optimal motion vector described above can be performed within a predefined exploration area.

[0028] In the image coding method applicable to the present invention, the motion vector of the reconstructed region can be candidate motion information of the merging pattern.

[0029] In the image encoding method to which this invention is applied, the candidate motion information of the above-mentioned merging mode may include at least one of the motion information of the spatial candidate block and the motion information of the temporal candidate block of the current block.

[0030] In the image coding method to which this invention is applied, the aforementioned optimal motion vector can be used in the derivation of motion vectors for blocks encoded after the current block.

[0031] An image decoding method applicable to one embodiment of the present invention may include: a step of determining an initial motion vector of a current block using motion vectors of a reconstructed region; a step of exploring motion vectors of the current block based on the initial motion vectors; and a step of generating a prediction sample of the current block using the motion vectors; wherein the initial motion vectors may include motion vectors in historical directions and motion vectors in future directions.

[0032] In the above image decoding method, the step of exploring motion vectors can be achieved by utilizing the difference between the prediction block of the historical direction indicated by the motion vector of the historical direction and the prediction block of the future direction indicated by the motion vector of the future direction.

[0033] In the above image decoding method, the step of exploring motion vectors can be based on the prediction error generated by applying the difference between the motion vectors in the historical direction and the predicted blocks in the future direction to the sum of absolute errors (SAD).

[0034] In the above image decoding method, the step of exploring motion vectors can be performed within a predefined exploration area.

[0035] In the above image decoding method, the motion vector of the reconstructed region can be candidate motion information of the merging pattern.

[0036] In the above image decoding method, the candidate motion information of the above merging mode may include at least one of the motion information of the spatial candidate block and the motion information of the temporal candidate block of the current block.

[0037] The above image decoding method may further include: a step of decoding decoder-side motion vector derivation (DMVD) mode motion information; wherein, when the above decoder-side motion vector derivation (DMVD) mode motion information indicates decoder-side motion vector derivation (DMVD) mode motion, a step of exploring the motion vector of the current block may be performed based on the above initial motion vector.

[0038] In the above image decoding method, the motion vectors explored above can be used in the derivation of motion vectors for blocks decoded after the current block.

[0039] An image encoding method applicable to one embodiment of the present invention may include: a step of determining an initial motion vector of the current block using the motion vector of the reconstructed region; a step of exploring an optimal motion vector for the current block based on the initial motion vector; and a step of decoding the motion information of the decoder-side motion vector derivation (DMVD) mode by determining whether the optimal motion vector is used; wherein the initial motion vector may include motion vectors in historical directions and motion vectors in future directions.

[0040] In the above image coding method, the step of exploring the optimal motion vector can be achieved by utilizing the difference between the prediction block of the historical direction indicated by the motion vector of the historical direction and the prediction block of the future direction indicated by the motion vector of the future direction.

[0041] In the above image coding method, the step of exploring the optimal motion vector can be based on the prediction error generated by applying the difference between the motion vector in the historical direction and the predicted block in the future direction to the sum of absolute differences (SAD) calculation method.

[0042] In the above image coding method, the step of exploring the optimal motion vector can be carried out within a predefined exploration area.

[0043] In the above image coding method, the motion vector of the reconstructed region can be candidate motion information of the merging pattern.

[0044] In the above image encoding method, the candidate motion information of the above merging mode may include at least one of the motion information of the spatial candidate blocks of the current block and the motion information of the temporal candidate blocks.

[0045] In the above image coding method, the optimal motion vector can be used in the derivation of motion vectors for blocks coded after the current block.

[0046] In a computer-readable non-transitory recording medium containing a bitstream used in image decoding to which one embodiment of the present invention applies, the bitstream may include Decoder-side Motion Vector Derivation (DMVD) mode motion information. In the image decoding, when the DMVD mode motion information indicates DMVD mode action, the exploration of motion vectors for the current block can be performed based on an initial motion vector. The initial motion vector may be determined using motion vectors of the reconstructed region, and the initial motion vector may include motion vectors in historical directions and motion vectors in future directions.

[0047] An image decoding method according to another embodiment of the present invention may include: a step of deriving an initial motion vector of a current block; a step of decoding motion vector difference information of the initial motion vector; a step of deriving motion vector difference of the motion vector using the motion vector difference information; and a step of deriving a final motion vector of the current block by adding the initial motion vector and the motion vector difference; wherein the motion vector difference information may include information related to a symbol group composed of one of the X component symbols and Y component symbols of the motion vector difference.

[0048] In the above image decoding method, the motion vector difference information may include at least one of the absolute value information of the X component of the motion vector difference or the absolute value information of the Y component of the motion vector difference.

[0049] In the above image decoding method, the symbol group information can be one of the symbols in a symbol group consisting of one of the X component symbols and one of the Y component symbols.

[0050] In the above image decoding method, binary information can be assigned to each symbol group.

[0051] In the above image decoding method, the symbol group information can be an index used to indicate a symbol group consisting of one of the X component symbols and one of the Y component symbols.

[0052] In the above image decoding method, the initial motion vector can be derived using the current block's merging candidate list.

[0053] An image encoding method applicable to another embodiment of the present invention may include: a step of determining the final motion vector of the current block; a step of determining the motion vector difference of the final motion vector; and a step of encoding the motion vector difference information of the motion vector difference; wherein the motion vector difference and the initial motion vector of the current block may be added together and used to derive the final motion vector, and the motion vector difference information may include information related to a symbol group composed of one of the X component symbols and Y component symbols of the motion vector difference.

[0054] In the above image coding method, the motion vector difference information may include at least one of the absolute value information of the X component of the motion vector difference or the absolute value information of the Y component of the motion vector difference.

[0055] In the above image coding method, the symbol group information can be one of the symbols in a symbol group consisting of one of the X component symbols and one of the Y component symbols.

[0056] In the above image encoding method, binary information can be assigned to each symbol group.

[0057] In the above image coding method, the symbol group information can be an index used to indicate one of the symbols in a symbol group consisting of one of the X component symbols and one of the Y component symbols.

[0058] In the above image coding method, the initial motion vector can be derived using the merging candidate list of the current block.

[0059] A computer-readable recording medium according to another embodiment of the present invention can store a bitstream used when a current block contained in an image is received by an image decoding device and reconstructed. The bitstream may include initial motion vector information of the current block and motion vector difference information of the initial motion vector. The initial motion vector information can be used to derive the initial motion vector of the current block, and the motion vector difference information can be used to derive the motion vector difference of the current block. The final motion vector of the current block can be derived by adding the initial motion vector and the motion vector difference. The motion vector difference information may include information related to a symbol group consisting of one of the X component symbols and the Y component symbols of the motion vector difference.

[0060] A computer-readable recording medium applicable to another aspect of the present invention can store bitstreams generated by the image encoding method and / or apparatus applicable to the present invention.

[0061] A computer-readable recording medium applicable to another aspect of the present invention can store bitstreams decoded by an image decoding method and / or apparatus applicable to the present invention.

[0062] Beneficial effects The present invention can provide an image encoding / decoding method and apparatus with improved compression efficiency.

[0063] Furthermore, the present invention can provide a method and apparatus for predicting the current block using multiple prediction blocks generated based on multiple prediction information.

[0064] Furthermore, this invention can improve the prediction performance between images by more accurately deriving motion information.

[0065] Furthermore, the present invention can improve coding efficiency by reducing the amount of encoded information.

[0066] Furthermore, this invention allows for the design of an efficient parallel processing encoding / decoding structure by defining a template region that can be used within the reconstruction area surrounding the current block used during motion information correction, when designing a method for correcting motion information in an image encoding / decoding apparatus using the same approach without transmitting motion information. This significantly reduces the time consumed in encoding images and thereby reduces the complexity of the encoding apparatus.

[0067] Furthermore, the present invention can provide an encoding / decoding method, apparatus, and recording medium storing bit streams that efficiently allocate corresponding binary information using template matching technology.

[0068] Furthermore, the present invention can provide a method, apparatus, and recording medium storing bit streams for efficiently encoding / decoding motion vector difference information.

[0069] Furthermore, the present invention can provide a computer-readable recording medium for storing a bitstream generated by an image encoding method / apparatus to which the present invention is applicable.

[0070] Furthermore, the present invention can provide a computer-readable recording medium for storing bitstreams decoded by an image decoding method and / or apparatus to which the present invention is applicable. Attached Figure Description

[0071] Figure 1 This is an example diagram illustrating the general structure of the image encoding device 100.

[0072] Figure 2 This is a schematic diagram illustrating one embodiment of the prediction unit of an image encoding apparatus.

[0073] Figure 3 This is a schematic diagram illustrating the method for generating motion information candidate groups in either SKIP or MERGE mode.

[0074] Figure 4 It is a schematic diagram used to illustrate the location of spatial candidate blocks and temporal candidate blocks.

[0075] Figure 5 This is a schematic diagram illustrating the method for determining motion information for time candidates.

[0076] Figure 6 This is a schematic diagram used to illustrate the method for generating in-frame prediction blocks using the mean (DC) mode.

[0077] Figure 7This is a schematic diagram used to illustrate the method for generating in-frame predictive blocks in Planar mode.

[0078] Figure 8 This is a schematic diagram illustrating N directional prediction patterns.

[0079] Figure 9 This is a schematic diagram illustrating the method for constructing a list of most probable pattern (MPM) candidate patterns.

[0080] Figure 10 This is a schematic diagram illustrating an embodiment of encoding information in an image encoding apparatus.

[0081] Figure 11 This is a schematic diagram illustrating the configuration of the image decoding apparatus to which the present invention is applied.

[0082] Figure 12 This is a schematic diagram used to explain the prediction unit of an image decoding device.

[0083] Figure 13 This is a schematic diagram illustrating an embodiment of decoding encoded information in an image decoding apparatus.

[0084] Figure 14 This is a schematic diagram used to explain the prediction unit of an image encoding device.

[0085] Figure 15 This is a schematic diagram used to explain the prediction unit of an image decoding device.

[0086] Figure 16 This is a method for describing an encoding method for encoded information applicable to one embodiment of the present invention.

[0087] Figure 17 This is a method for describing a decoding method for encoded information applicable to one embodiment of the present invention.

[0088] Figure 18 This is an example diagram illustrating the results of the initial motion information exploration unit in the decoder-side motion vector derivation (DMVD) mode of the prediction unit within the image encoding / decoding device.

[0089] Figure 19 This is a schematic diagram used to explain the operation of the motion prediction unit of the decoder-side motion vector derivation (DMVD) in the prediction unit of an image encoding / decoding device.

[0090] Figure 20 This is a diagram used to illustrate template matching modes and bidirectional matching modes.

[0091] Figure 21 This is an example diagram illustrating the method for setting template blocks.

[0092] Figure 22 This is a schematic diagram used to illustrate the use of template matching technology in the reconstruction area around the current block in the in-picture prediction mode.

[0093] Figure 23 This is a schematic diagram used to illustrate the Linear Mode (LM) applicable to color difference blocks.

[0094] Figure 24 This is a schematic diagram illustrating a method for applying a single weighting value to a predicted block.

[0095] Figure 25 This is a schematic diagram illustrating a method for applying multiple weighted values ​​to a predicted block.

[0096] Figure 26 This is a schematic diagram illustrating another method for applying multiple weighted values ​​to opposing prediction blocks.

[0097] Figure 27 This is a schematic diagram illustrating a method for applying weighted values ​​only to a portion of a prediction block.

[0098] Figure 28 This is a schematic diagram illustrating another method for applying weighted values ​​only to a portion of the prediction block.

[0099] Figure 29 This is a flowchart illustrating an image encoding device.

[0100] Figure 30 This is a schematic diagram used to provide a detailed explanation of the prediction unit of an image encoding device.

[0101] Figure 31 This is a flowchart illustrating the derivation method of candidate motion information for skip and merge modes.

[0102] Figure 32 This is a flowchart illustrating the derivation method of candidate motion information for the Advanced Motion Vector Prediction (AMVP) model.

[0103] Figure 33 This is a schematic diagram illustrating the location of the reconstruction blocks used to deduce spatial / temporal candidates from candidate motion information in Skip, Merge, and Advanced Motion Vector Prediction (AMVP) modes.

[0104] Figure 34This is a schematic diagram illustrating the method for deriving temporal candidates from candidate motion information in Skip, Merge, and Advanced Motion Vector Prediction (AMVP) modes.

[0105] Figure 35 This is a schematic diagram illustrating the derivation method of combining bidirectional candidate patterns in candidate motion information of skip and merge modes.

[0106] Figure 36 This is a flowchart illustrating the encoding method for predictive information.

[0107] Figure 37 This is a flowchart providing a schematic diagram of an image decoding device.

[0108] Figure 38 This is a schematic diagram used to explain the prediction unit of an image decoding device.

[0109] Figure 39 This is a flowchart illustrating the method for decoding predicted information.

[0110] Figure 40 This is a schematic diagram illustrating the prediction unit of an image encoding apparatus to which one embodiment of the present invention is applied.

[0111] Figure 41 This is a schematic diagram illustrating the prediction unit of an image decoding apparatus to which one embodiment of the present invention is applied.

[0112] Figure 42 This is a table used to describe the initial motion information exploration unit of the decoder-side motion vector derivation (DMVD) in the prediction unit of an image encoding / decoding apparatus to which one embodiment of the present invention applies.

[0113] Figure 43 This is a schematic diagram for explaining the decoder-side motion vector derivation (DMVD) motion prediction unit in the prediction unit of an image encoding / decoding apparatus to which one embodiment of the present invention applies.

[0114] Figure 44 This is a schematic diagram illustrating a template matching mode for a decoder-side motion vector derivation (DMVD) mode applicable to one embodiment of the present invention.

[0115] Figure 45 This is a schematic diagram illustrating a bidirectional matching mode of a decoder-side motion vector derivation (DMVD) mode applicable to one embodiment of the present invention.

[0116] Figure 46This is a schematic diagram illustrating a motion inference method for a decoder-side motion vector derivation (DMVD) mode applicable to one embodiment of the present invention.

[0117] Figure 47 This is a flowchart illustrating the encoding process of prediction information applicable to one embodiment of the present invention.

[0118] Figure 48 This is a flowchart illustrating the decoding process of prediction information applicable to one embodiment of the present invention.

[0119] Figure 49 This is a schematic diagram illustrating a method for using the best motion information of a template matching pattern as candidate motion information for a skip (SKIP) or merge (MERGE) pattern, according to one embodiment of the present invention.

[0120] Figure 50 This is a schematic diagram illustrating a method for using the best motion information of a template matching pattern as candidate motion information for an advanced motion vector prediction (AMVP) pattern, applicable to one embodiment of the present invention.

[0121] Figure 51 This is a schematic diagram illustrating a method for determining motion information using multiple templates when deriving optimal motion information for a template matching pattern, according to one embodiment of the present invention.

[0122] Figure 52 This is a schematic diagram illustrating a method for correcting motion information in a candidate motion list using a template matching pattern according to an embodiment of the present invention, which is applicable to the method of correcting motion information in a candidate motion list using a template matching pattern.

[0123] Figure 53 This is a schematic diagram illustrating a method for modifying a reference image when correcting candidate motion information for skip and merge modes according to one embodiment of the present invention.

[0124] Figure 54 This is a schematic diagram illustrating a method for rearranging motion information in a candidate motion list using a template matching pattern according to an embodiment of the present invention, employing a template matching pattern.

[0125] Figure 55 This is a schematic diagram illustrating a method for dividing a template of a template matching pattern into sub-templates and applying weighted values ​​of different cost values ​​to each sub-template, according to one embodiment of the present invention.

[0126] Figure 56This is a flowchart illustrating an image encoding method applicable to one embodiment of the present invention.

[0127] Figure 57 This is a flowchart illustrating an image encoding method applicable to one embodiment of the present invention.

[0128] Figure 58 This is a flowchart illustrating an image encoding device.

[0129] Figure 59 This is a schematic diagram used to explain the prediction unit of an image encoding device.

[0130] Figure 60 This is a flowchart illustrating the encoding method for predictive information.

[0131] Figure 61 This is a flowchart illustrating the image decoding device.

[0132] Figure 62 This is a schematic diagram used to explain the prediction unit of an image decoding device.

[0133] Figure 63 This is a flowchart illustrating the method for decoding predicted information.

[0134] Figure 64 This is a schematic diagram illustrating the prediction unit of an image encoding apparatus to which one embodiment of the present invention is applied.

[0135] Figure 65 This is a schematic diagram illustrating the prediction unit of an image decoding apparatus to which one embodiment of the present invention is applied.

[0136] Figure 66 This is a flowchart illustrating an encoding method for encoding and decoding information applicable to one embodiment of the present invention.

[0137] Figure 67 This is a flowchart illustrating a decoding method for encoded and decoded information applicable to one embodiment of the present invention.

[0138] Figure 68 This is a flowchart illustrating a method for encoding / decoding motion vector difference information applicable to one embodiment of the present invention.

[0139] Figure 69 This is a table illustrating a method for determining binary information using template matching cost when encoding motion vector difference information, according to one embodiment of the present invention.

[0140] Figure 70 This is a schematic diagram illustrating a template matching technique for determining binary information of motion vector difference information when an embodiment of the present invention is applied.

[0141] Figure 71 This is a schematic diagram illustrating a method for allocating binary information to each candidate in the most probable mode (MPM) candidate mode list using a template matching technique applicable to the decoder-side intra-mode derivation (DIMD) mode according to an embodiment of the present invention, and a method for allocating binary information to chromatic aberration (Chroma) candidate modes.

[0142] Figure 72 This is a schematic diagram illustrating a template matching technique for a decoder-side intra-frame mode derivation (DIMD) mode applicable to one embodiment of the present invention.

[0143] Figure 73 This is a schematic diagram illustrating a method for determining whether to use a template in the reconstruction area surrounding a predicted block based on whether a template is used in the reconstruction area surrounding the current block when performing a regional brightness compensation technique according to one embodiment of the present invention.

[0144] Figure 74 It is a transformation unit of an image encoding apparatus to which one embodiment of the present invention is applied.

[0145] Figure 75 It is the inverse transformation unit of an image encoding / decoding apparatus applicable to one embodiment of the present invention.

[0146] Figure 76 This is a schematic diagram illustrating a method using template matching technology to determine binary information of the transformation type in the transformation / inverse transformation unit of an image encoding / decoding apparatus, according to one embodiment of the present invention.

[0147] Figure 77 This is a flowchart illustrating an image decoding method applicable to one embodiment of the present invention.

[0148] Figure 78 This is a flowchart illustrating an image encoding method applicable to one embodiment of the present invention. Detailed Implementation

[0149] The embodiments to which the present invention applies will now be described in detail with reference to the accompanying drawings, so that those skilled in the art can easily implement the invention. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Furthermore, irrelevant parts have been omitted in the drawings for clarity of explanation, and similar drawing numbers have been assigned to similar parts throughout the specification.

[0150] Throughout the specification, when a part is described as being "connected" to other parts, this includes not only direct connections but also connections where other components are present between them and are electrically connected.

[0151] Furthermore, throughout the specification, when a part is described as "including" a certain component, unless otherwise expressly stated to the contrary, it does not mean that other components are excluded, but rather that other components may also be included.

[0152] Furthermore, terms such as "first" and "second" may be used when describing different constituent elements, but these terms do not limit the constituent elements. These terms are merely used to distinguish one constituent element from others.

[0153] Furthermore, in embodiments relating to the apparatus and methods described herein, some components of the apparatus or some steps of the method may be omitted. Furthermore, the order of some components of the apparatus or some steps of the method may be changed. Furthermore, other components or other steps may be inserted into some components of the apparatus or some steps of the method.

[0154] Furthermore, a portion of the components or steps applicable to the first embodiment of the present invention may be added to, or replace a portion of the components or steps applicable to the second embodiment of the present invention.

[0155] Furthermore, the constituent parts included in the embodiments of the present invention are illustrated separately only to show different specific functions, and do not represent that each constituent part is composed of a separate hardware or software unit. That is, although the constituent parts are listed for ease of explanation, at least two constituent parts can be combined into one constituent part, or a constituent part can be divided into multiple constituent parts and each part can perform a corresponding function. The combined and divided embodiments of the constituent parts described above are also included within the scope of the claims of the present invention without departing from the essence of the present invention.

[0156] First, the terminology used in this application will be briefly explained below.

[0157] The video decoding device described below can be included in service terminals such as civilian security cameras, civilian security systems, military security cameras, military security systems, personal computers (PCs), laptops, portable multimedia players (PMPs), wireless communication terminals, smartphones, television (TV) application services, and service servers. It can be a device equipped with a communication modem for communicating with various devices such as user terminals and wired and wireless communication networks, a memory for storing various programs and data for performing video decoding or performing inter- or in-screen prediction for decoding, and a microprocessor for performing calculations and control by executing programs.

[0158] Furthermore, images encoded into a bitstream using an encoder can be transmitted in real-time or non-real-time to an image decoding device via wired and wireless communication networks such as the Internet, short-range wireless communication networks, wireless local area networks, wireless broadband access service (WiBro) networks, and mobile communication networks, or via various communication interfaces such as cables and Universal Serial Bus (USB), and then decoded and reconstructed into images for playback. Alternatively, the bitstream generated by the encoder can be stored in a memory. This memory can include volatile memory and non-volatile memory. In this specification, the memory can refer to a recording medium storing the bitstream.

[0159] Typically, video can be composed of a series of pictures, and each picture can be divided into encoding / decoding units such as blocks. Furthermore, those skilled in the art to which this embodiment pertains should understand that the term "picture" used in the following description can be replaced with other terms with the same meaning, such as image or frame. Similarly, those skilled in the art to which this embodiment pertains should understand that the term "encoding / decoding unit" used in the following description can also be replaced with other terms with the same meaning, such as unit block or block.

[0160] Next, embodiments to which the present invention is applied will be described in more detail with reference to the accompanying drawings. In the process of describing the present invention, repeated descriptions of the same constituent elements will be omitted.

[0161] Figure 1This is an example diagram illustrating the general structure of the image encoding device 100.

[0162] The image encoding apparatus 100 is an apparatus for encoding images, and generally includes an image segmentation unit 101, a prediction unit 102, 103, a transformation unit 105, a quantization unit 106, an entropy encoding unit 107, an inverse quantization unit 108, an inverse transformation unit 109, an addition unit 110, a loop filtering unit 111, a memory unit 112, and a subtraction unit 104.

[0163] The image segmentation unit 101 can segment the input image step-by-step from the largest coded object block (hereinafter referred to as "largest coded block") into the smallest coded object block (hereinafter referred to as "smallest coded block"). Various segmentation methods can be used. Quad-tree segmentation (hereinafter referred to as "Quad-Tree segmentation") is a segmentation method that accurately divides the current coded block into four parts. Binary-tree segmentation (hereinafter referred to as "Binary-Tree segmentation") is a segmentation method that accurately divides the coded block into two parts along the horizontal or vertical direction. In addition, various other segmentation methods can be used. Furthermore, a method that simultaneously utilizes multiple segmentation methods can also be employed.

[0164] Prediction units 102 and 103 can generate prediction blocks using pixels surrounding the current prediction target block within the current original block or pixels within a previously encoded / decoded reference image. For a single encoded block, one or more prediction blocks can be generated. When there is only one prediction block for an encoded block, the prediction block can have the same shape as the encoded block.

[0165] Video signal prediction techniques can be broadly categorized into intra-frame prediction and inter-frame prediction. Intra-frame prediction utilizes the surrounding pixels of the current block to generate a prediction block. Inter-frame prediction identifies the most similar block to the current block from previously encoded / decoded reference images and generates a prediction block accordingly.

[0166] After generating the prediction block, a residual block can be generated by subtracting the prediction block from the original block. For the generated residual block, the optimal prediction mode for generating the prediction block can be determined by using various techniques such as Rate-Distortion Optimization (RDO). For example, the calculation formula for the Rate-Distortion Optimization (RDO) cost can be shown in Equation 1.

[0167]

Mathematical Formula 1

[0168] In Equation 1 above, D, R, and J represent the degradation caused by quantization, the bit rate of the compressed stream, and the rate-distortion (RD) cost, respectively. Furthermore, Φ is the encoding mode, and λ is the Lagrangian multiplier. λ is used as a scaling factor to compensate for the unit difference between the error and the bit rate. To select the optimal encoding mode during the encoding process, the rate-distortion (RD) cost when using the appropriate mode should be less than that when using other modes. As shown in Equation 1 above, the rate-distortion (RD) cost should be calculated while simultaneously considering both the bit rate (R) and the error (D).

[0169] Figure 2 This is a schematic diagram illustrating one embodiment of the prediction unit of an image encoding apparatus.

[0170] The in-frame prediction unit 201 can perform in-frame prediction using raw information and reconstructed information. For example, it can perform actions such as selecting at least one reference pixel row from a plurality of reference pixel rows, generating reference pixels, filtering reference pixels, generating prediction blocks using reference pixels, and / or filtering the generated prediction blocks. The in-frame prediction mode detection unit 202 can determine, for example, the optimal in-frame prediction mode using the rate-distortion (RD) cost calculated according to each prediction mode, and then generate or select prediction blocks based on this and finally output them.

[0171] Inter-frame prediction unit 203 can perform inter-frame prediction using raw information and reconstructed information. Inter-frame prediction unit 203 can calculate rate-distortion (RD) costs for multiple inter-frame prediction modes, including SKIP mode, MERGE mode, and Advanced Motion Vector Prediction (AMVP) mode. Merge candidate exploration unit 204 can construct candidate motion information groups for SKIP and MERGE modes. Merge candidate decision unit 205 can determine the optimal motion information from the candidate motion information groups using the rate-distortion (RD) costs. Advanced Motion Vector Prediction (AMVP) candidate exploration unit 206 can construct candidate motion information groups for Advanced Motion Vector Prediction (AMVP) mode. Motion inference unit 207 can perform motion inference using the candidate motion information groups and determine the optimal motion information. Motion compensation unit 208 can generate prediction blocks by performing motion compensation using the optimal motion information. Alternatively, motion compensation unit 208 can ultimately select / output the generated prediction blocks associated with the optimal motion information.

[0172] The aforementioned inter-frame prediction modes can include three modes: Skip mode, Merge mode, and Advanced Motion Vector Prediction (AMVP) mode. Each prediction mode can generate a predicted block for the current block using motion information (prediction direction information, reference image information, and motion vectors). Furthermore, there can also be additional prediction modes that utilize motion information.

[0173] In the SKIP mode, motion information from the reconstructed regions can be used to determine the prediction information (e.g., the best prediction information) for the current block. Motion information candidate groups can be constructed within the reconstructed regions, and the candidate with the lowest rate-distortion (RD) cost among the corresponding candidate groups is used as the prediction information for the current block. The method for constructing the motion information candidate groups in the SKIP mode is the same as the method for constructing the motion information candidate groups in the merge mode, which will be described later.

[0174] In the MERGE mode, the process of using motion information from the reconstructed region to determine the prediction information (e.g., the best prediction information) for the current block is the same as in the SKIP mode. However, the difference lies in that the SKIP mode explores motion information with a prediction error of 0 from the motion information candidate group, while the MERGE mode explores motion information with a prediction error of non-zero from the motion information candidate group. Similar to the SKIP mode, motion information candidate groups can be constructed within the reconstructed region, and the predicted block is generated by using the candidate with the lowest rate-distortion (RD) cost in the corresponding candidate group as the prediction information for the current block.

[0175] Figure 3 This is a schematic diagram illustrating the method for generating motion information candidate groups in either SKIP or MERGE mode.

[0176] Figure 4 It is a schematic diagram used to illustrate the location of spatial candidate blocks and temporal candidate blocks.

[0177] Figure 5 This is a schematic diagram illustrating the method for determining motion information for time candidates.

[0178] The maximum number of motion information candidate groups can be determined to be the same value in both the image encoding device and the image decoding device, or the corresponding quantity information can be transmitted in the upper-level header of the image encoding device (the upper-level header refers to the parameters transmitted at the level above the block, such as video parameter set, sequence parameter set, image parameter set, parallel block header, strip header, etc.).

[0179] In steps S301 and S302, only when spatial candidate blocks and temporal candidate blocks are encoded in an inter-frame prediction mode can the motion information derived from the corresponding motion information be included in the motion information candidate group.

[0180] In step S301, spatial candidates can be selected from the vicinity of the current block within the same image. For example, four out of five spatial candidate blocks at specific locations can be selected as spatial candidates. The locations of the spatial candidate blocks can be as follows: Figure 4 The positions A1 to A5 are shown. However, the number and position of spatial candidate blocks are not limited to this and can be any block within the reconstructed area. Spatial candidates can be considered in the order of A1, A2, A3, A4, A5, and the motion information of the most available spatial candidate blocks can be used to determine the spatial candidates. However, the order in which multiple candidates are considered is not limited to the order described above. In the case of duplicate motion information among multiple spatial candidates, only the motion information of the candidate with the higher priority can be considered.

[0181] In step S302, a time candidate can be selected from an image encoded / decoded prior to the current image. For example, one of two time candidate blocks within a collocated image can be selected as a time candidate. The location of the time candidate block can be as follows: Figure 4 The positions of B1 and B2 are shown. The position of each candidate is determined based on a block within the image that is at the same position as the current block in the current image. The juxtaposed images can be set under the same conditions in both the image encoding and decoding apparatus. For example, a reference image corresponding to a specific reference image index can be selected as the juxtaposed image. Alternatively, signaling can be applied to the index information used to indicate the juxtaposed image. Temporal candidates can be considered in the order of blocks B1 and B2, and motion information of the preferred candidate blocks can be used to determine the temporal candidates. However, the number, position, and consideration order of temporal candidate blocks are not limited to the embodiments described above.

[0182] like Figure 4 as well as Figure 5 As shown, for example, the motion information of a candidate block (B1 or B2) within the juxtaposed image can refer to a predicted block located within the reference image B. The reference images for each candidate block may be different, but for ease of explanation in this specification, they are all represented as reference image B. The corresponding motion vector can be determined as a temporal candidate motion vector by scaling it using the ratio of the distance (TD) between the juxtaposed image and reference image B to the distance (TB) between the current image and reference image A. For example, the scaling described above can be performed using the following mathematical formula 2.

[0183]

Mathematical Formula 2

[0184] Wherein, MV represents the motion vector of the motion information of the time candidate block, MV scale Let TB represent the scaling motion vector, TD represent the temporal distance between the juxtaposed image and reference image B, and TD represent the temporal distance between the current image and reference image A. In this case, reference image A and reference image B can be the same reference image. The scaling motion vector can be used as a temporal candidate motion vector. Alternatively, the reference image information for the temporal candidate motion information can be used as the reference image for the current image, and the temporal candidate motion information can be derived.

[0185] Step S303 can be performed only if the number of motion information candidates derived in steps S301 and S302 is less than the maximum number. In step S303, duplicate candidates can be deleted. Furthermore, new bidirectional motion information candidates can be added using combinations of motion information candidates derived in steps S301 and S302. Bidirectional motion information candidates refer to new candidates derived by sequentially selecting and combining previously derived historical or future direction motion information.

[0186] Table 1 below shows the priority order of candidate combinations of bidirectional motion information. However, Table 1 is only an example, and combinations different from those in Table 1 may exist.

[0187] Table 1

[0188] If the maximum number of motion information candidates cannot be filled even using bidirectional motion information candidates, step S304 will be executed. In step S304, the motion vector of the motion information candidate can be fixed to a zero motion vector, and the maximum number of motion information candidates can be filled by changing the reference image for different prediction directions. The Advanced Motion Vector Prediction (AMVP) mode determines the motion information (e.g., optimal motion information) of the current block by performing motion inference on the reference image for different prediction directions. The prediction direction can be a single direction using only one of the historical / future directions, or a bidirectional direction using both historical and future directions. Using the motion information determined by motion inference, a prediction block can be generated by performing motion compensation. Specifically, motion information candidate groups for motion inference are derived for the reference image for different prediction directions. The corresponding motion information candidate groups are used as the starting position for motion inference.

[0189] The method for deriving motion information candidate sets for motion inference in advanced motion vector prediction (AMVP) models is described in the reference. Figure 3 The explanation method is similar. The maximum number of motion information candidates in Advanced Motion Vector Prediction (AMVP) mode can be determined to be the same value in both the image encoding and decoding devices, or the corresponding number of candidates can be transmitted at the next higher level of the block. In AMVP mode, motion information derived from the corresponding motion information can only be included in the motion information candidate group when spatial candidate blocks and temporal candidate blocks are encoded using an inter-frame prediction mode.

[0190] In the Advanced Motion Vector Prediction (AMVP) mode, the number of spatial candidate blocks can be two, and the priority order for selecting spatial candidates can also differ from that in the MERGE mode. However, the remaining explanations can be the same as those for step S301. Furthermore, the temporal candidates for the AMVP mode can be derived in the same manner as described in step S306.

[0191] In Advanced Motion Vector Prediction (AMVP) mode, duplicate motion information can be removed from currently derived candidate selections. Furthermore, if the number of derived candidates is less than the maximum number of candidates, zero-motion-information candidates can be added in the same manner as in step S308.

[0192] One of the motion information candidates derived in the manner described above can be selected. For example, the motion information candidate with the lowest rate-distortion (RD) cost can be selected as the best motion information candidate, and the best motion information of the advanced motion vector prediction (AMVP) mode can be obtained through the motion inference process based on the corresponding motion information.

[0193] Figure 6 This is a schematic diagram used to illustrate the method for generating in-frame prediction blocks using the mean (DC) mode.

[0194] The in-frame prediction modes can include Planar mode, DC mode, and N directional modes (Angular mode or directional mode), totaling (N+2) prediction modes. In the DC mode, regions R1 to R4 constituting the current block can be predicted using different methods. For example, the average value of reference pixels a to s can be calculated and applied to all predicted pixels in regions R1 to R4. Next, in region R1, the final prediction block can be generated by using finite impulse response (FIR) filtering with two adjacent reference pixels a and j. Furthermore, in regions R2 and R3, the final prediction block can be generated by using finite impulse response (FIR) filtering with one adjacent reference pixel (one of b to h, k to q).

[0195] Figure 7 This is a schematic diagram used to illustrate the method for generating in-frame predictive blocks in Planar mode.

[0196] The Planar mode generates the final prediction block at different prediction pixel positions by using linear interpolation of reference pixels at the top and left positions and copied reference pixels at the bottom and right positions.

[0197] Figure 8 This is a schematic diagram illustrating N directional prediction patterns.

[0198] The directional prediction mode generates the final prediction block by applying the values ​​of adjacent reference pixels in different prediction directions to the prediction block.

[0199] For efficient encoding / decoding of predicted modes within the frame, the Most Probable Mode (MPM) can be used. To construct a candidate mode list for the Most Probable Mode (MPM) of the current block, candidate modes can be derived from the spatial candidate blocks surrounding the current block. The available spatial candidate blocks can then be compared with... Figure 4 The spatial candidate blocks are the same. However, this is not a limitation; the number and / or location of spatial candidate blocks can also be changed.

[0200] Figure 9 This is a schematic diagram illustrating the method for constructing a list of most probable pattern (MPM) candidate patterns.

[0201] In step S901, the reconstructed intra-frame prediction modes of spatial candidate blocks are first derived as the most probable mode (MPM) candidate modes. The intra-frame prediction modes of each block can be used as the most probable mode (MPM) candidate modes in the order of blocks A1, A2, A3, A4, and A5. Candidate blocks for which no intra-frame prediction mode has been reconstructed can be excluded. Furthermore, as described above, the priority order of spatial candidate blocks can be changed.

[0202] In step S902, the candidate in-frame prediction modes derived in step S901 are added as the most probable mode (MPM) candidate modes in descending order of priority. For example, if the candidate modes in step S901 are directional mode 4 and directional mode 11, the directional modes adjacent to directional mode 4 (i.e., modes 3 and 5) and the directional modes adjacent to directional mode 11 (i.e., modes 10 and 12) can be added as the most probable mode (MPM) candidate modes.

[0203] In step S903, a non-directional pattern can be added as a candidate pattern for the most likely pattern (MPM).

[0204] In step S904, it is determined whether the number of current most likely mode (MPM) candidate modes is greater than or equal to the number of maximum most likely mode (MPM) candidate modes. If, in step S904, it is determined that the number of current most likely mode (MPM) candidate modes is less than the number of maximum most likely mode (MPM) candidate modes, a list of most likely mode (MPM) candidate modes can be constructed by adding default candidate modes up to the number of maximum most likely mode (MPM) candidate modes. In this case, the default candidate modes can include any in-frame prediction mode such as vertical mode, horizontal mode, mean (DC) mode, or planar mode. The priority order among the default candidate modes is not limited to the order described above, but can be changed. If it is determined that the number of current most likely mode (MPM) candidate modes is greater than or equal to the number of maximum most likely mode (MPM) candidate modes, a list of most likely mode (MPM) candidate modes can be constructed using the most likely mode (MPM) candidate modes corresponding to the number of maximum most likely mode (MPM) candidate modes in descending order of priority.

[0205] The transform unit 105 generates a transform block by transforming the residual block, which is the difference between the original block and the prediction block. The transform block is the smallest unit used in the transform and quantization process. The transform unit 105 generates a transform block with transform coefficients by transforming the residual signal from a spatial region to a frequency region. Various transform techniques, such as Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), and Karhunen-Loeve Transform (KLT), can be used to transform the residual signal into a frequency region. By performing the transform, the residual signal can be transformed into a frequency region, thereby generating transform coefficients. To facilitate the use of transform techniques, matrix operations using basis vectors are performed. Depending on the prediction mode used when encoding the prediction block, multiple different transform techniques can be mixed during matrix operations. For example, when performing in-frame prediction, a Discrete Cosine Transform can be used in the horizontal direction and a Discrete Sine Transform in the vertical direction, depending on the prediction mode. Alternatively, different transformation techniques can be used in the horizontal and vertical directions based on the aspect ratio, size (length of one or both sides in the horizontal or vertical direction), shape, prediction mode (inter-frame prediction or intra-frame prediction), and inter-frame prediction mode of the current block.

[0206] The quantization unit 106 can generate a quantized transform block by quantizing the transform block. That is, the quantization unit can generate a quantized transform block with quantized transform coefficients by quantizing the transform coefficients of the transform block generated from the transform unit 105. As a quantization method, methods such as Dead Zone Uniform Threshold Quantization (DZUTQ) or Quantization Weighted Matrix can be used. However, it is not limited to the above examples; for example, various quantization methods that are improvements on the above quantization methods can also be used.

[0207] In the above description, the image encoding apparatus including a transform unit and a quantization unit has been illustrated and explained. However, the image encoding apparatus may selectively include a transform unit and / or a quantization unit. That is, the image encoding apparatus may generate a transform block by transforming the residual block but without performing a quantization process. Alternatively, the image encoding apparatus may perform only quantization without transforming the residual block. Alternatively, the image encoding apparatus may not perform both transform and quantization. Even when only a part of the transform unit or quantization unit is performed in the image encoding apparatus, or when all processes are not performed, the block input to the entropy encoding unit 107 is generally referred to as a "quantization transform block".

[0208] The entropy encoding unit 107 can encode the quantization transform block and output a bitstream. That is, the entropy encoding unit can use various encoding techniques such as entropy encoding to encode the coefficients of the quantization transform block output from the quantization unit, and generate and output a bitstream containing additional information (e.g., prediction mode-related information (information related to the prediction mode refers to information belonging to intra-frame prediction or inter-frame prediction), quantization coefficients, etc.) required for decoding the corresponding block in the image decoding device described later.

[0209] The inverse quantization unit 108 can reconstruct the inverse quantization transform block by reversing the quantization techniques used in the quantization process on the quantization transform block.

[0210] The inverse transform unit 109 reconstructs the residual block by performing an inverse transform on the inverse quantization transform block using the same method as that used in the transform process. The inverse transform can be performed by reversing the transform technique used in the transform unit.

[0211] Specifically, the inverse quantization unit and the inverse transform unit can perform inverse quantization and inverse transform by reversing the quantization and transform methods used in the quantization unit and the transform unit. Alternatively, if only quantization is performed without transform in the encoding step, only inverse quantization can be performed without inverse transform in the reconstruction step. Furthermore, if no transform or inverse transform is performed in the encoding step, inverse transform and inverse quantization can be omitted in the decoding step. Alternatively, information related to the corresponding block can be omitted from the image encoding device.

[0212] The addition unit 110 can reconstruct the current block by performing an addition operation on the residual signal generated in the inverse transform unit and the prediction block generated by prediction.

[0213] The filtering unit 111 performs filtering on the entire image after reconstructing all blocks within the current image. This filtering can include deblocking filtering and Sample Adaptive Offset (SAO). Block filtering refers to filtering used to reduce block distortion generated during the encoding of an image in block units. Sample Adaptive Offset (SAO) refers to filtering that minimizes the difference between the reconstructed image and the original image by performing addition or subtraction operations on specific values ​​at the reconstructed pixels.

[0214] The memory unit 112 stores the current block of reconstruction after adding the residual signal generated in the inverse transform unit and the prediction block generated by prediction to the loop filtering unit, and can be used to predict the next block or the next image, etc.

[0215] The subtraction unit 104 generates a residual block by performing a subtraction operation between the current original block and the predicted block.

[0216] Figure 10 This is a schematic diagram illustrating an embodiment of encoding information in an image encoding apparatus.

[0217] In step S1001, the action information for the Skip mode is encoded. In Skip mode, the Merge mode is used when predicting the current block, and the predicted block of the current block is used as the reconstructed block in the decoding device. In step S1002 of inter-frame prediction, it is determined whether Skip mode is active. If Skip mode is active in step S1002 (Yes), in step S1007, the Merge candidate index information for Skip mode is encoded, and then the flowchart ends. If Skip mode is not active in step S1002 (No), the prediction mode is encoded in step S1003. In step S1004, it is determined whether the prediction mode is an inter-frame prediction mode or an intra-frame prediction mode. When the prediction mode is determined to be inter-frame prediction mode in S1004 (Yes), the action information of the MERGE mode is encoded in step S1005. In step S1006, the action of the MERGE mode is determined. When the MERGE mode is determined to be active in step S1006 (Yes), the process jumps to step S1007 and the MERGE candidate index information for the MERGE mode is encoded, then the flowchart ends. When the MERGE mode is determined not to be active in step S1006 (No), the prediction direction is encoded in step S1008. The prediction direction can be a historical direction, a future direction, or one of both. In step S1009, the prediction direction is determined to be a future direction. When the prediction direction is determined not to be a future direction in step S1009 (Yes), the reference image index information of the historical direction is encoded in step S1010. In step S1011, the motion vector difference (MVD) information of the historical direction is encoded. In step S1012, the motion vector prediction (MVP) information of the historical direction is encoded. If it is determined in step S1009 that the predicted direction is a future direction or both directions (No), after completing step S1012, in step S1013, it is determined whether the predicted direction is a historical direction. If it is determined in step S1013 that the predicted direction is not a historical direction (Yes), in step S1014, the reference image index information of the future direction is encoded. In step S1015, the motion vector difference (MVD) information of the future direction is encoded. In step S1016, the motion vector prediction (MVP) information of the future direction is encoded, and then the flowchart ends.

[0218] When the prediction mode is determined to be an in-frame prediction mode in S1004 (No), in step S1017, the action information of the Most Probable Mode (MPM) is encoded. The Most Probable Mode (MPM) refers to a method of transmitting the index information of the candidate in-frame prediction modes when, after constructing candidate in-frame prediction modes using reconstructed in-frame prediction modes from the surrounding blocks, the candidate in-frame prediction modes include the best in-frame prediction mode information for the current block determined in the image encoding device. In step S1018, it is determined whether the Most Probable Mode (MPM) is active. The Most Probable Mode (MPM) is a mode in which the index information is encoded when the prediction modes of reconstructed blocks surrounding the current block include the same mode as the prediction mode for the current block. When the most likely mode (MPM) is determined to be active in step S1018 (Yes), the index information of the most likely mode (MPM) candidate in-frame prediction mode is encoded in step S1019. When the most likely mode (MPM) is determined not to be active in step S1018 (No), the best in-frame prediction mode information for luminance (Luma) among the remaining in-frame prediction modes other than the most likely mode (MPM) candidate in-frame prediction mode is encoded in step S1020.

[0219] After steps S1019 and S1020, the optimal in-frame prediction mode information for chromatic aberration is encoded, and then the flowchart ends. The module executing steps S1008 to S1016 can be referred to as the motion information encoding unit. Furthermore, the module executing steps S1017 to S1020 can be referred to as the luminance in-frame prediction mode encoding unit.

[0220] Figure 11 This is a schematic diagram illustrating the configuration of the image decoding apparatus to which the present invention is applied.

[0221] An image decoding device is a device used to decode images, and may include an entropy decoding unit, an inverse quantization unit, an inverse transform unit, a prediction unit, an addition unit, a loop filtering unit, and a memory unit. The encoding block in an image encoding device can be called a decoding block in an image decoding device.

[0222] The entropy decoding unit 1101 can reconstruct various information and quantization transformation coefficients required for decoding the corresponding block by parsing the bit stream transmitted from the image encoding device.

[0223] The inverse quantization unit 1102 can reconstruct the inverse quantization block with inverse quantization coefficients by reversing the quantization technique used in the quantization process on the quantization coefficients decoded in the entropy decoding unit.

[0224] The inverse transform unit 1103 reconstructs the residual block with the difference signal by performing an inverse transform on the inverse quantization transform block using the same method as that used in the transform process. The inverse transform can be performed by reversing the transform technique used in the transform unit.

[0225] Prediction units 1107 and 1108 generate prediction blocks using prediction mode information decoded in the entropy decoding unit. At this time, the same prediction method as that performed in the prediction unit of the image encoding device can be used.

[0226] The addition unit 1104 can reconstruct the current block by performing an addition operation on the residual signal reconstructed in the inverse transform unit and the prediction block generated by prediction.

[0227] The filtering unit 1105 performs filtering on the entire image area after reconstructing all blocks within the current image. The filtering may include deblocking filtering and Sample Adaptive Offset (SAO), etc. The details are the same as those described for the filtering unit 111 of the image encoding apparatus described above.

[0228] The memory 1106 stores the current block of reconstruction after adding the residual signal generated in the inverse transform unit and the prediction block generated by prediction to the loop filtering unit, and can be used to predict the next block or the next image, etc.

[0229] Figure 12 This is a schematic diagram used to explain the prediction unit of an image decoding device.

[0230] In the in-screen prediction unit 1201, when the prediction mode is in-screen prediction, the in-screen prediction mode determination unit 1202 can determine the in-screen prediction mode information and generate a prediction block by performing in-screen prediction.

[0231] In the inter-frame prediction unit 1203, when the prediction mode is inter-frame prediction, the prediction mode of the current block can be reconstructed in Skip, Merge, or Advanced Motion Vector Prediction (AMVP) modes. In Skip or Merge mode, a candidate motion information group for Skip and Merge modes can be constructed in the Merge candidate exploration unit 1204. The Merge candidate decision unit 1205 can determine the motion information of the current block from the corresponding candidate motion information group. In the case of Advanced Motion Vector Prediction (AMVP), the Advanced Motion Vector Prediction (AMVP) candidate exploration unit 1206 can construct a candidate motion information group for Advanced Motion Vector Prediction (AMVP) mode. The motion information decision unit 1207 can determine the motion information of the current block from the corresponding candidate motion information group using the transmitted motion vector prediction (MVP) information. Next, a prediction block can be generated by performing motion compensation in the motion compensation unit 1208 using the motion information of the current block determined in each mode.

[0232] Figure 13 This is a schematic diagram illustrating an embodiment of decoding encoded information in an image decoding apparatus.

[0233] In step S1301, the action information for the Skip mode is decoded. In step S1302, it is determined whether the Skip mode is active. If it is determined in step S1302 that the Skip mode is active (Yes), in step S1307, the Merge candidate index information for the Skip mode is decoded, and then the flowchart ends. If it is determined in step S1302 that the Skip mode is inactive (No), in step S1303, the prediction mode is decoded. In step S1304, it is determined whether the prediction mode is an inter-frame prediction mode or an intra-frame prediction mode. If it is determined in S1304 that the prediction mode is an inter-frame prediction mode (Yes), in step S1305, the action information for the Merge mode is decoded. In step S1306, it is determined whether the Merge mode is active. When it is determined in step S1306 that the MERGE mode is active (Yes), the process jumps to step S1307 and decodes the MERGE candidate index information used for the MERGE mode, then ends the flowchart. When it is determined in step S1306 that the MERGE mode is not active (No), in step S1308, the predicted direction is decoded. The predicted direction can be a historical direction, a future direction, or one of both. In step S1309, it is determined whether the predicted direction is a future direction. When it is determined in step S1309 that the predicted direction is not a future direction (Yes), in step S1310, the reference image index information for the historical direction is decoded. In step S1311, the motion vector difference (MVD) information for the historical direction is decoded. In step S1312, the motion vector prediction (MVP) information for the historical direction is decoded. If, in step S1309, the predicted direction is determined to be a future direction or both directions (No), after completing step S1312, in step S1313, it will be determined whether the predicted direction is a historical direction. If, in step S1313, the predicted direction is determined not to be a historical direction (Yes), in step S1314, the reference image index information for the future direction will be decoded. In step S1315, the motion vector difference (MVD) information for the future direction will be decoded. In step S1316, the motion vector prediction (MVP) information for the future direction will be decoded, and then this flowchart ends. If, in step S1304, the prediction mode is determined to be an in-frame prediction mode (No), in step S1317, the motion information of the most probable mode (MPM) will be decoded.In step S1318, it is determined whether the most likely mode (MPM) is active. If it is determined in step S1318 that the most likely mode (MPM) is active (Yes), in step S1319, the index information of the in-frame prediction mode of the candidate most likely mode (MPM) is decoded. If it is determined in step S1318 that the most likely mode (MPM) is not active (No), in step S1320, the optimal in-frame prediction mode information for luminance (Luma) among the remaining in-frame prediction modes other than the candidate most likely mode (MPM) is decoded. After steps S1319 and S1320, the optimal in-frame prediction mode information for chroma (Chroma) is decoded, and then this flowchart ends. The module used to execute steps S1308 to S1316 can be referred to as the motion information decoding unit. Furthermore, the module used to execute steps S1317 to S1320 can be referred to as the luminance in-frame prediction mode decoding unit.

[0234] Next, an example of generating a weighted prediction block by weighting a prediction block generated using two or more prediction information will be described. Figures 14 to 17 The accompanying drawings are generally applicable to the embodiments described below.

[0235] Figure 14 This is a schematic diagram used to explain the prediction unit of an image encoding device.

[0236] The intra-frame prediction unit 1401 can perform intra-frame prediction using the original information and the reconstructed information. It determines whether to use the decoder-side intra-frame mode derivation (DIMD) mode to determine the intra-frame prediction mode based on the rate-distortion (RD) cost. Without using the decoder-side intra-frame mode derivation (DIMD) mode, the luminance intra-frame prediction mode exploration unit 1402 can use the rate-distortion (RD) cost to determine the most suitable intra-frame prediction mode for the luminance block from among the intra-frame prediction modes. When using the decoder-side intra-frame mode derivation (DIMD) mode, the luminance intra-frame prediction mode detection unit 1403 using DIMD can set a template region in the reconstructed area surrounding the current block, and use the reconstructed pixels surrounding the template region to deduce the most suitable luminance intra-frame prediction mode for the template region, thereby determining the most suitable intra-frame prediction mode for the template region as the optimal luminance intra-frame prediction mode for the current block. Next, the chrominance intra-frame prediction mode exploration unit 1404 can use the rate-distortion (RD) cost to explore the most suitable intra-frame prediction mode for the chrominance block.

[0237] The inter-frame prediction unit 1405 can perform inter-frame prediction using both raw and reconstructed information. It can calculate the rate-distortion (RD) cost for skip (DKIP) mode, merge mode, advanced motion vector prediction (AMVP) mode, and decoder-side motion vector derivation (DMVD) mode, and determine the optimal prediction mode. (See Figures 1406 to 1409 for further details.) Figure 2 The descriptions of Figures 204 to 207 are identical. The Decoder-Side Motion Vector Derivation (DMVD) initial motion exploration unit 1410 can derive initial motion information applicable to the DMVD mode using inter-frame prediction modes of the reconstructed blocks surrounding the current block and within the reference image. The DMVD motion estimation unit 1411 can determine optimal motion information by performing DMVD motion estimation using the derived initial motion information. The motion compensation unit 1412 can generate a prediction block by performing motion compensation using the optimal motion information determined in each mode. The motion information may include at least one of motion vectors, reference image indexes, and prediction direction information.

[0238] The prediction information exploration unit 1413, used to generate weighted prediction blocks, can explore additional prediction information (hereinafter referred to as "Nth auxiliary prediction information," where N is an integer of 1 or more) that is weighted and summed with the best prediction information (hereinafter referred to as "main prediction information") determined using in-frame prediction modes and inter-frame prediction modes. There can be one or more auxiliary prediction information, and they can correspond to any prediction information in the in-frame prediction modes and inter-frame prediction modes. For example, when the main prediction information is the planar mode in the in-frame prediction modes, the first auxiliary prediction information can be the first motion information in the merge candidate list, and the second auxiliary prediction information can be the vertical mode in the in-frame prediction modes. Furthermore, the weighting information of different prediction blocks of each prediction information can also be determined by rate-distortion (RD) cost and included in the additional prediction information for transmission.

[0239] Figure 15 This is a schematic diagram used to explain the prediction unit of an image decoding device.

[0240] The intra-frame prediction unit 1501 can use the reconstruction information to determine whether to use the decoder-side intra-frame mode derivation (DIMD) mode to determine the intra-frame prediction mode. When not using the decoder-side intra-frame mode derivation (DIMD) mode, the luminance intra-frame prediction mode determination unit 1502 can use the reconstruction information to determine the intra-frame prediction mode for the luminance block. When using the decoder-side intra-frame mode derivation (DIMD) mode, the luminance intra-frame prediction mode detection unit 1503 using DIMD can set a template region in the reconstruction area surrounding the current block, and use the reconstructed pixels surrounding the template region to deduce the most suitable luminance intra-frame prediction mode for the template region, thereby determining the most suitable intra-frame prediction mode for the template region as the luminance intra-frame prediction mode for the current block. Next, the chrominance intra-frame prediction mode determination unit 1504 can determine the intra-frame prediction mode for the chrominance block.

[0241] The inter-frame prediction unit 1505 can determine one of the following prediction modes when performing inter-frame prediction using reconstructed information: SKIP mode, MERGE mode, Advanced Motion Vector Prediction (AMVP) mode, and Decoder-Side Motion Vector Derivation (DMVD). The descriptions of figures 1506 to 1509 are consistent with... Figure 12 The descriptions for figures 1204 to 1207 are the same. The descriptions for figures 1510 and 1511 are the same as those for… Figure 14 The descriptions of figures 1410 and 1411 are identical. The motion compensation unit 1512 can generate prediction blocks by performing motion compensation using motion information determined in each mode. The prediction information determination unit 1513, which determines whether to apply weighted prediction and generates weighted prediction blocks, can determine N auxiliary prediction information (i.e., the Nth auxiliary prediction information) for weighted summation with the best prediction information (hereinafter referred to as "main prediction information") determined using the intra-frame prediction mode and the inter-frame prediction mode.

[0242] Figure 16 This is a method for describing an encoding method for encoded information applicable to one embodiment of the present invention.

[0243] In step S1601, the action information for the Skip mode is encoded. In step S1602, it is determined whether the Skip mode is active. When the Skip mode is determined to be active, in step S1607, the action information for the Decoder-Side Motion Vector Derivation (DMVD) mode is encoded. In step S1608, it is determined whether the Decoder-Side Motion Vector Derivation (DMVD) mode is active. When the Decoder-Side Motion Vector Derivation (DMVD) mode is determined to be active, in step S1609, the Decoder-Side Motion Vector Derivation (DMVD) mode information is encoded. When the Decoder-Side Motion Vector Derivation (DMVD) mode is determined not to be active, in step S1610, the MERGE candidate index information for the Skip mode is encoded. When the Skip mode is determined not to be active, in step S1603, the prediction mode is encoded. In step S1604, it is determined whether the prediction mode is inter-frame prediction. When the prediction mode is determined to be inter-frame prediction, in step S1605, the motion information of the MERGE mode is encoded. In step S1606, it is determined whether the MERGE mode is active. When the MERGE mode is determined to be active, steps S1607 to S1610 can be executed, as described above. The MERGE candidate index information in step S1610 is candidate index information applicable to the MERGE mode, not for the SKIP mode. When the MERGE mode is determined not to be active, in step S1620, the motion information encoding unit can encode the motion information. Step S1620 can correspond to... Figure 10 The process from steps S1008 to S1010. When the prediction mode is intra-frame prediction mode rather than inter-frame prediction, in step S1611, the decoder-side intra-frame mode derivation (DIMD) mode action information is encoded. In step S1612, it is determined whether the decoder-side intra-frame mode derivation (DIMD) mode is active. When it is determined that the decoder-side intra-frame mode derivation (DIMD) mode is not active, in step S1630, the luminance intra-frame prediction mode encoding unit can encode the intra-frame prediction mode information. Step S1630 can correspond to Figure 10 The process from steps S1017 to S1020. In step S1613, the prediction mode information within the color difference image can be encoded. The main prediction information can be encoded using the encoding method described above.

[0244] In step S1614, the weighted sum prediction action information used to determine whether to perform a weighted sum with the main prediction information can be encoded. The weighted sum prediction can be applied unconditionally (the weighted sum prediction action is executed unconditionally) without encoding the corresponding information, or it can be not applied (the weighted sum prediction action is not executed unconditionally). In step S1615, a determination is made regarding whether to perform the weighted sum prediction action. If it is determined that the weighted sum prediction action will not be executed, this flowchart ends. If it is determined that the weighted sum prediction action will be executed, the weighted sum prediction information is encoded in step S1616. The weighted sum prediction information may include weighted value information allocated to each prediction block generated using the main prediction information and the auxiliary prediction information, as well as the weighting method, etc.

[0245] Figure 17 This is a method for describing a decoding method for encoded information applicable to one embodiment of the present invention.

[0246] In step S1701, the action information for the Skip mode is decoded. In step S1702, it is determined whether the Skip mode is active. If the Skip mode is active, in step S1707, the action information for the Decoder-Side Motion Vector Derivation (DMVD) mode is decoded. In step S1708, it is determined whether the Decoder-Side Motion Vector Derivation (DMVD) mode is active. If the Decoder-Side Motion Vector Derivation (DMVD) mode is active, in step S1709, the Decoder-Side Motion Vector Derivation (DMVD) mode information is decoded. If the Decoder-Side Motion Vector Derivation (DMVD) mode is inactive, in step S1710, the MERGE candidate index information for the Skip mode is decoded. If the Skip mode is inactive, in step S1703, the prediction mode is decoded. In step S1704, it is determined whether the prediction mode is inter-frame prediction. When the prediction mode is determined to be inter-frame prediction, in step S1705, the motion information of the MERGE mode is decoded. In step S1706, it is determined whether the MERGE mode is active. When the MERGE mode is determined to be active, steps S1707 to S1710 can be executed, as described above. The MERGE candidate index information in step S1710 is candidate index information applicable to the MERGE mode, not for the SKIP mode. When the MERGE mode is determined not to be active, in step S1720, the motion information decoding unit can decode the motion information. Step S1720 can correspond to... Figure 13The process from steps S1308 to S1310. When the prediction mode is intra-frame prediction mode rather than inter-frame prediction, in step S1711, the decoder-side intra-frame mode derivation (DIMD) mode action information is decoded. In step S1712, it is determined whether the decoder-side intra-frame mode derivation (DIMD) mode is active. When it is determined that the decoder-side intra-frame mode derivation (DIMD) mode is not active, in step S1730, the luminance intra-frame prediction mode decoding unit can decode the luminance intra-frame prediction mode information. Step S1730 can correspond to Figure 13 The process from steps S1317 to S1320 is as follows. In step S1713, the prediction mode information within the color difference image can be decoded. In step S1714, the weighted sum prediction action information used to determine whether to perform a weighted sum with the main prediction information can be decoded. When it is determined that the weighted sum prediction action will not be performed, this flowchart will end; when it is determined that the weighted sum prediction action will be performed, the weighted sum prediction information will be decoded in step S1716.

[0247] (Example 1) Example 1 relates to a method for generating weighted sum prediction blocks using prediction information from inter-frame prediction modes (SKIP mode, MERGE mode, Advanced Motion Vector Prediction (AMVP) mode, Decoder-Side Motion Vector Derivation (DMVD) mode) and additional inter-frame prediction mode prediction information.

[0248] Before detailing the weighted and prediction block generation methods, the prediction modes used to generate the prediction information required in this embodiment will be explained in detail. The Skip mode, Merge mode, and Advanced Motion Vector Prediction (AMVP) mode are explained as described above. The Decoder-Side Motion Vector Derivation (DMVD) mode refers to a method of generating prediction blocks by generating the same motion information without encoding / decoding motion information in both the image encoding and decoding devices.

[0249] Figure 18 This is an example diagram illustrating the results of the initial motion information exploration unit in the decoder-side motion vector derivation (DMVD) mode of the prediction unit within the image encoding / decoding device.

[0250] Additional motion information not listed in this list may also be included in the initial motion information list.

[0251] Figure 19 This is a schematic diagram used to explain the operation of the motion prediction unit of the decoder-side motion vector derivation (DMVD) in the prediction unit of an image encoding / decoding device.

[0252] The optimal motion information can be determined after executing the Decoder-Side Motion Vector Derivation (DMVD) mode using the initial motion information determined by the Decoder-Side Motion Vector Derivation (DMVD) initial motion information exploration unit within the image encoding / decoding device. Executing the Decoder-Side Motion Vector Derivation (DMVD) mode can refer to correcting the initial motion information.

[0253] Decoder-side motion vector derivation (DMVD) modes include modes that use templates (hereinafter referred to as "template matching modes") and modes that do not use templates (hereinafter referred to as "bidirectional matching modes").

[0254] When using the bidirectional matching mode 1901, the unidirectional motion vectors of each initial motion information are linearly scaled to a reference image in the opposite prediction direction. The scaling of the motion vectors is performed in a manner proportional to the distance between the current image and the reference images in each direction. After determining the motion vectors in both directions 1902 as described above, the motion vectors in each direction with the smallest difference between the prediction blocks in the historical and future directions are determined as the optimal motion information 1903.

[0255] Figure 20 This is a diagram used to illustrate template matching modes and bidirectional matching modes.

[0256] In the bidirectional matching mode 2002, after generating the historical and future direction motion vectors of the current block in a linear manner, the prediction block of the current block can be generated by the tie of the two prediction blocks in both directions.

[0257] When using template matching mode 1904, the number of template blocks can be determined in the reconstructed area. When using a single template block 1905 (hereinafter referred to as "single template matching mode"), the reconstructed areas on the left and top adjacent to the current block can be determined as template blocks, and the optimal motion information is determined by using the motion inference of the corresponding template blocks 1907.

[0258] After identifying the predicted block of the template block most similar to the template block in the single template matching mode 2001, the block adjacent to the corresponding template block can be determined as the predicted block of the current block. The cost of motion inference is the sum of the prediction error and the hypothetical bits of motion information. The prediction error can be obtained through various calculation methods such as the sum of absolute errors (SAD), the sum of absolute variation errors (SATD), and the sum of square errors (SSD). Equations 3, 4, and 5 describe the calculation methods of the sum of absolute errors (SAD), the sum of absolute variation errors (SATD), and the sum of square errors (SSD), respectively.

[0259]

Mathematical Expression 3

[0260]

Mathematical Expression 4

[0261]

Mathematical Expression 5

[0262] In the above mathematical formulas, i and j represent pixel positions, Template(i, j) represents the pixel of the template block, and PredBlk(i, j) represents the pixel of the prediction block. The HT() function in formula 4 represents the function value obtained by performing a Hadamard transform on the difference block between the template block and the prediction block. The hypothetical bit value of motion information is not the information actually transmitted, but rather a value calculated from the hypothetical bit value of motion information anticipated in the image encoding and decoding devices using the same method. For example, the hypothetical bit value can be determined by calculating the difference vector between the motion vector of the initial motion information and the motion vector within the motion information currently being inferred. Alternatively, the hypothetical bit value of motion information can be calculated using the bit value of the reference image information. In the case of using multiple template blocks 1906 (hereinafter referred to as "multi-template matching mode"), multiple template blocks can be determined in the reconstructed region.

[0263] Figure 21 This is an example diagram illustrating the method for setting template blocks.

[0264] Template blocks can be arranged according to Figure 21The settings are as shown in the example diagrams numbered 2101 to 2103 in the attached figures. In template block setting method A 2101, template blocks can be determined separately at the lower left (template A), upper left (template B), upper left side (template C), and upper right side (template D) of the current block. Each template block can be determined to have various sizes and shapes. In template block setting method B 2102, template blocks can be determined at the lower left (template A), upper left (template B), upper left side (template C), and upper right side (template D) of the current block in the same way as template block setting method A, but the difference is that all reconstructed areas on the left and upper sides adjacent to the current block are used. In template block setting method C 2103, template blocks can be generated by considering both template block setting method A and template block setting method B simultaneously. In addition, the reconstructed areas on the left and upper sides adjacent to the current block can also be determined as a single template block. See also... Figure 21 The description states that template blocks can be generated from the reconstructed area surrounding the current block using various methods. However, it can also be used by transmitting information representing the shape and size of the template blocks from the image encoding device. After performing motion inference 1907 to explore the prediction blocks most similar to the determined template blocks in the reference image, motion information most suitable for each template block is inferred, and the optimal motion information is determined by exploring the motion information most suitable for the current block among the corresponding motion information. The index information of the aforementioned optimal motion information can also be transmitted simultaneously (i.e., equivalent to transmitting the index information of the template blocks).

[0265] As mentioned above, four prediction modes can be used to generate primary prediction information in the inter-frame prediction mode. In addition, supplementary inter-frame prediction modes can be used to generate the primary prediction information. The supplementary supplementary prediction information can be derived using various methods.

[0266] When the primary prediction information is motion information generated through the SKIP or MERGE modes, method 1-a can be used to perform a weighted sum with other candidate motion information in the candidate motion information list of the SKIP or MERGE modes. In this case, the other candidate motion information corresponding to the secondary prediction information can either transmit the index information of the candidate motion information or not transmit it, instead using a pre-defined method such as agreeing to use the motion information of the next order of the candidate motion information corresponding to the prediction information to determine the pre-defined candidate motion information as the secondary prediction information. Method 1-b can be used by using the reconstructed motion information surrounding the current block as initial motion information and performing additional motion inference to perform a weighted sum with the determined motion information. In this case, the motion information corresponding to the secondary prediction information can be expressed using parameters such as non-prediction direction, reference image, and two-dimensional (2D) motion vectors. For example, it can be expressed using three-dimensional (3D) motion vectors (inter-image distance, X coordinate, Y coordinate). Alternatively, only a portion of the motion information can be transmitted. For example, instead of transmitting the prediction direction, it can be unconditionally determined to be the direction opposite to the prediction direction of motion information in skip and merge modes. As method 1-c, it can be weighted and summed with motion information generated by the decoder-side motion vector derivation (DMVD) mode. In this case, the motion information corresponding to the auxiliary prediction information can be derived using the template matching mode and bidirectional matching mode of the decoder-side motion vector derivation (DMVD) mode. It can also be determined which mode's best motion information among the motion information determined according to each mode should be weighted and summed, and the corresponding mode information should be transmitted.

[0267] When the primary prediction information is motion information generated through the Advanced Motion Vector Prediction (AMVP) mode, as in Method 2-a, one of the candidate motion information from the Skip or Merge modes can be weighted and summed. In this case, either the index information of the candidate motion information corresponding to the secondary prediction information can be transmitted, or pre-defined candidate motion information can be used unconditionally. For example, it can be unconditionally weighted and summed with the candidate motion information with the highest priority. As in Method 2-b, the motion information determined by the additional motion inference can be determined as secondary prediction information in the same way as in Method 1-b. In this case, motion inference can be performed using initial motion information other than the initial motion information used when deriving the motion information of the primary prediction information. Otherwise, it is the same as Method 1-b. Method 2-c is the same as Method 1-c.

[0268] When the primary prediction information is motion information generated through the decoder-side motion vector derivation (DMVD) mode, Method 3-a is the same as Method 1-a. Method 3-b is the same as Method 1-b. As Method 3-c, when the decoder-side motion vector derivation (DMVD) mode for determining the primary prediction information is a bidirectional matching mode, the best motion information of the single-template block matching mode is determined as the secondary prediction information. When the primary prediction information is a single-template block matching mode, conversely, the best motion information of the bidirectional matching mode can be determined as the secondary prediction information, or the motion information of the sub-template block most suitable for the current block in the multi-template matching mode can be determined as the secondary prediction information. When the primary prediction information is a multi-template block matching mode, the motion information of other sub-template blocks besides the motion information of the sub-template block corresponding to the primary prediction information can be determined as the secondary prediction information.

[0269] In the above method, the primary prediction information and the secondary prediction information should be distinct. Furthermore, two or more secondary prediction information sets can be used instead of just one. The weighting values, applicable to each prediction block generated using the primary and secondary prediction information respectively, can be determined by transmitting any group of information from a pre-defined weighting value set for the primary and secondary prediction information. The weighting values ​​can also be complex numbers, and the sum of the weighting values ​​assigned to each prediction block is 1.

[0270] Figures 24 to 28 This is a schematic diagram illustrating a weighted summation method for prediction blocks using both primary and secondary prediction information when there is only one secondary prediction information. In the corresponding figure, prediction block A is the prediction information generated using the primary prediction information, while prediction block B is the prediction information generated using the secondary prediction information. Furthermore, it is assumed that prediction blocks A and B are 4×4 in size, and the predicted pixels at the same position in prediction blocks A and B are weighted and summed using the applicable weighting values.

[0271] Figure 24 This is a schematic diagram illustrating a method for applying a single weighting value to a predicted block.

[0272] exist Figure 24 In this process, a weighted sum can be performed by applying a weighted value A1 to all pixels in the prediction block A and a weighted value W2 to all pixels in the prediction block B, thereby generating the final prediction block.

[0273] Figure 25 This is a schematic diagram illustrating a method for applying multiple weighted values ​​to a predicted block.

[0274] exist Figure 25In this example, the predicted block A can be divided into N arbitrary regions along the vertical direction, and different weighting values ​​can be applied to each arbitrary region. The block A is divided into four arbitrary regions, and weighting values ​​W1, W2, W3, and W4 are applied sequentially from left to right to each region. Similarly, weighting values ​​W5, W6, W7, and W8 are applied sequentially from left to right to each region of the predicted block B. The sums of W1 and W5, W2 and W6, W3 and W7, and W4 and W5 are all 1. After determining the weighting values ​​as described above, a weighted sum is performed to generate the final predicted block.

[0275] Figure 26 This is a schematic diagram illustrating another method for applying multiple weighted values ​​to opposing prediction blocks.

[0276] Figure 26 The method in [the text], besides dividing into N arbitrary regions along the horizontal direction, is similar to... Figure 25 The method is the same as in [the previous section].

[0277] Figure 27 as well as Figure 28 This is an example diagram illustrating a method that applies a weighted sum to a portion of the area rather than to the entire predicted block.

[0278] Figure 27 This is a schematic diagram illustrating a method for applying weighted values ​​only to a portion of a prediction block.

[0279] exist Figure 27 In this case, the weighting can be applied only to the upper left region of predicted blocks A and B. See also... Figure 27 The four pixels at the top left can be determined as the top left region, and the final prediction block is generated by weighting the prediction pixels corresponding to the corresponding region in prediction blocks A and B.

[0280] Figure 28 This is a schematic diagram illustrating another method for applying weighted values ​​only to a portion of the prediction block.

[0281] Figure 28 The method in the middle, besides using the weighted sum region as the lower right region of the predicted block, is similar to... Figure 27 The method is the same as in [the previous section].

[0282] In addition, Figure 27 , Figure 28 The weighted values ​​W1 and W2 are applicable to the predicted pixels of the prediction blocks A and B, respectively, and the sum of their weighted values ​​is 1.

[0283] In the weighted sum method, a pre-defined weighted sum method and weighting value information can be used based on the types of primary and secondary prediction information. Furthermore, when weighting more than three prediction blocks, it is not necessary to weight each prediction block all at once. Instead, methods such as first weighting the primary prediction information and the first secondary prediction information, and then sequentially weighting the result along with the second and third secondary prediction information (only two prediction blocks at a time) can be used.

[0284] (Example 2) In this embodiment, a method for generating weighted sum prediction blocks using prediction information from in-frame prediction modes and additional prediction information from in-frame prediction modes will be described in detail.

[0285] Before detailing the weighted and prediction block generation method, the prediction mode used to generate the prediction information required in this embodiment will be explained in detail. The decoder-side intra-frame pattern derivation (DIMD) mode refers to a method of generating prediction blocks by generating the same intra-frame prediction mode without encoding / decoding intra-frame prediction information in the image encoding and decoding devices.

[0286] Figure 22 This is a schematic diagram used to illustrate the use of template matching technology in the reconstruction area around the current block in the in-picture prediction mode.

[0287] exist Figure 22 In the process, the template region is determined from the reconstructed area surrounding the current block. The template region can be determined to have different sizes. Next, a reference region for the template region is determined from its surrounding area. The reconstructed pixels of the aforementioned reference region are used to perform in-frame prediction of the template region, and the difference between the predicted value and the reconstructed value of the template region, i.e., the prediction error, is determined as the template matching cost. As a method for calculating the prediction error, various different methods can be used, such as the sum of absolute errors (SAD), sum of absolute variation errors (SATD), and sum of squared errors (SSD) as described above.

[0288] Figure 23 This is a schematic diagram used to illustrate the Linear Mode (LM) applicable to color difference blocks.

[0289] The predicted blocks in the color difference blocks using the linear mode (LM) can be generated by referring to mathematical formula 6.

[0290]

Mathematical Expression 6

[0291] Here, predC(i, j) represents the predicted pixels at different positions (i, j) in the chromatic difference block, and rec'L(i, j) represents the downsampled reconstructed pixels obtained using the best in-frame prediction mode of the luminance block. α and β are linear regression parameters used to minimize the difference between the reconstructed pixels in the template region surrounding the luminance block and the reconstructed pixels in the template region surrounding the chromatic difference block. The linear regression parameters α and β can be derived using Equation 7.

[0292]

Mathematical Expression 7

[0293] Where L(n) represents the reconstructed pixels in the template region surrounding the downsampled luminance block, and C(n) represents the reconstructed pixels in the template region surrounding the chrominance block. The mode for generating the predicted block of the chrominance block using the method described above is the linear mode (LM). The template regions surrounding the luminance block and the chrominance block can be determined as shown in Figure 2301.

[0294] As mentioned above, in the intra-frame prediction mode, four prediction modes can be used to generate the primary prediction information for the luma block: mean (DC), planar, directional mode (hereinafter referred to as "Normal Intra"), and decoder-side intra-frame mode derivation (DIMD). In the chromatic aberration block, the primary prediction information can also be generated using the linear (LM) mode. In addition, supplementary inter-frame prediction modes can be used to generate the primary prediction information. The supplementary auxiliary prediction information can be derived using various methods.

[0295] When the primary prediction information for the luma block is an intra-frame prediction mode generated through the Normal Intra mode, as method 4-a, other intra-frame prediction modes besides the intra-frame prediction mode corresponding to the primary prediction information can be determined as secondary prediction information. In this case, the intra-frame prediction mode used as secondary prediction information can be selected from the Most Probable Mode (MPM) candidate list. As described above, candidate index information can be transmitted, or the candidate intra-frame prediction mode with the highest priority can be fixed and used as secondary prediction information. As method 4-b, in addition to the intra-frame prediction mode corresponding to the primary prediction information, an intra-frame prediction mode determined through the decoder-side Intra-Frame Mode Derivation (DIMD) mode can be used as secondary prediction information.

[0296] When the primary prediction information for the luma block is an intra-frame prediction mode generated through decoder-side intra-mode derivation (DIMD) mode, as method 5-a, one of the intra-frame prediction modes included in the most probable mode (MPM) candidate list, excluding the intra-frame prediction information corresponding to the primary prediction information, can be designated as secondary prediction information. In this case, the most probable mode (MPM) candidate mode information can be transmitted, or the candidate intra-frame prediction mode with the highest priority can be consistently used as secondary prediction information. As method 5-b, other intra-frame prediction modes derived through decoder-side intra-frame mode derivation (DIMD) can be used as secondary prediction information. In this case, the intra-frame prediction mode used as secondary prediction information can be derived using a portion of the template used when executing decoder-side intra-frame mode derivation (DIMD) mode, or it can be derived using all templates.

[0297] When the main prediction information for the color difference area is an in-frame prediction mode generated through the linear mode (LM), as for method 6-a, please refer to [reference needed]. Figure 23 Figure 2302 illustrates the derivation of two sets of linear regression parameters, α and β, in Linear Mode (LM). The reconstructed pixels of the template region surrounding the luminance block are divided using arbitrary thresholds. Linear regression parameters are then derived from the luminance reconstructed pixels in region A (divided by the threshold) and pixels in the template region surrounding the chrominance block that occupy the same positions as the reconstructed pixels in the template region surrounding the luminance block (containing the luminance reconstructed pixels in region A). The linear regression parameters for region B are derived using the same method. Predictive blocks are generated using the two derived sets of linear regression parameters and Equation 6, and a final predictive block is generated by performing a weighted sum among the generated predictive blocks.

[0298] In the above method, the primary prediction information and the secondary prediction information should be distinct. Furthermore, two or more secondary prediction information sets can be used instead of just one. The weighting values, applicable to each prediction block generated using the primary and secondary prediction information respectively, can be determined by transmitting any group of information from a pre-defined weighting value set for the primary and secondary prediction information. The weighting values ​​can also be complex numbers, and the sum of the weighting values ​​assigned to each prediction block is 1.

[0299] right Figures 24 to 28The explanation is the same as above. The determined main prediction information, auxiliary prediction information, weighting method, and weighting value information can be determined as weighted prediction information and transmitted. However, some information can also be used after being preset in the image encoding device and the image decoding device.

[0300] In the weighted sum method, a pre-defined weighted sum method and weighting value information can be used based on the types of primary and secondary prediction information. Furthermore, when weighting more than three prediction blocks, it is not necessary to weight each prediction block all at once. Instead, methods such as first weighting the primary prediction information and the first secondary prediction information, and then sequentially weighting the result along with the second and third secondary prediction information (only two prediction blocks at a time) can be used.

[0301] (Example 3) In this embodiment, a method for generating weighted prediction blocks using prediction information from inter-frame prediction modes and additional intra-frame prediction modes, or prediction information from intra-frame prediction modes and additional inter-frame prediction information, will be described in detail.

[0302] When the primary prediction information is an in-frame prediction mode generated through the Normal Intra mode or the Decoder-Side Intra-Motion Derivation (DIMD) mode, the motion information of the corresponding mode can be determined as the secondary prediction information after selecting the best prediction mode from among Skip, Merge, Advanced Motion Vector Prediction (AMVP), and Decoder-Side Motion Vector Derivation (DMVD) modes. The motion information of the secondary prediction information can be transmitted entirely in the same manner as the primary prediction mode, or only a portion of the motion information can be transmitted, or it can be determined as the secondary prediction information in the image encoding and decoding devices using only pre-set prediction information without transmission.

[0303] When the primary prediction information is motion information generated through Skip, Merge, Advanced Motion Vector Prediction (AMVP), or Decoder-Side Motion Vector Derivation (DMVD) modes, the intra-frame prediction mode determined by the NormalIntra mode or Decoder-Side Intra-Frame Mode Derivation (DIMD) mode can be used as secondary prediction information. For example, in the case of Merge mode, the primary prediction information could be the merge index. The intra-frame prediction mode information used as secondary prediction information can be determined only from a subset of intra-frame prediction modes by reducing the number of intra-frame prediction modes. A subset of intra-frame prediction modes can refer to a portion of all available intra-frame prediction modes. For example, a subset of intra-frame prediction modes could refer to non-directional modes. Alternatively, only one fixed intra-frame prediction mode can be used as the default mode. For example, the default mode could be the Planar mode.

[0304] Two or more auxiliary prediction information can be used instead of one. The weighting values, applied separately to each prediction block generated using the primary and auxiliary prediction information, can be determined by transmitting any group of information from the pre-defined weighting value groups of the primary and auxiliary prediction information. The weighting values ​​can also be complex numbers, and the sum of the weighting values ​​assigned to each prediction block is 1.

[0305] Alternatively, the weighting can be determined based on the prediction patterns of the surrounding blocks of the current block. For example, the surrounding blocks could be the top surrounding block and the left surrounding block. For example, the top surrounding block could be the rightmost block in the top block of the current block. For example, when the top-left coordinate of the current block of size W×H is (0, 0), the top surrounding block could be a block containing pixels with coordinates (-1, H-1). For example, the left surrounding block could be the bottommost block in the left block of the current block. For example, when the top-left coordinate of the current block of size W×H is (0, 0), the left surrounding block could be a block containing pixels with coordinates (W-1, -1). For example, the weighting can be determined based on the number of intra-frame prediction blocks and inter-frame prediction blocks in the surrounding blocks. For example, if there are more intra-frame prediction blocks in the surrounding blocks, a larger weighting can be assigned to the intra-frame prediction blocks of the current block than to the inter-frame prediction blocks. For example, if both the top and left peripheral blocks are intra-frame predictions, a weighting of 0.75 and 0.25 can be assigned to the intra-frame and inter-frame prediction blocks of the current block, respectively. Conversely, if both the top and left peripheral blocks are inter-frame predictions, a weighting of 0.25 and 0.75 can be assigned to the intra-frame and inter-frame prediction blocks of the current block, respectively. If the top and left peripheral blocks are inter-frame and intra-frame predictions, respectively, a weighting of 0.5 and 0.5 can be assigned to the intra-frame and inter-frame prediction blocks of the current block, respectively.

[0306] right Figures 24 to 28 The explanation is the same as above. The determined main prediction information, auxiliary prediction information, weighting method, and weighting value information can be determined as weighted prediction information and transmitted. However, some information can also be used after being preset in the image encoding device and the image decoding device.

[0307] In the weighted sum method, a pre-defined weighted sum method and weighting value information can be used based on the types of primary and secondary prediction information. Furthermore, when weighting more than three prediction blocks, it is not necessary to weight each prediction block all at once. Instead, methods such as first weighting the primary prediction information and the first secondary prediction information, and then sequentially weighting the result along with the second and third secondary prediction information (only two prediction blocks at a time) can be used.

[0308] exist Figure 25 as well as Figure 26In the case where one of the main or auxiliary prediction modes is a directional in-image prediction mode, the weighting information can be pre-set according to the direction of the directional mode. For example, when weighting motion information with a horizontal mode, a method such as... Figure 25 The weighted sum method shown assigns higher weights in the prediction block corresponding to the directional mode, in the order of W1, W2, W3, W4, while in the prediction block generated from motion information, it assigns higher weights in the order of W4, W3, W2, W1. In the case of the vertical direction mode, a method such as... Figure 26 The weighted sum method is shown, while other explanations are the same as those in the examples above.

[0309] Figure 29 It is a block diagram that provides a general illustration of the structure of an image encoding device.

[0310] See Figure 29 The image encoding apparatus is a device for encoding images, and may include a block segmentation unit 2901, a prediction unit 2902, a transformation unit 2903, a quantization unit 2904, an entropy encoding unit 2905, an inverse quantization unit 2906, an inverse transformation unit 2907, an addition unit 2908, a loop filtering unit 2909, a memory unit 2910, and a subtraction unit 2911.

[0311] Figure 29 The various components illustrated are shown separately to illustrate different features and functions of the image encoding apparatus, and do not represent that each component is composed of separate hardware or software units. That is, although the various components are listed for ease of explanation, at least two components can be combined into one component, or a component can be divided into multiple components to perform corresponding functions. The embodiments where the various components are integrated or separated as described above are included within the scope of the claims of this invention without departing from the essence of the invention.

[0312] Furthermore, some constituent elements may not be essential for performing the essential functions of this invention, but rather optional elements used to improve performance. This invention may include only the constituent parts essential for realizing the essence of the invention, excluding those merely used to improve performance, and structures including essential constituent elements other than optional elements used to improve performance are also included within the scope of the claims of this invention.

[0313] The block segmentation unit 2901 can segment the input image into at least one block. The input image can have various shapes and sizes, such as images, stripes, parallel blocks, tiles, or fragments. A block can refer to a coding unit (CU), a prediction unit (PU), or a transform unit (TU). The segmentation can be performed based on at least one of quadtree, binary tree, and ternary tree segmentation.

[0314] A quadtree is a partitioning method that accurately divides the current coded block into four parts. A binary tree is a partitioning method that accurately divides the coded block into two parts along a horizontal or vertical direction. A ternary tree is a partitioning method that divides the above block into three lower-level blocks. For example, the above three lower-level blocks can be obtained by dividing the width or height of the above upper-level block in a 1:2:1 ratio. By performing partitioning based on a binary tree as described above, the block can have not only a square shape but also a non-square shape. In addition, a method that uses multiple partitioning methods simultaneously can also be used.

[0315] The block segmentation unit 2901 can segment the largest block that needs to be encoded (hereinafter referred to as the largest encoded block) into the smallest block that needs to be encoded (hereinafter referred to as the smallest encoded block).

[0316] The prediction unit 2902 can generate a prediction block using the surrounding pixels of the block that needs to be predicted in the current original block (hereinafter referred to as the prediction block) or pixels in a reference image that has been previously encoded / decoded.

[0317] As a prediction block, one or more prediction blocks can be generated within the coding block. When there is only one prediction block within the coding block, the prediction block can have the same form as the coding block.

[0318] Video signal prediction techniques include intra-frame prediction and inter-frame prediction.

[0319] In-frame prediction is a method of generating prediction blocks using the surrounding pixels of the current block.

[0320] Inter-image prediction is a method of finding the most similar block to the current block from previously encoded / decoded reference images and generating a predicted block.

[0321] Furthermore, the prediction unit 2902 can determine the optimal prediction mode for the prediction block from the residual block obtained by subtracting the prediction block from the original block after generating the prediction block, using various techniques such as rate-distortion optimization (RDO). The calculation formula for the rate-distortion optimization (RDO) cost is shown in mathematical formula 8.

[0322]

Mathematical Expression 8

[0323] Where D, R, and J represent the degradation caused by quantization, the bit rate of the compressed stream, and the rate-distortion (RD) cost, respectively; Φ represents the encoding mode; and λ represents the Lagrange multiplier, which can be used as a scaling factor to match the units between error and bit count. To select the optimal encoding mode during the encoding process, the rate-distortion (RD) cost when using the appropriate mode should be less than when using other modes. The formula for calculating the rate-distortion (RD) cost can be performed while simultaneously considering both bit rate and error.

[0324] Figure 30 This is a flowchart explaining the process within the prediction section of the image encoding device.

[0325] When performing in-frame prediction 3001 using the original information and the reconstructed information, the optimal in-frame prediction mode 3002 can be determined by using the rate-distortion (RD) cost for each prediction mode and a prediction block can be generated.

[0326] When performing inter-frame prediction 3003 using raw and reconstructed information, the rate-distortion (RD) cost can be calculated for Skip mode, Merge mode, and Advanced Motion Vector Prediction (AMVP) mode.

[0327] In the MERGE candidate exploration section 3004, candidate motion information groups for the SKIP and MERGE modes can be constructed. Furthermore, in the corresponding candidate motion information groups, the optimal motion information 3005 can be determined using rate-distortion (RD) cost.

[0328] In the Advanced Motion Vector Prediction (AMVP) candidate exploration unit 3006, a group of candidate motion information can be constructed for the AMVP mode. Furthermore, the optimal motion information can be determined by performing motion inference 3007 using the corresponding group of candidate motion information.

[0329] Predictive blocks can be generated by performing motion compensation 3008 using the best motion information determined in each mode.

[0330] The aforementioned inter-frame prediction can be achieved using three modes: Skip mode, Merge mode, and Advanced Motion Vector Prediction (AMVP) mode. Motion information (predicted direction information, reference image information, and motion vectors) can be encoded according to each prediction mode.

[0331] The Skip mode utilizes motion information from reconstructed regions to determine the optimal prediction information. The Skip mode constructs motion information candidate groups within the reconstructed regions and generates prediction blocks by using the candidate with the lowest rate-distortion (RD) cost from the corresponding candidate group as the prediction information. Since the method for constructing motion information candidate groups is the same as the method for constructing motion information candidate groups in the MERGE mode described below, it will be omitted in this description.

[0332] The MERGE mode is similar to the SKIP mode in utilizing motion information from reconstructed regions to determine the best prediction information. However, the difference lies in that the SKIP mode explores motion information with zero prediction error from a set of motion information candidates, while the MERGE mode explores motion information with non-zero prediction error from a set of motion information candidates. Similar to the SKIP mode, motion information candidate sets can be constructed within the reconstructed region, and prediction blocks are generated by using the candidate with the lowest rate-distortion (RD) cost from the corresponding candidate set as the prediction information.

[0333] Figure 31 The methods for generating motion information candidate groups in SKIP and MERGE modes are illustrated. The maximum number of motion information candidate groups can be determined to be the same value in both the image encoding and decoding devices, or the corresponding quantity information can be pre-transmitted in the upper-level header of the image encoding device (the upper-level header refers to parameters transmitted at the level above the block, such as video parameter level, sequence parameter level, and image parameter level).

[0334] See Figure 31 In step S3101, four candidates can be selected from five spatial selection blocks surrounding the current block within the same image. Figure 33This is a schematic diagram illustrating the locations of spatial candidates. Furthermore, the location of a spatial candidate can be changed to any block within the reconstructed area. Spatial candidates can be considered in the order of A1, A2, A3, A4, A5, and the motion information of the most available spatial candidate blocks can be used to determine the spatial candidate. In cases of duplicate motion information, only the motion information of the higher-priority candidate can be considered.

[0335] In step S3102, one candidate can be selected from two time-based selection blocks. Figure 33 The positions of the time candidates are illustrated. The position of each candidate can be determined based on a block within the collocated image that is at the same position as the current block in the current image. The collocated image can be set under the same conditions in both the image encoding and decoding devices within the reconstructed image. Time candidates can be considered in the order of blocks B1, B2, etc., and the motion information of the most available candidate blocks can be used to determine the time candidates.

[0336] For methods regarding motion information for determining time candidates, please refer to [link / reference needed]. Figure 34 .

[0337] See Figure 34 The motion information of candidate blocks (B1, B2) within the collocated image refers to the predicted blocks located within the reference image B. (The reference images for each candidate block can be different. For convenience, all are represented as reference image B in this specification.) Regarding the corresponding motion vector, the motion vector derived by scaling the motion vector of the candidate blocks according to the ratio of the distance between the collocated image and the reference image B to the distance between the current image and the reference image A is determined as the motion vector of the temporal candidate motion information. Equation 9 is the scaling mathematical expression.

[0338]

Mathematical Expression 9

[0339] Here, MV represents the motion vector of the temporal candidate block, MVscale represents the scaled motion vector, TB represents the temporal distance between the collocated image and the reference image B, and TD represents the temporal distance between the current image and the reference image A. Furthermore, reference image A and reference image B can be the same reference image. The motion vector scaled as described above can be used to determine the temporal candidate motion vector, and the reference image information of the temporal candidate motion information can be used to determine the reference image of the current image, thereby deriving the temporal candidate motion information.

[0340] Furthermore, in steps S3101 and S3102, the corresponding motion information can only be included in the motion information candidate group if the spatial candidate block and the temporal candidate block are encoded in the inter-frame prediction mode.

[0341] Step S3103 is executed only if the maximum number of motion information candidate groups cannot be derived in steps S3101 and S3102. It is a step of adding new bidirectional motion information candidate groups by using the combination of motion information candidates derived in the previous steps. Bidirectional motion information candidates refer to selecting one of the previously derived historical or future direction motion information in sequence and combining it into a new candidate.

[0342] Figure 35 The table provides the priority order for candidate combinations of bidirectional motion information. (Except for...) Figure 25 In addition to the combinations in the table, other additional combinations may exist; this table only illustrates one example. Even when using bidirectional motion information candidates, if the maximum number of motion information candidate groups cannot be satisfied, step S3104 can be executed. In step S3104, the motion vector of the motion information candidate can be fixed to a zero motion vector, and the maximum number of motion information candidate groups can be satisfied by changing the reference image in different prediction directions.

[0343] The Advanced Motion Vector Prediction (AMVP) mode determines the optimal motion information by performing motion inference on reference images for different prediction directions. The prediction direction can be a single direction using only one of the historical / future directions, or a combination of both. Predicted blocks are generated by performing motion compensation using the optimal motion information determined through motion inference. Specifically, candidate sets of motion information for performing motion inference can be derived from the reference images for different prediction directions. These candidate sets of motion information can then be used as the starting positions for motion inference.

[0344] Figure 32 This is a flowchart illustrating the derivation method of motion information candidate groups for motion inference in Advanced Motion Vector Prediction (AMVP) mode. The maximum number of motion information candidate groups can be determined to be the same value in both the image encoding and decoding devices, or the corresponding number of groups can be pre-transmitted in the upper-level header of the image encoding device.

[0345] See Figure 32 Step S3201 and the Figure 31The description of step S3101 may differ, the number of spatial candidates derived (2) may differ, and the priority order for selecting spatial candidates may also differ. Furthermore, since the description of S3201 is the same as that of step S3101, repeated descriptions will be omitted.

[0346] Since step S3202 is the same as the description of step S3102, the repeated description will be omitted.

[0347] In step S3203, if the currently derived subsequent selection contains duplicate motion information, it can be deleted.

[0348] Since step S3204 is the same as the description of step S3104, the repeated description will be omitted.

[0349] Among the motion information candidates derived in the manner described above, the motion information candidate with the lowest rate-distortion (RD) cost can be selected as the best motion information candidate, and the best motion information of the advanced motion vector prediction (AMVP) mode can be obtained through the motion inference process based on the corresponding motion information.

[0350] Furthermore, in the description of steps S3201 and S3202, only when spatial candidate blocks and temporal candidate blocks are encoded in an inter-frame prediction mode can motion information derived from the corresponding motion information be included in the motion information candidate group.

[0351] The transform unit 2903 generates a transform block by transforming the residual block, which is the difference between the original block and the prediction block. The transform block is the smallest unit used in the transform and quantization processes. The transform unit 2903 generates a transform block with transform coefficients by transforming the residual signal into a frequency region. Various transform techniques can be used to transform the residual signal into a frequency region, such as Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), and Karhunen-Loeve Transform (KLT), thereby transforming the residual signal into a frequency region and generating transform coefficients. To facilitate the use of transform techniques, matrix operations using basis vectors are performed. Depending on the prediction mode used when encoding the prediction block, multiple different transform techniques can be mixed during matrix operations. For example, when performing in-frame prediction, a Discrete Cosine Transform can be used in the horizontal direction and a Discrete Sine Transform in the vertical direction, depending on the prediction mode.

[0352] The quantization unit 2904 can generate a quantized transform block by quantizing the transform block. That is, the quantization unit 2904 can generate a quantized transform block with quantized transform coefficients by quantizing the transform coefficients of the transform block generated from the transform unit 2903. As a quantization method, various quantization methods can be used, such as Dead Zone Uniform Threshold Quantization (DZUTQ) or Quantization Weighted Matrix, or improved quantization methods.

[0353] Furthermore, the above description illustrates and explains the case where the image encoding apparatus includes a transform unit 2903 and a quantization unit 2904. However, the transform unit and quantization unit can also be selectively included in the image encoding apparatus. That is, the image encoding apparatus can generate a transform block by transforming the residual block but without performing a quantization process, or it can perform a quantization process without transforming the residual block into frequency coefficients, or it can even omit both the transform and quantization processes. Even when only a part of the transform unit or quantization unit is performed in the image encoding apparatus, or when all processes are not performed, the block input to the entropy encoding unit is usually referred to as a "quantization transform block".

[0354] The entropy encoding unit 2905 can encode the quantization transform block and output a bitstream. That is, the entropy encoding unit 2905 can use various encoding techniques, such as entropy encoding, to encode the coefficients of the quantization transform block output from the quantization unit 2904, and generate and output a bitstream containing additional information (e.g., prediction mode-related information, quantization coefficients, etc.) required for decoding the corresponding block in the image decoding apparatus described later. In other words, the entropy encoding unit 2905 can use various encoding techniques, such as entropy encoding, to encode the coefficients of the quantization transform block output from the quantization unit 2904, and generate and output a bitstream containing additional information (e.g., prediction mode-related information (which may include motion information or in-frame prediction mode information determined in the prediction unit, etc.), quantization coefficients, etc.) required for decoding the corresponding block in the image decoding apparatus described later.

[0355] The inverse quantization unit 2906 can reconstruct the inverse quantization transform block by reversing the quantization techniques used in the quantization process on the quantization transform block.

[0356] The inverse transform unit 2907 reconstructs the residual block by performing an inverse transform on the inverse quantization transform block using the same method used in the transform process. The inverse transform can be performed by reversing the transform technique used in the transform unit.

[0357] Furthermore, the aforementioned inverse quantization unit and inverse transform unit can perform inverse quantization and inverse transform by reversing the quantization and transform methods used in the quantization unit and the transform unit. Alternatively, if only quantization is performed in the transform unit and quantization unit without transformation, only inverse quantization can be performed without inverse transform. Moreover, if neither transformation nor quantization is performed, the inverse quantization unit and inverse transform unit may not perform inverse transform and inverse quantization, or may be omitted directly from the image encoding device.

[0358] The addition unit 2908 can reconstruct the current block by performing an addition operation on the residual signal generated in the inverse transform unit and the prediction block generated by prediction.

[0359] The filtering unit 2909 performs a filtering process on the entire image after reconstructing all blocks within the current image. This can include deblocking filtering and Sample Adaptive Offset (SAO). Block filtering refers to the operation used to reduce block distortion generated during the encoding of the image in block units, while Sample Adaptive Offset (SAO) refers to the operation of minimizing the difference between the reconstructed image and the original image by performing addition or subtraction operations on the reconstructed pixels using specific values.

[0360] The memory unit 2910 stores the current block of reconstruction after adding the residual signal generated in the inverse transform unit and the prediction block generated by prediction to the loop filtering unit, and can be used to predict the next block or the next image, etc.

[0361] The subtraction unit 2911 generates a residual block by performing a subtraction operation between the current original block and the predicted block.

[0362] Figure 36 This is a flowchart illustrating the encoding process of encoded information in an image encoding device.

[0363] See Figure 36 In step S3601, the action information for the skip (SKIP) mode will be encoded.

[0364] Next, in step S3602, it will be determined whether the SKIP mode action is enabled. When the SKIP mode action is enabled in step S3602, in step S3607, the MERGE candidate index information used for the SKIP mode can be encoded, and then this flowchart will end.

[0365] Conversely, when it is determined in step S3602 that the skip (SKIP) mode is not active, the prediction mode will be encoded in step S3603.

[0366] Next, in step S3604, it will be determined whether the prediction mode is an inter-frame prediction mode or an intra-frame prediction mode. When it is determined in S3604 that the prediction mode is an intra-frame prediction mode, in step S3608, the intra-frame prediction mode information can be encoded, and then this flowchart ends.

[0367] Conversely, when the prediction mode is determined to be the inter-frame prediction mode in S3604, the motion information of the MERGE mode will be encoded in step S3605.

[0368] Next, in step S3606, it will be determined whether the merge mode action is enabled. When the merge mode action is enabled in step S3606, the process can jump to step S3607 and encode the merge candidate index information used for the merge mode, and then end this flowchart.

[0369] Conversely, if it is determined in step S3606 that the MERGE mode is not active, the predicted direction will be encoded in step S3609. The predicted direction can be a historical direction, a future direction, or one of both.

[0370] Next, in step S3610, it is determined whether the predicted direction is a future direction. If it is determined in step S3610 that the predicted direction is not a future direction, in step S3611, the reference image index information of the historical direction is encoded. Next, in step S3612, the motion vector difference (MVD) information of the historical direction is encoded. Next, in step S3613, the motion vector prediction (MVP) information of the historical direction is encoded.

[0371] Conversely, when it is determined in step S3610 that the predicted direction is a future direction or both, after completing step S3613, in step S1313, it will be determined whether the predicted direction is a historical direction. Next, when it is determined in step S3614 that the predicted direction is not a historical direction, in step S3615, the reference image index information for the future direction will be encoded. Next, in step S3616, the motion vector difference (MVD) information for the future direction will be encoded. In step S3617, the motion vector prediction (MVP) information for the future direction can be encoded, and then this flowchart ends.

[0372] Figure 37 It is a block diagram that provides a general illustration of the structure of an image decoding device.

[0373] See Figure 37 The image decoding device is a device for decoding images, and may include an entropy decoding unit 3701, an inverse quantization unit 3702, an inverse transform unit 3703, a prediction unit 3704, an addition unit 3705, a loop filter unit 3706, and a memory unit 3707.

[0374] Figure 37 The various components illustrated are shown separately to illustrate different features and functions of the image decoding device, and do not represent that each component is composed of separate hardware or software units. That is, although the various components are listed for ease of explanation, at least two components can be combined into one component, or a component can be divided into multiple components to perform corresponding functions. The embodiments where the various components are integrated or separated as described above are included within the scope of the claims of this invention without departing from the essence of the invention.

[0375] Furthermore, some constituent elements may not be essential for performing the essential functions of this invention, but rather optional elements used to improve performance. This invention may include only the constituent parts essential for realizing the essence of the invention, excluding those merely used to improve performance, and structures including essential constituent elements other than optional elements used to improve performance are also included within the scope of the claims of this invention.

[0376] The encoding block in an image encoding device can be called the decoding block or the encoding block in an image decoding device.

[0377] The entropy decoding unit 3701 can read various information and quantization transformation coefficients required for decoding the corresponding block by parsing the bit stream transmitted from the image encoding device.

[0378] The inverse quantization unit 3702 can reconstruct the inverse quantization block with inverse quantization coefficients by reversing the quantization techniques used in the quantization process on the quantization coefficients decoded in the entropy decoding unit.

[0379] The inverse transform unit 3703 reconstructs the residual block with the difference signal by performing an inverse transform on the inverse quantization transform block using the same method as that used in the transform process. The inverse transform can be performed by reversing the transform technique used in the transform unit.

[0380] The prediction unit 3704 generates prediction blocks using the prediction mode information decoded in the entropy decoding unit. At this time, the same prediction method as that performed in the prediction unit of the image encoding device can be used.

[0381] The addition unit 3705 can reconstruct the current block by performing an addition operation on the residual signal reconstructed in the inverse transform unit and the prediction block generated by prediction.

[0382] The filtering unit 3706 performs a filtering process on the entire image area after reconstructing all blocks in the current image. It may include deblocking filtering and sample adaptive offset (SAO), etc. The details are the same as those described for the loop filtering unit of the image coding device.

[0383] The memory unit 3707 stores the current block of reconstruction after adding the residual signal generated in the inverse transform unit and the prediction block generated by prediction to the loop filtering unit, and can be used to predict the next block or the next image, etc.

[0384] Figure 38 This is a flowchart explaining the process within the prediction section of the image decoding device.

[0385] See Figure 38 When the prediction mode is in-screen prediction, the optimal in-screen prediction mode information 3801 can be determined, and a prediction block can be generated by performing in-screen prediction 3802.

[0386] When the prediction mode is inter-frame prediction, the optimal prediction mode 3803 can be determined from the skip, merge, and advanced motion vector prediction (AMVP) modes. Next, when decoding via the skip or merge mode, candidate motion information groups for the skip and merge modes can be constructed in the merge candidate exploration unit 3804. From the corresponding candidate motion information groups, the optimal motion information 3805 can be determined.

[0387] When decoding via Advanced Motion Vector Prediction (AMVP), candidate motion information groups for AMVP modes can be constructed in the AMVP candidate exploration unit 3806. Within each candidate motion information group, the transmitted motion vector prediction (MVP) information can be used to determine the optimal motion information 3807. Next, a prediction block can be generated by performing motion compensation 3808 using the optimal motion information determined in each mode.

[0388] Figure 39 This is a flowchart illustrating the decoding process of encoding and decoding information in an image decoding device.

[0389] See Figure 39 In step S3901, the action information of the skip (SKIP) mode will be decoded.

[0390] Next, in step S3902, it will be determined whether the SKIP mode action is active. When the SKIP mode action is determined in step S3902, in step S3907, the MERGE candidate index information used for the SKIP mode can be decoded, and then this flowchart ends.

[0391] Conversely, if it is determined in step S3902 that the skip (SKIP) mode is not active, the prediction mode will be decoded in step S3903.

[0392] Next, in step S3904, it will be determined whether the prediction mode is an inter-frame prediction mode or an intra-frame prediction mode. If it is determined in S3904 that the prediction mode is an intra-frame prediction mode, in step S3908, the intra-frame prediction mode information can be decoded, and then this flowchart ends.

[0393] Conversely, when the prediction mode is determined to be the inter-frame prediction mode in S3904, the motion information of the MERGE mode will be decoded in step S3905.

[0394] Next, in step S3906, it will be determined whether the merge mode action has been performed. When the merge mode action is determined in step S3906, the process can jump to step S3907 and decode the merge candidate index information used for the merge mode, and then end this flowchart.

[0395] Conversely, if it is determined in step S3906 that the MERGE mode is not active, the predicted direction will be decoded in step S3909. The predicted direction can be a historical direction, a future direction, or one of both.

[0396] Next, in step S3910, it is determined whether the predicted direction is a future direction. If it is determined in step S3910 that the predicted direction is not a future direction, in step S3911, the reference image index information of the historical direction is decoded. Next, in step S3912, the motion vector difference (MVD) information of the historical direction is decoded. Next, in step S3913, the motion vector prediction (MVP) information of the historical direction is decoded.

[0397] Furthermore, when it is determined in step S3910 that the predicted direction is a future direction or both, after completing step S3913, in step S1913, it will be determined whether the predicted direction is a historical direction. Next, when it is determined in step S3914 that the predicted direction is not a historical direction, in step S3915, the reference image index information for the future direction will be decoded. Next, in step S3916, the motion vector difference (MVD) information for the future direction will be decoded. Next, in step S3917, the motion vector prediction (MVP) information for the future direction can be decoded, and then this flowchart ends.

[0398] Next, various embodiments of the present invention will be described in more detail with reference to the accompanying drawings.

[0399] The image encoding / decoding method applicable to one embodiment of the present invention can be used as an inter-frame prediction mode including a decoder-side motion vector derivation (DMVD) mode.

[0400] Decoder-side motion vector derivation (DMVD) mode can refer to a mode in which some or all of the motion information is directly derived in the image decoding device without transmitting some or all of the motion information from the image encoding device.

[0401] That is, decoder-side motion vector derivation (DMVD) mode can refer to a mode in which the image decoding device derives motion information on its own. Alternatively, decoder-side motion vector derivation (DMVD) mode can refer to a mode in which the image decoding device corrects the initial motion information derived based on the information transmitted from the image encoding device.

[0402] Figure 40 This is a schematic diagram illustrating the prediction unit within an image encoding apparatus to which one embodiment of the present invention is applied.

[0403] See Figure 40 In steps 4001 to 4007, the following can be executed: Figure 30 The process is the same as steps 3001 to 3007. That is, a prediction block can be generated using the optimal in-frame prediction mode determined by steps 4001 to 4007, or motion compensation 4010 can be performed and a prediction block can be generated using the optimal motion information determined by steps 4003 to 4007.

[0404] Figure 40 The image encoding apparatus can perform a motion information determination method using the decoder-side motion vector derivation (DMVD) mode when additionally performing inter-frame prediction. To perform motion inference in the same manner within the image encoding / apparatus, initial motion information can be determined using motion information from the reconstructed region (DMVD initial motion information exploration unit 4008). Next, optimal motion information can be determined by performing motion inference using the determined initial motion information (DMVD motion inference unit 4009), and a prediction block can be generated by performing motion compensation 4010 using the determined optimal motion information. The motion information may include at least one of motion vectors, reference image indexes, and prediction direction information.

[0405] Figure 41 This is a schematic diagram illustrating the prediction unit within an image decoding apparatus according to one embodiment.

[0406] See Figure 41 In steps 4101 to 4107, the following can be executed: Figure 38 The process is the same as steps 3801 to 3807. That is, a prediction block can be generated using the in-frame prediction mode determined by steps 4101 to 4102, or motion compensation 4110 can be performed and a prediction block can be generated using the motion information determined by steps 4103 to 4007.

[0407] Figure 41The image decoding apparatus can perform a motion information determination method using decoder-side motion vector derivation (DMVD) mode when additionally performing inter-frame prediction. To perform motion inference in the same manner within the image encoding / apparatus, initial motion information 4108 can be determined using motion information from the reconstructed region. Next, motion information can be determined by performing motion inference 4109 using the determined initial motion information, and a prediction block can be generated by performing motion compensation 4110 using the determined motion information. That is, in decoder-side motion vector derivation (DMVD) mode, the image decoding apparatus can automatically correct the initial motion information and derive motion information for motion compensation without signaling information.

[0408] also, Figure 41 The Decoder-Side Motion Vector Derivation (DMVD) Initial Motion Information Exploration Unit 4108 can determine the motion information of the merged candidate derived through steps 4104 to 4105 as the Decoder-Side Motion Vector Derivation (DMVD) Initial Motion Information, or it can determine the motion information derived through steps 4106 to 4107 as the Decoder-Side Motion Vector Derivation (DMVD) Initial Motion Information.

[0409] Figure 42 This is a schematic diagram used to explain the initial motion information exploration units 4008 and 4108 of the decoder-side motion vector derivation (DMVD).

[0410] See Figure 42 The table, Figure 42 The indexes in the table can refer to the priority order of the initial motion information. As initial motion information, at least one of the following can be used: candidate motion information from the Skip mode, candidate motion information from the Advanced Motion Vector Prediction (AMVP) mode, candidate motion information from the Merge mode, motion information from sub-blocks within the reconstruction regions located above, to the left, to the upper left, to the upper right, and to the lower left of the current block, or zero motion information. Furthermore, various different motion information candidates derived from other reconstruction information can be used.

[0411] As mentioned above, initial motion information can be obtained by utilizing, for example, in Figures 31 to 35 The motion information derived from inter-frame prediction modes, such as Skip mode, Advanced Motion Vector Prediction (AMVP) mode, or Merge mode, is explained in the text.

[0412] Figure 43 This is a schematic diagram illustrating the motion prediction unit 4009 and 4109 of the decoder-side motion vector derivation (DMVD) unit in the prediction unit of the image encoding / decoding apparatus.

[0413] The image encoding device can determine the optimal motion information after executing the decoder-side motion vector derivation (DMVD) mode using the initial motion information determined by the decoder-side motion vector derivation (DMVD) initial motion information exploration unit.

[0414] The image decoding device can determine the motion information needed for motion compensation after executing the decoder-side motion vector derivation (DMVD) mode using the initial motion information determined by the decoder-side motion vector derivation (DMVD) initial motion information exploration unit. Executing the decoder-side motion vector derivation (DMVD) mode can refer to correcting the initial motion information.

[0415] Decoder-side motion vector derivation (DMVD) modes can include modes that use templates (hereinafter referred to as "template matching modes") and modes that do not use templates (hereinafter referred to as "bidirectional matching modes").

[0416] When using the bidirectional matching mode 4301, the unidirectional motion vectors of each initial motion information can be linearly scaled to a reference image in the opposite prediction direction. The scaling of the motion vectors can be performed in a manner proportional to the distance between the current image and the reference images in each direction. After determining the motion vectors 4302 in both directions as described above, the motion vectors in each direction with the smallest difference between the prediction blocks in the historical and future directions can be determined as the optimal motion information 4304.

[0417] Furthermore, when the initial motion information is bidirectional motion information that simultaneously contains motion information in both historical and future directions, the motion information with the smallest difference between the predicted blocks indicated by the motion information in each direction can be determined as the optimal motion information 4304.

[0418] As an example, the optimal motion vector can be determined by minimizing the difference between the predicted block indicated by the initial motion vector in the historical direction and the predicted block indicated by the initial motion vector in the future direction. This optimal motion vector can then be used as the motion information for the current block.

[0419] Figure 45 This is a schematic diagram illustrating a method for exploring motion vectors in a two-way matching mode, based on the predicted blocks indicated by the historical and future direction motion vectors of the current block. It can be used to explore... Figure 45 After identifying the location with the smallest difference between predicted blocks in each direction, the motion vectors in each direction are corrected to point to the previously explored location. At this point, the corrected motion vectors can be used as the optimal motion vector or as the motion vector for performing motion compensation (or inter-frame prediction).

[0420] When using template matching mode 4305, the template region 4306 can be determined from the reconstructed region. The size and shape of the template region can be determined using various methods, such as determining the reconstructed regions adjacent to the current region on its left and top edges as template regions. Optimal motion information can be determined by using the motion inference 4307 of the corresponding template region. Figure 44 This is a schematic diagram illustrating the method of determining the block adjacent to the corresponding template block as the prediction block of the current block after exploring the predicted block of the template block most similar to the template block in the single template matching mode.

[0421] In exploring the optimal motion information for both bidirectional matching patterns and template matching patterns, the exploration pattern can be determined in several different ways. Among them, the process of exploring the final motion information can be a motion inference process.

[0422] Figure 46 This is a schematic diagram illustrating the process of conducting an exploration using a diamond-shaped exploration pattern.

[0423] See Figure 46 P1 can be used as the starting point for exploration, and the cost of motion inference for pixels S1 located inside the diamond shape, including P1, can be calculated. Here, P1 can be determined based on the initial motion information.

[0424] After calculation, the cost value of the pixel S2 located inside the diamond shape can be recalculated based on the position with the minimum cost value, i.e., the reference point P2, to determine the position P3 with the minimum cost value. This process is repeated within the exploration area until the cost value of the central pixel inside the diamond shape is minimized, thus determining the motion information indicating the optimal position as either template matching mode or bidirectional matching mode motion information. Furthermore, Figure 46 The process of exploring the position with the minimum cost within the pattern can be repeated a predefined number of times.

[0425] like Figure 44 as well as Figure 45 As shown, when exploring (or speculating) the best motion information in both bidirectional matching mode and template matching mode, the exploration area (or motion speculation area) can be limited within the reference image 4303.

[0426] In particular, when executing the template matching mode, the template-configurable region (hereinafter referred to as the "template speculation region") of the current block in the current image can also be limited 4303.

[0427] If the exploration area is not limited in the image decoding device and the exploration is repeatedly performed until the optimal motion position is found, the computational complexity of the image decoding device will inevitably become very high. Moreover, if the template inference area is not limited, the parallel processing design of the current image will become very difficult, so it is necessary to improve the processing speed by limiting it.

[0428] The constraint can be set such that, in bidirectional matching mode, the predicted blocks in the historical / future directions must not exceed the motion inference region, while in template matching mode, the template block of the predicted block within the reference image must not exceed the motion inference region. Furthermore, the constraint can be set such that, in template matching mode, the template block of the current block within the current image must not exceed the template inference region.

[0429] Furthermore, the exploration area can be set to a window of any size (I×J). The size of this window can be changed based on the distance between the current image and the reference image, or it can be transmitted in the parent header (such as video parameter level, sequence parameter level, image parameter level, and stripe level).

[0430] In addition, a pre-set exploration area of ​​the same size can be used in both the image encoding device and the image decoding device.

[0431] Furthermore, the size of the exploration region cannot be determined beyond the segmentation boundaries determined in higher-level headers such as stripes and parallel blocks.

[0432] However, in the case of bidirectional matching mode, since the premise is that the motion vectors in different directions have linear orientation, the exploration areas in different directions can be different.

[0433] In addition, the exploration area can be determined as the largest coded block containing the current block.

[0434] Furthermore, windows of arbitrary size (K×L) can be set in the same manner as the exploration region, and the corresponding window can be determined as the template prediction exploration region. The size of the aforementioned window can be transmitted in the upper-level header, or a pre-set size can be used in the image encoding / decoding device. However, it is not possible to determine a size exceeding the segmentation boundaries determined in upper-level headers such as stripes and parallel blocks. Template regions not included in the template prediction region cannot be used as templates for template matching patterns.

[0435] The cost of the motion information exploration process refers to the sum of the prediction error and the hypothetical bit amount of motion information. The prediction error can be obtained through various calculation methods, such as the sum of absolute errors (SAD), the sum of absolute variation errors (SATD), and the sum of square errors (SSD). Equations 10, 11, and 12 describe the calculation methods of the sum of absolute errors (SAD), the sum of absolute variation errors (SATD), and the sum of square errors (SSD), respectively.

[0436]

Mathematical Formula 10

[0437]

Mathematical Expression 11

[0438]

Mathematical Expression 12

[0439] Here, i and j represent pixel positions, Template(i, j) represents the pixels in the template block, and PredBlk(i, j) represents the pixels in the prediction block. The HT() function in Equation 4 refers to the function value obtained by performing a Hadamard transform on the difference block between the template block and the prediction block. The hypothetical bit value of motion information is not the information actually transmitted, but rather a value calculated from the hypothetical bit value of motion information anticipated in the image encoding and decoding devices using the same method. For example, the hypothetical bit value can be determined by calculating the difference vector between the motion vector of the initial motion information and the motion vector within the motion information currently being predicted. Alternatively, the hypothetical bit value of motion information can also be calculated using the bit value of the reference image information.

[0440] Figure 47 This is a flowchart of encoding codec information according to one embodiment.

[0441] See Figure 47In step S4701, the action information for the Skip mode is encoded. In step S4702, it is determined whether the Skip mode action has occurred. When the Skip mode action is determined in step S4702, the action information for the Decoder-Side Motion Vector Derivation (DMVD) mode is encoded in step S4703. In step S4704, it is determined whether the Decoder-Side Motion Vector Derivation (DMVD) mode action has occurred. When the Decoder-Side Motion Vector Derivation (DMVD) mode action is determined in step S4704, the Decoder-Side Motion Vector Derivation (DMVD) information is encoded in step S4705, and then the flowchart ends. The Decoder-Side Motion Vector Derivation (DMVD) information refers to the flag information used to indicate whether it belongs to a bidirectional matching mode or a template matching mode. If the upper / left template blocks surrounding the current block are not included in the template prediction region, the decoder-side motion vector derivation (DMVD) information will not be encoded, and the decoder-side motion vector derivation (DMVD) information will be determined as a bidirectional matching mode. When it is determined in step S4704 that the decoder-side motion vector derivation (DMVD) mode is inactive, in step S4706, the merge candidate index information for the SKIP mode will be encoded, and then the flowchart will end. When it is determined in step S4702 that the SKIP mode is inactive, in step S4707, the prediction mode information will be encoded. In step S4708, it will be determined whether the prediction mode is inter-frame prediction. When it is determined in S4708 that the prediction mode is intra-frame prediction mode rather than inter-frame prediction, in step S4709, the intra-frame prediction information will be encoded, and then the flowchart will end. When the prediction mode is determined to be inter-frame prediction in step S4708, the MERGE mode action information is encoded in step S4710. In step S4711, the presence or absence of MERGE mode action is determined. When MERGE mode action is determined in step S4711, the decoder-side motion vector derivation (DMVD) mode action information is encoded in step S4712. When DMVD mode action is determined in step S4713, the decoder-side motion vector derivation (DMVD) information is encoded in step S4714, and then the flowchart ends. When DMVD mode action is determined not to occur in step S4713, the MERGE candidate index information used for MERGE mode is encoded in step S4706, and then the flowchart ends. If it is determined in step S4711 that the MERGE mode is inactive, steps S4715 to S4723 will be executed. This is consistent with the previous steps. Figure 36The descriptions of steps S3609 to S3617 are the same. This flowchart will end after completing step S4723.

[0442] Figure 48 This is a flowchart of decoding encoded and decoded information according to one embodiment.

[0443] See Figure 48 In step S4801, the action information for the Skip mode is decoded. In step S4802, it is determined whether the Skip mode action has occurred. If the Skip mode action is determined in step S4802, the action information for the Decoder-Side Motion Vector Derivation (DMVD) mode is decoded in step S4803. In step S4804, it is determined whether the Decoder-Side Motion Vector Derivation (DMVD) mode action has occurred. If the Decoder-Side Motion Vector Derivation (DMVD) mode action is determined in step S4804, the Decoder-Side Motion Vector Derivation (DMVD) information is decoded in step S4805, and then the flowchart ends. If the Decoder-Side Motion Vector Derivation (DMVD) mode action is not determined in step S4804, the MERGE candidate index information for the Skip mode is decoded in step S4806, and then the flowchart ends. If it is determined in step S4802 that the Skip mode is inactive, the prediction mode information will be decoded in step S4807. In step S4808, it will be determined whether the prediction mode is inter-frame prediction. If it is determined in S4808 that the prediction mode is intra-frame prediction rather than inter-frame prediction, the intra-frame prediction information will be decoded in step S4809, and then the flowchart will end. If it is determined in S4808 that the prediction mode is inter-frame prediction, the MERGE mode action information will be decoded in step S4710. In step S4811, it will be determined whether the MERGE mode is active. If it is determined in step S4811 that the MERGE mode is active, the decoder-side Motion Vector Derivation (DMVD) mode action information will be decoded in step S4812. When it is determined in step S4813 that the decoder-side motion vector derivation (DMVD) mode is active, in step S4814, the decoder-side motion vector derivation (DMVD) information will be decoded, and then the flowchart will end. When it is determined in step S4813 that the decoder-side motion vector derivation (DMVD) mode is not active, in step S4806, the merge candidate index information used for the merge mode will be decoded, and then the flowchart will end. When it is determined in step S4811 that the merge mode is not active, steps S4815 to S4823 will be executed, which is consistent with the previous steps. Figure 39 The descriptions of steps S3909 to S3917 are the same. This flowchart will end after step S4823 is completed.

[0444] Next, when determining the template region from the reconstructed region surrounding the current block in this embodiment, multiple sub-template blocks can be used instead of a single template block. The method for deriving the motion information of each sub-template block using the template matching pattern described above and using the corresponding motion information as candidate motion information for skip, merge, and advanced motion vector prediction (AMVP) predictions will be explained in detail below.

[0445] Figure 49 This is a flowchart illustrating a method for deriving candidate motion information for skip (SKIP) and merge (MERGE) modes using a template matching pattern, applicable to one embodiment of the present invention.

[0446] See Figure 49 In step S4901, the reconstructed motion information in the sub-template block is used as the initial motion information and processed as described above. Figure 43 The processes 4305, 4306, 4304, and 4307 in the code derive the optimal motion information for the sub-template blocks. At this point, the number of sub-template blocks can be L (L is an integer ≥ 1).

[0447] Figure 51 This is a schematic diagram illustrating an example of determining sub-template blocks from the reconstruction area surrounding the current block.

[0448] See Figure 51 The sub-template blocks can be set according to the examples in 5101 to 5103.

[0449] exist Figure 51 In 5101, sub-template blocks can be determined at the bottom left (template A), top left (template B), top left (template C), and top right (template D) of the current block. Each sub-template block can be determined to have a variety of different sizes and shapes.

[0450] Figure 51 The method in 5102 is the same as that in 5101. The sub-template blocks can be determined from the lower left (template A), upper left (template B), upper left (template C), and upper right (template D) of the current block. However, the difference is that all the reconstruction areas on the left and upper sides adjacent to the current block are used.

[0451] Figure 51 Method 5103 is a method for generating sub-template blocks that takes into account both the sub-template block generation methods in 5101 and 5102.

[0452] In addition, sub-template blocks can be generated from the reconstruction areas surrounding the current block using various methods, such as determining the reconstruction areas to the left and top of the current block as a sub-template block.

[0453] In step S4902, M space candidates can be selected (M is an integer ≥ 1). The generation method is similar to that used for... Figure 31 The description of step S3101 is the same. In step S4903, N time candidates can be selected (N is an integer ≥ 1). The generation method is the same as for... Figure 31 The description of step S3102 is the same. Steps S4904 and S4905 are the same as those in the previous section. Figure 31 The descriptions of steps S3103 and S3104 in the flowchart are the same. The order of each step can be changed in this flowchart. Furthermore, if previously derived motion information in each step is the same as the currently derived motion information, the corresponding sufficient candidates can be deleted.

[0454] Figure 50 This is a flowchart illustrating a method for deriving candidate motion information for an Advanced Motion Vector Prediction (AMVP) mode using a template matching pattern, applicable to one embodiment of the present invention.

[0455] See Figure 50 The explanation of step S5001 and the Figure 49 The description of step S4901 is the same. In step S5002, M space candidates can be selected (M is an integer ≥ 1). The generation method is the same as for... Figure 32 The description of step S3201 is the same. In step S5003, N time candidates can be selected (N is an integer ≥ 1). The generation method is the same as for... Figure 32 The description of step S3202 is the same. Step S5004 is the same as that of... Figure 32 The description of step S3204 is the same. The order of each step can be changed in this flowchart. Furthermore, if previously derived motion information in each step is the same as the currently derived motion information, the corresponding sufficient candidates can be deleted.

[0456] Next, in this embodiment, we will describe the method of deriving corrected candidate motion information from candidate motion information in skip and merge modes using decoder-side motion vector derivation (DMVD) mode (template matching mode or bidirectional matching mode), and the method of rearranging candidate motion information in skip and merge modes using the cost of template matching.

[0457] Figure 52 This is a table explaining the method of correcting candidate motion information for skip and merge modes using the decoder-side motion vector derivation (DMVD) mode.

[0458] See Figure 52 , Figure 52 Table 5201 in the table is used Figure 31 or Figure 49 The method described above derives the results of candidate motion information for skip and merge patterns. In this example, there are 5 candidate motion information. The derived candidate motion information can include historical / future direction motion information.

[0459] Different candidate motion information can be obtained through the above... Figure 43 The process obtains corrected motion information. Figure 52 In Table 5202, a portion of the candidate motion information in Table 5201 has been corrected. This correction can be performed on all candidate motion information, or it can be done on only a subset of candidate motion information, such as correcting only those with higher priority.

[0460] exist Figure 52 In Table 5202, corrections were made to the three candidate motion information from the higher level. Figure 52 In Table 5202, the motion information of candidate motion information 1 and 3 was corrected, but that of candidate 2 was not corrected. This indicates that correction of candidate motion information 2 is not necessary. As exemplified by candidate motion information 1, correction can be performed only on the motion vector within the motion information. Similarly, as exemplified by candidate motion information 3, correction can be performed on both the motion vector and the reference image index within the motion information.

[0461] As described above, the candidate motion information corrected using the decoder-side motion vector derivation (DMVD) mode can be used for motion compensation in the current block. Furthermore, for use in blocks encoded / decoded after the current block, the candidate motion information corrected using the decoder-side motion vector derivation (DMVD) mode can be stored. In the case described above, the candidate motion information corrected using the decoder-side motion vector derivation (DMVD) mode can be used to derive motion information for blocks encoded / decoded after the current block.

[0462] Furthermore, in the event of a change in the reference image, the above Figure 43 The motion inference detection area in step 4303 can also be changed compared to the original reference image. For example, see... Figure 53In addition to using template matching to correct the original candidate motion information in the skip and merge modes, the reference image can also be corrected. At this point, the motion inference exploration region can also be changed; specifically, it can be moved based on the linear orientation of the motion vector. When the reference image becomes farther away from the current image, the motion inference exploration region can be further expanded or reduced while still including the current motion inference exploration region.

[0463] Figure 54 This is a table illustrating the method of rearranging candidate motion information for skip and merge patterns using template matching.

[0464] Figure 54 Table 5401 and Figure 52 The same as Table 5201 in the table is used. Figure 31 or Figure 49 The method described in the paper derives the results of candidate motion information for skip and merge patterns.

[0465] exist Figure 54 Table 5402 uses the candidate motion information derived from Table 5401 to calculate the cost value of the template matching pattern (hereinafter referred to as "cost value"). Table 5402 shows the result of rearranging the candidate motion information in ascending order of corresponding cost value. When the candidate motion information includes both historical and future motion information, the cost value is calculated using candidate motion information in different directions, and the average cost value in different directions is determined as the cost value of the corresponding candidate motion information. Alternatively, the cost value of candidate motion information with lower cost values ​​in different directions can be determined as the cost value of the corresponding candidate motion information.

[0466] See the above. Figure 32 In the candidate motion information of the Advanced Motion Vector Prediction (AMVP) mode, the best candidate motion information, i.e., the motion vector prediction (MVP) information, can be encoded in the entropy coding unit and decoded in the entropy decoding unit. Following the method described above, the template matching mode cost value is calculated for each candidate motion information of the AMVP mode using template matching mode. Next, the template matching mode costs are arranged in ascending order, and the candidate motion information with the lowest cost value is determined as the motion vector prediction (MVP) information. Therefore, the motion vector prediction (MVP) information can be reconstructed in the image decoding device without encoding it in the entropy coding unit.

[0467] Next, in this embodiment, the method for calculating the weighted value based on the template region when calculating the template matching pattern value in the template matching pattern mentioned in the above embodiments will be explained in detail.

[0468] Figure 55 This is a schematic diagram illustrating a method for dividing a template of a template matching pattern into sub-templates and applying weighted values ​​of different cost values ​​to each sub-template, according to one embodiment of the present invention.

[0469] The weighted values ​​of the cost values ​​calculated in different sub-templates can be different. Formula 13 is the general mathematical formula used to calculate the cost value.

[0470]

Mathematical Expression 13

[0471] In Equation 13, TempDist refers to the prediction error in the cost of motion inference for template matching patterns. The prediction error is equal to the sum of the products of the prediction error of each sub-template block (SubTempDisti) and the weighted value of the corresponding sub-template block (Wi). i represents the index information of the sub-template block.

[0472] Figure 55 5501 in the diagram is an example of a template block constructed using sub-template blocks of the same size. Sub-template blocks can be set to the same size in a variety of different ways.

[0473] Figure 55 5502 is an example diagram of a template block constructed using sub-templates based on the actual dividing boundary line of the reconstructed block.

[0474] Figure 55 Figure 5503 is an example of constructing a template block using sub-template blocks with similar reconstructed pixel characteristics. The reconstructed pixel characteristics can be the reconstructed pixel value itself or the motion information of the reconstructed pixel. In addition, template blocks can also be constructed using sub-template blocks segmented by various different methods.

[0475] The weighting of the prediction error can be changed based on the position of the sub-template block determined as described above.

[0476] As one embodiment of the weighting method applicable to this invention, the weighting value of the error can be reduced when the sub-template block is closer to the pixel located at the upper left end of the current block, and increased when it is farther away. The opposite approach can also be used.

[0477] As another embodiment of the weighting value determination method applicable to the present invention, the weighting value of each sub-template block can be changed according to the similarity between the motion information of each sub-template block and the motion information currently being performed for motion prediction. The weighting value can be decreased when the similarity is higher and increased when the similarity is lower, or vice versa.

[0478] Figure 56 This is a flowchart illustrating an image decoding method applicable to one embodiment of the present invention.

[0479] See Figure 56 In step 5601, the image decoding device can use the motion vector of the reconstructed region to determine the initial motion vector of the current block.

[0480] The initial motion vector can include motion vectors from historical directions and motion vectors from future directions. That is, the initial motion vector can be predicted motion information in both directions.

[0481] Furthermore, the motion vector of the reconstructed region can be candidate motion information from the MERGE pattern. Specifically, the candidate motion information from the MERGE pattern can include at least one of the motion information of spatial candidate blocks and the motion information of temporal candidate blocks for the current block.

[0482] Next, in step 5602, the image decoding device can explore the motion vector of the current block based on the initial motion vector determined in step 5601.

[0483] Specifically, the steps for exploring motion vectors can be to explore motion vectors by utilizing the difference between the predicted blocks of the historical direction indicated by the motion vectors of the historical direction and the predicted blocks of the future direction indicated by the motion vectors of the future direction.

[0484] As an example, the steps for exploring motion vectors can be based on the prediction error generated by applying the difference between motion vectors in historical directions and predicted blocks in future directions to the sum of absolute differences (SAD) calculation method.

[0485] Furthermore, the step of exploring motion vectors can be performed within a predefined exploration area.

[0486] Next, in step 5603, the image decoding device can generate a prediction sample of the current block based on the motion vectors explored in step 5602.

[0487] also, Figure 56The image decoding method may further include a step of decoding Decoder-side Motion Vector Derivation (DMVD) mode motion information. This step can be performed after step S2801. If the DMVD mode motion information indicates DMVD mode motion, a step of exploring the motion vector of the current block based on the initial motion vector can be performed.

[0488] exist Figure 56 The motion vectors explored in the image decoding method can be used in the motion vector derivation of blocks to be decoded after the current block. The explored motion vectors can refer to those corrected using the decoder-side motion vector derivation (DMVD) mode.

[0489] Figure 57 This is a flowchart illustrating an image encoding method applicable to one embodiment of the present invention.

[0490] See Figure 57 In step 5701, the image encoding device can use the motion vector of the reconstructed region to determine the initial motion vector of the current block.

[0491] The initial motion vector can include motion vectors from historical directions and motion vectors from future directions. That is, the initial motion vector can be predicted motion information in both directions.

[0492] Furthermore, the motion vector of the reconstructed region can be candidate motion information from the MERGE pattern. Specifically, the candidate motion information from the MERGE pattern can include at least one of the motion information of spatial candidate blocks and the motion information of temporal candidate blocks for the current block.

[0493] Next, in step 5702, the image encoding device can explore the optimal motion vector for the current block based on the initial motion vector.

[0494] Specifically, the steps to explore the optimal motion vector can be to utilize the difference between the predicted block of the historical direction indicated by the motion vector of the historical direction and the predicted block of the future direction indicated by the motion vector of the future direction.

[0495] As an example, the steps for exploring the optimal motion vector can be based on the prediction error generated by applying the difference between the motion vector in the historical direction and the predicted block in the future direction to the sum of absolute errors (SAD) calculation method.

[0496] Furthermore, the step of exploring the optimal motion vector can be performed within a predefined exploration area.

[0497] Next, in step 5703, the image encoding device can encode the motion information of the decoder-side motion vector derivation (DMVD) mode by determining whether or not the optimal motion vector is used.

[0498] exist Figure 57 The optimal motion vector discovered in the image coding method can be used in the motion vector derivation of blocks encoded after the current block. The optimal motion vector discovered can refer to the motion vector corrected using the decoder-side motion vector derivation (DMVD) mode.

[0499] pass Figure 57 The bitstream (or encoded data) generated by the image encoding method can be stored on a computer-readable, non-transitory recording medium. Furthermore, the bitstream may also include decoder-side motion vector derivation (DMVD) mode motion information.

[0500] Figure 58 This is a flowchart illustrating an image encoding device.

[0501] See Figure 58 The image encoding apparatus is an apparatus for encoding images, and may include a block segmentation unit 5801, a prediction unit 5802, a transform unit 5803, a quantization unit 5804, an entropy encoding unit 5805, an inverse quantization unit 5806, an inverse transform unit 5807, an addition unit 5808, a loop filter unit 5809, a memory unit 5810, and a subtraction unit 5811.

[0502] Figure 58 The various components illustrated are shown separately to illustrate different features and functions of the image encoding apparatus, and do not represent that each component is composed of separate hardware or software units. That is, although the various components are listed for ease of explanation, at least two components can be combined into one component, or a component can be divided into multiple components to perform corresponding functions. The embodiments where the various components are integrated or separated as described above are included within the scope of the claims of this invention without departing from the essence of the invention.

[0503] Furthermore, some constituent elements may not be essential for performing the essential functions of this invention, but rather optional elements used to improve performance. This invention may include only the constituent parts essential for realizing the essence of the invention, excluding those merely used to improve performance, and structures including essential constituent elements other than optional elements used to improve performance are also included within the scope of the claims of this invention.

[0504] The block segmentation unit 5801 can segment the input image into at least one block. The input image can have various shapes and sizes, such as images, stripes, parallel blocks, tiles, or fragments. A block can refer to a coding unit (CU), a prediction unit (PU), or a transform unit (TU). The segmentation can be performed based on at least one of quadtree, binary tree, and ternary tree segmentation.

[0505] A quadtree is a partitioning method that accurately divides the current coded block into four parts. A binary tree is a partitioning method that accurately divides the coded block into two parts along a horizontal or vertical direction. A ternary tree is a partitioning method that divides the above block into three lower-level blocks. For example, the above three lower-level blocks can be obtained by dividing the width or height of the above upper-level block in a 1:2:1 ratio. By performing partitioning based on a binary tree as described above, the block can have not only a square shape but also a non-square shape. In addition, a method that uses multiple partitioning methods simultaneously can also be used.

[0506] The prediction unit 5802 can generate a prediction block using the surrounding pixels of the block that needs to be predicted in the current original block (hereinafter referred to as the prediction block) or pixels in a reference image that has been previously encoded / decoded.

[0507] As a prediction block, one or more prediction blocks can be generated within the coding block. When there is only one prediction block within the coding block, the prediction block can have the same form as the coding block.

[0508] Video signal prediction techniques include intra-frame prediction and inter-frame prediction.

[0509] In-frame prediction is a method of generating prediction blocks using the surrounding pixels of the current block.

[0510] Inter-image prediction is a method of finding the most similar block to the current block from previously encoded / decoded reference images and generating a predicted block.

[0511] Furthermore, the prediction unit 5802 can determine the optimal prediction mode for the prediction block from the residual block obtained by subtracting the prediction block from the original block after generating the prediction block, using various techniques such as rate-distortion optimization (RDO). The calculation formula for the rate-distortion optimization (RDO) cost is shown in mathematical formula 14.

[0512]

Mathematical Expression 14

[0513] Where D, R, and J represent the degradation caused by quantization, the bit rate of the compressed stream, and the rate-distortion (RD) cost, respectively; Φ represents the encoding mode; and λ represents the Lagrange multiplier, which can be used as a scaling factor to match the units between error and bit count. To select the optimal encoding mode during the encoding process, the rate-distortion (RD) cost when using the appropriate mode should be less than when using other modes. The formula for calculating the rate-distortion (RD) cost can be performed while simultaneously considering both bit rate and error.

[0514] Figure 59 This is a schematic diagram used to explain the prediction unit of an image encoding device.

[0515] When performing in-frame prediction 5901 using the original information and the reconstructed information, the optimal in-frame prediction mode 5902 can be determined by using the rate-distortion (RD) cost for each prediction mode and a prediction block can be generated.

[0516] When performing inter-frame prediction 5903 using raw and reconstructed information, the rate-distortion (RD) cost can be calculated for Skip mode, Merge mode, and Advanced Motion Vector Prediction (AMVP) mode.

[0517] In the MERGE candidate exploration section 5904, candidate motion information groups for both the SKIP and MERGE modes can be constructed. Furthermore, within each candidate motion information group, rate-distortion (RD) cost can be used to determine the optimal motion information 5905.

[0518] In the Advanced Motion Vector Prediction (AMVP) candidate exploration unit 5906, a group of candidate motion information can be constructed for the AMVP mode. Furthermore, the optimal motion information can be determined by performing motion inference 5907 using the corresponding group of candidate motion information.

[0519] Predictive blocks can be generated by performing motion compensation 5908 using the best motion information determined in each mode.

[0520] The aforementioned inter-frame prediction can be achieved using three modes: Skip mode, Merge mode, and Advanced Motion Vector Prediction (AMVP) mode. Motion information (predicted direction information, reference image information, and motion vectors) can be encoded according to each prediction mode.

[0521] The Skip mode utilizes motion information from reconstructed regions to determine the optimal prediction information. The Skip mode constructs motion information candidate groups within the reconstructed regions and generates prediction blocks by using the candidate with the lowest rate-distortion (RD) cost from the corresponding candidate group as the prediction information. Since the method for constructing motion information candidate groups is the same as the method for constructing motion information candidate groups in the MERGE mode described below, it will be omitted in this description.

[0522] The MERGE mode is similar to the SKIP mode in utilizing motion information from reconstructed regions to determine the best prediction information. However, the difference lies in that the SKIP mode explores motion information with zero prediction error from a set of motion information candidates, while the MERGE mode explores motion information with non-zero prediction error from a set of motion information candidates. Similar to the SKIP mode, motion information candidate sets can be constructed within the reconstructed region, and prediction blocks are generated by using the candidate with the lowest rate-distortion (RD) cost from the corresponding candidate set as the prediction information.

[0523] The Advanced Motion Vector Prediction (AMVP) mode determines the optimal motion information by performing motion inference on reference images for different prediction directions. The prediction direction can be a single direction using only one of the historical / future directions, or a combination of both. Predicted blocks are generated by performing motion compensation using the optimal motion information determined through motion inference. Specifically, candidate sets of motion information for performing motion inference can be derived for reference images of different prediction directions. These candidate sets of motion information can then be used as the starting positions for motion inference.

[0524] For efficient encoding / decoding of the optimal intra-frame prediction mode, the Most Probable Mode (MPM) can be used for intra-frame prediction. The Most Probable Mode (MPM) is a candidate list constructed using reconstructed intra-frame prediction modes surrounding the current block. When the optimal intra-frame prediction mode for the current block is one of the intra-frame prediction modes included in the Most Probable Mode (MPM) candidate list, only the candidate index information of the Most Probable Mode (MPM) is transmitted, and the intra-frame prediction mode is encoded. The Most Probable Mode (MPM) candidate list can be used to encode the optimal luminance intra-frame prediction mode. Conversely, when encoding chrominance intra-frame prediction modes, a chrominance candidate list can be constructed using luminance intra-frame prediction modes. Within the constructed chrominance candidate list, the prediction block for the optimal chrominance intra-frame prediction mode can be generated through rate-distortion (RD) cost calculation.

[0525] The transform unit 5803 generates a transform block by transforming the residual block, which is the difference between the original block and the prediction block. The transform block is the smallest unit used in the transform and quantization process. The transform unit 5803 generates a transform block with transform coefficients by transforming the residual signal into a frequency region. Various transform techniques can be used to transform the residual signal into a frequency region, such as Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), and Karhunen-Loeve Transform (KLT), thereby transforming the residual signal into a frequency region and generating transform coefficients. To facilitate the use of transform techniques, matrix operations using basis vectors are performed. Depending on the prediction mode used when encoding the prediction block, multiple different transform techniques can be mixed during matrix operations. For example, when performing in-frame prediction, a Discrete Cosine Transform can be used in the horizontal direction and a Discrete Sine Transform in the vertical direction, depending on the prediction mode.

[0526] The quantization unit 5804 can generate a quantized transform block by quantizing the transform block. That is, the quantization unit 5804 can generate a quantized transform block with quantized transform coefficients by quantizing the transform coefficients of the transform block generated from the transform unit 5803. As a quantization method, various quantization methods can be used, such as Dead Zone Uniform Threshold Quantization (DZUTQ) or Quantization Weighted Matrix, or improved quantization methods.

[0527] Furthermore, the above description illustrates and explains the case where the image encoding apparatus includes a transform unit 5803 and a quantization unit 5804. However, the transform unit and quantization unit can also be selectively included in the image encoding apparatus. That is, the image encoding apparatus can generate a transform block by transforming the residual block but without performing a quantization process, or it can perform a quantization process without transforming the residual block into frequency coefficients, or it can even omit both the transform and quantization processes. Even when only a part of the transform unit or quantization unit is performed in the image encoding apparatus, or when all processes are not performed, the block input to the entropy encoding unit is usually referred to as a "quantization transform block".

[0528] The entropy encoding unit 5805 can encode the quantization transform block and output a bitstream. That is, the entropy encoding unit 5805 can use various encoding techniques, such as entropy encoding, to encode the coefficients of the quantization transform block output from the quantization unit 5804, and generate and output a bitstream containing additional information (e.g., prediction mode-related information, quantization coefficients, etc.) required for decoding the corresponding block in the image decoding apparatus described later. Similarly, the entropy encoding unit 2905 can use various encoding techniques, such as entropy encoding, to encode the coefficients of the quantization transform block output from the quantization unit 2904, and generate and output a bitstream containing additional information (e.g., prediction mode-related information (which may include motion information or in-frame prediction mode information determined in the prediction unit, quantization coefficients, etc.) required for decoding the corresponding block in the image decoding apparatus described later.

[0529] The inverse quantization unit 5806 can reconstruct the inverse quantization transform block by reversing the quantization techniques used in the quantization process on the quantization transform block.

[0530] The inverse transform unit 5807 reconstructs the residual block by performing an inverse transform on the inverse quantization transform block using the same method as that used in the transform process. The inverse transform can be performed by reversing the transform technique used in the transform unit.

[0531] Furthermore, the aforementioned inverse quantization unit and inverse transform unit can perform inverse quantization and inverse transform by reversing the quantization and transform methods used in the quantization unit and the transform unit. Alternatively, if only quantization is performed in the transform unit and quantization unit without transformation, only inverse quantization can be performed without inverse transform. Moreover, if neither transformation nor quantization is performed, the inverse quantization unit and inverse transform unit may not perform inverse transform and inverse quantization, or may be omitted directly from the image encoding device.

[0532] The addition unit 5808 can reconstruct the current block by performing an addition operation on the residual signal generated in the inverse transform unit and the prediction block generated by prediction.

[0533] The filtering unit 5809 performs a filtering process on the entire image after reconstructing all blocks within the current image. This can include deblocking filtering and Sample Adaptive Offset (SAO). Block filtering refers to the operation used to reduce block distortion generated during the encoding of the image in block units, while Sample Adaptive Offset (SAO) refers to the operation of minimizing the difference between the reconstructed image and the original image by performing addition or subtraction operations on the reconstructed pixels using specific values.

[0534] The memory unit 5810 stores the current block of reconstruction after adding the residual signal generated in the inverse transform unit and the prediction block generated by prediction to the loop filtering unit, and can be used to predict the next block or the next image, etc.

[0535] The subtraction unit 5811 can generate a residual block by performing a subtraction operation between the current original block and the predicted block.

[0536] Figure 60 This is a flowchart illustrating the encoding method for predictive information.

[0537] See Figure 60In step S6001, the action information for the Skip mode is encoded. In step S6002, it is determined whether the Skip mode action has occurred. When the Skip mode action is determined in step S6002, in step S6007, the MERGE candidate index information for the Skip mode is encoded, and then the flowchart ends.

[0538] If it is determined in step S6002 that the Skip mode is inactive, the prediction mode will be encoded in step S6003. In step S6003, it will be determined whether the prediction mode is an inter-frame prediction mode or an intra-frame prediction mode. If it is determined in step S6004 that the prediction mode is an inter-frame prediction mode, the action information for the Merge mode will be encoded in step S6005.

[0539] In step S6006, it is determined whether the MERGE mode is active. If the MERGE mode is active in step S6006, the process jumps to step S6007 and the MERGE candidate index information is encoded, then the flowchart ends. If the MERGE mode is inactive in step S6006, the prediction direction is encoded in step S6008. The prediction direction can be a historical direction, a future direction, or one of both.

[0540] In step S6009, it is determined whether the predicted direction is a future direction. If it is determined in step S6009 that the predicted direction is not a future direction, in step S6010, the reference image index information of the historical direction is encoded.

[0541] In step S6011, the motion vector difference (MVD) information of the historical direction is encoded. In step S6012, the motion vector prediction (MVP) information of the historical direction is encoded.

[0542] When it is determined in step S6009 that the predicted direction is a future direction or both directions, after completing step S6012, in step S6013, it will be determined whether the predicted direction is a historical direction. When it is determined in step S6013 that the predicted direction is not a historical direction, in step S6014, the reference image index information for the future direction will be encoded.

[0543] In step S6015, the motion vector difference (MVD) information for the future direction is encoded. In step S6016, the motion vector prediction (MVP) information for the future direction is encoded, and then the flowchart ends.

[0544] When the prediction mode is determined to be an in-frame prediction mode in S6004, the motion information of the Most Probable Mode (MPM) is encoded in step S6017. The Most Probable Mode (MPM) refers to a method of transmitting the index information of the candidate in-frame prediction modes when, after constructing candidate in-frame prediction modes using reconstructed in-frame prediction modes from the surrounding area of ​​the current block, the candidate in-frame prediction modes include the best in-frame prediction mode information for the current block determined in the image encoding device.

[0545] In step S6018, it is determined whether the most likely mode (MPM) is active. If it is determined that the most likely mode (MPM) is active in step S6018, in step S6019, the index information of the in-frame prediction mode of the most likely mode (MPM) candidate is encoded. If it is determined that the most likely mode (MPM) is not active in step S6018, in step S6020, the best in-frame prediction mode information of luminance (Luma) among the remaining in-frame prediction modes other than the most likely mode (MPM) candidate in-frame prediction mode is encoded.

[0546] After steps S6019 and S6020 are completed, in step S6021, the optimal in-frame prediction mode information for chroma difference is encoded, and then this flowchart ends. The unit executing steps S6008 to S6016 is referred to as motion information encoding unit 601. Furthermore, the unit executing steps S6017 to S6020 is referred to as luminance in-frame prediction mode encoding unit 602.

[0547] Figure 61 This is a flowchart illustrating the image decoding device.

[0548] See Figure 61 The image decoding device is a device for decoding images, and may include an entropy decoding unit 6101, an inverse quantization unit 6102, an inverse transform unit 6103, a prediction unit 6104, an addition unit 6105, a loop filter unit 6106, and a memory unit 6107.

[0549] Figure 61The various components illustrated are shown separately to illustrate different features and functions of the image decoding device, and do not represent that each component is composed of separate hardware or software units. That is, although the various components are listed for ease of explanation, at least two components can be combined into one component, or a component can be divided into multiple components to perform corresponding functions. The embodiments where the various components are integrated or separated as described above are included within the scope of the claims of this invention without departing from the essence of the invention.

[0550] Furthermore, some constituent elements may not be essential for performing the essential functions of this invention, but rather optional elements used to improve performance. This invention may include only the constituent parts essential for realizing the essence of the invention, excluding those merely used to improve performance, and structures including essential constituent elements other than optional elements used to improve performance are also included within the scope of the claims of this invention.

[0551] The encoding block in an image encoding device can be called the decoding block or the encoding block in an image decoding device.

[0552] The entropy decoding unit 6101 can read various information and quantization transformation coefficients required for decoding the corresponding block by parsing the bit stream transmitted from the image encoding device.

[0553] The inverse quantization unit 6102 can reconstruct the inverse quantization block with inverse quantization coefficients by reversing the quantization technique used in the quantization process on the quantization coefficients decoded in the entropy decoding unit.

[0554] The inverse transform unit 6103 reconstructs the residual block with the difference signal by performing an inverse transform on the inverse quantization transform block using the same method as that used in the transform process. The inverse transform can be performed by reversing the transform technique used in the transform unit.

[0555] The prediction unit 6104 generates prediction blocks using the prediction mode information decoded in the entropy decoding unit. At this time, the same prediction method as that performed in the prediction unit of the image encoding device can be used.

[0556] The addition unit 6105 can reconstruct the current block by performing an addition operation on the residual signal reconstructed in the inverse transform unit and the prediction block generated by prediction.

[0557] The filtering unit 6106 performs a filtering process on the entire image area after reconstructing all blocks in the current image. It may include deblocking filtering and sample adaptive offset (SAO), etc. The details are the same as those described for the loop filtering unit of the image coding device.

[0558] The memory unit 6107 stores the current block of reconstruction after adding the residual signal generated in the inverse transform unit and the prediction block generated by prediction to the loop filtering unit, and can be used to predict the next block or the next image, etc.

[0559] Figure 62 This is a schematic diagram used to explain the prediction unit of an image decoding device.

[0560] See Figure 62 When the prediction mode is in-screen prediction, the optimal in-screen prediction mode information 6201 can be determined, and a prediction block can be generated by performing in-screen prediction 6202.

[0561] When the prediction mode is inter-frame prediction, the optimal prediction mode 6203 can be determined from the skip, merge, and advanced motion vector prediction (AMVP) modes. Next, when decoding via the skip or merge mode, candidate motion information groups for the skip and merge modes can be constructed in the merge candidate exploration unit 6204. From the corresponding candidate motion information groups, the optimal motion information 6205 can be determined.

[0562] When decoding via Advanced Motion Vector Prediction (AMVP), candidate motion information groups for AMVP modes can be constructed in the AMVP candidate exploration unit 6206. Within each candidate motion information group, the transmitted motion vector prediction (MVP) information can be used to determine the optimal motion information 6207. Next, a prediction block can be generated by performing motion compensation 6208 using the optimal motion information determined in each mode.

[0563] Figure 63 This is a flowchart illustrating the method for decoding predicted information.

[0564] See Figure 63In step S6301, the action information for the Skip mode is decoded. In step S6302, it is determined whether the Skip mode action has occurred. When the Skip mode action is determined in step S6302, in step S6307, the MERGE candidate index information for the Skip mode is decoded, and then the flowchart ends.

[0565] If it is determined in step S6302 that the Skip mode is inactive, the prediction mode will be decoded in step S6303. In step S6303, it will be determined whether the prediction mode is an inter-frame prediction mode or an intra-frame prediction mode. If it is determined in step S6304 that the prediction mode is an inter-frame prediction mode, the action information of the Merge mode will be decoded in step S6305.

[0566] In step S6306, it is determined whether the MERGE mode is active. If the MERGE mode is active in step S6306, the process can jump to step S6307 and decode the MERGE candidate index information used for the MERGE mode, then end the flowchart. If the MERGE mode is not active in step S6306, the prediction direction is decoded in step S6308. The prediction direction can be a historical direction, a future direction, or one of both. In step S6309, it is determined whether the prediction direction is a future direction. If the prediction direction is not a future direction in step S6309, the reference image index information of the historical direction is decoded in step S6310.

[0567] In step S6311, the motion vector difference (MVD) information of the historical direction is decoded. In step S6312, the motion vector prediction (MVP) information of the historical direction is decoded.

[0568] When it is determined in step S6309 that the predicted direction is a future direction or both directions, after completing step S6312, in step S6313, it will be determined whether the predicted direction is a historical direction. When it is determined in step S6313 that the predicted direction is not a historical direction, in step S6314, the reference image index information of the future direction will be decoded.

[0569] In step S6315, the motion vector difference (MVD) information for the future direction is decoded. In step S6316, the motion vector prediction (MVP) information for the future direction is decoded, and then this flowchart ends.

[0570] When the prediction mode is determined to be an in-frame prediction mode in step S6304, the action information of the most likely mode (MPM) is decoded in step S6317. In step S6318, the action or non-action of the most likely mode (MPM) is determined. When the most likely mode (MPM) is determined to be in action in step S6318, the index information of the candidate in-frame prediction modes of the most likely mode (MPM) is decoded in step S6319. When the most likely mode (MPM) is determined not to be in action in step S6318, the optimal in-frame prediction mode information for luminance (Luma) among the remaining in-frame prediction modes other than the candidate in-frame prediction mode of the most likely mode (MPM) is decoded in step S6320.

[0571] After steps S6319 and S6320 are completed, in step S6321, the optimal in-frame prediction mode information for chroma is decoded, and then this flowchart ends. The unit executing steps S6308 to S6316 is called the motion information decoding unit 631. Furthermore, the unit executing steps S6317 to S6320 is called the luma in-frame prediction mode encoding unit 632.

[0572] Next, various embodiments of the present invention will be described in more detail with reference to the accompanying drawings.

[0573] Figure 64 This is a schematic diagram illustrating the prediction unit of an image encoding apparatus to which one embodiment of the present invention is applied.

[0574] When performing intra-frame prediction 6401 using the original and reconstructed information, the rate-distortion (RD) cost can be used to determine whether the decoder-side intraprediction mode (DIMD) is needed to decide the intra-frame prediction mode. Without using DIMD, the luma intra-frame prediction mode 6402 will be determined. In this case, the intra-frame prediction mode with the lowest rate-distortion (RD) cost can be selected from the intra-frame prediction modes.

[0575] Conversely, when using decoder-side intra-frame mode derivation (DIMD) mode, intra-frame prediction mode information can be omitted. Instead, the intra-frame prediction mode for luminance (Luma) is determined using the same prediction method as the image decoding device. For details on decoder-side intra-frame mode derivation (DIMD) prediction methods, please refer to [link to relevant documentation]. Figure 72 A detailed explanation will be provided.

[0576] Next, the candidate modes for the chromatic prediction mode can be determined using the luminance (Luma) intra-frame prediction mode, and the intra-frame prediction mode with the lowest rate-distortion (RD) cost among the corresponding candidate modes is determined as the chromatic prediction mode 6404. The prediction unit 6400 of the image encoding apparatus can generate the intra-frame prediction block of the current block using the determined luminance (Luma) and chromatic prediction modes.

[0577] By using raw and reconstructed information to perform inter-frame prediction 6405, the rate-distortion (RD) cost can be calculated for Skip mode, Merge mode, and Advanced Motion Vector Prediction (AMVP) mode.

[0578] In the MERGE candidate exploration unit 6406, candidate motion information groups 6406 can be constructed for the SKIP mode and the MERGE mode. Among them, the optimal motion information 6407 can be determined by using rate distortion (RD) cost in the constructed candidate motion information group.

[0579] In the Advanced Motion Vector Prediction (AMVP) candidate exploration unit 6408, a candidate motion information group 6408 for the Advanced Motion Vector Prediction (AMVP) mode can be constructed. Motion inference 6409 can be performed using the constructed candidate motion information group to determine the optimal motion information.

[0580] After determining the optimal motion information for the Skip, Merge, or Advanced Motion Vector Prediction (AMVP) modes, the applicability of regional brightness compensation can be decided. In Skip and Merge modes, regional brightness compensation can be applied directly to the predicted blocks reconstructed using the optimal candidate motion information, or it can be determined by comparing the rate-distortion (RD) cost of applying regional brightness compensation with the rate-distortion (RD) cost of not applying it. In Advanced Motion Vector Prediction (AMVP) mode, regional brightness compensation can be determined by comparing the rate-distortion (RD) cost of applying regional brightness compensation with the rate-distortion (RD) cost of not applying it.

[0581] After determining the optimal motion information (including motion vector accuracy) and whether regional brightness compensation is applicable, motion compensation 6411 can be performed using the determined information. The prediction unit 6400 of the image encoding device can generate inter-frame prediction blocks through motion compensation 6411.

[0582] Figure 65 This is a schematic diagram illustrating the prediction unit of an image decoding apparatus to which one embodiment of the present invention is applied.

[0583] When performing intra-frame prediction 6501 using reconstructed information, it can be determined whether the intra-frame prediction mode needs to be determined using the decoder-side intra-prediction mode derivation (DIMD).

[0584] At this point, without using the decoder-side intra-frame mode derivation (DIMD) mode, the intra-frame prediction mode 6502 for luminance (Luma) can be determined using the reconstruction information. However, when using the decoder-side intra-frame mode derivation (DIMD) mode, the intra-frame prediction mode for luminance (Luma) can be determined using the reconstruction information and the same prediction method as the image encoding device.

[0585] Next, the in-frame prediction mode 6504 for chromatic aberration can be determined using the reconstructed information. The prediction unit 6500 of the image decoding device can generate an in-frame prediction block for the current block using the determined in-frame prediction modes for luminance and chromatic aberration.

[0586] When performing inter-frame prediction 6505 using reconstruction information, it is necessary to determine whether the current prediction mode in the inter-frame prediction mode is the SKIP mode, the MERGE mode, or the Advanced Motion Vector Prediction (AMVP) mode.

[0587] If the current prediction mode is either the SKIP mode or the MERGE mode, candidate motion information groups for the SKIP mode and the MERGE mode can be constructed in the MERGE candidate exploration unit 6506. By reconstructing the information, the optimal motion information 6507 can be determined from the corresponding candidate motion information groups.

[0588] In the Advanced Motion Vector Prediction (AMVP) candidate exploration unit 6508, a group of candidate motion information can be constructed for the AMVP mode. By reconstructing the information, the best candidate can be selected from the corresponding group of candidate motion information, and the best motion information 6509 is determined by the reconstructed motion information.

[0589] After determining the optimal motion information for Skip, Merge, and Advanced Motion Vector Prediction (AMVP) modes, the applicability of regional brightness compensation can be determined using the reconstructed information 6510. After determining the optimal motion information (including motion vector accuracy) and the applicability of regional brightness compensation, motion compensation can be performed using the determined information 6511. The prediction unit 6500 of the image decoding device can generate inter-frame prediction blocks using motion compensation.

[0590] Figure 66 This is a flowchart illustrating an encoding method for encoding and decoding information applicable to one embodiment of the present invention.

[0591] Next, please refer to Figure 66 A method for encoding various encoding and decoding information in the entropy encoding unit of an image encoding apparatus, applicable to one embodiment of this patent, will be described in detail. Figure 66 The descriptions of steps S6601 to S6607 in the text can be compared with those of... Figure 60 The descriptions of steps S6001 to S6007 are the same.

[0592] If it is determined in step S6606 that the MERGE mode is not in operation, or after completing step S6607, the regional brightness compensation action information can be encoded in step S6608. However, if the prediction mode is SKIP or MERGE mode and regional brightness compensation is directly applied to the predicted block reconstructed through the best candidate motion information, this step can be omitted.

[0593] After completing step S6608, the motion information encoding unit 601 ( Figure 60 Steps S6008 to S6016 in the process encode the motion information.

[0594] When the prediction mode in step S6604 is an intra-frame prediction mode rather than an inter-frame prediction mode, in step S6609, the decoder-side intra-frame mode derivation (DIMD) mode action information can be encoded. In step S6610, it is determined whether the decoder-side intra-frame mode derivation (DIMD) mode is active. When the decoder-side intra-frame mode derivation (DIMD) mode is active, the additional intra-frame prediction mode information does not need to be encoded. When the decoder-side intra-frame mode derivation (DIMD) mode is not active, the luminance (Luma) intra-frame prediction mode encoding unit 602 ( Figure 60 Steps S6017 to S6020 encode the optimal in-frame prediction mode for luminance (Luma).

[0595] In step S6611, the optimal in-frame prediction mode for chromatic aberration can be encoded. Next, in step S6612, information regarding the applicability of the transformation type can be encoded.

[0596] In step S6613, it can be determined whether the transformation type is fixed. When using a fixed transformation type, this flowchart can end; however, when adaptively determining the transformation type, this flowchart can end after encoding the transformation type information for different transformation directions.

[0597] Figure 67 This is a flowchart illustrating a decoding method for encoded and decoded information applicable to one embodiment of the present invention.

[0598] Next, please refer to Figure 67 A method for decoding various encoding and decoding information in the entropy decoding unit of an image decoding apparatus, applicable to one embodiment of this patent, will be described in detail. The description of steps S6701 to S6707 can be compared with the description of... Figure 63 The descriptions of steps S6301 to S6307 are the same.

[0599] If it is determined in step S6706 that the MERGE mode is not in operation, or after completing step S6707, the regional brightness compensation action information can be decoded in step S6708. However, if the prediction mode is SKIP or MERGE mode and regional brightness compensation for the predicted block reconstructed through the best candidate motion information is applied directly, this step can be omitted.

[0600] After completing step S6708, the motion information decoding unit 631 ( Figure 63 In steps S6308 to S6316, motion information is decoded. When the prediction mode in step S6704 is the in-frame prediction mode, in step S6709, the motion information of the decoder-side intra-frame mode derivation (DIMD) mode can be decoded.

[0601] In step S6710, it can be determined whether the decoder-side intra-frame mode derivation (DIMD) mode is active. When the decoder-side intra-frame mode derivation (DIMD) mode is active, the additional intra-frame prediction mode information can be left undecoded. When the decoder-side intra-frame mode derivation (DIMD) mode is not active, the in-frame prediction mode encoding unit 632 can be used to determine whether the luminance (Luma) mode is active. Figure 63 Steps S6317 to S6320 in the process decode the in-frame prediction mode for optimal brightness (Luma).

[0602] In step S6711, the optimal in-frame prediction mode for the best chromatic aberration can be decoded. Next, in step S6712, the information on whether the transformation type is fixed or not can be decoded.

[0603] In step S6713, it can be determined whether the transformation type is fixed. When using a fixed transformation type, this flowchart can end; however, when adaptively determining the transformation type, this flowchart can end after decoding the transformation type information for different transformation directions.

[0604] Next, we will explain in detail the method of encoding / decoding motion vector difference (MVD) using motion vector difference information in the inter-frame prediction mode.

[0605] Next, the motion vector difference can refer to the vector value used to derive the final motion vector. For example, the motion vector difference of the current block can be added to the initial motion vector of the current block to derive the final motion vector of the current block.

[0606] Motion vector difference information can be used to derive the motion vector difference of the current block. The image decoding device can decode the motion vector difference information in order to derive the motion vector difference of the current block.

[0607] The inter-frame prediction mode applicable to this embodiment can be one of the following: MERGE mode, Advanced Motion Vector Prediction (AMVP) mode, or SKIP mode.

[0608] As an example, when the inter-frame prediction mode of the current block is Advanced Motion Vector Prediction (AMVP) mode, the motion vector difference can be added to the motion vector prediction (MVP) to derive the final motion vector of the current block.

[0609] As another example, when the inter-frame prediction mode for the current block is MERGE mode, the motion vector difference can be added to the initial motion vector derived through MERGE mode to derive the final motion vector for the current block. That is, the motion vector difference can be a vector value used to correct the initial motion vector derived through MERGE mode. For example, the motion vector difference can refer to the accuracy information of the initial motion vector. In MERGE mode, the initial motion vector can be derived using the MERGE candidate list for the current block.

[0610] Figure 68This is a flowchart illustrating the method of encoding / decoding motion vector difference information in inter-frame prediction mode.

[0611] exist Figure 68 In methods 6801 and 6802, the methods for encoding (steps S6011 and S6015) and decoding (steps S6311 and S6315) motion vector difference information in motion information encoding unit 601 and motion information decoding unit 631, respectively, are illustrated in detail.

[0612] As another example, in Figure 68 In methods 6801 and 6802, the actions in encoding (step S305) and decoding (step S605) of MERGE mode action information are illustrated in detail.

[0613] As an example, motion vector difference information can include information indicating whether motion vector differences exist in the current block. Information related to the presence or absence of motion vector differences can be expressed using flag values.

[0614] As an example, motion vector difference information may include at least one of the absolute values ​​of the X component or the absolute values ​​of the Y component of the motion vector difference.

[0615] In step S6803, the image encoding device can encode the absolute value information of the X component of the motion vector difference, and in step S6804, it can encode the absolute value information of the Y component. As another example, the image encoding device can first encode the absolute value information of the Y component of the motion vector difference, and then encode the absolute value information of the X component of the motion vector difference.

[0616] Conversely, in step S6807, the image decoding device can decode the absolute value information of the X component of the motion vector difference, and in step S6808, it can decode the absolute value information of the Y component. As another example, the image decoding device can first decode the absolute value information of the Y component of the motion vector difference, and then decode the absolute value information of the X component of the motion vector difference.

[0617] As another example, the image encoding / decoding apparatus can simultaneously encode / decode the absolute value information of both the X and Y components of the motion vector difference. To simultaneously encode / decode both the X and Y components of the motion vector difference, the motion vector difference information may include a group of absolute motion vector values ​​consisting of at least one of the absolute values ​​of the X component or the absolute value information of the Y component.

[0618] Furthermore, the motion vector difference information may include symbol group information consisting of one of the X component symbols and one of the Y component symbols of the motion vector difference. For example, the symbol group information may be information used to indicate one of the symbols in a symbol group consisting of one of the X component symbols and one of the Y component symbols of the motion vector difference. In this case, binary information can be assigned to different symbol groups, and the symbol group information can be expressed using an index used to indicate one of the aforementioned symbol groups.

[0619] As an example, symbol group information can be an index used to indicate a symbol group as defined in Table 2 below.

[0620] Table 2

[0621] In step S6805, the encoding device can calculate the template matching cost for different symbol groups of motion vector differences, and assign binary information or indexes to different symbol groups of motion vector differences.

[0622] Figure 69 This is a table illustrating a method for determining binary information using template matching cost when encoding motion vector difference information, according to one embodiment of the present invention.

[0623] As an example, when the absolute values ​​of the X and Y components of the motion vector difference are both non-zero, it can be calculated according to... Figure 69 In this context, 6901 assigns binary information to the motion vector difference symbol group. At this point, the template matching cost can be calculated separately for each different motion vector difference symbol group.

[0624] As another example, when the absolute value of either the X or Y component of the motion vector difference is 0, binary information can be assigned to the motion vector difference symbol group according to 6902 and 6903. 6902 can represent the case where the absolute value of the X component of the motion vector difference is 0, while 6903 can represent the case where the absolute value of the Y component of the motion vector difference is 0. In the cases described above, the template matching cost can be calculated for only the two symbol groups.

[0625] The image encoding device can efficiently determine the binary information allocated to each symbol group according to the calculated template matching cost value in ascending order under different conditions 6901, 6902, and 6903, thereby improving its encoding and decoding efficiency. In cases as shown in 6902 and 6903, the binary number of the allocated binary information is always 1 bit in different motion vector difference symbol groups. However, by using the probability of binary information with a lower template matching cost value as context-adaptive binary arithmetic coding (CABAC) probability information corresponding to each binary information in the entropy coding unit and entropy decoding unit, its encoding and decoding efficiency can be improved.

[0626] Figure 69 The template matching value and the assigned binary information are just examples; their values ​​can be changed, and the method of assigning binary information can also be changed.

[0627] Figure 70 This is a schematic diagram illustrating a template matching technique for determining binary information of motion vector difference information when an embodiment of the present invention is applied.

[0628] Next, please refer to Figure 70 The template matching technique described above is explained in detail. The template region can be determined within the reconstructed region of the current block in the current image. The template region is as follows: Figure 70 As shown, the template area can be determined within the current block's (Lh×Lw, where Lh and Lw are integers greater than 0) and the upper (Aw×Ah, where Aw and Ah are integers greater than 0) regions. In this case, the template area can be determined to have different shapes, positions, and sizes.

[0629] The region most similar to the template in the current image, explored within the motion inference exploration region of the reference image, can be used as the template region of the reference image. The error between the template regions surrounding the current block and the template regions within the motion inference exploration region can then be determined as the template matching cost. This error can be derived using methods such as the Sum of Absolute Difference (SAD) and the Sum of Square Difference (SSD). Equations 15 and 16 below describe the calculation methods for the Sum of Absolute Difference (SAD) and the Sum of Square Difference (SSD), respectively.

[0630]

Mathematical Expression 15

[0631]

Mathematical Expression 16

[0632] Where i and j represent the pixel positions, Template(i, j) represents the pixels of the template block, and PredBlk(i, j) represents the pixels of the prediction block.

[0633] Next, in steps S6806 and S6810, the binary information or index of the sign group of the determined motion vector difference can be encoded / decoded.

[0634] Next, we will explain in detail the method of determining binary information using template matching technology when encoding / decoding the optimal luminance (Luma) and chromatic aberration (Chroma) in the in-frame prediction mode.

[0635] Figure 71 This is a schematic diagram illustrating a method for allocating binary information to each candidate in the most probable mode (MPM) candidate mode list using a template matching technique applicable to the decoder-side intra-mode derivation (DIMD) mode according to an embodiment of the present invention, and a method for allocating binary information to chromatic aberration (Chroma) candidate modes.

[0636] Figure 71 Is Figure 60 The in-frame predictive mode coding unit for luminance (Luma) and Figure 63 In the case of the Most Probable Mode (MPM) operation in the intra-frame prediction mode decoding unit for luminance (Luma), an illustrative diagram is provided illustrating a method for determining the binary information of the index information based on the template matching cost when encoding the MPM index information in step S6019 and decoding the MPM index information in step S6319. The template matching technique in the intra-frame prediction mode and the template matching technique in the inter-frame prediction mode can be different.

[0637] Figure 72 This is a schematic diagram illustrating a template matching technique for a decoder-side intra-frame mode derivation (DIMD) mode applicable to one embodiment of the present invention.

[0638] Next, please refer to Figure 72 This section explains the use of template matching technology for template regions in the reconstruction area surrounding the current block in in-image prediction mode. For example... Figure 72 As shown, the template area can be determined from the reconstruction area surrounding the current block.

[0639] The template region can be of different sizes. Next, a reference region for the template region can be determined from its periphery. The reconstructed pixels of this reference region can be used to perform in-frame prediction of the template region, and the difference between the predicted value and the reconstructed value of the template region—the prediction error—is determined as the template matching cost. Various methods can be used to calculate the prediction error, such as the sum of absolute errors (SAD) and the sum of squared errors (SSD) as described above.

[0640] exist Figure 71 In 7101, when encoding the optimal brightness (Luma) intra-frame prediction mode, the template matching cost of the intra-frame prediction modes included in the most probable mode (MPM) candidate list can be calculated first.

[0641] Next, smaller binary information can be assigned to the prediction patterns within the image in ascending order of template matching cost.

[0642] For example, as shown in Table 7101, if the most likely mode (MPM) candidate list includes the vertical (Ver) mode, the mean (DC) mode, the planar mode, the horizontal (Hor) mode, and the 20th angular mode, and the template matching cost values ​​for the predicted modes within each frame are 100, 120, 80, 140, and 150, the assigned binary information can be determined as 0, 10, 110, 1110, and 1111 in ascending order of template matching cost value. The template matching cost values ​​and assigned binary information in Table 7101 are only illustrative examples; the cost values ​​can be changed, and the method of assigning binary information can also be changed.

[0643] exist Figure 71 In 7102, when encoding the best chromatic difference (Chroma) intra-frame prediction mode, the template matching cost of the intra-frame prediction mode in the chromatic difference (Chroma) candidate mode list can be calculated first.

[0644] Next, smaller binary information can be assigned to the prediction modes within the chroma image according to the template matching cost from smallest to largest.

[0645] As an intra-frame prediction mode in the list of chroma candidate modes, an intra-frame prediction mode reconstructed in the luma block can be used. In 4:2:0 format images, the chroma block is 1 / 4 the size of the luma block. Furthermore, because block segmentation of the luma and chroma blocks can be performed independently in intra-only images, as shown in 7102, the block segmentation of the luma and chroma blocks can be different.

[0646] When constructing the chroma candidate mode list, the intra-frame prediction modes reconstructed from five distinct luminance blocks (A1, A2, A3, A4, A5) can be used. Within this list, the template matching cost of the intra-frame prediction modes for blocks A1, A2, A3, A4, and A5 can be calculated, and smaller binary information is assigned according to the ascending order of the corresponding cost, as shown in Table 7102. The template matching costs and assigned binary information in Table 7102 are merely illustrative; the costs and the method of assigning binary information can be modified.

[0647] Next, we will explain in detail the method of using template regions in the reconstruction area around the current block when performing the regional brightness compensation process in the inter-picture prediction mode.

[0648] Figure 73 This is a schematic diagram illustrating a method for determining whether to use a template in the reconstruction area surrounding a predicted block based on whether a template is used in the reconstruction area surrounding the current block when performing a regional brightness compensation technique according to one embodiment of the present invention.

[0649] When performing regional brightness compensation, a compensation process can be performed to minimize the difference between the template region of the reconstructed area surrounding the predicted block pointed to by the best motion information and the template region of the reconstructed area surrounding the current block. This compensation process can be derived using linear regression analysis to minimize the least squares sum of parameters α and β. Regional brightness compensation using α and β can be performed using mathematical formula 17.

[0650]

Mathematical Expression 17

[0651] In deriving α and β, we can follow the steps as follows: Figure 73The method shown defines the template area for the reconstruction region surrounding the current block. This definition, as described above, is to design an efficient image parallel processing structure.

[0652] When deciding whether to use a template region for the reconstruction area surrounding the current block, regions exceeding the largest encoded block containing the current block can be excluded from the template region. Furthermore, a window of arbitrary size (K×L) can be set, and template regions exceeding the corresponding window size can be avoided. The size of this window can be transmitted in the parent header (such as video parameter level, sequence parameter level, image parameter level, and stripe level), or a pre-defined size can be used in the video encoding / decoding device.

[0653] Furthermore, the use of template regions should not be determined by infringing upon the state of segmentation boundaries such as stripes and parallel blocks.

[0654] In the template area for the reconstruction area around the predicted block, you can also use only the available template area around the current block.

[0655] Figure 73 This is a schematic diagram illustrating a method for determining whether to use a template in the reconstruction area surrounding a predicted block based on whether a template is used in the reconstruction area surrounding the current block when performing a regional brightness compensation technique according to one embodiment of the present invention.

[0656] Figure 73 7301 in the diagram is an example illustrating the situation where the template area of ​​the upper region of the predicted block cannot be used because the upper region of the reconstruction area surrounding the current block cannot be used as a template area. Figure 73 7302 in the diagram is an example illustrating the state where the template area of ​​the left side of the predicted block cannot be used because the left side of the reconstruction area surrounding the current block cannot be used as a template area. Figure 73 7303 in the diagram is an example illustrating the situation where the template areas of the upper and left sides of the current block cannot be used because the upper and left sides of the reconstruction area surrounding the current block cannot be used as template areas, which in turn makes it impossible to use the template areas of the upper and left sides of the predicted block.

[0657] In the case illustrated in 7303, the local brightness compensation process may not be performed. In the case described above, in Figure 66 In step S6608, the area brightness compensation action information may not be encoded, and in Figure 67 In step S6708, the area brightness compensation action information may not need to be decoded. In this case, the area brightness compensation process action information can be determined as false.

[0658] Next, we will explain in detail the method of using template matching to determine the binary information of the transformation type when encoding / decoding the transformation type for the purpose of transforming the image into block units.

[0659] Figure 74 It is a transformation unit of an image encoding apparatus to which one embodiment of the present invention is applied.

[0660] The transformation unit 7400 of the image encoding apparatus can derive the transformation coefficients by transforming the residual information into the frequency domain. When performing residual information transformation, the transformation type can be fixedly determined 7410 or adaptively determined 7402.

[0661] With a fixed transform type, the transform coefficients can be obtained by performing a horizontal transform 7404 on the residual signal using a fixed transform type, followed by a vertical transform 7405. In this case, one of several different transform types, such as Discrete Cosine Transform-2 (DCT-2), can be used as the fixed transform type.

[0662] Conversely, in the case of adaptive transformation type determination, the transformation type can be adaptively determined from a variety of different transformation types. For example, one of the following transformation types can be adaptively determined: Discrete Cosine Transform-2 (DCT-2), Discrete Cosine Transform-5 (DCT-5), Discrete Cosine Transform-8 (DCT-8), Discrete Sine Transform-1 (DST-1), and Discrete Sine Transform (DST-7). In this case, the transformation type 7403 can be determined in different transformation directions. By performing horizontal transformation 7404 and vertical transformation 7405 using the determined transformation types in different transformation directions, the transformation coefficients can be derived. In this case, the transformation types in different directions can be transmitted in block units, or they can be preset and used in the upper-level header, taking into account the size and shape of the transformation block.

[0663] Figure 75 It is the inverse transformation unit of an image encoding / decoding apparatus applicable to one embodiment of the present invention.

[0664] In the inverse transform unit 7500 of the image encoding / decoding apparatus, the reconstructed residual coefficients can be derived by transforming the inverse quantization coefficients into the pixel domain. When performing the inverse transform on the inverse quantization coefficients, the inverse transform type can be fixedly determined 7410 or adaptively determined 7502.

[0665] If a transformation is performed in the transformation unit 7400 of the image encoding apparatus using a fixed transformation type, then an inverse transformation can also be performed in the inverse transformation unit 7500 of the image encoding / decoding apparatus using a fixed inverse transformation type. As the inverse transformation type, the reconstruction residual coefficients can be derived by performing a vertical inverse transformation 7504 and a horizontal inverse transformation 7505, both of which are determined in the transformation unit.

[0666] When the transformation type is adaptively determined, the reconstruction residual coefficients can be derived by performing the vertical inverse transformation 7504 and the horizontal inverse transformation 7505 of the transformation type determined in the transformation section for different transformation directions.

[0667] Figure 76 This is a schematic diagram illustrating a method using template matching technology to determine binary information of the transformation type in the transformation / inverse transformation unit of an image encoding / decoding apparatus, according to one embodiment of the present invention.

[0668] Next, please refer to Figure 76 The method for determining binary information when encoding / decoding transformation type information in different transformation directions in the case of adaptively determining the transformation type in different transformation directions in the transformation unit, and the method for determining the applicability of template regions from the reconstructed region around the current block when applying template matching technology in the process of determining binary information, are explained in detail.

[0669] As illustrated in example 7604, five transformation type groups are defined, and the horizontal / vertical transformation type is determined for each transformation type group. In this case, the template matching cost can be calculated for each transformation group using template matching techniques.

[0670] In template matching techniques within the transform unit, to calculate the template matching cost, it may only be necessary to use the template region of the reconstructed area surrounding the current block. In examples 7601-7603, the reconstructed pixel values ​​of the left and top rows of the current block can be calculated using various transform type groups. By transforming / quantizing the residual values ​​corresponding to the reconstructed area and then adding the predicted values ​​after inverse quantization / inverse transform, the reconstructed pixel values ​​of the reconstructed area can be obtained. The template matching cost can be calculated using the reconstructed area and the template region using the following mathematical formula 18.

[0671]

Mathematical Expression 18

[0672] In equation 18, TempDist represents the template matching cost. Here, i represents the coordinates of the left template region, and j represents the coordinates of the upper template region. Width and Height represent the horizontal and vertical lengths of the current block, respectively.

[0673] Using mathematical formula 5, the template matching cost can be calculated for different transformation type groups, and smaller binary information can be allocated according to the order of the corresponding cost values ​​from smallest to largest. The transformation types, template matching costs, and allocated binary information for horizontal / vertical transformations in Table 7604 are only illustrative examples. The transformation types can be changed to various combinations and their costs can be changed. Furthermore, the method of allocating binary information can also be changed.

[0674] When calculating the template matching cost, the template region for the reconstructed area surrounding the current block can be limited according to examples such as 7601, 7602, and 7603. Regions exceeding the largest coded block containing the current block can be excluded from the template region. Furthermore, a window of arbitrary size (K×L) can be set, and template regions exceeding the corresponding window size can be excluded. The size of this window can be transmitted in the upper-level header or a pre-defined size can be used in the image encoding / decoding device.

[0675] Furthermore, it is impossible to determine the size of the segmentation boundaries that exceed those determined in higher-level headers such as stripes and parallel blocks.

[0676] Figure 76 Example 7601 illustrates the situation where the upper region of the current block's surrounding reconstruction area cannot be used as a template region. Figure 76 Example 7602 illustrates the situation where the left-hand area in the reconstruction area surrounding the current block cannot be used as a template area. Figure 76 Example 7603 illustrates the situation where the upper and left sides of the current block's surrounding reconstruction area cannot be used as template areas.

[0677] In the example of 7603, the transformation type can be used consistently. Therefore, in Figure 66 In step S6612, the information regarding the applicability of the transformation type can be omitted, and the transformation type can be fixed. Figure 67 In step S6712, the information on whether the transformation type is fixed or not can be decoded instead of using the transformation type.

[0678] Figure 77 This is a flowchart illustrating an image decoding method applicable to one embodiment of the present invention.

[0679] See Figure 77In step S7701, the image decoding device can derive the initial motion vector of the current block.

[0680] The initial motion vector can be derived using the merge candidate list of the current block. For example, the initial motion vector can be derived using candidate motion information derived from the merge pattern. The merge candidate list can be derived using at least one of the motion information of spatial candidate blocks and temporal candidate blocks of the current block.

[0681] Next, in step S7702, the image decoding device can decode the motion vector difference information of the initial motion vector derived in step S7701. In step S7703, the decoded motion vector difference information can be used to derive the motion vector difference of the initial motion vector. Next, in step S7704, the image decoding device can add the initial motion vector derived in step S7701 and the motion vector difference derived in step S7703 to derive the final motion vector of the current block.

[0682] The motion vector difference information may include a group of symbols consisting of one of the X-component symbols and one of the Y-component symbols of the motion vector difference. Furthermore, the motion vector difference information may include at least one of the absolute values ​​of the X-component or the absolute values ​​of the Y-component of the motion vector difference.

[0683] At this point, the symbol group information can indicate a specific symbol in a symbol group consisting of either the X component symbol or the Y component symbol, and can be expressed in the form of an index.

[0684] Figure 78 This is a flowchart illustrating an image encoding method applicable to one embodiment of the present invention.

[0685] See Figure 78 In step S7801, the image decoding device can determine the final motion vector of the current block.

[0686] At this point, the image encoding device can derive the final motion vector of the current block through methods such as rate-distortion (RD) cost comparison.

[0687] Next, in step S7802, the image encoding device can determine the motion vector difference of the final motion vector determined in step S7801. The image encoding device can encode the motion vector difference information of the motion vector difference determined in step S7801.

[0688] The motion vector difference can be added to the initial motion vector of the current block in the image decoding device, thereby being used to derive the final motion vector determined in step S7801.

[0689] The motion vector difference information may include a group of symbols consisting of one of the X-component symbols and one of the Y-component symbols of the motion vector difference. Furthermore, the motion vector difference information may include at least one of the absolute values ​​of the X-component or the absolute values ​​of the Y-component of the motion vector difference.

[0690] At this point, the symbol group information can indicate a specific symbol in a symbol group consisting of either the X component symbol or the Y component symbol, and can be expressed in the form of an index.

[0691] pass Figure 78 The bitstream (or encoded data) generated by the image encoding method can be stored on a computer-readable recording medium. Furthermore, the bitstream may also include initial motion vector information and motion vector difference information between the initial motion vectors.

[0692] The above embodiments can be executed in the same or corresponding methods in the image encoding device and the image decoding device.

[0693] Images can be encoded / decoded using at least one or a combination of at least one of the above embodiments.

[0694] The order in which the above embodiments are applied may be different in the image encoding apparatus and the image decoding apparatus, and the order in which the above embodiments are applied may be the same in the image encoding apparatus and the image decoding apparatus.

[0695] The above embodiments can be performed separately for the luminance and chromatic difference signals, or the above embodiments can be performed in the same way for the luminance and chromatic difference signals.

[0696] The exemplary methods in this disclosure are described as a sequence of actions for clarity of explanation, but this is not intended to limit the order in which the steps are executed. The steps may be executed simultaneously or in different orders if necessary. To implement the methods in this disclosure, additional steps may be added to the example steps, or only the remaining steps may be included, or additional steps may be added after excluding a portion of the steps.

[0697] The various embodiments described herein are not a list of all possible combinations, but are merely illustrative of representative forms of the disclosure. The matters described in the various embodiments may be applied independently or in combination of two or more.

[0698] Furthermore, the various embodiments described in this disclosure can be implemented using hardware, firmware, software, or a combination thereof. When implemented in hardware, they can be implemented using one or more application-specific integrated circuits (ACICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, and microprocessors.

[0699] The scope of this disclosure includes software or device-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) that are executable on a device or computer and are applicable to actions in methods applicable to various embodiments, as well as a device or computer-executable non-transitory computer-readable medium storing the software or instructions as described above.

[0700] Industrial availability This invention can be used to encode / decode images.

Claims

1. An image decoding method, comprising the following steps: The first prediction block is generated by performing inter-screen predictions on the current block; A second prediction block is generated by performing in-screen prediction on the current block; The final prediction block is generated by performing a weighted sum on the first prediction block and the second prediction block; as well as A reconstructed block is generated based on the final predicted block. In this configuration, the first prediction block is predicted in merge mode, and the second prediction block is predicted using a fixed in-frame prediction mode. The weighted sum is performed based on the weighted values. The weighting value is determined based on the prediction patterns of the surrounding blocks of the current block. The fixed in-frame prediction mode is a planar mode.

2. An image coding method, comprising the following steps: The first prediction block is generated by performing inter-screen predictions on the current block; A second prediction block is generated by performing in-frame prediction on the current block; and The final prediction block is generated by performing a weighted sum on the first prediction block and the second prediction block. in, The first prediction block is predicted in merge mode, and the second prediction block is predicted using a fixed in-frame prediction mode. The weighted sum is performed based on the weighted values. The weighting value is determined based on the prediction patterns of the surrounding blocks of the current block. The fixed in-frame prediction mode is a planar mode.

3. A method for transmitting a bit stream, the method comprising the following steps: Transmit the bitstream generated by the image encoding method. in, The image encoding method includes the following steps: The first prediction block is generated by performing inter-screen predictions on the current block; A second prediction block is generated by performing in-screen prediction on the current block; as well as The final prediction block is generated by performing a weighted sum on the first prediction block and the second prediction block. In this configuration, the first prediction block is predicted in merge mode, and the second prediction block is predicted using a fixed in-frame prediction mode. The weighted sum is performed based on the weighted values. The weighting value is determined based on the prediction patterns of the surrounding blocks of the current block. The fixed in-frame prediction mode is a planar mode.

Images