Video encoding and decoding method and apparatus using improved intra prediction structure

Abstract

The present invention relates to encoding and decoding of digital video, and relates to a method for encoding and decoding of digital video, a method for recording data thereof, and components, devices and systems for implementing the method. The image decoding method according to an embodiment of the present invention can be characterized by comprising: a step of obtaining information about a current block from a bitstream; a step of determining a template located in a neighboring region of the current block; a step of deriving a first reference vector of the current block using the template; a step of modifying the first reference vector to derive a modified first reference vector; a step of generating a prediction block for the current block using a region pointed to by the modified first reference vector; and a step of restoring the current block using the prediction block, wherein the modification of the first reference vector is achieved by template matching within a predetermined range around the first reference vector.

Description

Technical Field

[0001] This invention relates to the field of digital video encoding and decoding, and to a digital video encoding and decoding method, a method for recording its data, and components, apparatus and systems for implementing the method.

[0002] This invention may be in the same technical field as at least one known digital video compression technology standard (e.g., MPEG-2, MPEG-4 Video, H.263, H.264 / AVC, H.265 / HEVC, H.266 / VVC, VC-1, AV1, QuickTime, VP-9, VP-10, and Motion JPEG), or may be used to improve the inherent efficiency of such standards, or may be used to improve or replace such standards. Background Technology

[0003] Digital video encoding and decoding are widely used in various digital video applications. For example, digital television broadcasting, image transmission via communication networks, video calls, video conversations, video chats, video content recorded and played using optical discs (including VCDs (video compact discs) / DVDs (digital versatile discs) / Blu-ray discs), the entire process of video content production, editing, collection, and distribution, as well as video shooting devices and cameras used in video shooting and recording activities (including personal, commercial, industrial, and security monitoring purposes), all rely on video encoding and decoding technologies.

[0004] Therefore, physical devices that can be called digital video encoders and decoders can constitute a part of a wide range of devices related to the generation, recording, and playback of digital video, including digital television, digital broadcasting systems, wireless broadcasting systems, laptops, desktop computers, tablets, e-book readers, digital cameras, digital video recording devices, digital multimedia playback devices, video game devices / terminals / control consoles, mobile phones (including smartphones) with multimedia playback capabilities, equipment for video conferencing, and others.

[0005] The aforementioned digital video encoder and decoder can be implemented using digital video compression standards widely used and understood by those skilled in the art. These digital video compression standards may include at least one known compression standard, such as MPEG-2, MPEG-4 Video, H.263, H.264 / AVC, H.265 / HEVC, H.266 / VVC, VC-1, AV1, QuickTime, VP-9, VP-10, and Motion JPEG.

[0006] Video encoders and decoders can encode or decode digital video information more efficiently by conforming to the aforementioned standards, or by improving or modifying them. Attempts to modify these standards may also lead to new standards. A well-known example is the Enhanced Compression Model (ECM), developed by the Joint Video Experts Team (JVET), a joint international standards organization comprised of ISO, IEC, and ITU-T, aimed at improving upon and replacing the existing H.266 / VVC standard. Summary of the Invention

[0007] The problem the invention aims to solve

[0008] As is well known, digital video, in its uncompressed state, requires a large amount of information to describe its content. Therefore, recording or transmitting this information in its raw form can be inefficient. Consequently, digital video needs to be compressed using various methods before recording or transmission. These compression methods include lossy coding and lossless coding. Lossy coding achieves high compression performance by sacrificing some image quality, while lossless coding prevents image quality degradation by sacrificing some compression performance. Regardless of the coding method used, to meet the demand for high-quality digital video within limited storage capacity and communication bandwidth, it is necessary to develop techniques that can achieve high compression rates while minimizing the sacrifice of image quality.

[0009] As mentioned above, the encoding process for compression requires various operations, such as: spatial segmentation of digital video, segmentation and / or processing of color channels, removal of spatial redundancy, removal of temporal redundancy, tracking of motion vectors within the video, encoding of differential images, quantization, coefficient scan, run-length coding, entropy coding, and a series of processes including loop filtering. These encoding operations typically consume computational resources and require a certain amount of time to complete. Similarly, the decoding operations that perform the above encoding operations also require certain computational resources and time. The main goal of video encoding and decoding technologies is to ensure that the aforementioned resource consumption and time consumption do not affect the production, recording, distribution, and viewing of digital video.

[0010] Therefore, the present invention addresses the aforementioned technical problems in the field of video encoding and decoding by providing a new technology that can function in at least one of the following aspects: improving encoding efficiency, improving decoding efficiency, improving video quality, reducing computational load, reducing software size, reducing hardware size, and improving other performance related to encoding and decoding.

[0011] Problem Solving Methods

[0012] This invention provides a video coding and decoding method and apparatus utilizing an improved intra-frame prediction structure, applicable to the aforementioned fields of video coding and decoding.

[0013] To solve the above-mentioned technical problems, an image decoding method according to an embodiment of the present invention may include the following steps: obtaining information about a current block from a bitstream; determining a template located in a region adjacent to the current block; deriving a first reference vector of the current block using the template; correcting the first reference vector to derive a corrected first reference vector; generating a prediction block for the current block using the region pointed to by the corrected first reference vector; and recovering the current block using the prediction block, wherein the correction of the first reference vector is achieved by template matching within a predetermined range around the first reference vector.

[0014] The above method may further include: the step of obtaining a second reference vector from the region pointed to by the modified first reference vector; the step of modifying the second reference vector to derive a modified second reference vector; and the step of generating a prediction block for the current block using the region pointed to by the modified second reference vector.

[0015] The feature could be that the correction for the first reference vector and the correction for the second reference vector are performed using different correction sample ranges.

[0016] The above method can be characterized by using a 1×1 sample range for the correction of the first reference vector and a 3×3 sample range for the correction of the second reference vector.

[0017] The step of obtaining the second reference vector can be characterized by the following steps: when there are multiple reference vectors in the region pointed to by the modified first reference vector, template matching is performed on each of the multiple reference vectors, and the results are compared with each other to obtain the second reference vector.

[0018] The above method may include, further, the step of: obtaining information from the bit stream indicating whether the image is screen content; and, based on the information indicating whether the image is screen content, determining whether the correction to the first reference vector is performed in integer pixel units or in sub-pixel units.

[0019] The above method may be characterized in that, when the image is screen content, the correction of the first reference vector is performed in integer pixel units; when the image is not screen content, the correction of the first reference vector is performed in at least one of integer pixel units and sub-pixel units.

[0020] The feature may be that the above template matching can be performed using at least one of a plurality of calculation methods for calculating the differences between templates for the above template matching, the plurality of calculation methods including: simple difference method, calculation method of adding the absolute values ​​of the differences between pixels (Sum of Absolute Differences, SAD), calculation method based on difference using transformation matrix (Sum of Absolute Transformed Differences, SATD), mean absolute difference (MAD), and sum of squared differences (SSD).

[0021] The step of deriving the first reference vector of the current block using the above template may include: configuring a candidate list of reference vectors for deriving the first reference vector; and removing at least one reference vector from the candidate list of reference vectors by comparing the reference vectors.

[0022] The step of deriving the first reference vector of the current block using the above template may include: configuring a candidate list of reference vectors for deriving the first reference vector; reading information from the bit stream for selecting at least one of the candidate list of reference vectors; and deriving the first reference vector based on the information for selecting at least one of the candidate list of reference vectors.

[0023] As a result of correcting the first reference vector, the difference vector can be characterized by a combination of direction information and magnitude information, wherein the direction information indicates one of a finite number of directions and the magnitude information indicates one of a finite number of magnitude values.

[0024] The above method can be characterized in that the search range for the template matching is set by referring to the reference vector of the surrounding blocks of the current block, wherein the surrounding blocks include at least one of the left block and the top block of the current block.

[0025] The above method may be characterized by comparing at least one of the width, height, area, or number of pixels in the current block with a predetermined threshold, and determining the applicability of the template matching prediction technique and at least one of the applicable methods based on the comparison result.

[0026] The above method may include the following steps: obtaining a first reference vector from the region pointed to by the modified first reference vector; modifying the first reference vector to update it; and generating a prediction block for the current block using the region pointed to by the first reference vector, wherein the step of updating the first reference vector is recursively performed multiple times, and at least one of the recursive steps involves a different method for modifying the reference vector.

[0027] To solve the above-mentioned technical problems, the image coding method according to an embodiment of the present invention may include the following steps: determining a template located in an adjacent region of a current block; deriving a first reference vector of the current block using the template; correcting the first reference vector to derive a corrected first reference vector; generating a prediction block for the current block using the region pointed to by the corrected first reference vector; generating a residual signal using the current block and the prediction block; and encoding the residual signal to generate a bitstream, wherein the correction of the first reference vector is achieved by template matching within a predetermined range around the first reference vector.

[0028] The above method may include the following steps: obtaining a second reference vector from the region pointed to by the modified first reference vector; modifying the second reference vector to derive a modified second reference vector; and generating a prediction block for the current block using the region pointed to by the modified second reference vector.

[0029] The above method may include, further, a step of determining whether the current encoded object image is screen content; and a step of including information indicating the determination result in the bitstream, wherein, depending on whether it is screen content, it is determined whether the correction for the first reference vector is performed in integer pixel units or in sub-pixel units.

[0030] The above method may include, further comprising: a step of determining a calculation method for the template matching; and a step of including information indicating the determined calculation method in the bitstream, wherein the calculation method includes at least one of: a simple difference method, a calculation method that adds the absolute values ​​of the differences between pixels (Sum of Absolute Differences, SAD), a calculation method based on the difference using the transformation matrix (Sum of Absolute Transformed Differences, SATD), a mean absolute difference (MAD), and a sum of squared differences (SSD).

[0031] The above method may further include: the step of generating a difference vector representing the difference between the first reference vector and the modified first reference vector; and the step of including the difference vector in the bit stream, wherein the difference vector is represented by a combination of direction information and magnitude information, the direction information indicating one of a finite number of directions, and the magnitude information indicating one of a finite number of magnitude values.

[0032] The step of deriving the first reference vector of the current block using the above template may include: configuring a reference vector candidate list for deriving the first reference vector; calculating the coding cost of the current block based on a predicted block obtained from at least one reference vector included in the reference vector candidate list; selecting at least one reference vector candidate from the candidate list based on the calculation of the coding cost to derive the first reference vector; and generating a bitstream from the candidate list containing relevant information for identifying the selected candidate.

[0033] The above method may include the following steps: obtaining a first reference vector from the region pointed to by the modified first reference vector; modifying the first reference vector to update it; and generating a prediction block for the current block using the region pointed to by the first reference vector, wherein the step of updating the first reference vector is recursively performed multiple times, and at least one of the recursive steps involves a different method for modifying the reference vector.

[0034] To solve the above-mentioned technical problems, an image decoding apparatus according to an embodiment of the present invention may include: a processor, a memory connected to the processor, a receiving unit for receiving bitstreams, and an image decoding unit, wherein the image decoding unit is configured to: obtain information about a current block from the bitstream; determine a template located in a region adjacent to the current block; derive a first reference vector of the current block using the template; correct the first reference vector to derive a corrected first reference vector; generate a prediction block for the current block using the region pointed to by the corrected first reference vector; and recover the current block using the prediction block, wherein the correction of the first reference vector is achieved by template matching within a predetermined range around the first reference vector.

[0035] The "reference vector" used in this invention includes the concept of block vector in intra-frame prediction and motion vector in inter-frame prediction. The methods for deriving and correcting reference vectors provided in this invention can be applied to both block vectors and motion vectors.

[0036] Invention Effects

[0037] According to the present invention, the complexity of template matching for intra-frame prediction in video encoders and decoders is reduced, thereby achieving efficient and high-speed video encoding and decoding.

[0038] According to the present invention, by employing a recursive reference vector configuration with a chain-like reference vector structure and a differentiated correction method, prediction accuracy can be significantly improved compared to existing template matching methods. In particular, in the various reference vector corrections from the first to the Nth iteration, by applying different correction sample ranges and calculation methods according to the number of iterations, optimized predictions can be achieved based on the local characteristics and complexity of the image. This ultimately leads to improved coding efficiency and a reduction in bitstream size. Attached Figure Description

[0039] Figure 1 This is a conceptual diagram of a video communication system according to an embodiment of the present invention.

[0040] Figure 2 This is a conceptual diagram of the arrangement of encoders and decoders in a real-time video streaming environment according to an embodiment of the present invention.

[0041] Figure 3 This is a conceptual diagram of the functional unit of a video decoder according to an embodiment of the present invention.

[0042] Figure 4This is a conceptual diagram of the functional unit of a video encoder according to an embodiment of the present invention.

[0043] Figure 5 This is a conceptual diagram of a frame type according to an embodiment of the present invention.

[0044] Figure 6 This is a conceptual diagram illustrating the structure of a video encoder according to other embodiments of the present invention.

[0045] Figure 7 This is a conceptual diagram illustrating a template matching prediction method according to one embodiment.

[0046] Figure 8 This is a conceptual diagram of the storage and use of block vectors in a template matching prediction method according to one embodiment.

[0047] Figure 9 This is an example diagram illustrating the setting of the block vector search range according to an embodiment of the present invention.

[0048] Figure 10 This is an example diagram of a modified block vector determination method according to an embodiment of the present invention.

[0049] Figure 11 This is an example diagram of a second block vector template matching method according to an embodiment of the present invention.

[0050] Figure 12 This is an example diagram of multiple block vectors according to an embodiment of the present invention. Detailed Implementation

[0051] This invention can be modified and has various embodiments, with specific embodiments illustrated in the accompanying drawings and described in detail. However, this is not intended to limit the invention to specific implementations, but rather to encompass all modifications, equivalents, and substitutions that fall within the scope of the invention's ideas and techniques.

[0052] While terms such as "first" and "second" may be used to describe various components, the components described above should not be limited by these terms. The purpose of these terms is to distinguish one component from another. For example, a first component may be named a second component without departing from the scope of the invention, and similarly, a second component may be named a first component. The term "and / or" includes a combination of or one of several related described items and is not exclusive unless otherwise stated. The items listed in this specification are merely illustrative descriptions intended to facilitate understanding of the concepts and possible implementations of this specification, and are therefore not intended to limit the scope of the embodiments of the invention.

[0053] In this specification, "A or B" can mean "A only", "B only", or "A and B". In other words, "A or B" in this specification can also be interpreted as "A and / or B". For example, "A, B or C" in this specification can mean "A only", "B only", "C only", or "any combination of A, B and C".

[0054] The forward slash ( / ) or comma used in this specification can mean "and / or". For example, "A / B" can mean "A and / or B". Accordingly, "A / B" can mean "A only", "B only", or "A and B". For example, "A, B, C" can mean "A, B, or C".

[0055] In this specification, "at least one of A and B" can mean "A only", "B only", or "A and B". Furthermore, the expressions "at least one of A or B" or "at least one of A and / or B" in this specification can be interpreted as "at least one of A and B".

[0056] Additionally, in this specification, "at least one of A, B and C" can mean "A only", "B only", "C only" or "any combination of all A, B and C". Furthermore, "at least one of A, B or C" or "at least one of A, B and / or C" can mean "at least one of A, B and C".

[0057] When a component is described as "connected" or "accessed" to another component, it should be understood that it may be directly connected to or accessed by another component, or there may be other components in between. Conversely, when a component is described as "directly connected" or "directly accessed" to another component, it should be understood that there are no other components in between.

[0058] The terminology used in this specification is illustrative of particular embodiments only and is not intended to limit the invention. Where no singular is explicitly indicated in the context, the singular designation includes the meaning of the plural. In this specification, terms such as "comprising" or "having" indicate the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, without precluding the presence or additional possibility of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0059] Unless otherwise specified, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms that are generally used identically to those defined in dictionaries have the same meaning as in the context of the relevant art, and unless explicitly defined, do not have an ideal or excessive meaning in this specification.

[0060] In describing the present invention in this specification, embodiments may be described or illustrated in the form of unit blocks that perform the described functions or functionalities. In this specification, the aforementioned blocks may represent one or more devices, units, modules, components, etc. The aforementioned blocks may be implemented in hardware using one or more logic gates, integrated circuits, processors, controllers, memories, electronic components, or information processing hardware methods including but not limited to these. Alternatively, the aforementioned blocks may be implemented in software using application software, operating system software, firmware, or information processing software methods including but not limited to these. A block may be divided into multiple blocks performing the same function, and conversely, the function of multiple blocks may be performed by a single block. The aforementioned blocks may also be implemented as physically separated or combined according to any standard. The aforementioned blocks may also be implemented to operate in an environment where their physical location is not specified and they are isolated from each other via communication networks, the Internet, cloud services, or communication methods including but not limited to these. All the above implementation methods fall within the scope of various embodiments that can be adopted by those skilled in the art of information and communication technology to achieve the same technical idea. Therefore, any specific implementation method should be interpreted as falling within the scope of the inventive technical idea of ​​this specification.

[0061] The preferred embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. To facilitate a comprehensive understanding of the invention, the same components will be denoted by the same reference numerals, and repeated descriptions of the same components will be omitted. Furthermore, the various embodiments are not mutually exclusive, and some embodiments can be combined with one or more other embodiments to form new embodiments.

[0062] Digital video codec

[0063] Figure 1This is a conceptual diagram of a video communication system according to an embodiment of the present invention. The video communication system 100 described above can be configured to include at least two terminals 110 and 120 interconnected via a network 105.

[0064] In one embodiment of the present invention, the above Figure 1 This can be represented as a block diagram for configuring a one-way video communication network. To transmit video data 111 via network 105, the first terminal 110 can encode the video data. The second terminal 120 can be configured to receive the encoded video data 121 via the network, decode it, and display it.

[0065] In another embodiment of the present invention, the above Figure 1 This can be represented as a block diagram for configuring a two-way video communication network. To achieve the aforementioned two-way video communication, each of the terminals 110 and 120 can be configured to encode its own acquired video data to transmit the video 112 and 122 to other terminals via the network. Each terminal can also be configured to receive video data sent via the network by other terminals 113 and 123, decode it, and display the decoded video data.

[0066] Figure 1 The terminals 110 and 120 shown above can be devices such as server computers, personal computers, laptops, and smartphones, but are not limited thereto, according to the embodiments. This invention is applicable to any environment where a one-way or two-way video communication network can be established, and it should be understood that the network 105 described above can be configured to transmit encoded video data between the terminals 110 and 120 by any means.

[0067] In one embodiment of the present invention, the network 105 may refer to a wired or wireless communication network. According to this embodiment, the network can be configured to communicate information using any communication standard, including packet-based communication. Packet communication can be understood to include, for example, TCP or UDP packets.

[0068] However, in another embodiment of the present invention, the network 105 described above can also be understood to include the process of transmitting information using a reading recording medium. In this case, the configuration of the network is not limited to communication media, but should be understood to include the process of temporary storage and physical transmission in hard disks, solid-state drives (SSDs), flash memory, CDs (compact discs), DVDs (digital versatile discs), Blu-ray discs, and other mechanical, electronic, or optical recording media.

[0069] Furthermore, regardless of the information communication or transmission means used, as long as its structure supports the transmission and decoding of video data in an encoded state, it can be considered to fall within the scope of the embodiments of this invention. Therefore, apart from the examples listed above, all known or future information communication or transmission means can be included within the scope of this invention.

[0070] Figure 2 This is a conceptual diagram of the arrangement of encoders and decoders in a real-time video streaming environment according to an embodiment of the present invention. Figure 2 The streaming media system 200 shown can be applied to video data communication networks, including, for example, digital broadcasting, video telephony, and video conferencing. However, even when information transmission is involved via recording media, the same or similar technical structures as described above can be applied.

[0071] According to one embodiment of the present invention, the streaming media system described above may include a video source 210 that generates the video stream. The video source may include a digital video capture unit 212 for capturing uncompressed raw video; for example, the digital video capture unit may be configured as a digital camera or other device. The raw video stream 215 may have a large capacity and therefore can be compressed by combining it with or connecting it to a video encoder 217 connected to the video source.

[0072] The encoder 217 described above can be configured as a unit including hardware, software, or a combination of both, for implementing an image encoding method and / or its implementation method according to an embodiment of the present invention.

[0073] The encoder 217 described above can output an encoded bitstream 219 with a reduced capacity compared to the original video stream. The bitstream 219 can be provided in real time for communication via a relay device (e.g., streaming media server 220) and / or stored on the recording medium 225 of the streaming media server 220 for later use.

[0074] The aforementioned streaming media system 200 may include at least one streaming media client 230, 240 that accesses the aforementioned streaming media server 220 to receive in real time or acquire later the aforementioned encoded bitstream 229. The aforementioned streaming media client may include a video decoder 232, which acquires the aforementioned encoded bitstream 229 (which may also be regarded as a copy of the bitstream 219 received by the aforementioned streaming media server), decodes the aforementioned bitstream 229, and outputs the resulting video data in a form that can be displayed on a display 235 or other visual, auditory, or other sensory display means.

[0075] As mentioned above, the encoding and decoding functions of video data are collectively referred to as a coder-decoder system, or video codec.

[0076] Figure 3 This is a conceptual diagram of the functional unit of a video decoder according to an embodiment of the present invention. Figure 3 As shown, the receiving unit 310 can receive at least one encoded video data to be decoded by the decoder 305. In one embodiment of the present invention, each received encoded video data may be independent, and the decoding process of each independent video data may be independent of the decoding process of other video data. The encoded video data can be received to the receiving unit 310 via a hardware or software connection 315 with a storage device. As described above, the storage device may refer to a streaming media server located at the other end of a communication network, or it may be a physical storage medium, but is not limited thereto.

[0077] The receiving unit 310 can receive the encoded video data and other accompanying data, such as encoded audio data or other auxiliary data, and each of the above data can be separated from the video data and provided to other appropriate processing function units 312 other than the video decoder.

[0078] When receiving the aforementioned video data via a communication network, a buffer memory 320 can be connected between the receiving unit 310 and the decoder 305 to minimize latency and interruptions caused by the network environment. The buffer memory 320 can be a computer-readable recording medium used to temporarily store the received video data and stably provide it to the parser 330 corresponding to the input of the decoder 305. However, if the bandwidth of the communication network is sufficient, or if the video data is read from a local and not physically isolated recording medium, or if communication latency is not expected in other environments, the buffer memory may not be necessary.

[0079] The video decoder 305 may include the parser 330 as its input for parsing the encoded video data. The parser can separate (parsing) various types of information stored in bitstream form within the encoded video data according to predetermined rules, and, when necessary, perform entropy decoding 335 on the entropy-coded video data, thereby reconstructing symbols 338 as segments of video encoded information. The symbols 338 may include any information for controlling the operation of the decoder 305, and / or may further include information for controlling devices attached to the decoder 305 that perform operations (such as display devices). Control information for controlling the display device may include information in a format called supplementary enhancement information (SEI) or video usability information (VUI).

[0080] As described above, the parser 330 can be configured to perform entropy decoding 335 on the encoded video data. The entropy encoding method for the encoded video data may vary depending on the encoding standard, and the decoding operation should be performed accordingly. Representative examples of the entropy encoding standards may include variable length coding, Huffman coding, and arithmetic coding. These various encoding methods may be context-adaptive or context-sensitive methods according to the standard, or they may be based on principles well known to those skilled in the art.

[0081] The parser 330 described above can be configured to extract at least one picture from the encoded video data described above. The definition of the picture may vary depending on the encoding standard described above. Depending on the standard, one of the following examples may apply, or multiple examples may overlap. For example, the picture may be grouped or defined and / or segmented into the following encoding / decoding units: group of pictures (GOPs), pictures / frames, tiles, slices, macroblocks, blocks, subblocks, transform units (TUs), prediction units (PUs), etc.

[0082] The parser 330 can be configured to extract encoded information from the encoded video data, such as transform coefficients, quantization parameters (QPs), and / or motion vectors. The parser 330 can be configured to perform entropy decoding 335 and parsing operations on the video data received from the buffer memory, and selectively decode symbols 338 representing the encoded information. Furthermore, the parser 330 can also be configured to selectively provide specific symbols 338 to specific decoding functions within the decoder 305, such as inverse quantization and inverse transform units 340, intra-frame prediction units 350, inter-frame prediction units 355, or loop filter units 360. The control over the provision of such information can be determined by the arrangement of information contained in the encoded video and may vary depending on the encoding standard, and is not limited to the scope of this embodiment, nor is it described in detail in this conceptual diagram.

[0083] The decoder 305 described above may be configured with multiple conceptual functional units for receiving and processing the encoded information from the parser 330 described above. It should be understood that these conceptual functional units can be combined or further subdivided according to implementation needs. For example, they can be further separated for ease of implementation; they can be integrated together to improve operational efficiency. In either case, the individual functional units can be configured to interact closely with each other. Despite the possibility of such integration or separation, the following description will still focus on the combination of conceptual functional units to clarify the decoding process of video data applied as an embodiment of the present invention.

[0084] The decoder described above may include an inverse quantization and inverse transform unit 340. The inverse quantization and inverse transform unit 340 may be configured to receive encoded information from the parser 330, the encoded information including a method for numerical transformation, block size, quantization coefficients for recovering quantized information, and distinguishing information of a quantization matrix that simplifies and represents the quantization coefficients, and may be configured to output block value 341 that can be input to the aggregator 370 as a result of processing the encoded information.

[0085] In one embodiment of the present invention, the output value of the inverse quantization and inverse transform unit 340 may include intra-frame prediction encoded block values. These intra-frame prediction block values ​​refer to values ​​obtained by decoding using prediction information of the currently being decoded image (e.g., within the current frame) without using previously decoded images (e.g., prediction information from the previous frame).

[0086] The prediction information in the current image can be provided by the intra-frame prediction unit 350. According to an embodiment of the present invention, the intra-frame prediction unit 350 uses image information of spatially adjacent regions extracted from the currently being decoded and partially decoded image to generate block values ​​as prediction information that are in the same block format as the block being decoded. This image information can be provided 381 by the current image buffer (i.e., the so-called line buffer 380). According to an embodiment, the merging unit 370 can be configured to merge the prediction information 351 generated by the intra-frame prediction unit 350 with the block values ​​341 provided by the inverse quantization and inverse transform unit 340.

[0087] In another embodiment, the output value of the inverse quantization and inverse transform unit 340 may be a block value encoded for inter-frame prediction. In some cases, the output value may include a block value that has undergone motion compensation. In this case, the inter-frame prediction unit 355 can extract and use sample information 386 for motion-based prediction from the reference picture buffer 385. The information 356 obtained by performing motion compensation on the sample information based on the symbols 338 contained in the block value as the output value can be configured to be merged by the merging unit 370 with the block value 341 provided by the inverse quantization and inverse transform unit 340. In this case, the block value 341 may be referred to as a differential value or a residual value.

[0088] The positional information in the memory is used by the inter-frame prediction unit 355 to extract the sample information from the reference image. This information can be determined, for example, by combining X, Y, and other symbols 338 used to indicate specific positions in the reference image, and by a motion vector transmitted to the inter-frame prediction unit 355. The inter-frame prediction unit 355 may also include the following functions: when providing motion vectors that support "subsampling," it can process the sample values ​​using interpolation; and it can predict and enhance the motion vector values.

[0089] The output value 371 of the merging unit 370 can be provided to the loop filter unit 360 and processed by various loop filtering methods. The loop filter unit 360 can also be configured to receive not only the block unit output 371 of the merging unit 370, but also the symbol 338 from the parser 330 to control its operation. The output of the loop filter unit 360 can be transmitted to an external display device such as the display device described above via the output connection 390. However, for subsequent prediction and parsing of intra-frame or inter-frame coded block values, 361 can be stored in the line buffer 380, and then stored in the reference image buffer 385.

[0090] Once a specific image (e.g., a frame) has been decoded, it can be used as a reference image for predictive decoding in subsequent decoding processes. An image (or frame) can be progressively accumulated into line buffer 380 and decoded, and when the decoding of a frame is complete, the contents of the aforementioned line buffer 380 can be transferred 383 to the aforementioned reference image buffer 385, and a new line buffer 380 is allocated for decoding the new frame.

[0091] The aforementioned video decoder 305 can be configured to perform decoding operations according to a predetermined video compression technology, which can be documented based on various international standards or commercial standards. These standards may include, for example, international standard recommendations such as H.264, H.265, and H.266 defined by the International Telecommunication Union Standardization Sector (ITU-T). Those skilled in the art should understand that these recommendations are equivalent to international standards jointly developed by the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC). The encoded video data may conform to a specific bitstream syntax defined by the relevant standards. This requirement is explicitly specified in the video compression specification document and standard document, and is specifically defined and constrained through the profiles and levels specified therein. To ensure compliance with the aforementioned profiles and levels, the complexity of the encoded video data also needs to be limited to a certain level. For example, certain profiles or levels may configure limitations such as maximum image size, maximum decoding rate, and maximum reference image size. In some embodiments, the aforementioned limitations can be further limited by the hypothetical reference decoder (HRD) and the HRD buffer management metadata signals included in the encoded video data.

[0092] According to one embodiment of the present invention, the receiving unit 310 may receive additional redundant data along with the encoded video. This additional data may be considered part of the encoded video data. The additional data may include information for use by the decoder 305 to correctly decode the data or more accurately reconstruct an image close to the pre-encoded image. The additional data may be provided in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, and forward error correction codes.

[0093] Figure 4 This is a conceptual diagram of a functional unit of a video encoder according to an embodiment of the present invention. The encoder 405 described above can be configured to receive raw video information 402 from a video source 401 and perform encoding.

[0094] The aforementioned raw video information 402 can have any suitable bit depth, such as 8-bit, 10-bit, 12-bit, etc. Furthermore, the aforementioned raw video information 402 can have any suitable color space, such as R / G / B, Y / U / V, Y / Cb / Cr, etc. Additionally, the aforementioned raw video information 402 can have any suitable sampling structure corresponding to the aforementioned color space, such as Y / Cb / Cr 4:2:0, Y / Cb / Cr 4:4:4, etc. The raw video information 402 with this predetermined format can be provided to the encoder in the form of a digital video stream.

[0095] In a one-way video communication network, the aforementioned raw video information 402 can be obtained from a recording medium storing pre-prepared original video footage. In a two-way video communication network, the aforementioned raw video information 402 can be obtained from an image acquisition device (e.g., a camera) included in the two-way video communication that generates at least one video transmission stream.

[0096] The video data, including the aforementioned raw video information 402, can be configured to play in chronological order to simulate multiple images of motion. These images can be represented by concepts such as frames, in addition to individual images. Depending on the sampling structure, color space, and other types used, these images may include more than one sample. Those skilled in the art will understand that the aforementioned sample is a term closely related to the "pixel" in digital imaging. The working principle of the encoder will be explained below in relation to such samples.

[0097] According to one embodiment of the present invention, encoder 405 may be configured to encode and compress the images (and / or their grouped or segmented information) constituting the original video information 402 in real time (or other time requirements as required by the implementation method) into the form of encoded video information.

[0098] In the encoder 405 described above, the control unit 450 can be a functional unit for appropriately controlling the encoding speed. As described below, the control unit 450 can be configured to control other functional units and combine functions with them. The parameters set by the control unit 450 may include parameters related to bitrate control, such as image skipping, quantizer values, and variable values ​​for applying image quality optimization techniques. Furthermore, it may include values ​​such as image size, group of pictures (GOP) structure, and the maximum search range of motion vectors. Those skilled in the art will understand that the control unit 450 may have various other functions, which can be added or removed depending on the video encoder design optimized for individual system designs.

[0099] According to one embodiment of the present invention, the encoder 405 can be configured to operate in a structure known to those skilled in the art, such as a "coding loop". For simplicity, the encoding loop can, exemplarily, consist of an internal encoder (also called a "source encoder") 410 and an internal decoder 420. The internal encoder is responsible for receiving the image as the object of encoding and generating symbols based on at least one previously encoded reference image. The internal decoder is configured to connect to the internal encoder. The internal decoder 420 can be configured to, by receiving the output of the internal encoder 410, perform a reproduction operation of sample data generated after a remote decoder 490, which is actually located remotely, receives encoded video information from the encoder 405.

[0100] The video data, composed of sample data reconstructed by the internal decoder 420, can be configured to be input into the reference image buffer of the encoder 405. As described above, since the internal decoder 420 is implemented to reproduce the output of the encoder 405 and the result of decoding by a remote decoder, the video data recorded in the reference image buffer can be identical in bit units to the information in the reference image buffer of the remote decoder. That is, the prediction function included in the encoder 405 can read the same value from the encoder 405 reference image buffer as the previous frame sample value that the subsequent decoder will reference during the decoding process.

[0101] As described above, the principle of achieving consistency of the reference image buffer between encoder 405 and decoder 490 through the internal decoder 420 at encoder 405 is well known to those skilled in the art. Furthermore, those skilled in the art are also familiar with methods for dealing with environments where such consistency cannot be guaranteed (e.g., information loss due to communication failures).

[0102] Referenced Figure 3 An embodiment of the operation method of the aforementioned internal decoder 420 has been described in detail. Figure 3 The decoder in the code can be considered as the "remote" decoder 490 described above. The internal decoder 420 described above may not include lossless encoding and decoding parts, such as the parser 330 or entropy decoder 335. This is because the implementation of the internal encoder 405 is merely to reproduce the actions of the remote decoder, thus allowing direct decoding of symbols without the need for symbol compression and decompression. Therefore, as... Figure 3 As shown, the parser and entropy decoder may not have the previous functional parts, or at least partially implement the previous functional parts.

[0103] As described above, according to a preferred embodiment of the present invention, any decoder functional unit (excluding the parser and entropy decoder) present in the decoder can naturally exist as substantially the same functional unit in the corresponding encoder 405.

[0104] The operation of the encoding function unit that may be included in the encoder 405 described above can be regarded as the inverse operation of the decoder function unit described above. Therefore, this embodiment can generally be explained by reversing the operation of the decoder function unit described above. For example, quantization and transform function units corresponding to inverse quantization and inverse transform units can be provided, as well as inter-frame prediction coding units corresponding to inter-frame prediction units can be provided. In addition, some additional explanations are provided below.

[0105] The internal encoder 410 described above can be configured to encode input image information (e.g., an input frame) for at least one reference image information, such as a video data designated as a reference frame, according to a predictive coding method performed by the predictive coding unit 440 operating via the reference image buffer 430. In this case, the encoder 405 described above can be configured to encode the difference between the sample blocks constituting the input image and the sample blocks constituting the reference image.

[0106] The internal decoder 420 can decode video data that can be designated as the reference image from the symbols generated by the internal encoder 410. As described above, since this video data is the same as the decoding action performed by the remote decoder, the video data used as the reference image can be provided to the encoder 405 in a lossy compressed and somewhat corrupted form, which is intended to ensure consistency with the decoder's action.

[0107] The predictive coding unit 440 can be configured to perform a predictive search operation within the encoder 405. This predictive search operation can refer to the inter-frame prediction or intra-frame prediction operation described in the decoder description. For input image information that is planned to be newly encoded, in order to obtain the motion vector, block shape, and metadata and sample blocks that can provide suitable predictive reference information for the new image information, including these, as well as the actual reference, of the reference image's positional information, the predictive unit can access the reference image buffer 430 and retrieve the information. The predictive coding unit 440 can operate based on a so-called "sample block by pixel block" reference to obtain suitable predictive reference information. According to an embodiment of the present invention, as can be determined based on the search results obtained by the predictive coding unit 440, at least one predictive reference information can be specified for the input image to point to at least one reference image information stored in the reference image buffer 430.

[0108] In one embodiment of the present invention, the control unit 450 may be configured to include setting parameters for video data encoding to manage the overall encoding operation of the internal encoder 410.

[0109] The outputs of all the aforementioned functional units can be processed by entropy coding 460 to achieve the final output. For the symbols generated by each of the aforementioned functional units, the entropy coding 460 employs various entropy coding techniques described above (which may include, for example, variable-length coding, Huffman coding, and arithmetic coding). Depending on different standards, these coding methods can be context-adaptive or context-sensitive, and can also be implemented based on principles well known to those skilled in the art. This entropy coding 460 typically achieves lossless compression; therefore, at least one symbol generated by the aforementioned functional units can be converted into encoded video data.

[0110] In controlling the operation of the encoder 405, the control unit 450 can apply a specific encoding type to each image (or frame) within the encoding range. The encoding method of the image is also affected by the type. According to an embodiment, the type may include a "frame type" as distinguished below.

[0111] Figure 5 This is a conceptual diagram of a frame type according to an embodiment of the present invention. The following references... Figure 5 Please provide an explanation.

[0112] Intra ("I") images 510 can refer to images that are encoded or decoded using only their own information without requiring predictive coding of other image information in the reference video data. According to video coding standards, the aforementioned "I" images can refer to key frames, independent / instantaneous decoder refresh (IDR) frames, clean random-access (CRA) frames, etc. As mentioned above, the various names for "I" images can have various variations and applications within the limits allowed by each standard, and there can be some differences between them. In addition to the methods listed above, various application methods for implementing "I" images can be methods known to those skilled in the art or newly provided.

[0113] The prediction ("P") image 520 can refer to an image used to predict sample values ​​constituting a block of the aforementioned image. It can be at least one prediction information specifying at least one reference image, and / or an image encoded or decoded based on motion vectors and via intra-frame or inter-frame prediction. According to video coding standards, the aforementioned "P" image can be configured to reference only one reference frame, or refer to more than one reference frame. When referencing more than one reference frame, sample information and / or related metadata derived from multiple reference images can be used to reconstruct a single block. However, generally, an image designated as a "P" image can be understood as limited to referencing images that are temporally earlier.

[0114] A bidirectional prediction (B) image 530 may refer to at least one prediction information from at least two reference images specified for predicting sample values ​​of blocks constituting the image, and / or an image encoded or decoded based on motion vectors and through intra-frame or inter-frame prediction. Typically, an image designated as a "B" image is distinct from an image designated as a "P" image and can be understood as an image that performs the reference, not limited to temporally previous images.

[0115] During encoding and decoding, video data can be spatially divided according to multiple sample blocks and encoded according to the aforementioned block units. It is well known that these block units may include, for example, sizes in horizontal / vertical pixels such as 4x4, 8x8, 4x8, or 16x16, but are not limited to these. These blocks can be encoded using predictive coding methods, depending on the respective images to which they belong and, according to the specified type (allowed and / or restricted), by referencing any other (already encoded) blocks. For example, blocks in "I" image 510 may not use predictive coding methods, or may be encoded by referencing already encoded blocks in the same local image. That is, only so-called intra-frame prediction methods may be used. In contrast, "P" image 520 may further refer to at least one reference image encoded in a previous time unit, and therefore may also be encoded using inter-frame prediction along with intra-frame prediction. "B" image 530 may also refer to reference images previously encoded in the encoding sequence but executed later in time unit. However, it is well known that even within "P" or "B" images, there may be blocks encoded without relying on predictive coding. However, it is well known that there may be blocks in a “P” or “B” image that are not encoded using predictive coding.

[0116] The video encoder 405 described above can be configured to perform encoding operations according to predetermined video compression techniques documented according to various international or commercial standards. Examples of such standards may include all the standards described in the decoder described above.

[0117] According to one embodiment of the present invention, the transmitting unit 470 can buffer the encoded video data generated by the entropy encoding described above, so as to provide / send the video data (ultimately transmitted to the remote decoder 490) to the device storing the encoded video data via a hardware or software connection 495. According to the embodiment, when the transmitting unit 470 receives the encoded video data provided / sent from the video encoder 405, it can receive and merge other data (e.g., encoded audio data or other auxiliary data) accompanying the encoded video data from a separate source 480.

[0118] According to one embodiment of the present invention, the transmitting unit 470 may also be configured to transmit additional data along with the encoded video. This additional data can be considered as part of the encoded video data. The additional data may include information for the decoder, which is used to correctly decode the data or to more accurately reconstruct an image closer to the pre-encoded image. Examples of the additional data may include all the examples described above with respect to the decoder receiving unit 310.

[0119] As described above, the present invention can be implemented using digital video compression standards widely used and understood by those skilled in the art. These digital video compression standards may include at least one known compression standard, such as MPEG-2, MPEG-4 video, H.263, H.264 / AVC, H.265 / HEVC, H.266 / VVC, VC-1, AV1, QuickTime, VP-9, VP-10, MotionJPEG, etc.

[0120] Figure 6 This is a conceptual diagram illustrating the structure of a video encoder according to other embodiments of the present invention. Figure 6 The structure shown is equivalent to well-known standard encodings such as ITU-T H.266 and ISO / IEC 23090-3, as well as the general structure of video encoders known as MPEG-I Part 3 or Versatile Video Coding (VVC).

[0121] according to Figure 6 The video encoder 605 can be configured to receive an encoded bitstream 602 as input and output uncompressed or unencoded raw video data 601. During intra-frame encoding, the video data 601 can be directly provided to the luma mapping unit 610a, or provided to the luma mapping unit 610b via the inter-frame prediction unit 620, which includes motion vector extraction. In the case of intra-frame encoding, the mapped luma signal can be provided not only separately to the output combiner 606, but also by selecting either the intra-frame prediction encoded signal output by the intra-frame prediction unit 625 (608) or the inter-frame prediction encoded signal output from the luma mapping unit 610b via the inter-frame prediction unit 620. The result of the output combiner can be applied to the chroma scaling unit 615 (the operations of the luma mapping unit 610 and the chroma scaling unit 615 are collectively referred to as the luma mapping / chroma scaling (LMCS) process). The scaled color difference signal can be provided to the transform unit 630, which in particular can perform an adaptive color transform on the color difference signal. The coefficients obtained from the transformation are applied to the quantization unit 640 and quantized. Thus, lossy compression can be achieved, and the result of the lossy compression can be output as a bitstream 602 using a lossless compression method based on a multi-hypothesis CABAC unit 650.

[0122] On the other hand, in order to generate the coding loop, the result of the aforementioned lossy compression can undergo inverse quantization 645, inverse transform 635, and luma signal amplification 617 before entering the actual decoding process. The result of the aforementioned luma signal amplification can be provided to the internal combiner 607 together with the result of the selection 608 of at least one previously generated intra-frame predictive coding signal or inter-frame predictive coding signal. After undergoing inverse luma mapping 617, the result of the aforementioned internal combiner can be processed by a deblocking filter 660, a sample adaptive offset (SAO) 670, an adaptive loop filter (ALF), etc., for reproducing the image quality improvement process in the decoder. As described above, the result of the reproduction action in the decoder can be applied to the reference image buffer 690 and can be reused for predictive coding by the aforementioned inter-frame prediction unit 620.

[0123] This invention can also be implemented or combined with the next-generation video codec, namely the enhanced compression model (ECM), developed by the Joint Video Experts Team (JVET), an international standardization expert group. The enhanced compression model may include improved intra-frame predictive coding methods, improved inter-frame predictive coding methods, improved transform and transform coefficient coding methods, improved adaptive loop filtering methods, bilateral filtering methods, a novel sample adaptive offset (SAO) method for image quality improvement, extended entropy coding methods, and improved gradual decoding refresh (GDR) technology.

[0124] Template matching prediction

[0125] This invention relates to a video encoding and decoding method and apparatus that utilizes an improved intra-frame prediction method and can be used in the field of video encoding and decoding, including the embodiments described above.

[0126] Figure 7 This is a conceptual diagram illustrating a template matching prediction method according to one embodiment. The template matching prediction method according to this embodiment can be performed by the video decoder and / or video encoder disclosed in this specification.

[0127] Reference Figure 7The current image may include: a reconstructed area 790, which is a region within which a previous decoding (or encoding) has been performed; and a current block 710, which is an encoding / decoding unit currently being decoded (or encoded). According to the implementation, the current image can be understood as including at least one concept such as slice, frame, subpicture, or tile. According to the implementation, the current block can be understood as a block, subblock, prediction unit, or coding unit.

[0128] On the other hand, in the samples (i.e., pixels) that constitute the recovery region 790, the set of samples that are adjacent to the current block 710 can be defined as template 715.

[0129] The template 715 may include samples that belong to and are adjacent to the current block 710, including the left block, top block, and upper left block. It can also be understood as an L-shaped set of samples directly adjacent to the current block 710 and having a predetermined sample thickness 717. The sample thickness 717 can be specified as any value greater than or less than one pixel. However, for ease of understanding, this specification assumes it to be one pixel. It will be readily understood by those skilled in the art that the present invention can be substantially similarly modified for templates with various sample thicknesses 717, which may consist of two or more pixels, or less than one pixel, and may sometimes include sub-pixel units. Furthermore, the sample thickness 717 may be non-fixed, variable, or vary depending on the orientation or range of the template. Additionally, the template 715 may also include samples contained in the lower left block and / or upper right block of the current block 710.

[0130] According to the template matching prediction method of this embodiment, a block 720 corresponding to the current block 710 can be searched within the recovery region 790, and the corresponding block 720 can be used as a prediction block. Specifically, the template matching prediction method of this embodiment may include: searching for sample occurrence positions similar to the samples constituting the template 715 within the recovery region 790 to perform template matching 735; deriving a prediction template 725 based on the template matching 735; and using the block corresponding to the current block 710 (or any set of samples corresponding to it) adjacent to the prediction template 725 as the prediction block 720.

[0131] At this point, the information representing the direction and distance from the current block 710 to the predicted block 720 can be defined as a block vector, serving as a reference vector. The pixel information of the predicted block 720 can then be used in conjunction with various common coding methods during the encoding process of the current block 710. For example, it can be used for differential coding.

[0132] Figure 8 This is a conceptual diagram illustrating the storage and use of block vectors in a template matching prediction method according to one embodiment. The template matching prediction method according to this embodiment can be executed by the video decoder and / or video encoder disclosed in this specification.

[0133] In the template matching prediction method according to this embodiment, when using the aforementioned prediction block, the distance between the current block 810 and the prediction block 820 is defined as block vector 830, which can be used in subsequent encoding and decoding. The block vector 830 can be used not only in the template matching prediction method but also in intra-frame block copy encoding and decoding methods and is stored. Furthermore, for the decoding (or encoding) of the next block (e.g., 840) of the current block 810, it can be configured to refer to the aforementioned block vector 850. According to an embodiment of the present invention, the aforementioned block vector can be a value determined by calculating the same method in both the encoder and decoder. In this case, the aforementioned block vector may not be included in the encoded bitstream.

[0134] On the other hand, excessive computation in the recovery region during template matching can increase the complexity of the encoder / decoder. This is because the search range for template matching in the recovery region is too large, or the number of pixels within the template is too large, or the number of block vectors used as reference objects in the candidate list for searching block vectors is too large. Therefore, to perform template matching prediction with low complexity, it is necessary to define the computation method, search range, and block vector configuration method of the template matching operation.

[0135] The structure of the present invention

[0136] According to one embodiment of the present invention, the template matching of the remaining block vectors can be terminated early by comparing multiple calculation results when performing template matching on block vectors in the candidate block vector list. For example, in one embodiment, the remaining template matching can be terminated early if a specific calculation result is less than any value. As another example, in one embodiment, the remaining template matching can be terminated early if a specific calculation result is significantly smaller than other calculation results.

[0137] According to one embodiment of the present invention, template matching for sub-pixels can be prematurely terminated based on the calculation result of integer pixel units. For example, in one embodiment, it can be configured to prematurely terminate template matching for the remaining sub-pixel units if the calculation result of the integer pixel unit is less than any value. As another example, in one embodiment, it can be configured to prematurely terminate the remaining template matching if the calculation result of the integer pixel unit is less than the calculation results of some sub-pixels. Yet another example, in one embodiment, it can be configured to prematurely terminate template matching for other regions based on the calculation result of a portion of the region pointed to by the block vector.

[0138] According to one embodiment of the present invention, template matching can be configured to terminate early if the calculation result gradually increases when starting from a region close to the current block. However, according to the embodiment, early termination may not be applied, and various configurations and methods described below may be included instead.

[0139] According to one embodiment of the present invention, it can be configured to: when performing template matching on block vectors in a candidate list of block vectors, reduce the number of block vectors or remove specific block vectors by comparing the block vectors in the candidate list. For example, in one embodiment, it can be configured to remove remaining block vectors by comparing the calculation results of specific block vectors in the candidate list. As another example, in another embodiment, it can be configured to: determine whether to perform template matching on remaining block vectors by comparing specific block vectors or comparing the calculation results of block vectors. Yet another example, in one embodiment, it can be configured to reduce the number of block vectors or remove specific block vectors based on compression information of surrounding blocks.

[0140] According to one embodiment of the present invention, when performing template matching calculations on block vectors in a block vector candidate list, the calculation can be configured to be performed in different ways based on pixel units, sub-pixel units, and / or characteristics of the block vectors. In one embodiment, preferably, the calculation method can be transmitted from the encoder to the decoder using symbols contained in the syntax information of the bitstream. If only one calculation method is used, transmission is not necessary. However, when various different calculation methods are applied, for example, the calculation methods can be transmitted using flags and / or indices indicating the calculation methods and combinations of these expressions. The various calculation methods described above are not limited.

[0141] According to one embodiment of the present invention, the above calculation method may include a simple difference method. According to another embodiment of the present invention, the above calculation method may include a calculation method that adds the absolute values ​​of the differences between pixels (e.g., Sum of Absolute Differences, SAD). According to yet another embodiment of the present invention, the above calculation method may include a calculation method based on difference using a transformation matrix incorporating Hada code transformation (e.g., Sum of Absolute Transformed Differences, SATD). According to yet another embodiment of the present invention, the above calculation method may be based on other functions or algorithms used for calculating other cost weights or differences, such as Mean Absolute Difference (MAD), Mean Removed Sum of Absolute Differences (MRSAD), and Sum of Squared Differences (SSD), and is not limited to the examples described above.

[0142] According to one embodiment of the present invention, the calculation method can be configured to be set and transmitted the same or different for compression units such as video sequences, frames, slices, sub-images, tiles, macroblocks, or coding tree units (CTUs). As described above, when multiple template matching calculation methods are implemented, the encoder can be configured to record, as symbols contained in the above-mentioned syntax information, which calculation method is applied to a specific sequence, frame, slice, sub-image, or tile in the bitstream, and the decoder can be configured to derive the above calculation method by reading the above symbols for use.

[0143] According to one embodiment of the present invention, when setting the search range for template matching prediction, the search range can be set to be centered on the block vector of the search range generated from the block vector of the surrounding blocks or the block vector candidate list. In this case, the search range can be set to integer pixel units or sub-pixel units.

[0144] Figure 9 This is an example diagram illustrating the setting of the block vector search range according to an embodiment of the present invention. (Refer to...) Figure 9 The adjacent blocks adjacent to the current block 930 may include at least one left block 932 and an upper block 934. For example, each of the aforementioned adjacent blocks 932, 934 may be a block with a block vector, and / or a block with a candidate list for block vector operations.

[0145] A block vector search method according to an embodiment of the present invention may include the step of generating a search range block vector 910 based on the current block 930, by referring to the block vector 915 of the left block (e.g., by copying it in the same direction and distance, reflecting the distance between blocks). Furthermore, it may be configured to calculate a search range 940 of a certain size upward / downward / leftward / right, based on the position 917 pointed to by the search range block vector 910 (or centered on the position).

[0146] A block vector search method according to another embodiment of the present invention may include the steps of generating a search range block vector 920 by referring to the block vector 925 of the upper block; and calculating a search range 945 of a certain size based on the position 927 pointed to by the block vector 920. Thus, searching and calculating all pixels in the frame can be avoided, thereby reducing complexity, while the accuracy of prediction can be ensured by referring to adjacent block vectors.

[0147] Figure 10 This is an example diagram of a modified block vector determination method according to an embodiment of the present invention. (Refer to...) Figure 10 The current block 1020 and its adjacent templates 1025 are located within the recovery region 1010. An initial block vector 1030 selected from the block vector candidate list points to a specific location within the recovery region 1010. According to the present invention, a modified block vector 1040 can be derived by applying a modification to the initial block vector 1030. This modification can be achieved by template matching within a predetermined range 1035 around the initial block vector 1030. The modified block vector 1040 can be represented as a differential vector 1045 relative to the initial block vector 1030, and this differential vector 1045 can be transmitted via signaling through a bit stream. A reference block 1050 and template 1055 can be determined based on the modified block vector 1040.

[0148] According to another embodiment of the present invention, the predicted vector used for the template matching described above can be configured to reference block vectors in the surrounding blocks or block vector candidate list and be used after individual correction. When correcting the block vectors in the surrounding blocks or block vector candidate list as described above, according to an embodiment of the present invention, it can be configured to transmit the difference between the block vector and the corrected block vector. Preferably, the correction can be performed within a search range set based on the block vector, and the search range can include sub-pixel units. For example, according to an embodiment of the present invention, it can be configured to perform additional template matching on eight 1 / 4 pixel units surrounding the block vector. Furthermore, various sub-pixel units such as 1 / 2 pixel and 1 / 8 pixel can be applied, and the search pattern used for correction can also be defined in multiple ways. For example, various search patterns such as diamond patterns, cross patterns, and square patterns can be used.

[0149] According to one embodiment of the present invention, information representing the differences can be represented by vector size, flag information, index information, or a combination of at least one of the information listed above. Preferably, the information is configured to be transmitted from the encoder to the decoder via symbols included in the syntax of the bitstream. For example, when represented by vector size, information describing the differential vector can be transmitted via symbols. As another example, when represented by a flag, the flag information can be transmitted via symbols. Yet another example, when represented by an index, index information representing a table of multiple possibilities for the differential vector can be transmitted via symbols.

[0150] According to one embodiment of the present invention, the correction information for the block vector can be implemented using a method with a configuration substantially the same as or similar to that of the existing merge mode with motion vector difference (MMVD) method. For example, it can include a technique that may be called merge mode with block vector difference (MBVD). For example, according to one embodiment of the present invention, it can be configured to generate a more accurate block vector by adding the aforementioned difference vector to the initial block vector, and the aforementioned difference vector can be designed to have a defined direction (e.g., 4-way or 8-way) and a defined magnitude.

[0151] Figure 11 This is an example diagram of a second block vector template matching method according to an embodiment of the present invention. (Refer to...) Figure 11 The current block 1120 and its adjacent templates 1125 are located within the recovery region 1110. A first block vector 1130 for the current block 1120 points to a first reference block 1140 within the recovery region 1110. According to an embodiment of the invention, a second reference block 1160 can be searched using a second block vector 1150 associated with the first reference block 1140. The second reference block 1160 can serve as another candidate for template matching, thereby enabling more accurate predictions.

[0152] According to one embodiment of the present invention, when searching for a block vector for template matching or configuring a block vector list, it can be configured to use a second block vector stored in the block pointed to by the first block vector. The first and second block vectors can be recursively used in the same way; for example, it can be configured to use a third block vector of the block pointed to by the second block vector, stored in the block pointed to by the first block vector. The recursive reference to the aforementioned block vectors can be called a chained motion vector, and should be understood as a technical configuration compatible with various block vector rereference configurations that encompass this name or concept. In this invention, there is no limitation on the number of recursions; the use of the Nth block vector can be determined based on any condition. According to one embodiment of the present invention, for any number N, whether to use the Nth block vector can be determined based on the calculation result of the (N-1)th block vector or the block vector. According to one embodiment of the present invention, for any number N, it can be configured to start from the Nth block vector and set the correction process, calculation method, or template matching at the sub-pixel unit in different ways.

[0153] According to one embodiment of the present invention, the block vector can be corrected at most twice, and can be configured to be corrected with a 1×1 sample range in the first correction and a 3×3 sample range in the second correction. However, in the present invention, as described above, the recursion number of the Nth (times) block vector is not limited, and it is obvious that the specific method of block vector correction can be changed in each number of block vector re-corrections. For example, different correction sample ranges can be applied in each number of corrections. For three, four, or more N corrections, at least some of the different correction sample ranges can be applied. Furthermore, for ease of understanding, the above reference number is represented in the order of first, second, third, etc., but other enumeration methods can be used, such as: first, second, third, etc., 0th, 1st, 2nd, etc., or BV0, BV1, BV2, etc. This is obvious to those skilled in the art, and therefore, as long as the technical concept of the present invention remains unchanged, these differences in representation do not limit the scope of the present invention.

[0154] That is, according to an embodiment of the present invention, it should be understood that the first block vector can be continuously referenced and corrected in the template matching prediction method. Specifically, the first reference block vector derived from the first block vector can be updated through a correction process, and this reference and update process can be recursively performed multiple times. The aforementioned first reference block vector can be understood to encompass any Nth block vector obtained from the region pointed to by the aforementioned first block vector, at least twice, as described in the above embodiment. During these recursions, the correction method for the block vector can be changed at least once. This can refer to various changes in correction methods, including correcting differences in the sample range, changing the processing method for integer pixels and sub-pixel units, switching the calculation algorithm, or adjusting the template matching parameters.

[0155] According to one embodiment of the present invention, whether to perform block vector correction in integer or sub-pixel units can be determined based on whether the encoded / decoded object image is screen content. Screen content here can refer to non-natural images, such as computer screens, game images, or other artificial image content, but is not limited thereto. Preferably, when it is screen content, sub-pixel re-correction may not be performed, but it can also be configured in other ways depending on the characteristics of the image or the encoder / decoder configuration method. According to the embodiment, whether it is screen content can be determined by signal transmission contained in the high-level bitstream syntax information (HLS), or by signal transmission contained in other units of bitstream syntax, or implicitly inferred from other signals of the bitstream syntax or the characteristics of the encoded / decoded object video.

[0156] Figure 12 This is an example diagram of multiple block vectors according to an embodiment of the present invention. (Refer to...) Figure 12 The current block 1220 and its adjacent template 1225 are located within the recovery region 1210. The Nth block vector 1230 points to the Nth reference block 1240 within the recovery region 1210. (As mentioned above, the Nth block vector 1230 can be derived through multiple block vector chains.) Another feature of the present invention is to provide a method for utilizing multiple block vectors 1250 that may exist within the Nth reference block 1240. These multiple block vectors point to different positions and can be selectively or entirely utilized as needed.

[0157] According to one embodiment of the present invention, for any number N, when multiple block vectors exist within the block pointed to by the Nth block vector, it can be configured to select at least one specific block vector for operation. That is, when multiple block vectors are extracted from the block pointed to by the Nth block vector through template matching, it can be configured to modify and compare each vector with each other for calculation. For example, according to one embodiment, it can be configured to use them sequentially in any order. For example, according to another embodiment, it can be configured to determine whether to use the remaining block vectors using the calculation results of any number of block vectors. For example, according to another embodiment, it can be configured to perform template matching down to the sub-pixel unit only on any number of block vectors. For example, according to yet another embodiment, it can be configured to use only the block vectors stored in the region closest to the pixel pointed to by the block vector.

[0158] According to one embodiment of the present invention, it can be determined whether to apply template matching prediction technology to the aforementioned Nth block by considering at least one of the block's width or height, area, number of pixels within the block, and / or information of surrounding blocks. In this case, it can be configured to select and use the template corresponding to the left side, top side, or both the left and top sides of the aforementioned Nth block.

[0159] According to one embodiment, if the width of a block is greater than a first threshold and the height is greater than a second threshold, it can be configured to specify any N, and template matching will only proceed to the Nth block. As another example, if the width of a block is greater than a third threshold and the height is greater than a fourth threshold, it can be configured to specify any N, and template matching will only proceed to the Nth block. As another example, if the number of pixels in a block is greater than or less than a fifth threshold, it can be configured to specify any N, and template matching will only proceed to the Nth block. As yet another example, if the area of ​​a block is greater than or less than a sixth threshold, it can be configured to specify any N, and template matching will only proceed to the Nth block.

[0160] According to an embodiment of the present invention, when multiple calculation methods are used in template matching prediction, and / or when comparing the current block and the predicted block between multiple block vectors in template matching prediction, and / or when setting the maximum number of block vectors in template matching prediction, and / or when setting whether to perform on integer pixels or sub-pixel units in template matching prediction, and / or when setting the search range area in template matching prediction, the information of the above settings can be transmitted by a combination of flags or indexes, and the information of the above settings can be set and transmitted in units of sequence, tile, sub-image, frame, slice or block.

[0161] For ease of explanation, the reference vector type has been described in the above embodiments of the present invention based on block vectors in intra-frame prediction. However, those skilled in the art should understand that the various application methods of the block vectors described above are also applicable to motion vectors in inter-frame prediction. That is, similar to block vectors, motion vectors can also be derived, recursively derived, and / or modified using the methods disclosed in this invention. Furthermore, it is also obvious that block vectors and motion vectors can be derived, recursively derived, and / or modified in combination. Therefore, the present invention, as applicable to both intra-frame and inter-frame prediction methods, should be understood as a technical concept universally applicable to general reference vectors.

[0162] Encoder and decoder

[0163] Clearly, the method according to the invention can be applied equally to both encoders and decoders. Furthermore, as... Figure 4 As shown in the diagram, this decoding process can also be implemented inside the encoder to predict the state of the decoder, through the internal decoder 420 and the encoding loop including it.

[0164] The encoding method according to the present invention described above can be implemented using an encoder as a device. The encoder, as a device, can maintain the previously passed... Figures 1 to 6The existing encoder structure shown, or its implementation in a form that specifies its application, is not necessarily limited to the example shown. Any encoder structure that can be used as a video encoder as long as it implements the technical idea of ​​the present invention should be regarded as an encoder according to the present invention.

[0165] Furthermore, the decoding method for the encoding result according to the present invention can be implemented using a decoder as a device. The decoder, as a device, can maintain the previously obtained... Figures 1 to 6 The existing decoder structures shown, or those implemented in a form that specifies their application, are not necessarily limited to the examples shown. Any form of decoder structure that can be used as a video decoder as long as it implements the technical idea of ​​the present invention should be regarded as a decoder according to the present invention.

[0166] Those skilled in the art will readily understand that a bitstream encoded using the methods and apparatus described above can be decoded by applying methods symmetrical to and / or opposite to the encoding method described above. In one embodiment, when reading information for decoding from the encoded bitstream, at least one variable-length coding syntax contained in the encoded bitstream can be parsed. Furthermore, in one embodiment, the variable-length coding can be implemented using an entropy coding method. The technical details and application methods for implementing this decoding step can be readily understood from the encoding steps described above.

[0167] The processors described in this specification that may be included in the encoders and / or decoders may refer to one or more general-purpose or special-purpose computers, such as processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field-programmable gate arrays (FPAs), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions.

[0168] For ease of understanding, even though the processor is referred to in the singular, those skilled in the art will understand that the processor may include multiple processing elements and / or various types of processing elements. For example, an apparatus according to an embodiment of the present invention may include multiple processors, or a processor and a controller, as the processor 130 described above. Furthermore, the processor may be implemented using various processing configurations, such as a parallel processor or a multi-core processor.

[0169] The processor described above can be configured to execute an operating system (OS) and one or more software programs running on the operating system. Furthermore, the processor can access, store, manipulate, process, and generate data in response to the execution of the software.

[0170] The aforementioned software may include computer programs, code, instructions, or a combination thereof, and can control a processor to operate in a desired manner. It can also be configured to issue instructions to the processor independently or collectively. The software may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, for parsing by the processor or to provide instructions or data to the processor. The software may be distributed across multiple computer systems connected to the aforementioned network and stored or executed in a distributed manner.

[0171] The aforementioned software can also be implemented in the form of program instructions executable by various computer means, and recorded or stored in the aforementioned memory. The aforementioned memory can be a computer-readable recording medium, in which program commands, data files, and data structures can be recorded individually or in combination. The program commands stored in the aforementioned memory can be implemented based on a command system specifically designed and configured for embodiments of the present invention, or can follow command systems known and available to those skilled in the art of computer software, such as command systems for assembly language, C, C++, Java, Python, etc. It should be understood that the aforementioned command system and the program commands it generates include not only machine language code generated by a compiler, but also high-level language code that can be executed by an interpreter or other means of apparatus and / or the aforementioned processor according to an embodiment of the present invention.

[0172] The computer-readable recording medium constituting an embodiment of the apparatus of the present invention includes the storage described above, and may include: temporary or volatile recording media such as processor cache, random access memory (RAM), flash memory, etc., which are maintained only during the operation of the processor; or relatively non-volatile or long-term recordable recording media such as hard disk, floppy disk, magnetic tape, etc.; optical recording media such as CD-ROM, DVD, etc.; magneto-optical media such as floppy disk; or solid state memory; or read-only recording media such as read-only memory (ROM) provided in hardware. Furthermore, since the circuit wiring is configured to execute a series of program instructions and equivalent actions through a hard-wired structure, it can be considered that the various steps involved in executing the actions described in the embodiments of the present invention can be regarded as being recorded through the connection and arrangement of the aforementioned hardware components. Therefore, it is obvious to those skilled in the art that this connection and arrangement method can be regarded as equivalent to the aforementioned memory.

[0173] The above embodiments regarding the processor and memory are not exclusive and can be selected or combined as needed. For example, a hardware device can be configured to execute the actions of an embodiment of the present invention as a module composed of one or more of the above-described software, and vice versa. As another example, in this specification, all or part of the actions assigned to a certain functional unit can be implemented by one or more of the above-described software stored in a device according to an embodiment of the present invention (preferably stored in a recording medium belonging to the scope of the above-described memory), and configured to be executed by the above-described processor. In this case, such a functional unit can be referred to as a functional unit "included" in the above-described processor.

[0174] The present invention has been described above with reference to the accompanying drawings and embodiments. However, as stated above, this does not mean that the scope of protection of the present invention is limited to the drawings or embodiments disclosed above. Those skilled in the art should understand that various modifications and changes can be made to the present invention without departing from the inventive spirit and scope set forth in the claims of this patent.

Claims

1. An image decoding method, characterized in that, include: The steps to obtain information about the current block from the bit stream; The step of determining the template located in the adjacent region of the current block; The step of deriving the first reference vector of the current block using the template; The step of modifying the first reference vector to derive the modified first reference vector; The step of generating a prediction block for the current block using the region pointed to by the corrected first reference vector; as well as, The step of recovering the current block using the predicted block. The correction to the first reference vector is achieved by performing template matching within a predetermined range around the first reference vector.

2. The image decoding method according to claim 1, characterized in that, Also includes: The step of obtaining the second reference vector from the region pointed to by the modified first reference vector; The step of modifying the second reference vector to derive the modified second reference vector; as well as, The step of generating a prediction block for the current block using the region pointed to by the corrected second reference vector.

3. The image decoding method according to claim 2, characterized in that, The corrections for the first reference vector and the corrections for the second reference vector are performed using different correction sample ranges.

4. The image decoding method according to claim 3, characterized in that, The correction of the first reference vector is performed using a 1×1 sample range; The correction for the second reference vector is performed using a 3×3 sample range.

5. The image decoding method according to claim 1, characterized in that, Also includes: The step of obtaining information from the bitstream indicating whether the image is screen content. Based on the information indicating whether the image is screen content, it is determined whether the correction to the first reference vector is performed on an integer pixel unit or a sub-pixel unit.

6. The image decoding method according to claim 5, characterized in that, When the image is screen content, the correction to the first reference vector is performed in integer pixel units; When the image is not screen content, the correction to the first reference vector is performed in at least one of integer pixel units and sub-pixel units.

7. The image decoding method according to claim 1, characterized in that, The template matching is performed using at least one of several calculation methods for calculating the differences between templates for the purpose of template matching. The plurality of calculation methods include at least one of the following: simple difference method, calculation method of adding the absolute values ​​of the differences between pixels, calculation method based on difference using transformation matrix, average absolute difference, sum of absolute differences after removing the mean, and sum of squared differences.

8. The image decoding method according to claim 7, wherein, The template matching is performed by using two or more of the plurality of calculation methods, and includes: The steps of reading information from the bitstream for selecting at least one of the plurality of calculation methods; and, The step of determining the calculation method for template matching based on the information used to select at least one of the plurality of calculation methods.

9. The image decoding method according to claim 1, wherein, The step of deriving the first reference vector of the current block using the template includes: The step of configuring a candidate list of reference vectors for exporting the first reference vector; and, The step of removing at least one reference vector from the candidate reference vector list by comparing the reference vectors.

10. The image decoding method according to claim 1, wherein, The step of deriving the first reference vector of the current block using the template includes: The step of configuring a candidate list of reference vectors for exporting the first reference vector; The steps of reading information from the bitstream for selecting at least one of the candidate reference vectors; and... The step of deriving the first reference vector based on the information used to select at least one of the candidate reference vectors in the list.

11. The image decoding method according to claim 1, characterized in that, The width, height, area, or number of pixels in the current block are compared with a predetermined threshold, and the applicability of the template matching prediction technique and at least one of the applicable methods are determined based on the comparison result.

12. The image decoding method according to claim 1, characterized in that, Also includes: The step of obtaining the first reference vector from the region pointed to by the modified first reference vector; The step of modifying the first reference vector to update the first reference vector; as well as, The step of generating a prediction block for the current block using the region pointed to by the first reference vector. The step of updating the first reference vector is recursively performed multiple times, and at least one of the recursive steps involves a different method for modifying the reference vector.

13. An image encoding method, characterized in that, include: The steps to determine the template located in the adjacent region of the current block; The step of deriving the first reference vector of the current block using the template; The step of modifying the first reference vector to derive the modified first reference vector; The step of generating a prediction block for the current block using the region pointed to by the corrected first reference vector; The step of generating a residual signal using the current block and the prediction block; as well as, The step of encoding the residual signal to generate a bit stream, The correction of the first reference vector is achieved by performing template matching within a predetermined range around the first reference vector.

14. The image encoding method according to claim 13, characterized in that, Also includes: The step of obtaining the second reference vector from the region pointed to by the modified first reference vector; The step of modifying the second reference vector to derive the modified second reference vector; as well as, The step of generating a prediction block for the current block using the region pointed to by the corrected second reference vector.

15. The image encoding method according to claim 13, characterized in that, Also includes: The steps to determine whether the currently encoded object image is screen content; as well as, The step of including information indicating the determination result in the bitstream. Specifically, depending on whether it is screen content, it is determined whether the correction for the first reference vector is performed on an integer pixel unit or a sub-pixel unit.

16. The image encoding method according to claim 13, characterized in that, Also includes: The step of determining the calculation method used for template matching; as well as, The step of including information indicating the determined calculation method in the bitstream. The calculation method includes at least one of the following: simple difference method, calculation method of adding the absolute values ​​of the differences between pixels, calculation method based on difference using transformation matrix, average absolute difference, sum of absolute differences after removing the mean, and sum of squared differences.

17. The image encoding method according to claim 13, characterized in that, Also includes: The step of generating a difference vector representing the difference between the first reference vector and the modified first reference vector; as well as, The step of including the difference vector in the bitstream. The difference vector is represented by a combination of direction and magnitude information. The direction information indicates one of a finite number of directions. The size information indicates one of a finite number of size values.

18. The image encoding method according to claim 13, wherein, The step of deriving the first reference vector of the current block using the template includes: The step of configuring a candidate list of reference vectors for exporting the first reference vector; The step of calculating the coding cost of the current block based on the predicted block obtained from at least one reference vector included in the candidate list of reference vectors; Based on the calculation of the encoding cost, at least one reference vector candidate is selected from the candidate list to derive the first reference vector; and, The step of generating a bitstream containing relevant information for identifying the selected candidate from the candidate list.

19. The image encoding method according to claim 13, characterized in that, Also includes: The step of obtaining the first reference vector from the region pointed to by the modified first reference vector; The step of modifying the first reference vector to update the first reference vector; as well as, The step of generating a prediction block for the current block using the region pointed to by the first reference vector. The step of updating the first reference vector is recursively performed multiple times, and at least one of the recursive steps involves a different method for modifying the reference vector.

20. An image decoding device, characterized in that, include: processor; Memory connected to the processor; A receiving unit that receives bit streams; and, Image decoding department, The image decoding unit is configured to: obtain information about the current block from the bitstream; determine a template located in a region adjacent to the current block; derive a first reference vector for the current block using the template; correct the first reference vector to derive a corrected first reference vector; generate a prediction block for the current block using the region pointed to by the corrected first reference vector; and recover the current block using the prediction block. The correction to the first reference vector is achieved by performing template matching within a predetermined range around the first reference vector.

Images