Video coding and decoding method and device based on improved local brightness compensation cross filter

By improving local brightness compensation technology and using cross filters and filter coefficients to process video blocks, the low efficiency of existing LIC technology is solved, and more efficient video encoding and decoding effects are achieved.

CN121925839APending Publication Date: 2026-04-24JIAWEN GRP CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202480061083.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-09-26
Filing Date
2024-09-26
Publication Date
2026-04-24

AI Technical Summary

Technical Problem

Existing Local Luminance Compensation (LIC) technology has limited efficiency in video encoding and decoding and needs to be improved to increase encoding efficiency.

Method used

An improved local brightness compensation technique is adopted. By defining templates and filter coefficients, the reference block is filtered using a cross filter to generate prediction samples to restore the current block, thereby improving prediction performance and minimizing the information capacity of the encoded bitstream.

Benefits of technology

It improves the efficiency of video encoding and decoding, reduces computational load and hardware size, and enhances video quality and the compression performance of encoded bitstreams.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121925839A_ABST
    Figure CN121925839A_ABST
Patent Text Reader

Abstract

The invention relates to a video compression technology, in particular to an improvement of a local luminance compensation (LIC) technology contained in a predictive coding technology for improving the compression performance of a video encoder and a video decoder, and discloses a method for improving the compression performance of the video encoder and the compression performance of the video decoder, and particularly relates to the improvement of the local luminance compensation (LIC) technology contained in the predictive coding technology for improving the compression performance of the video encoder and the video decoder. A method for decoding an encoded video bitstream, according to one embodiment of the present invention, may comprise: a step for acquiring, from the bitstream, at least one prediction vector for a current block that is currently being decoded; a step of determining at least one reference block on the basis of the prediction vector; a step of defining at least one of the following templates: a first template that takes the current block as a reference block and includes at least one pixel adjacent to the current block; the second template takes the reference block as a reference block and comprises at least one pixel adjacent to the reference block; a step of extracting parameter information related to an LIC with reference to at least one piece of pixel information derived from at least one of the first template and the second template; a step of determining at least one coefficient applied to a first filter on the basis of the parameter information; acquiring a filtering result value for the reference block by using the first filter to which the coefficient is applied; a step of generating a prediction sample for the current block on the basis of the filtering result value; and a step of restoring the current block using the prediction sample.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to video compression technology, and more specifically, to an improvement in local illuminance compensation (LIC) technology included in predictive coding techniques to enhance the compression performance of video encoders and decoders.

[0002] This invention may be in the same technical field as at least one known digital video compression technology standard (e.g., MPEG-2, MPEG-4 Video, H.263, H.264 / AVC, H.265 / HEVC, H.266 / VVC, VC-1, AV1, QuickTime, VP-9, VP-10, and Motion JPEG), or may be used to improve the inherent efficiency of such standards, or may be used to improve or replace such standards. Background Technology

[0003] Digital video encoding and decoding are widely used in various digital video applications. For example, digital television broadcasting, video transmission via communication networks, video calls, video conversations, video chats, video content recorded and played using optical discs (including VCDs (video compact discs) / DVDs (digital versatile discs) / Blu-ray discs), the entire process of video content production, editing, collection, and distribution, and video shooting devices and cameras used in video shooting and recording activities (including personal, commercial, industrial, and security monitoring purposes) all rely on video encoding and decoding technologies.

[0004] Therefore, physical devices that can be called digital video encoders and decoders can constitute a part of a wide range of devices related to the generation, recording, and playback of digital video, including digital television, digital broadcasting systems, wireless broadcasting systems, laptops, desktop computers, tablets, e-book readers, digital cameras, digital video recording devices, digital multimedia playback devices, video game devices / terminals / control consoles, mobile phones (including smartphones) with multimedia playback capabilities, equipment for video conferencing, and others.

[0005] The aforementioned digital video encoder and decoder can be implemented using digital video compression standards widely used and understood by those skilled in the art. These digital video compression standards may include at least one known compression standard, such as MPEG-2, MPEG-4 Video, H.263, H.264 / AVC, H.265 / HEVC, H.266 / VVC, VC-1, AV1, QuickTime, VP-9, VP-10, and Motion JPEG.

[0006] Video encoders and decoders can encode or decode digital video information more efficiently by conforming to the aforementioned standards, or by improving or modifying them. Attempts to modify these standards may also lead to new standards. A well-known example is the Enhanced Compression Model (ECM), developed by the Joint Video Experts Team (JVET), a joint international standards organization comprised of ISO, IEC, and ITU-T, aimed at improving upon and replacing the existing H.266 / VVC standard.

[0007] Among the various detailed techniques applied to existing video encoders and decoders, there exists a technique collectively known as local illuminance compensation (LIC). LIC refers to the technique of performing illuminance correction (i.e., compensation) on the values ​​of selected prediction blocks in order to calculate prediction residuals during inter-frame predictive coding (i.e., inter-coding) and / or intra-frame predictive coding (i.e., intra-coding), and decoding corresponding to the aforementioned coding methods. Specifically, it can refer to performing the compensation locally, for example, on a block-by-block basis. Summary of the Invention

[0008] The problem the invention aims to solve

[0009] Existing LIC technology has limited application methods, and it is expected that various improvements can enhance coding efficiency. The purpose of this invention is to address this technical problem and provide improved coding efficiency.

[0010] Problem Solving Methods

[0011] To address the aforementioned technical problems, a decoding method according to an embodiment of the present invention, as a decoding method for an encoded video bitstream, includes the following steps: obtaining at least one prediction vector from the bitstream for a current block currently being decoded; determining at least one reference block based on the prediction vector; defining at least one template: a first template that uses the current block as a reference block and includes at least one pixel adjacent to the current block; and a second template that uses the reference block as a reference block and includes at least one pixel adjacent to the reference block; extracting parameter information related to Local Illumination Compensation (LIC) by referring to at least one pixel information derived from at least one of the first template and the second template; determining at least one coefficient applied to a first filter based on the parameter information; performing filtering on the reference block using the first filter with the applied coefficient and obtaining a filtering result value; generating a prediction sample for the current block based on the filtering result value; and reconstructing the current block using the prediction sample.

[0012] The first filter described above may be a cross filter, configured to receive at least one left pixel, at least one right pixel, at least one top pixel, at least one bottom pixel and a center pixel as input, and having at least one coefficient that can be applied to each of the aforementioned pixels.

[0013] The first filter described above may be a 3×3 cross filter configured to receive five pixels as input, including a left pixel, a right pixel, a top pixel, a bottom pixel, and a center pixel; and has five coefficients that can be applied to each of the aforementioned pixels.

[0014] The characteristics of the first filter described above can be determined according to the following mathematical formula:

[0015]

[0016] Wherein, Pred is the filtered result value of the first filter, A represents the input value of the left pixel, B represents the input value of the right pixel, C represents the input value of the top pixel, D represents the input value of the bottom pixel, E represents the input value of the center pixel, and C1, C2, C3, C4, and C5 represent the coefficients applied to each of the above inputs.

[0017] The characteristics of the first filter described above can be determined according to the following mathematical formula:

[0018]

[0019] Wherein, Pred is the filtered result value of the first filter, A represents the input value of the left pixel, B represents the input value of the right pixel, C represents the input value of the top pixel, D represents the input value of the bottom pixel, E represents the input value of the center pixel, F represents the nonlinear term based on E, and V... mid It is the additive median constant, the above V div It is the equalization division constant, and C1, C2, C3, C4, C5, and C6 above represent the coefficients applied to each of the above inputs.

[0020] The above V mid The feature could be the median (“bitDepth_mid”) of the bit depth of the pixels for the current block, and is determined by the aforementioned V. div The division operation is replaced by a right binary shift operation based on the bit depth value (">>bitDepth".

[0021] The characteristics of the first filter described above can be determined according to the following mathematical formula:

[0022]

[0023] Wherein, Pred is the filtered result value of the first filter, A represents the input value of the left pixel, B represents the input value of the right pixel, C represents the input value of the top pixel, D represents the input value of the bottom pixel, E represents the input value of the center pixel, F represents the nonlinear term based on E, and V... mid It is the additive median constant, the above V div It is the equalization division constant, the above V const These are batch addition constants, and C1, C2, C3, C4, C5, C6, and C7 above represent the coefficients applied to each of the above inputs.

[0024] The above V mid And the above V const The feature could be the median ("bitDepth_mid") representing the bit depth of the pixels for the current block, and is determined by the aforementioned V. div The division operation is replaced by a right binary shift operation based on the bit depth value (">>bitDepth".

[0025] The first filter described above may be an image processing filter configured to receive a group of pixels centered on a center pixel and having a width of a first pixel and a height of a second pixel as input; and having at least one coefficient applicable to each of the aforementioned pixels.

[0026] The step of determining at least one coefficient applied to the first filter can be performed by a minimum mean squared error (MSE) minimization method based on the Wiener filter, which is calculated based on at least one of the restored pixel values ​​on the upper and left sides of the current block.

[0027] To address the aforementioned technical problems, an encoding method according to an embodiment of the present invention, as an encoding method for encoding a bitstream generated from video, includes the following steps: determining at least one prediction vector for a current block currently being encoded; determining at least one reference block based on the prediction vector; defining at least one template: a first template that uses the current block as a reference block and includes at least one pixel adjacent to the current block; and a second template that uses the reference block as a reference block and includes at least one pixel adjacent to the reference block; extracting parameter information related to Local Illumination Compensation (LIC) by referring to at least one pixel information derived from at least one of the first template and the second template; determining at least one coefficient applied to a first filter based on the parameter information; performing filtering on the reference block using the first filter with the applied coefficient to obtain a filtering result value; generating a prediction sample for the current block based on the filtering result value; encoding the current block using the prediction sample; and recording the encoded result into the bitstream.

[0028] The first filter described above may be a cross filter, configured to receive at least one left pixel, at least one right pixel, at least one top pixel, at least one bottom pixel, and a center pixel as input; and has at least one coefficient that can be applied to each of the aforementioned pixels.

[0029] The first filter described above may be a 3×3 cross filter configured to receive five pixels as input, including a left pixel, a right pixel, a top pixel, a bottom pixel, and a center pixel; and has five coefficients that can be applied to each of the aforementioned pixels.

[0030] The characteristics of the first filter described above can be determined according to the following mathematical formula:

[0031]

[0032] Wherein, Pred is the filtered result value of the first filter, A represents the input value of the left pixel, B represents the input value of the right pixel, C represents the input value of the top pixel, D represents the input value of the bottom pixel, E represents the input value of the center pixel, and C1, C2, C3, C4, and C5 represent the coefficients applied to each of the above inputs.

[0033] The characteristics of the first filter described above can be determined according to the following mathematical formula:

[0034]

[0035] Wherein, Pred is the filtered result value of the first filter, A represents the input value of the left pixel, B represents the input value of the right pixel, C represents the input value of the top pixel, D represents the input value of the bottom pixel, E represents the input value of the center pixel, F represents the nonlinear term based on E, and V... mid It is the additive median constant, the above V div It is the equalization division constant, and C1, C2, C3, C4, C5, and C6 above represent the coefficients applied to each of the above inputs.

[0036] The above V mid The feature could be the median (“bitDepth_mid”) of the bit depth of the pixels for the current block, and is determined by the aforementioned V. div The division operation is replaced by a right binary shift operation based on the bit depth value (">>bitDepth".

[0037] The characteristics of the first filter described above can be determined according to the following mathematical formula:

[0038]

[0039] Wherein, Pred is the filtered result value of the first filter, A represents the input value of the left pixel, B represents the input value of the right pixel, C represents the input value of the top pixel, D represents the input value of the bottom pixel, E represents the input value of the center pixel, F represents the nonlinear term based on E, and V... mid It is the additive median constant, the above V div It is the equalization division constant, the above V constThese are batch addition constants, and C1, C2, C3, C4, C5, C6, and C7 above represent the coefficients applied to each of the above inputs.

[0040] The above V mid And the above V const The feature could be the median ("bitDepth_mid") representing the bit depth of the pixels for the current block, and is determined by the aforementioned V. div The division operation is replaced by a right binary shift operation based on the bit depth value (">>bitDepth".

[0041] The first filter described above may be an image processing filter configured to receive a group of pixels centered on a center pixel and having a width of a first pixel and a height of a second pixel as input; and having at least one coefficient applicable to each of the aforementioned pixels.

[0042] The step of determining at least one coefficient applied to the first filter can be performed by a minimum mean squared error (MSE) minimization method based on the Wiener filter, which is calculated based on at least one of the restored pixel values ​​on the upper and left sides of the current block.

[0043] To address the aforementioned technical problems, a decoder apparatus according to an embodiment of the present invention is provided, configured to decode an encoded video bitstream using a computing device, comprising: a receiving unit for receiving the bitstream; an output unit for outputting the decoded video; a reference buffer for storing at least one decoded picture information; a processor; and a memory for storing instructions executable by the processor; wherein the instructions are configured to include the following instructions: an instruction to obtain at least one prediction vector from the bitstream for a currently being decoded block; an instruction to determine at least one reference block based on the prediction vector; and an instruction to define at least one template: a first template that uses the current block as a reference block and includes at least one pixel adjacent to the current block; and a second template that uses the reference block as a reference block and includes at least one pixel adjacent to the reference block; and an extraction of local illumination compensation information based on at least one pixel information derived from at least one of the first template and the second template. Instructions for obtaining parameter information related to Compensation (LIC); instructions for determining at least one coefficient applied to a first filter based on the parameter information; instructions for performing filtering on the reference block using the first filter with the applied coefficient to obtain a filtering result value; instructions for generating a prediction sample for the current block based on the filtering result value; and instructions for reconstructing the current block using the prediction sample.

[0044] The first filter described above may be a cross filter, configured to receive at least one left pixel, at least one right pixel, at least one top pixel, at least one bottom pixel, and a center pixel as input; and has at least one coefficient that can be applied to each of the aforementioned pixels.

[0045] The step of determining at least one coefficient applied to the first filter can be performed by a minimum mean squared error (MSE) minimization method based on the Wiener filter, which is calculated based on at least one of the restored pixel values ​​on the upper and left sides of the current block.

[0046] Invention Effects

[0047] According to the present invention, it has the beneficial effect of improving prediction performance through LIC technology while minimizing the overhead of information capacity of the encoded bitstream.

[0048] According to the present invention, video encoding and decoding can achieve at least one of the following effects: improved encoding efficiency, improved decoding efficiency, improved video quality, reduced computational load, reduced software size, reduced hardware size, and other performance improvements related to encoding and decoding. Attached Figure Description

[0049] Figure 1 This is a conceptual diagram of a video communication system according to an embodiment of the present invention.

[0050] Figure 2 This is a conceptual diagram of the arrangement of encoders and decoders in a real-time video streaming environment according to an embodiment of the present invention.

[0051] Figure 3 This is a conceptual diagram of the functional unit of a video decoder according to an embodiment of the present invention.

[0052] Figure 5 This is a conceptual diagram of a frame type according to an embodiment of the present invention.

[0053] Figure 6 This is a conceptual diagram showing the structure of a video encoder according to the H.266 / VVC standard.

[0054] Figure 7 This is a conceptual diagram illustrating a conventional LIC application method according to an embodiment of the present invention.

[0055] Figure 8 This is a conceptual diagram illustrating the shape and coefficient positions of a cross filter according to an embodiment of the present invention.

[0056] Figure 9 This is an example diagram illustrating an example of a template application pattern according to an embodiment of the present invention. Detailed Implementation

[0057] This invention can be modified and has various embodiments, with specific embodiments illustrated in the accompanying drawings and described in detail. However, this is not intended to limit the invention to specific implementations, but rather to encompass all modifications, equivalents, and substitutions that fall within the scope of the invention's ideas and techniques.

[0058] While terms such as "first" and "second" may be used to describe various components, the components described above should not be limited by these terms. The purpose of these terms is to distinguish one component from another. For example, a first component may be named a second component without departing from the scope of the invention, and similarly, a second component may be named a first component. The term "and / or" includes a combination of or one of several related described items and is not exclusive unless otherwise stated. The items listed in this application are merely illustrative descriptions intended to facilitate understanding of the concepts and possible implementations of the invention, and are therefore not intended to limit the scope of the embodiments of the invention.

[0059] In this specification, "A or B" can mean "A only", "B only", or "A and B". In other words, "A or B" in this specification can also be interpreted as "A and / or B". For example, "A, B or C" in this specification can mean "A only", "B only", "C only", or "any combination of A, B and C".

[0060] The forward slash ( / ) or comma used in this specification can mean "and / or". For example, "A / B" can mean "A and / or B". Accordingly, "A / B" can mean "A only", "B only", or "A and B". For example, "A, B, C" can mean "A, B, or C".

[0061] In this specification, "at least one of A and B" can mean "A only", "B only", or "A and B". Furthermore, the expressions "at least one of A or B" or "at least one of A and / or B" in this specification can be interpreted as "at least one of A and B".

[0062] Additionally, in this specification, "at least one of A, B and C" can mean "A only", "B only", "C only" or "any combination of all A, B and C". Furthermore, "at least one of A, B or C" or "at least one of A, B and / or C" can mean "at least one of A, B and C".

[0063] When a component is described as "connected" or "accessed" to another component, it should be understood that it may be directly connected to or accessed by another component, or there may be other components in between. Conversely, when a component is described as "directly connected" or "directly accessed" to another component, it should be understood that there are no other components in between.

[0064] The terminology used in this application is merely illustrative of specific embodiments and not limiting of the invention. Where there is no explicit indication of singularity in the context, the singular designation includes the meaning of plural. In this application, terms such as "comprising" or "possessing" indicate the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, rather than precluding the presence or additional possibility of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0065] Unless otherwise specified, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms that are generally used identically to those defined in dictionaries have the same meaning as in the context of the relevant art, and unless explicitly defined, do not have an ideal or excessive meaning in this application.

[0066] In describing the present invention in this application, embodiments may be described or illustrated in the form of unit blocks that perform the described functions or functionalities. In this application, the aforementioned blocks may represent one or more devices, units, modules, components, etc. The aforementioned blocks may be implemented in hardware using one or more logic gates, integrated circuits, processors, controllers, memories, electronic components, or information processing hardware methods including but not limited to these. Alternatively, the aforementioned blocks may be implemented in software using application software, operating system software, firmware, or information processing software methods including but not limited to these. A block may be divided into multiple blocks performing the same function, and conversely, the function of simultaneously implementing multiple blocks may be performed by a single block. The aforementioned blocks may also be implemented as physically separated or combined according to any standard. The aforementioned blocks may also be implemented to operate in an environment where their physical location is not specified and they are isolated from each other via communication networks, the Internet, cloud services, or communication methods including but not limited to these. All of the above implementation methods fall within the scope of various embodiments that can be adopted by those skilled in the art of information and communication technology to achieve the same technical concept. Therefore, any specific implementation method should be interpreted as falling within the scope of the inventive technical concept of this application.

[0067] The preferred embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. To facilitate a comprehensive understanding of the invention, the same components will be denoted by the same reference numerals, and repeated descriptions of the same components will be omitted. Furthermore, the various embodiments are not mutually exclusive, and some embodiments can be combined with one or more other embodiments to form new embodiments.

[0068] Digital video codec

[0069] Figure 1 This is a conceptual diagram of a video communication system according to an embodiment of the present invention. The video communication system 100 described above can be configured to include at least two terminals 110 and 120 interconnected via a network 105.

[0070] In one embodiment of the present invention, the above Figure 1 This can be represented as a block diagram for configuring a one-way video communication network. To transmit video data 111 via network 105, the first terminal 110 can encode the video data. The second terminal 120 can be configured to receive the encoded video data 121 via the network, decode it, and display it.

[0071] In another embodiment of the present invention, the above Figure 1This can be represented as a block diagram for configuring a two-way video communication network. To achieve the aforementioned two-way video communication, each of the terminals 110 and 120 can be configured to encode its own acquired video data to transmit the video 112 and 122 to other terminals via the network. Each terminal can also be configured to receive video data sent via the network by other terminals 113 and 123, decode it, and display the decoded video data.

[0072] Figure 1 The terminals 110 and 120 shown above, according to embodiments, can be devices such as server computers, personal computers, portable computers, and smartphones, but are not limited thereto; any commonly used computing device can be used. For example, according to embodiments of the present invention, the terminals 110 and 120 may refer to desktop computers, laptop computers, tablet PCs, mobile phones, smartphones, personal digital assistants (PDAs), workstations, electronic calculators, server computers, cloud computers, virtual computers, quantum computers, or any other electronic, electrical, or quantum computing device implemented in a portable or non-portable form. In particular, the above-mentioned devices can also be interpreted as any device designed to operate as a terminal device according to an embodiment of the present invention, or capable of operating as a terminal device according to an embodiment of the present invention, or capable of installing and / or running computer programs to perform corresponding methods.

[0073] The aforementioned terminals 110 and 120 can be implemented by multiple functional units, which are interconnected in various forms, such as buses, circuits, or relationships between routines and subroutines, and configured to exchange information within the terminals 110 and 120. Furthermore, through these interconnections, these terminals can be configured to include a processor 130 with computing capabilities, and a memory 140 connected to the processor, to execute or support the operation of functional units that primarily require computation between the aforementioned functional units.

[0074] The processor 130 described in this specification may refer to one or more general-purpose or special-purpose computers, such as processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field-programmable gate arrays (FPAs), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions.

[0075] For ease of understanding, even though the processor 130 is referred to only in the singular, those skilled in the art will understand that the processor 130 may include multiple processing elements and / or various types of processing elements. For example, an apparatus according to an embodiment of the present invention may include multiple processors, or a processor and a controller, as the processor 130. Furthermore, the processor 130 may be implemented using various processing configurations, such as a parallel processor or a multi-core processor.

[0076] The processor 130 can be configured to execute an operating system (OS) and one or more software programs running on the OS. Furthermore, the processor can access, store, manipulate, process, and generate data in response to the execution of the software. The software may include computer programs, code, instructions, or a combination thereof, and can be configured as needed to provide independent or collective instruction control over the processing device. The software can be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave for parsing by the processor 130 or to provide instructions or data to the processor. The software can be distributed among multiple computer systems (e.g., terminals 110 and 120) connected to the network 105 and stored or executed in a distributed manner.

[0077] The aforementioned software can also be implemented in the form of program instructions executable by various computer means, and recorded or stored in the aforementioned memory 140. The aforementioned memory 140 can be a computer-readable recording medium, in which program commands, data files, and data structures can be recorded individually or in combination. The program commands stored in the aforementioned memory 140 can be implemented based on a command system specifically designed and configured for embodiments of the present invention, or can follow command systems known and available to those skilled in the art of computer software, such as command systems for assembly language, C, C++, Java, Python, etc. It should be understood that the aforementioned command system and the program commands it generates include not only machine language code generated by a compiler, but also high-level language code that can be executed by an interpreter or other means of apparatus according to an embodiment of the present invention and / or the aforementioned processor 130.

[0078] The computer-readable recording medium constituting an embodiment of the apparatus of the present invention includes the memory 140 described herein, and may include: temporary or volatile recording media such as processor cache, random access memory (RAM), flash memory, etc., which are maintained only during the operation of the processor; or relatively non-volatile or long-term recording media such as hard disk, floppy disk, magnetic tape, etc.; optical recording media such as CD-ROM, DVD, etc.; magneto-optical media such as floppy disk; or solid state memory; or read-only recording media such as a hardware-installed read-only memory (ROM). Furthermore, since the circuit wiring is configured to execute a series of program instructions and equivalent actions through a hard-wired structure, it can be considered that the various steps involved in executing the actions described in the embodiments of the present invention can be regarded as being recorded through the connection and arrangement of the aforementioned hardware components. Therefore, it is obvious to those skilled in the art that this connection and arrangement method can be regarded as equivalent to the memory 140 described above.

[0079] The above embodiments regarding the processor 130 and the memory 140 are not exclusive and can be selected or combined as needed. For example, a hardware device can be configured to execute the actions of an embodiment of the present invention as a module composed of more than one of the above-described software, and vice versa. As another example, in this specification, all or part of the actions assigned to a certain functional unit can be implemented by more than one of the above-described software stored in a device according to an embodiment of the present invention (preferably stored in a recording medium belonging to the scope of the above-described memory), and configured to be executed by the processor. In this case, such a functional unit can be referred to as a functional unit "included" in the processor.

[0080] This invention is applicable to any environment in which a one-way or two-way video communication network is established, and it should be understood that the network 105 described above can be configured to transmit encoded video data between the terminals 110 and 120 by any means.

[0081] In one embodiment of the present invention, the network 105 may refer to a wired or wireless communication network. According to this embodiment, the network can be configured to communicate information using any communication standard, including packet-based communication. Packet communication can be understood to include, for example, TCP or UDP packets. The wired communication method of the network 105 may be a method of connecting to an external communication network via RJ-11 standard telephone lines, various types of Ethernet cables conforming to the RJ-45 standard, other coaxial cables, metallic cables, optical cables, and other various wired media. According to the embodiments, the wireless communication method of the network 105 described above may include short-range wireless communication methods, such as Bluetooth, Wi-Fi, Zigbee, and near field communication (NFC); ​​or may include long-range wireless communication methods, such as Wibro, WiMax, Global Systems for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long-Term Evolution (LTE), New Radio (NR), and other wireless communication technology names collectively referred to as 2G, 3G, 4G, 5G, and 6G communication standard generations, as well as international standard wireless communication standards such as IMT-2000, IMT-Advanced, IMT-2020, and IMT-2030. Of course, apart from this, any existing or newly developed wired or wireless communication means, methods, standards, and protocols applied to the implementation of the network 105, as long as they are configured for transmission and reception in the information communication devices such as terminals 110 and 120, will not affect the achievement of the purpose of this invention. Furthermore, it is obvious that a network 105 can be configured using a combination of more than one wired and / or wireless standard.

[0082] However, in another embodiment of the present invention, the network 105 described above can also be understood to include a process of transmitting information using a computer-readable recording medium. In this case, the configuration of the network is not limited to communication media, but should be understood to include a process of temporary storage and physical transmission in computer-readable storage and / or recording media. The computer-readable recording medium used for the above information transmission can be understood as a relatively non-volatile or long-term recordable recording medium, which is mainly used as a means of transmitting data between computing devices, especially magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floppy disks; or solid-state memory.

[0083] Furthermore, regardless of the information communication or transmission means used, as long as its structure supports the transmission and decoding of video data in an encoded state, it can be considered to fall within the scope of the embodiments of this invention. Therefore, apart from the examples listed above, all known or future information communication or transmission means can be included within the scope of this invention.

[0084] Figure 2 This is a conceptual diagram of the arrangement of encoders and decoders in a real-time video streaming environment according to an embodiment of the present invention. Figure 2 The streaming media system 200 shown can be applied to video data communication networks, including, for example, digital broadcasting, video telephony, and video conferencing. However, even when information transmission is involved via recording media, the same or similar technical structures as described above can be applied.

[0085] According to one embodiment of the present invention, the streaming media system described above may include a video source 210 that generates the video stream. The video source may include a digital video capture unit 212 for capturing uncompressed raw video; for example, the digital video capture unit may be configured as a digital camera or other device. The raw video stream 215 may have a large capacity and therefore can be compressed by combining it with or connecting it to a video encoder 217 connected to the video source.

[0086] The encoder 217 described above can be configured as a unit including hardware, software, or a combination of both, for implementing a video encoding method and / or its implementation method according to an embodiment of the present invention.

[0087] The encoder 217 described above can output an encoded bitstream 219 with a reduced capacity compared to the original video stream. The bitstream 219 can be provided in real time for communication via a relay device (e.g., streaming media server 220) and / or stored on the recording medium 225 of the streaming media server 220 for later use.

[0088] The aforementioned streaming media system 200 may include at least one streaming media client 230, 240 that accesses the aforementioned streaming media server 220 to receive in real time or acquire later the aforementioned encoded bitstream 229. The aforementioned streaming media client may include a video decoder 232, which acquires the aforementioned encoded bitstream 229 (which may also be regarded as a copy of the bitstream 219 received by the aforementioned streaming media server), decodes the aforementioned bitstream 229, and outputs the resulting video data in a form that can be displayed on a display 235 or other visual, auditory, or other sensory display means.

[0089] As mentioned above, the encoding and decoding functions of video data are collectively referred to as a coder-decoder system, or video codec.

[0090] Figure 3 This is a conceptual diagram of the functional unit of a video decoder according to an embodiment of the present invention. Figure 3 As shown, the receiving unit 310 can receive at least one encoded video data to be decoded by the decoder 305. In one embodiment of the present invention, each received encoded video data may be independent, and the decoding process of each independent video data may be independent of the decoding process of other video data. The encoded video data can be received to the receiving unit 310 via a hardware or software connection 315 with a storage device. As described above, the storage device may refer to a streaming media server located at the other end of a communication network, or it may be a physical storage medium, but is not limited thereto.

[0091] The receiving unit 310 can receive the encoded video data and other accompanying data, such as encoded audio data or other auxiliary data, and each of the above data can be separated from the video data and provided to other appropriate processing function units 312 other than the video decoder.

[0092] When receiving the aforementioned video data via a communication network, a buffer memory 320 can be connected between the receiving unit 310 and the decoder 305 to minimize latency and interruptions caused by the network environment. The buffer memory 320 can be a computer-readable recording medium used to temporarily store the received video data and stably provide it to the parser 330 corresponding to the input of the decoder 305. However, if the bandwidth of the communication network is sufficient, or if the video data is read from a local and not physically isolated recording medium, or if communication latency is not expected in other environments, the buffer memory may not be necessary.

[0093] The video decoder 305 may include the parser 330 as its input for parsing the encoded video data. The parser can separate (parsing) various types of information stored in bitstream form within the encoded video data according to predetermined rules, and, when necessary, perform entropy decoding 335 on the entropy-coded video data, thereby reconstructing symbols 338 as segments of video encoded information. The symbols 338 may include any information for controlling the operation of the decoder 305, and / or may further include information for controlling devices attached to the decoder 305 that perform operations (such as display devices). Control information for controlling the display device may include information in a format called supplementary enhancement information (SEI) or video usability information (VUI).

[0094] As described above, the parser 330 can be configured to perform entropy decoding 335 on the encoded video data. The entropy encoding method for the encoded video data may vary depending on the encoding standard, and the decoding operation should be performed accordingly. Representative examples of the entropy encoding standards may include variable length coding, Huffman coding, and arithmetic coding. These various encoding methods may be context-adaptive or context-sensitive methods according to the standard, or they may be based on principles well known to those skilled in the art.

[0095] The parser 330 described above can be configured to extract at least one local image from the encoded video data described above. The definition of the local image may vary depending on the encoding standard described above. Depending on the standard, one of the examples listed below may apply, or multiple examples may be applied simultaneously. For example, the local image may be defined as the following units: group of pictures (GOPs), pictures / frames, tiles, slices, macroblocks, blocks, subblocks, transform units (TUs), prediction units (PUs), etc.

[0096] The parser 330 can be configured to extract encoded information from the encoded video data, such as transform coefficients, quantization parameters (QPs), and / or motion vectors. The parser 330 can be configured to perform entropy decoding 335 and parsing operations on the video data received from the buffer memory, and selectively decode symbols 338 representing the encoded information. Furthermore, the parser 330 can also be configured to selectively provide specific symbols 338 to specific decoding functions within the decoder 305, such as inverse quantization and inverse transform units 340, intra-frame prediction units 350, inter-frame prediction units 355, or loop filter units 360. The control over the provision of such information can be determined by the arrangement of information contained in the encoded video and may vary depending on the encoding standard, and is not limited to the scope of this embodiment, nor is it described in detail in this conceptual diagram.

[0097] The decoder 305 described above may be configured with multiple conceptual functional units for receiving and processing the encoded information from the parser 330 described above. It should be understood that these conceptual functional units can be combined or further subdivided according to implementation needs. For example, they can be further separated for ease of implementation; they can be integrated together to improve operational efficiency. In either case, the individual functional units can be configured to interact closely with each other. Despite the possibility of such integration or separation, the following description will still focus on the combination of conceptual functional units to clarify the decoding process of video data applied as an embodiment of the present invention.

[0098] The decoder described above may include an inverse quantization and inverse transform unit 340. The inverse quantization and inverse transform unit 340 may be configured to receive encoded information from the parser 330, the encoded information including a method for numerical transformation, block size, quantization coefficients for recovering quantized information, and distinguishing information of a quantization matrix that simplifies and represents the quantization coefficients, and may be configured to output block value 341 that can be input to the aggregator 370 as a result of processing the encoded information.

[0099] In one embodiment of the present invention, the output value of the inverse quantization and inverse transform unit 340 may include intra-frame prediction encoded block values. These intra-frame prediction block values ​​refer to values ​​obtained by decoding using prediction information of the currently being decoded local image (e.g., within the current frame) without using previously decoded local images (e.g., prediction information from the previous frame).

[0100] The prediction information in the current local image can be provided by the intra-frame prediction unit 350. According to an embodiment of the present invention, the intra-frame prediction unit 350 uses image information of spatially adjacent regions extracted from the currently being decoded and partially decoded local image to generate block values ​​in the same block format as the being decoded as prediction information. The local image information can be provided 381 by the current image buffer (i.e., the so-called line buffer 380). According to an embodiment, the merging unit 370 can be configured to merge the prediction information 351 generated by the intra-frame prediction unit 350 with the block values ​​341 provided by the inverse quantization and inverse transform unit 340.

[0101] In another embodiment, the output value of the inverse quantization and inverse transform unit 340 may be a block value encoded for inter-frame prediction. In some cases, the output value may include a block value that has undergone motion compensation. In this case, the inter-frame prediction unit 355 can extract and use sample information 386 for motion-based prediction from the reference picture buffer 385. The information 356 obtained by performing motion compensation on the sample information based on the symbols 338 contained in the block value as the output value can be configured to be merged by the merging unit 370 with the block value 341 provided by the inverse quantization and inverse transform unit 340. In this case, the block value 341 may be referred to as a differential value or a residual value.

[0102] The positional information in the memory is used by the inter-frame prediction unit 355 to extract the sample information from the reference image. This information can be determined, for example, by combining X, Y, and other symbols 338 used to indicate specific positions in the reference image, and by a motion vector transmitted to the inter-frame prediction unit 355. The inter-frame prediction unit 355 may also include the following functions: when providing motion vectors that support "subsampling," it can process the sample values ​​using interpolation; and it can predict and enhance the motion vector values.

[0103] The output value 371 of the merging unit 370 can be provided to the loop filter unit 360 and processed by various loop filtering methods. The loop filter unit 360 can also be configured to receive not only the block unit output 371 of the merging unit 370, but also the symbol 338 from the parser 330 to control its operation. The output of the loop filter unit 360 can be transmitted to an external display device such as the display device described above via the output connection 390. However, for subsequent prediction and parsing of intra-frame or inter-frame coded block values, 361 can be stored in the line buffer 380, and then stored in the reference image buffer 385.

[0104] After decoding, a specific local image (e.g., a frame) can be used as a reference image for predictive decoding in subsequent decoding processes. A local image (e.g., a frame) can be progressively accumulated into line buffer 380 and decoded, and when the decoding of a frame is complete, the contents of the line buffer 380 can be transferred 383 to the reference image buffer 385, and a new line buffer 380 is allocated for decoding the new frame.

[0105] The aforementioned video decoder 305 can be configured to perform decoding operations according to a predetermined video compression technology, which can be documented based on various international standards or commercial standards. These standards may include, for example, international standard recommendations such as H.264, H.265, and H.266 defined by the International Telecommunication Union Standardization Sector (ITU-T). Those skilled in the art should understand that these recommendations are equivalent to international standards jointly developed by the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC). The encoded video data may conform to a specific bitstream syntax defined by the relevant standards. This requirement is explicitly specified in the video compression specification document and standard document, and is specifically defined and constrained through the profiles and levels specified therein. To ensure compliance with the aforementioned profiles and levels, the complexity of the encoded video data also needs to be limited to a certain level. For example, certain profiles or levels may configure limitations such as maximum image size, maximum decoding rate, and maximum reference image size. In some embodiments, the above limitations can be further limited by the hypothetical reference decoder (HRD) and the HRD buffer management metadata signals included in the encoded video data.

[0106] According to one embodiment of the present invention, the receiving unit 310 may receive additional redundant data along with the encoded video. This additional data may be considered part of the encoded video data. The additional data may include information for use by the decoder 305 to correctly decode the data or more accurately reconstruct an image close to the original image. The additional data may be provided in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, and forward error correction codes.

[0107] Figure 4 This is a conceptual diagram of a functional unit of a video encoder according to an embodiment of the present invention. The encoder 405 described above can be configured to receive raw video information 402 from a video source 401 and perform encoding.

[0108] The aforementioned raw video information 402 can have any suitable bit depth, such as 8-bit, 10-bit, 12-bit, etc. Furthermore, the aforementioned raw video information 402 can have any suitable color space, such as R / G / B, Y / U / V, Y / Cb / Cr, etc. Additionally, the aforementioned raw video source can have any suitable sampling structure corresponding to the aforementioned color space, such as Y / Cb / Cr4:2:0, Y / Cb / Cr 4:4:4, etc. A raw video source with this predetermined format can be provided to the aforementioned encoder in the form of a digital video stream.

[0109] In a one-way video communication network, the aforementioned raw video information 402 can be obtained from a recording medium storing pre-prepared original video footage. In a two-way video communication network, the aforementioned raw video information 402 can be obtained from an image acquisition device (e.g., a camera) included in the two-way video communication that generates at least one video transmission stream.

[0110] The video data, including the aforementioned raw video information 402, can be configured to be played sequentially in time to simulate multiple local images of motion. These local images can be represented using concepts such as pictures or frames. Depending on the sampling structure, color space, and other types used, each local image may include more than one sample. Those skilled in the art will understand that the term "sample" is closely related to the concept of a "pixel" in digital imaging. The working principle of the encoder will be explained below using examples of such samples.

[0111] According to one embodiment of the present invention, encoder 405 may be configured to encode and compress (partial) images constituting the original video information 402 in real time (or according to other time requirements required by the implementation method) into the form of encoded video information.

[0112] In the encoder 405 described above, the control unit 450 can be a functional unit for appropriately controlling the encoding speed. As described below, the control unit 450 can be configured to control other functional units and combine functions with them. The parameters set by the control unit 450 may include parameters related to bitrate control, such as image skipping, quantizer values, and variable values ​​for applying image quality optimization techniques. Furthermore, it may include values ​​such as image size, group of pictures (GOP) structure, and the maximum search range of motion vectors. Those skilled in the art will understand that the control unit 450 may have various other functions, which can be added or removed depending on the video encoder design optimized for individual system designs.

[0113] According to one embodiment of the present invention, the encoder 405 can be configured to operate in a structure known to those skilled in the art, such as a "coding loop". For simplicity, the encoding loop can, exemplarily, consist of an internal encoder (also called a "source encoder") 410 and an internal decoder 420. The internal encoder is responsible for receiving the image to be encoded and generating symbols based on at least one previously encoded reference image. The internal decoder is configured to connect to the internal encoder. The internal decoder 420 can be configured to, by receiving the output of the internal encoder 410, perform a reproduction operation of sample data generated after a remote decoder 490, which is actually located remotely, receives encoded video information from the encoder 405.

[0114] The video data, composed of sample data reconstructed by the internal decoder 420, can be configured to be input into the reference image buffer of the encoder 405. As described above, since the internal decoder 420 is implemented to reproduce the output of the encoder 405 and the result of decoding by a remote decoder, the video data recorded in the reference image buffer can be identical in bit units to the information in the reference image buffer of the remote decoder. That is, the prediction function included in the encoder 405 can read the same value from the encoder 405 reference image buffer as the previous frame sample value that the subsequent decoder will reference during the decoding process.

[0115] As described above, the principle of achieving consistency of the reference image buffer between encoder 405 and decoder 490 through the internal decoder 420 at encoder 405 is well known to those skilled in the art. Furthermore, those skilled in the art are also familiar with methods for dealing with environments where such consistency cannot be guaranteed (e.g., information loss due to communication failures).

[0116] Referenced Figure 3 An embodiment of the operation method of the aforementioned internal decoder 420 has been described in detail. Figure 3 The decoder in the code can be considered as the "remote" decoder 490 described above. The internal decoder 420 described above may not include lossless encoding and decoding parts, such as the parser 330 or entropy decoder 335. This is because the implementation of the internal encoder 405 is merely to reproduce the actions of the remote decoder, thus allowing direct decoding of symbols without the need for symbol compression and decompression. Therefore, as... Figure 3 As shown, the parser and entropy decoder may not have the previous functional parts, or at least partially implement the previous functional parts.

[0117] As described above, according to a preferred embodiment of the present invention, any decoder functional unit (excluding the parser and entropy decoder) present in the decoder can naturally exist as substantially the same functional unit in the corresponding encoder 405.

[0118] The operation of the encoding function unit that may be included in the encoder 405 described above can be viewed as the reverse process of the decoder function unit described above. Therefore, this embodiment can generally be explained by reversing the operation of the decoder function unit described above. For example, quantization and transform function units corresponding to inverse quantization and inverse transform units can be provided, as well as inter-frame prediction coding units corresponding to inter-frame prediction units can be provided. In addition, some further explanation is provided below.

[0119] The aforementioned internal encoder 410 can be configured to perform encoding of input image information (e.g., input frame) for at least one reference image information, such as a local image (e.g., a frame) that has been previously encoded in time sequence at least one distance from video data designated as a reference frame, according to a predictive coding method performed by the predictive coding unit 440 operating via the reference image buffer 430. In this case, the aforementioned encoder 405 can be configured to encode the difference between the sample blocks constituting the input image and the sample blocks constituting the reference image.

[0120] The internal decoder 420 can decode video data that can be designated as the reference image from the symbols generated by the internal encoder 410. As described above, since this video data is the same as the decoding action performed by the remote decoder, the video data used as the reference image can be provided to the encoder 405 in a lossy compressed and somewhat corrupted form, which is intended to ensure consistency with the decoder's action.

[0121] The predictive coding unit 440 can be configured to perform a predictive search operation within the encoder 405. This predictive search operation can refer to the inter-frame prediction or intra-frame prediction operation described in the decoder description. For input image information that is planned to be newly encoded, in order to obtain the motion vector, block shape, and metadata that may include these, as well as the actual reference sample block, of the reference image's positional information that can provide suitable predictive reference information for the new image information, and the sample block that is actually referenced, the predictive unit can access the reference image buffer 430 and retrieve the information. The predictive coding unit 440 can operate based on a so-called "sample block by pixel block" reference to obtain suitable predictive reference information. According to an embodiment of the present invention, as can be determined based on the search results obtained by the predictive coding unit 440, at least one predictive reference information can be specified for the input image to point to at least one reference image information stored in the reference image buffer 430.

[0122] In one embodiment of the present invention, the control unit 450 may be configured to include setting parameters for video data encoding to manage the overall encoding operation of the internal encoder 410.

[0123] The outputs of all the aforementioned functional units can be processed by entropy coding 460 to achieve the final output. For the symbols generated by each of the aforementioned functional units, the entropy coding 460 employs various entropy coding techniques described above (which may include, for example, variable-length coding, Huffman coding, and arithmetic coding). Depending on different standards, these coding methods can be context-adaptive or context-sensitive, and can also be implemented based on principles well known to those skilled in the art. This entropy coding 460 typically achieves lossless compression; therefore, at least one symbol generated by the aforementioned functional units can be converted into encoded video data.

[0124] In controlling the operation of the encoder 405, the control unit 450 can apply a specific encoding type to each local image (e.g., a frame or a scene) within the encoding range. The encoding method of the local image is also affected by the type. According to an embodiment, the type may include a "frame type" as distinguished below.

[0125] Figure 5 This is a conceptual diagram of a frame type according to an embodiment of the present invention. The following references... Figure 5 Please provide an explanation.

[0126] Intra-frame ("I") images 510 can refer to images encoded or decoded using only their own information without requiring predictive coding of other scene information in the reference video data. According to video coding standards, the aforementioned "I" images can refer to keyframes, independent / instantaneous decoder refresh (IDR) frames, clean random-access (CRA) frames, etc. As mentioned above, the various names for "I" images can have various variations and applications within the limits allowed by each standard, and there can be some differences between them. In addition to the methods listed above, various application methods for implementing "I" images can be methods known to those skilled in the art or newly provided methods.

[0127] The prediction ("P") image 520 can refer to an image used to predict sample values ​​constituting a block of the aforementioned image. It can be at least one prediction information specifying at least one reference image, and / or an image encoded or decoded based on motion vectors and via intra-frame or inter-frame prediction. According to video coding standards, the aforementioned "P" image can be configured to reference only one reference frame, or refer to more than one reference frame. When referencing more than one reference frame, sample information and / or related metadata derived from multiple reference images can be used to reconstruct a single block. However, generally, an image designated as a "P" image can be understood as limited to referencing images that are temporally earlier.

[0128] A bidirectional prediction (B) image 530 may refer to at least one prediction information from at least two reference images specified for predicting sample values ​​of blocks constituting the image, and / or an image encoded or decoded based on motion vectors and through intra-frame or inter-frame prediction. Typically, an image designated as a "B" image is distinct from an image designated as a "P" image and can be understood as an image that performs the reference, not limited to temporally previous images.

[0129] During encoding and decoding, video data can be spatially divided according to multiple sample blocks and encoded according to the aforementioned block units. It is well known that these block units may include, for example, sizes in horizontal / vertical pixels such as 4×4, 8×8, 4×8, or 16×16, but are not limited to these. These blocks can be encoded using predictive coding methods, based on their respective local images and, depending on the specified type (allowed and / or restricted), by referencing any other (already encoded) blocks. For example, blocks in "I" image 510 may not use predictive coding methods, or may be encoded by referencing already encoded blocks in the same local image. That is, only so-called intra-frame prediction methods may be used. In contrast, "P" image 520 may further refer to at least one reference image encoded in a previous time unit, and therefore may also be encoded using inter-frame prediction along with intra-frame prediction. "B" image 530 may also refer to reference images previously encoded in the encoding sequence but executed later in the time unit. However, it is well known that even within "P" or "B" images, there may be blocks encoded without relying on predictive coding. However, it is well known that there may be blocks in a “P” or “B” image that are not encoded using predictive coding.

[0130] The video encoder 405 described above can be configured to perform encoding operations according to predetermined video compression techniques documented according to various international or commercial standards. Examples of such standards may include all the standards described in the decoder described above.

[0131] According to one embodiment of the present invention, the transmitting unit 470 can buffer the encoded video data generated by the entropy encoding described above, so as to provide / send the video data (ultimately transmitted to the remote decoder 490) to the device storing the encoded video data via a hardware or software connection 495. According to the embodiment, when the transmitting unit 470 receives the encoded video data provided / sent from the video encoder 405, it can receive and merge other data (e.g., encoded audio data or other auxiliary data) accompanying the encoded video data from a separate source 480.

[0132] According to one embodiment of the present invention, the transmitting unit 470 may also be configured to transmit additional data along with the encoded video. This additional data can be considered as part of the encoded video data. The additional data may include information for the decoder, used to correctly decode the data or to more accurately reconstruct information closer to the pre-encoded image. Examples of the additional data may include all the examples described above with respect to the decoder receiving unit 310.

[0133] As described above, the present invention can be implemented using digital video compression standards widely used and understood by those skilled in the art. These digital video compression standards may include at least one known compression standard, such as MPEG-2, MPEG-4 video, H.263, H.264 / AVC, H.265 / HEVC, H.266 / VVC, VC-1, AV1, QuickTime, VP-9, VP-10, MotionJPEG, etc.

[0134] Figure 6 This is a conceptual diagram showing the structure of a video encoder using H.266 / VVC. Figure 6 The structure shown is equivalent to well-known standard encodings such as ITU-T H.266 and ISO / IEC 23090-3, as well as the general structure of video encoders known as MPEG-I Part 3 or Versatile Video Coding (VVC).

[0135] according to Figure 6 The video encoder 605 can be configured to receive an encoded bitstream 602 as input and output uncompressed or unencoded raw video data 601. During intra-frame encoding, the video data 601 can be directly provided to the luma mapping unit 610a, or provided to the luma mapping unit 610b via the inter-frame prediction unit 620, which includes motion vector extraction. In the case of intra-frame encoding, the mapped luma signal can be provided not only separately to the output combiner 606, but also by selecting either the intra-frame prediction encoded signal output by the intra-frame prediction unit 625 (608) or the inter-frame prediction encoded signal output from the luma mapping unit 610b via the inter-frame prediction unit 620. The result of the output combiner can be applied to the chroma scaling unit 615 (the operations of the luma mapping unit 610 and the chroma scaling unit 615 are collectively referred to as the luma mapping / chroma scaling (LMCS) process). The scaled color difference signal can be provided to the transform unit 630, which in particular can perform an adaptive color transform on the color difference signal. The coefficients obtained from the transformation are applied to the quantization unit 640 and quantized. Thus, lossy compression can be achieved, and the result of the lossy compression can be output as a bitstream 602 using a lossless compression method based on a multi-hypothesis CABAC unit 650.

[0136] On the other hand, in order to generate the coding loop, the result of the aforementioned lossy compression can undergo inverse quantization 645, inverse transform 635, and luma signal amplification 617 before entering the actual decoding process. The result of the aforementioned luma signal amplification can be provided to the internal combiner 607 together with the result of the selection 608 of at least one previously generated intra-frame predictive coding signal or inter-frame predictive coding signal. After undergoing inverse luma mapping 617, the result of the aforementioned internal combiner can be processed by a deblocking filter 660, a sample adaptive offset (SAO) 670, an adaptive loop filter (ALF), etc., for reproducing the image quality improvement process in the decoder. As described above, the result of the reproduction action in the decoder can be applied to the reference image buffer 690 and can be reused for predictive coding by the aforementioned inter-frame prediction unit 620.

[0137] This invention can also be implemented or combined with the enhanced compression model (ECM), a next-generation video codec developed by the Joint Video Experts Team (JVET) to improve H.266 / VVC. According to the aforementioned JVET standardization progress document, document number ISO / IEC JTC 1 / SC 29 / WG 5 N 190 (also document number JVET AC2025-v1), the enhanced compression model may include improved intra-frame predictive coding methods, improved inter-frame predictive coding methods, improved transform and transform coefficient coding methods, improved adaptive loop filtering methods, bilateral filtering methods, a novel sample adaptive offset (SAO) method for image quality improvement, extended entropy coding methods, and improved gradual decoding refresh (GDR) technology. These technologies are included as background technology in this invention.

[0138] Conventional LIC application methods

[0139] This invention provides an improved method for applying local illuminance compensation (LIC) technology and an apparatus for applying the method. The method and apparatus can be used in the field of video encoding and decoding, including the above embodiments.

[0140] Figure 7This is a conceptual diagram illustrating a conventional LIC application method according to an embodiment of the present invention. To address the problem of reduced compression efficiency caused by illumination differences between frames, Weighted Prediction (WP) is applied in traditional video coding standards. WP is applied on a frame-by-frame basis and is a technique that compensates for brightness variations in a reference block to accurately obtain the predicted value during the encoding of the current frame. WP is applied to H.264 / AVC and H.265 / HEVC, and each slice can be used in either explicit or implicit modes depending on whether brightness compensation parameters are transmitted.

[0141] The aforementioned WP-based technique can effectively handle brightness variations between frames and accurately correct the predicted values ​​of the reference frame for the entire image. However, due to the limitations of applying the technique on a frame-by-frame basis, it cannot properly perform brightness compensation for local brightness variations. To overcome this, the technique for inducing appropriate brightness compensation for local predicted brightness variations is the aforementioned LIC technique.

[0142] The aforementioned LIC is a technique for modeling local brightness variations between the final predicted block 770 for the current block 710. This technique is implemented through a function 750, which approximates the relationship between the template 715 adjacent to the current block 710 and the template 725 adjacent to the reference block 720 using a value 730 based on the current block 730 and a value 740 based on the reference sample 720. The parameters of the function can be represented by a scale α and an offset β forming a linear equation, which, as a linear equation for compensating for brightness variations, can be expressed in the form α×p[x]+β, where p[x] can be the reference sample pointed to by the position x of the motion vector (MV) 717 on the reference image in the inter-prediction method. In one implementation of the aforementioned LIC, when wrap-around motion compensation is activated, the offset of the spatial structure of the loop pixel can be taken into account when clipping the MV. Since α and β can be derived based on the current block template and the reference block template, no signaling overhead is required to represent the use of LIC as described above, except when LIC is represented in the bitstream using an indicator flag (preferably for Advanced Motion Vector Prediction (AMVP) mode).

[0143] However, the present invention does not preclude the possibility that the LIC technique described above can also be used for intra-frame prediction. For example, the block vector (BV) used for intra-frame prediction can be used instead of the MV 717 as the prediction vector of the current block, or the MV and / or BV can be derived from the decoder, or a chained vector can be used by continuously referencing the MV and / or BV, or any other method can be used to use any type of prediction vector. The method based on the LIC technique described above according to an embodiment of the present invention can also be applied in the same way.

[0144] In the application implementation method of the present invention, the implementation method based on the method proposed in JVET technical proposal JVET-O0066 can be modified as follows and used in the inter-frame coding block, such as the inter-frame coding unit (inter-coded CU).

[0145] According to one embodiment of the present invention, adjacent samples within a frame can be used to derive the aforementioned LIC parameters. Therefore, information from coded adjacent blocks within a frame can be utilized to improve the accuracy of the LIC.

[0146] According to one embodiment of the present invention, when the number of luminance samples is less than 32 blocks, LIC can be disabled. This reduces unnecessary computation for small blocks and improves coding efficiency.

[0147] According to one embodiment of the present invention, in non-subblock and affine modes, LIC parameters can be derived based on template block samples corresponding to the current CU (instead of partial template block samples corresponding to the first 16×16 element in the upper left corner). This allows for the utilization of more information to improve the accuracy of LIC.

[0148] According to one embodiment of the present invention, samples of a reference block template can be generated using the MV and motion compensation (MC) of the corresponding block without rounding to integer pixel precision. This enables more accurate prediction, thereby improving coding efficiency.

[0149] According to one embodiment of the present invention, in the case of bi-directional interprediction (bi-directional) CU, two sets of LIC parameters can be derived for L0 and L1 prediction samples respectively. The aforementioned L0 and L1 prediction samples can refer to samples obtained respectively in the two prediction directions used in bi-directional prediction. For example, according to one embodiment, the L0 prediction sample can generally refer to a prediction sample derived from a reference image that is temporally earlier than the current image. That is, it can represent prediction information obtained from past frames. Furthermore, according to one embodiment, the L1 prediction sample can generally refer to a prediction sample derived from a reference image that is temporally later than the current image. That is, it can represent prediction information obtained from future frames.

[0150] According to one embodiment of the present invention, the L0 and L1 prediction samples can be obtained from different reference images and can be located at different time positions. As described above, by using the L0 and L1 prediction samples to derive the LIC parameters respectively, more accurate brightness compensation that takes into account temporal orientation can be provided.

[0151] According to one embodiment of the present invention, the same method can be repeatedly applied to derive the aforementioned L0 and L1 LIC parameters. Specifically, the L0 LIC parameters are first derived by minimizing the difference between the L0 template prediction T0 and the template T, where the samples of T are updated by subtracting the corresponding samples of T0. Secondly, the L1 parameters that minimize the difference between the L1 template prediction T1 and the updated template can be calculated. Finally, the L0 parameters can be further refined in the same manner.

[0152] According to another embodiment of the present invention, in the process of deriving the LIC parameters of the L0 and L1 predicted samples respectively, a weighting considering the temporal characteristics of each direction can be applied. For example, when the L0 predicted sample is temporally closer to the current image, the L0 LIC parameter can be assigned a higher weight. Conversely, when the L1 predicted sample is closer, the L1 LIC parameter can be assigned a higher weight. Thus, temporal correlation can be utilized more effectively. The present invention proposes a method and apparatus for improving compression performance by appropriately performing LIC brightness compensation.

[0153] 3×3 cross-shaped filter

[0154] According to one embodiment of the present invention, when applying the LIC method, after setting an L-shaped template from the restored pixel values ​​on the upper and left sides of the current block being encoded / decoded, coefficients of size 3×3 that can be used for cross-shaped filters are generated from the template area and used. Figure 8This is a conceptual diagram representing the shape and coefficient positions of a cross-filter according to an embodiment of the present invention. (Refer to...) Figure 8 As mentioned above, the filter 900 with a cross-shaped configuration of size 3×3 can be represented by the positions of the coefficients A (left), B (right), C (top), D (bottom), and E (middle).

[0155] According to various embodiments of the present invention, the value filtered by the above-described 3×3 cross-shaped filter can be represented by one of the following mathematical formulas.

[0156]

Mathematical Formula 1

[0157]

[0158] The above mathematical formula 1 represents a basic linear filter, which can be calculated by multiplying the pixel values ​​at each position by pre-specified coefficients (C1 to C5) and then summing them.

[0159]

Mathematical Formula 2

[0160]

[0161] Equation 2 above represents the addition of a nonlinear term F to the basic linear filtering. F can refer to the square of the center pixel E, adjusted according to the bit depth value (bitDepth) of each pixel (in this case, "bitDepth_mid" can refer to the median of the range of values ​​used to express the pixel value based on the bit depth of each pixel). The ">>" operator in Equation 2 can be understood as a right binary shift operation. Of course, the elements configuring the nonlinear term F can vary depending on the embodiment. For example, the nonlinear term F can be based on other pixel values ​​instead of E. As another example, the nonlinear term F can be defined by equations of degree three or higher. Furthermore, the constant values ​​such as "bitDepth" and "bitDepth_mid" included in the calculation of the nonlinear term can also be replaced by other suitable constant values.

[0162]

Mathematical Expression 3

[0163]

[0164] Equation 3 above represents an additional constant term G on top of Equation 2 above. G, as "bitDepth_mid", can refer to the median of the range of pixel values. Of course, according to the embodiment, the elements configuring the constant term F can vary. For example, the value of the constant term F can be replaced by other suitable constant values.

[0165] According to one embodiment of the present invention, the coefficients C1 to C7 specified in the above mathematical formulas 1 to 3 can be derived from an L-shaped template derived from the restored pixel values ​​on the upper and / or left sides of the current block. Of course, the above-described various application methods are applicable together; for the use of the above template, the coefficients can be used only for the specified entire template region or a portion of the template region to derive the above coefficients. Furthermore, according to one embodiment of the present invention, to derive the above coefficients, the coefficient derivation method of the Wiener filter using the restored pixel values ​​on the upper and / or left sides of the current block can be applied in the same or applicable way. The Wiener filter can have the characteristic of deriving coefficients in a way that minimizes the minimum mean squared error (MSE), and the above coefficients can be obtained based on this characteristic.

[0166] Obviously, this invention is not limited to the embodiments described above, and various variations and applications related to LIC and predictive coding can be applied within the scope of this invention. For example, the size of the filter described above can be adjusted, and other types of filters besides non-cross-linked filters can also be used. For example, a 5×5 cross-linked filter or a 3×3 Sobel filter can also be used. Furthermore, it is obvious that any filter configured to receive a group of pixels with a width of a first pixel and a height of a second pixel as input can be applied. In addition, extensions can be made, such as dynamically adjusting the shape or size of the template, or introducing multiple functions or nonlinear functions into the linear model described above to model more complex lighting variations, etc. Alternatively, various methods for determining the prediction mode by explicit or implicit methods can be used to implement the above examples of this invention, or the invention can be implemented in conjunction with these other methods.

[0167] Template Determination Method

[0168] The LIC-based method described above can be configured to use a pre-defined template to find a reference block and use an L-shape template to generate a linear model of the brightness variation between the current block (e.g., CU) and the reference block. This linear model can be configured as a first-order function of α×p[x]+β. p[x], as the predicted pixel value, can be defined as a reference value, and α and β can represent the slope and bias of the derived linear model. Based on existing JVET standardization activities, two or more of these linear models can be used simultaneously. According to an embodiment of the present invention, the derivation method of the linear model parameters is not limited to a first-order linear model; it can also be set to a second-order or higher line fitting model. This helps to more accurately model more complex brightness variations.

[0169] According to one embodiment of the present invention, multiple modes can be provided by improving the LIC template setting method. That is, the L-shaped template does not have to be used uniformly. For example, a mode that uses only the top or left template can be used, or a mode that uses both the top and left templates but uses each template with the same weight or different weights can be used. Figure 9 This is an example diagram illustrating an example of a template application mode according to an embodiment of the present invention. In summary, the following modes may be included and used during the LIC template setup process, but are not limited to these examples.

[0170] (a) A mode that uses only the upper pixel 815 of the current block 810 as the template.

[0171] (b) A mode that uses only the left pixel 827 of the current block 820 as the template.

[0172] (c) A mode that uses the top 835 and left 837 pixels of the current block 830 as templates with the same weighting (1:1).

[0173] (d) A pattern that uses the top 845 and left 847 pixels of the current block 840 as templates with different weights (N:M).

[0174] According to one embodiment of the present invention, the mode of using only the upper pixel of the current block as the template can use at least one pixel selected within a distance of at least one pixel above the upper edge of the current block as the template. Furthermore, according to one embodiment of the present invention, the width of the template region corresponding to the upper pixel can be expanded along the left or right edge by the number of unused left-side pixels. For example, if the size of the current block is 16×16 and no left-side pixels are used in the template, the width of the upper template region can be expanded from 16 pixels to 32 pixels.

[0175] According to one embodiment of the present invention, the mode of using only the left-hand pixels of the current block as templates can use at least one pixel selected within a distance of at least one pixel on the left side of the current block as templates. Furthermore, according to one embodiment of the present invention, the height of the template region corresponding to the left-hand pixels can be extended along the top or bottom by the number of unused top-hand pixels. For example, if the size of the current block is 16×16 and top-hand pixels are not used in the template, the height of the left-hand template region can be extended from 16 pixels to 32 pixels.

[0176] According to one embodiment of the present invention, when setting a LIC template, at least one or more modes including the above-described modes can be selected and used. In this case, the method for selecting one of the above-described modes can use at least one of the following methods.

[0177] According to one embodiment of the present invention, as a method for selecting the above-described mode, the encoder can use the following approach: using Rate-Distortion Optimization (RDO) as an evaluation criterion, selects the mode that exhibits the best bit rate-distortion (RD) performance, and informs the decoder of this mode information. According to one embodiment, it is assumed that the above-described mode is supported. Figure 9 The four template patterns in the example can be configured to perform RDO evaluation on each of the patterns (a) to (d) above, selecting the pattern with the lowest encoding cost and / or good RD performance as the LIC usage pattern for the current block, and including this information in the bitstream for transmission to the decoder. In the decoder, after parsing the bitstream, the template can be configured according to the selected LIC usage pattern. Other LIC-based prediction block-related encoding and decoding processes can be performed using the same or similar steps as existing methods.

[0178] According to one embodiment of the present invention, a template-based method can be used as a method for selecting the above-described mode. According to one embodiment, the above-described mode can be selected based on calculations of various L-shaped templates (i.e., upper and left pixels) configured with the surrounding pixels of the current block and the reference block. According to one embodiment, the optimal template mode can be selected using methods such as: using the upper restored value of the L-shaped template of the current block and the upper value of the reference block (first mode); using the left restored value of the L-shaped template of the current block and the left value of the reference block (second mode); using the upper and left restored values ​​of the L-shaped template of the current block and the upper and left values ​​of the reference block with the same weighting (third mode, same weighting); or using the upper and left restored values ​​of the L-shaped template of the current block and the upper and left values ​​of the reference block with different weightings (fourth mode, different weighting). When selecting the optimal template mode, the evaluation benchmark can be the sum of absolute differences (SAD), the sum of absolute transformed differences (SATD), or the mean-removal SAD (MR-SAD) value calculated based on the values ​​of the template (and / or the template of the template) of the current block and the reference block, and / or their average value. According to a preferred embodiment of the invention, the evaluation method described above can be configured to operate similarly in both the encoder and decoder during the encoding / decoding process, based on previously encoded / decoded pixel regions. Therefore, when applying this method, since the template mode can be implicitly determined, there is no need to record additional information about which specific mode was used in the bitstream, and the mode can be determined automatically by the decoder.

[0179] According to one embodiment of the present invention, as a method for selecting the aforementioned mode, the LIC mode can be explicitly determined using information from surrounding blocks. According to one embodiment, when usable blocks exist on the upper and left sides, the LIC mode of the current block can be explicitly determined using prediction information from usable adjacent blocks. According to one embodiment of the present invention, the explicit determination method can include configuring a candidate set and selecting at least one candidate belonging to the candidate set as the aforementioned LIC mode. Various methods can be used to configure the candidate set. For example, if there are blocks using LIC around the current LIC block, the LIC mode can be included in the candidate set. Furthermore, the usage history information of LIC modes of blocks previously used as LIC modes can be stored and included in the candidate set. Additionally, even if surrounding blocks do not use LIC, if at least one surrounding block is encoded according to an intra-angular prediction mode, the angle of the angular prediction mode can be considered, and a suitable LIC mode can be included in the candidate set. For example, if the angular prediction mode has an angle of a horizontal system, then... Figure 9 Part (b) mainly references the patterns of pixels in the horizontal direction as LIC patterns more suitable for that direction and includes them in the candidate set. Furthermore, information related to the inter-frame prediction method can also be considered when configuring the candidate set. According to an embodiment of the present invention, when configuring the candidate set, if a specific number of candidate set values ​​cannot be selected, the patterns of the candidate set can be arbitrarily increased according to a predetermined order or information. According to an embodiment of the present invention, after the candidate set is configured, the order of the candidate set can be re-ordered using the template (or, a template of a template) (e.g., top row, left row, top and left row, etc.). The re-ordering can be performed based on the cost required for the encoded information calculated based on the template matching results. According to an embodiment, the re-ordering can be performed in a low-cost order. According to an embodiment, the cost required for the information can be calculated using SAD, SATD, or MR-SAD, and the re-ordering can also be based on this cost.

[0180] Once the candidate set is finalized, the optimal LIC mode can be selected via RDO or similar methods, and its index can be included in the bitstream and transmitted to the decoder. The decoder can be configured to configure the candidate set in the same way as the encoder and reconstruct the LIC mode used in the encoder using the information transmitted through the bitstream.

[0181] According to one embodiment of the present invention, as a method for selecting the above-mentioned mode, the LIC mode can be implicitly determined using information from surrounding blocks. According to one embodiment, when usable blocks exist on the upper and left sides, the LIC mode of the current block can be implicitly determined using usable surrounding prediction information. For example, when the current block is encoded using LIC, if there are surrounding blocks using LIC, the LIC mode of the current block can be determined based on the frequency of the LIC modes used by the surrounding blocks. Furthermore, even if surrounding blocks do not use LIC, if at least one surrounding block is encoded according to an intra-angular prediction mode, the angle of the aforementioned angular prediction mode can be considered to determine a suitable LIC mode. For example, if the aforementioned angular prediction mode has an angle of a horizontal system, then... Figure 9 (b) The LIC mode of the current block is determined based on the pattern of the pixels in the primary reference horizontal direction. According to a preferred embodiment of the invention, the above-described LIC mode selection method can be configured to operate similarly in the encoder and decoder during encoding / decoding based on previously encoded / decoded pixel regions. Therefore, when this method is applied, since there is no need to record additional information about which specific mode was used in the bitstream, it can be configured so that the decoder automatically determines the LIC mode.

[0182] According to an embodiment of the present invention, it is obvious that in the method of deriving the decoder-side mode using the implicit determination method in the above embodiments, various well-known or newly provided decoder-side mode derivation methods can be used or applied.

[0183] According to one embodiment of the present invention, in order to set the LIC template, a mode that uses the same weighting for both the top and left sides, or a mode that uses different weightings for the top and left sides, can be defined and used. Various methods can be used to define the aforementioned weighting. According to one embodiment, the same weighting used in existing conventional LICs can be applied for use when generating α and / or β of the aforementioned linear model. According to another embodiment, the top and left regions can be divided, and weighting can be applied separately to each region for use when generating the aforementioned α and / or β. According to one embodiment of the present invention, for the weighting of each of the aforementioned divided regions, values ​​determined in advance through experiments can be used, and the determined weighting values ​​can be tableded for use. Furthermore, an index pointing to a specific value in the table can be determined explicitly or implicitly and provided to the decoder.

[0184] According to one embodiment of the present invention, the above table can preferably be specified as a power of 2 for the sum of a weighted set. The purpose is to facilitate the use of shift operations instead of division operations in weighted summation operations. For example, if the weighted set is {3, 5}, then the sum is 8(2^3). 3 Therefore, the division operation required for the above weighted set can be replaced by a 3-bit right binary shift operation.

[0185] According to one embodiment of the present invention, the aforementioned weighting value can be determined on-the-fly using information from surrounding blocks. It is readily understood that this method belongs to an implicit signaling method. For example, when there are usable blocks on the top and left sides, the weighting value can be determined using the inter-frame prediction information, intra-frame prediction information, and block size information of the usable surrounding blocks. Similarly, in this case, for the purpose of facilitating the replacement of division operations with shift operations, the weighting value can be determined as a power of 2.

[0186] According to one embodiment of the present invention, in order to reduce the complexity of the method when selecting one of the above-mentioned modes when setting the above-mentioned LIC template, a method can be applied to limit the pixels used for the template to a portion of the pixels within the template area rather than all the pixels, thereby reducing their number. According to one embodiment, when generating the above-mentioned α and β after finding the reference block, only a portion of the pixels may be used instead of all the pixels. According to one embodiment, when finding the reference block, all the pixels of the template may be additionally used, while when generating the above-mentioned α and β, only a portion of the pixels may be used instead of all the pixels. According to one embodiment, when finding the reference block, pixels of a reduced template containing only a portion of the pixels rather than all the pixels may be additionally used, while when generating the above-mentioned α and β, all the pixels may be used.

[0187] As mentioned above, methods to reduce the number of pixels used in the template to only use a portion rather than all of them in order to reduce complexity may include, but are not limited to, the following methods.

[0188] According to one embodiment of the present invention, when using the entire template of the upper side, the left side, or the upper side and the left side, the number of pixels can be reduced by using the subsampling method corresponding to multiples of 2 such as 2:1, 4:1, etc., and the reduced pixels can be used to find reference blocks or generate the above-mentioned α and β as described above.

[0189] According to one embodiment of the present invention, when using only the upper template, the number of pixels is reduced by dividing the entire area of ​​the upper template into N equal parts in the horizontal direction and selecting M pixels. These reduced pixels are then used for finding reference blocks or generating α and β as described above. According to the embodiment, N and M can vary depending on the block size and the desired level of complexity. According to a preferred embodiment of the present invention, at least one of N or M can be defined as a power of 2 for ease of computation.

[0190] For example, the entire upper template area can be divided into four equal parts horizontally, and two pixels corresponding to the 1 / 4 and 3 / 4 regions can be selected. These selected pixels are then used to find reference blocks or generate the aforementioned α and β. As another example, the entire upper template area can be divided into eight equal parts horizontally, and three pixels corresponding to the 2 / 8, 4 / 8, and 6 / 8 regions can be selected. These selected pixels are then used to find reference blocks or generate the aforementioned α and β. As yet another example, the entire upper template area can be divided into eight equal parts horizontally, and four pixels corresponding to the 1 / 8, 3 / 8, 5 / 8, and 7 / 8 regions can be selected. These selected pixels are then used to find reference blocks or generate the aforementioned α and β. As yet another example, when using only the upper template, the entire upper template area can be divided into eight equal parts horizontally, and five pixels corresponding to the 0 / 8, 2 / 8, 4 / 8, 6 / 8, and 8 / 8 regions can be selected. These selected pixels are then used to find reference blocks or generate the aforementioned α and β.

[0191] According to one embodiment of the present invention, when using only the left template, the number of pixels is reduced by dividing the entire area of ​​the left template into N equal parts in the vertical direction and selecting M pixels. These reduced pixels are then used for finding reference blocks or generating α and β as described above. According to the embodiment, N and M can vary depending on the block size and the desired level of complexity. According to a preferred embodiment of the present invention, at least one of N or M can be defined as a power of 2 for ease of computation.

[0192] For example, the entire left template area can be vertically divided into 4 equal parts, and 2 pixels corresponding to the 1 / 4 and 3 / 4 regions can be selected. These selected pixels can then be used to find reference blocks or generate the aforementioned α and β. As another example, the entire left template area can be vertically divided into 8 equal parts, and 3 pixels corresponding to the 2 / 8, 4 / 8, and 6 / 8 regions can be selected. These selected pixels can then be used to find reference blocks or generate the aforementioned α and β. As yet another example, the entire left template area can be vertically divided into 8 equal parts, and 4 pixels corresponding to the 1 / 8, 3 / 8, 5 / 8, and 7 / 8 regions can be selected. These selected pixels can then be used to find reference blocks or generate the aforementioned α and β. As yet another example, the entire left template area can be vertically divided into 8 equal parts, and 5 pixels corresponding to the 0 / 8, 2 / 8, 4 / 8, 6 / 8, and 8 / 8 regions can be selected. These selected pixels can then be used to find reference blocks or generate the aforementioned α and β. In the various methods described above, the aforementioned entire upper template can refer to an upper template formed by extending the width of the current block to the left or right by a size equal to the height of the current block. Similarly, the aforementioned entire left template can refer to a left template formed by extending the height of the current block upwards or downwards by a size equal to the width of the current block.

[0193] Encoder and decoder

[0194] Clearly, the encoding method according to the invention can be equally applied to both the encoder and the decoder. The encoding method according to the invention can first be used as one of the methods for generating sample values ​​of prediction blocks in the encoder for intra-frame and / or inter-frame prediction. When residual signal information is encoded by the difference value with that prediction block, the decoder can generate sample values ​​of the same prediction block using a corresponding and / or symmetrical method, and obtain decoded samples by combining the difference value with that prediction block. Further, as... Figure 4 As shown in the diagram, this decoding process can also be implemented inside the encoder to predict the state of the decoder, through the internal decoder 420 and the encoding loop including it.

[0195] Furthermore, the encoding method according to the present invention described above can be implemented using an encoder as a device. The encoder, as a device, can maintain the previously passed... Figures 1 to 6 The existing encoder structure shown, or its implementation in a form that specifies its application, is not necessarily limited to the example shown. Any encoder structure that can be used as a video encoder as long as it implements the technical idea of ​​the present invention should be regarded as an encoder according to the present invention.

[0196] Furthermore, the decoding method for the encoding result according to the present invention can be implemented using a decoder as a device. The decoder, as a device, can maintain the previously obtained... Figures 1 to 6The existing decoder structures shown, or those implemented in a form that specifies their application, are not necessarily limited to the examples shown. Any form of decoder structure that can be used as a video decoder as long as it implements the technical idea of ​​the present invention should be regarded as a decoder according to the present invention.

[0197] Those skilled in the art will readily understand that the bitstream encoded by the above-described method and apparatus can be decoded by applying a method symmetrical to and / or opposite to the above-described encoding method. In one embodiment, when reading information for decoding from the above-described encoded bitstream, at least one variable-length coding syntax contained in the above-described encoded bitstream can be parsed. Furthermore, in one embodiment, the above-described variable-length coding can be implemented using an entropy coding method. The technical details and application methods for implementing this decoding step can be readily understood from the above-described encoding steps.

[0198] The encoders and / or decoders described in this specification are devices that include a processor and a memory, and preferably correspond to computing (operational) devices. The processors that may be included in the encoders and / or decoders described in this specification can refer to one or more general-purpose or special-purpose computers, such as processors, controllers, arithmetic logic units (ALUs), digital signal processors (DSPs), microcomputers, field-programmable gate arrays (FPAs), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions.

[0199] For ease of understanding, even though the processor is referred to in the singular, those skilled in the art will understand that the processor may include multiple processing elements and / or various types of processing elements. For example, an apparatus according to an embodiment of the present invention may include multiple processors, or a processor and a controller, as the processor 130 described above. Furthermore, the processor may be implemented using various processing configurations, such as a parallel processor or a multi-core processor.

[0200] The processor described above can be configured to execute an operating system (OS) and one or more software programs running on the operating system. Furthermore, the processor can access, store, manipulate, process, and generate data in response to the execution of the software.

[0201] The aforementioned software may include computer programs, code, instructions, or a combination of more than one of these, and may be configured to control a processing device as needed, or to provide independent or collective instruction control over the processing device. The aforementioned software may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, for parsing by the aforementioned processor, or for providing instructions or data to the aforementioned processor. The aforementioned software may be distributed across multiple computer systems connected to the aforementioned network and stored or executed in a distributed manner.

[0202] The aforementioned software can also be implemented in the form of program instructions executable by various computer means, and recorded or stored in the aforementioned memory. The aforementioned memory can be a computer-readable recording medium, in which program commands, data files, and data structures can be recorded individually or in combination. The program commands stored in the aforementioned memory can be implemented based on a command system specifically designed and configured for embodiments of the present invention, or can follow command systems known and available to those skilled in the art of computer software, such as command systems for assembly language, C, C++, Java, Python, etc. It should be understood that the aforementioned command system and the program commands it generates include not only machine language code generated by a compiler, but also high-level language code that can be executed by an interpreter or other means of apparatus and / or the aforementioned processor according to an embodiment of the present invention.

[0203] The computer-readable recording medium constituting an embodiment of the apparatus of the present invention includes the storage described above, and may include: temporary or volatile recording media such as processor cache, random access memory (RAM), flash memory, etc., which are maintained only during the operation of the processor; or relatively non-volatile or long-term recordable recording media such as hard disk, floppy disk, magnetic tape, etc.; optical recording media such as CD-ROM, DVD, etc.; magneto-optical media such as floppy disk; or solid state memory; or read-only recording media such as read-only memory (ROM) provided in hardware. Furthermore, since the circuit wiring is configured to execute a series of program instructions and equivalent actions through a hard-wired structure, it can be considered that the various steps involved in executing the actions described in the embodiments of the present invention can be regarded as being recorded through the connection and arrangement of the aforementioned hardware components. Therefore, it is obvious to those skilled in the art that this connection and arrangement method can be regarded as equivalent to the aforementioned memory.

[0204] The above embodiments regarding the processor and memory are not exclusive and can be selected or combined as needed. For example, a hardware device can be configured to execute the actions of an embodiment of the present invention as a module composed of one or more of the above-described software, and vice versa. As another example, in this specification, all or part of the actions assigned to a certain functional unit can be implemented by one or more of the above-described software stored in a device according to an embodiment of the present invention (preferably stored in a recording medium belonging to the scope of the above-described memory), and configured to be executed by the above-described processor. In this case, such a functional unit can be referred to as a functional unit "included" in the above-described processor.

[0205] The present invention has been described above with reference to the accompanying drawings and embodiments. However, as stated above, this does not mean that the scope of protection of the present invention is limited to the drawings or embodiments disclosed above. Those skilled in the art should understand that various modifications and changes can be made to the present invention without departing from the inventive spirit and scope set forth in the claims of this patent.

Claims

1. A decoding method, as a decoding method for an encoded video bitstream, comprising: The step of obtaining at least one prediction vector from the bitstream for the current block being decoded; The step of determining at least one reference block based on the prediction vector; The step of defining at least one of the following templates: a first template that uses the current block as a reference block and includes at least one pixel adjacent to the current block; And a second template, which uses the reference block as a base block and includes at least one pixel adjacent to the reference block; The step of extracting parameter information related to local brightness compensation by referring to at least one pixel information derived from at least one of the first template and the second template; Based on the parameter information, the step of determining at least one coefficient applied to the first filter; The step of performing filtering on the reference block using the first filter with the applied coefficients to obtain the filtering result value; The step of generating a predicted sample for the current block based on the filtered result value; as well as The step of reconstructing the current block using the predicted samples.

2. The decoding method according to claim 1, characterized in that, The first filter is a cross filter configured to receive at least one left pixel, at least one right pixel, at least one top pixel, at least one bottom pixel, and a center pixel as input; and has at least one coefficient that can be applied to each of the pixels.

3. The decoding method according to claim 2, characterized in that, The first filter is a 3×3 cross filter configured to receive five pixels as input, including a left pixel, a right pixel, a top pixel, a bottom pixel, and a center pixel; and has five coefficients that can be applied to each of the pixels.

4. The decoding method according to claim 3, characterized in that, The first filter operates according to the following mathematical formula: Wherein, Pred is the filtered result value of the first filter, A represents the input value of the left pixel, B represents the input value of the right pixel, C represents the input value of the top pixel, D represents the input value of the bottom pixel, E represents the input value of the center pixel, and C1, C2, C3, C4, and C5 represent the coefficients applied to each of the inputs.

5. The decoding method according to claim 3, characterized in that, The first filter operates according to the following mathematical formula: Wherein, Pred is the filtered result value of the first filter, A represents the input value of the left pixel, B represents the input value of the right pixel, C represents the input value of the top pixel, D represents the input value of the bottom pixel, E represents the input value of the center pixel, F represents the nonlinear term based on E, and V... mid It is the additive median constant, the V div It is the equalization division constant, and C1, C2, C3, C4, C5, and C6 represent the coefficients applied to each of the inputs.

6. The decoding method according to claim 5, characterized in that, The V mid This represents the median bit depth ("bitDepth_mid") of the pixels for the current block, and is determined by the V... div The division operation was replaced by a right binary shift operation (">>bitDepth").

7. The decoding method according to claim 3, characterized in that, The first filter operates according to the following mathematical formula: Wherein, Pred is the filtered result value of the first filter, A represents the input value of the left pixel, B represents the input value of the right pixel, C represents the input value of the top pixel, D represents the input value of the bottom pixel, E represents the input value of the center pixel, F represents the nonlinear term based on E, and V... mid It is the additive median constant, the V div It is the equalization division constant, the V const These are batch addition constants, and C1, C2, C3, C4, C5, C6, and C7 represent coefficients applied to each of the inputs.

8. The decoding method according to claim 7, characterized in that, The V mid and the V const This represents the median bit depth ("bitDepth_mid") of the pixels for the current block, and is determined by the V... div The division operation was replaced by a right binary shift operation (">>bitDepth").

9. The decoding method according to claim 1, characterized in that, The first filter is an image processing filter configured to receive as input a group of pixels centered at a center pixel and having a width of a first pixel and a height of a second pixel, comprising at least one shape of a quadrilateral, a rhombus, and a circle; and having at least one coefficient applicable to each of the pixels.

10. The decoding method according to claim 1, characterized in that, The step of determining at least one coefficient applied to the first filter is performed by a Wiener filter-based minimum mean square error minimization method, which is calculated based on at least one of the restored pixel values ​​on the upper and left sides of the current block.

11. An encoding method for encoding a bitstream generated from video, comprising: The step of determining at least one prediction vector for the current block being encoded; The step of determining at least one reference block based on the prediction vector; The step of defining at least one of the following templates: a first template that uses the current block as a reference block and includes at least one pixel adjacent to the current block; And a second template, which uses the reference block as a base block and includes at least one pixel adjacent to the reference block; The step of extracting parameter information related to local brightness compensation by referring to at least one pixel information derived from at least one of the first template and the second template; Based on the parameter information, the step of determining at least one coefficient applied to the first filter; The step of performing filtering on the reference block using the first filter with the applied coefficients to obtain the filtering result value; The step of generating a predicted sample for the current block based on the filtered result value; The step of encoding the current block using the predicted samples; as well as, The step of recording the encoded result into the bit stream.

12. The encoding method according to claim 11, characterized in that, The first filter is a cross filter configured to receive at least one left pixel, at least one right pixel, at least one top pixel, at least one bottom pixel, and a center pixel as input; and has at least one coefficient that can be applied to each of the pixels.

13. The encoding method according to claim 12, characterized in that, The first filter is a 3×3 cross filter configured to receive five pixels as input, including a left pixel, a right pixel, a top pixel, a bottom pixel, and a center pixel; and has five coefficients that can be applied to each of the pixels.

14. The encoding method according to claim 13, characterized in that, The first filter operates according to the following mathematical formula: Wherein, Pred is the filtered result value of the first filter, A represents the input value of the left pixel, B represents the input value of the right pixel, C represents the input value of the top pixel, D represents the input value of the bottom pixel, E represents the input value of the center pixel, and C1, C2, C3, C4, and C5 represent the coefficients applied to each of the inputs.

15. The encoding method according to claim 13, characterized in that, The first filter operates according to the following mathematical formula: Wherein, Pred is the filtered result value of the first filter, A represents the input value of the left pixel, B represents the input value of the right pixel, C represents the input value of the top pixel, D represents the input value of the bottom pixel, E represents the input value of the center pixel, F represents the nonlinear term based on E, and V... mid It is the additive median constant, the V div It is the equalization division constant, and C1, C2, C3, C4, C5, and C6 represent the coefficients applied to each of the inputs.

16. The encoding method according to claim 15, characterized in that, The V mid This represents the median bit depth ("bitDepth_mid") of the pixels for the current block, and is determined by the V... div The division operation was replaced by a right binary shift operation (">>bitDepth").

17. The encoding method according to claim 13, characterized in that, The first filter operates according to the following mathematical formula: Wherein, Pred is the filtered result value of the first filter, A represents the input value of the left pixel, B represents the input value of the right pixel, C represents the input value of the top pixel, D represents the input value of the bottom pixel, E represents the input value of the center pixel, F represents the nonlinear term based on E, and V... mid It is the additive median constant, the V div It is the equalization division constant, the V const These are batch addition constants, and C1, C2, C3, C4, C5, and C6 represent coefficients applied to each of the inputs.

18. The encoding method according to claim 17, characterized in that, The V mid and the V const This represents the median bit depth ("bitDepth_mid") of the pixels for the current block, and is determined by the V... div The division operation was replaced by a right binary shift operation (">>bitDepth").

19. The encoding method according to claim 11, characterized in that, The step of determining at least one coefficient applied to the first filter is performed by a Wiener filter-based minimum mean square error minimization method, which is calculated based on at least one of the restored pixel values ​​on the upper and left sides of the current block.

20. A decoder apparatus configured to decode an encoded video bitstream by a computing device, comprising: The receiving unit is used to receive bit streams; The output section is used to output the decoded video. A reference buffer is used to store at least one decoded image information; processor; as well as, Memory for storing instructions that can be executed by the processor, wherein, The instructions are configured to include the following: instructions to obtain at least one prediction vector from the bitstream for the currently being decoded block; instructions to determine at least one reference block based on the prediction vector; instructions for defining at least one template: a first template that uses the current block as a reference block and includes at least one pixel adjacent to the current block, and a second template that uses the reference block as a reference block and includes at least one pixel adjacent to the reference block; instructions to extract parameter information related to local brightness compensation by referring to at least one pixel information derived from at least one of the first template and the second template; instructions to determine at least one coefficient applied to a first filter based on the parameter information; instructions to perform filtering on the reference block using the first filter with the applied coefficient and obtain a filtering result value; instructions to generate a prediction sample for the current block based on the filtering result value; and instructions to reconstruct the current block using the prediction sample.