Enhanced gradual decoder refresh method and device, for video encoding and decoding

Abstract

The present invention relates to a video compression technique, and specifically to an enhanced application method for and utilization of a gradual decoder refresh (GDR) technique contributing to bit rate stabilization in a video encoder and decoder. A video encoding method including a GDR function, according to one embodiment of the present invention, comprises the steps of: setting a frame length of a GDR period in which GDR is to be performed; specifying, in at least one frame belonging to the GDR period, a refresh area including at least one image encoding unit; performing intra encoding for the at least one image encoding unit included in the refresh area; and recording the result of the encoding in a bit string, wherein all spatial areas within a video resolution in the GDR period are intra encoded at least once.

Description

Improved stepwise decoder updating method and device for video encoding and decoding

[0001] The present invention relates to video compression technology, and more particularly, to an improved application method and utilization of a gradual decoder referh technique that contributes to stabilization of a bit rate in a video encoder and decoder.

[0002] The present invention may be in the same technical field as at least one of the digital video compression technology standards known by the standard names such as MPEG-2, MPEG-4 Video, H.263, H.264 / AVC, H.265 / HEVC, H.266 / VVC, VC-1, AV-1, QuickTime, VP-9, VP-10, Motion JPEG, or may be in the technical field for improving the inherent efficiency of the standard, or may be in the technical field for improving or replacing the standard.

[0003] Digital video encoding and decoding are widely used in various digital video applications. For example, digital television broadcasting, video transmission through communication networks, video calls / video conversations / video chats, recording and providing video content using optical media including video compact discs (VCDs) / digital versatile discs (DVDs) / Blu-Rays, all processes for producing, editing, collecting, and distributing video content, and devices such as video recording devices and camcorders for shooting and recording video for various reasons including personal, commercial, industrial, and security purposes, all depend on video encoding and decoding technologies.

[0004] Accordingly, implementations that may be referred to as digital video encoders and decoders may form part of a wide range of devices related to the generation, recording, and provision of digital video, including digital televisions, digital broadcasting systems, wireless broadcasting systems, computers in the form of notebooks / desktops / tablets, e-book readers, digital cameras, digital recording devices, digital multimedia playback devices, video game devices / terminals / consoles, mobile phones (including smartphones) with multimedia playback capabilities, equipment for video conferencing, and other devices.

[0005] The above digital video encoders and decoders can be implemented by a digital video compression standard that is widely used and understood by those skilled in the art. The digital video compression standard may include at least one of compression standards known by a standard name such as MPEG-2, MPEG-4 Video, H.263, H.264 / AVC, H.265 / HEVC, H.266 / VVC, VC-1, AV1, QuickTime, VP-9, VP-10, and Motion JPEG.

[0006] Video encoders and decoders can be implemented to more efficiently encode or decode digital video information while complying with the above standards, or by improving or modifying the above standards. Attempts to modify the above standards can also lead to the development of new standards. A well-known example is the so-called enhanced compression model (ECM), an attempt to improve and replace the existing H.266 / VVC standard, currently being developed by the Joint Video Experts Team (JVET), a joint international standardization group of ISO, IEC, and ITU-T.

[0007] Among the various detailed technologies applied to conventional video encoders and decoders, there is a technology collectively called gradual decoder refresh (GDR). GDR is a technology proposed to improve the problem that the information content of random access frames, previously known as "intra frames," is much higher than that of highly compressed "inter frames" based on temporal prediction, and thus causes rapid bit rate changes when transmitted over ultra-high-speed communication networks, which hinders the guarantee of ultra-reliable low-latency communication (URLLC) performance.

[0008] Conventional GDR technology has had limitations, with its uniformly limited refresh direction to either horizontal or vertical, resulting in a refresh direction that is inappropriate for the actual video content, significantly reducing encoding efficiency. However, with the widespread adoption of URLLC communication, the need for easier GDR implementation is increasing. Therefore, a method is needed that improves upon the conventional, uniform GDR technology application while minimizing the additional data inserted due to the information used to dictate the refresh method.

[0009] According to an embodiment of the present invention for solving the above-described technical problem, a video encoding method including a gradual decoder refresh (GDR) function comprises the steps of: setting a frame length of a GDR section in which a gradual decoder refresh is to be performed; designating an update area including at least one picture coding unit in at least one frame belonging to the GDR section; performing intra-coding on at least one picture coding unit included in the update area; and recording a result of the encoding in a bit string, wherein all spatial areas within a video resolution within the GDR section are intra-coded at least once.

[0010] The step of specifying the above update area may include a step of specifying the update area from a spatial area derived by a function that uses a frame number as a variable.

[0011] The step of specifying the above update area may include a step of specifying the update area from a spatial area excluding an area derived from a function that inputs the n-1th frame number from an area derived from a function that inputs the nth frame number.

[0012] The above function may be characterized in that it is repeatedly executed for each image encoding unit, and each time it is executed, calculations are performed including the spatial position of the image encoding unit, the number of the current frame, and the length of the GDR section.

[0013] The above function may be characterized in that it is defined by normalization based on a lower limit and an upper limit, and the function may be characterized in that it is projected onto the frame to be encoded in order to derive the update area from the normalized function.

[0014] The lower limit is 0, the upper limit is 1, and the projection method may be characterized by including at least one of a projection that maintains the aspect ratio, a projection that does not maintain the aspect ratio, and an iterative projection.

[0015] An encoding method, characterized in that the above function is a function of a line whose area can be derived by integration, or a function of a closed curve whose area can be calculated.

[0016] The above function may be characterized in that it is applied by being transformed by at least one operator among an operator that inverts an area to be judged by area, a rotation operator, a skew operator, a scalaing operator, and an affine transform operator.

[0017] The above encoding method may further include a step of recording information related to the type of the function in the encoding bit string.

[0018] Information related to the type of the above function may be characterized as information indicating an index of the function selected from a list of functions.

[0019] Information related to the type of the above function may be characterized by including at least one of a reference point, an angle, a magnification, a tilt degree, an inward / outward transformation, and an affine transformation parameter for application of the above function.

[0020] Information related to the type of the above function may be characterized as information representing a mathematical expression of the above function.

[0021] Information related to the type of the above function may be characterized by being entropy encoded.

[0022] The step of specifying the above update area may include a step of specifying the update area from a spatial area specified by a change pattern whose order of change is determined according to the progress of the frame.

[0023] The above change pattern may be characterized by being defined by at least one bitmap image.

[0024] The above change pattern may be characterized by being defined by at least one vector graphic.

[0025] The above change pattern may be characterized in that it has a fixed resolution and is used by being upscaled or downscaled from the fixed resolution to the resolution of the actually applied video frame.

[0026] The above change pattern may be characterized in that the shape to be changed for each frame is independently defined in time order.

[0027] The above change pattern may be characterized in that the shape of the pattern to be changed for each frame is defined as a residual from a temporally preceding pattern.

[0028] The above change pattern may be characterized in that it is applied by being transformed by at least one operator among an operator that inverts an area to be judged by area, a rotation operator, a skew operator, a scalaing operator, and an affine transform operator.

[0029] The above encoding method may further include a step of recording information related to the type of the change pattern in the encoding bit string.

[0030] Information related to the type of the above change pattern may be characterized as information indicating an index of the above change pattern selected from a list of change patterns.

[0031] Information related to the type of the above change pattern may be characterized by including at least one of a reference point, an angle, a magnification, a tilt degree, an inner / outer transformation, and an affine transformation parameter for application of the above change pattern.

[0032] Information related to the type of the above change pattern may be characterized as information indicating at least one image constituting the above change pattern.

[0033] Information related to the type of the above change pattern may be characterized by being entropy encoded.

[0034] The step of specifying the above update area may include a step of specifying a spatial area so that all spatial areas within the video resolution within the GDR section can be intra-coded at least once.

[0035] The step of specifying the above update area may be characterized by specifying that, when the GDR section has a length of m frames, at least 1 / m of the spatial area can be intra-coded for each frame.

[0036] The above encoding method may further include a step of storing, in the encoder, cumulative update region information representing a spatial region that has been intra-coded at least once within the GDR section.

[0037] The step of designating the above update area may be characterized by designating the entire spatial area not included in the accumulated update area information in the GDR area as the update area when it is the last frame of the GDR area.

[0038] The step of specifying the above update area may include a step of specifying that at least one inter-encoding target image coding unit be encoded by changing it into an intra-encoding target image coding unit.

[0039] The step of changing and designating may be characterized in that it operates based on the encoding efficiency of the inter-encoding target image encoding unit, and is configured to change and designate when the encoding efficiency of the inter-encoding target image encoding unit is low.

[0040] The above encoding efficiency may be characterized in that it is predicted from at least one of encoding information including a division level of a coding tree unit (CTU), a distribution of DCT coefficient values, an encoding mode set for a surrounding image encoding unit, and an encoding result of an already encoded area.

[0041] The image encoding unit belonging to the above update area may be characterized in that it is encoded in a manner that does not refer to an image encoding unit outside the above update area.

[0042] According to an embodiment of the present invention for solving the above-described technical problem, a video decoding method including a gradual decoder refresh (GDR) function comprises the steps of: receiving an encoded bit stream; reading at least one frame constituting a GDR section from the bit stream; intra-decoding at least one image coding unit included in at least one frame belonging to the GDR section; and recording the intra-decoded result in an output buffer, wherein all spatial areas within a video resolution within the GDR section are intra-decoded at least once.

[0043] The above intra-decoded image area may be characterized in that it is included in a function area corresponding to a spatial area derived by a function that uses the frame number as a variable for each frame within the GDR section.

[0044] The above decryption method can be executed based on at least one function pre-stored in the decryptor.

[0045] The above decryption method may further include a step of reading function area information for at least one frame belonging to the GDR section from the bit string.

[0046] The above function area information may include information indicating the index of a function selected from a list of functions.

[0047] The above function domain information may include at least one of a reference point, an angle, a magnification, a tilt degree, an inner / outer transformation, and an affine transformation parameter for application of the function.

[0048] The above function domain information may include a mathematical expression of the function.

[0049] The above function domain information may be characterized by being entropy decoded.

[0050] The above intra-decoded image area may be characterized in that it is included in a change pattern area corresponding to a spatial area specified by a change pattern in which the order of change is determined according to the progression of frames in each frame within the GDR section.

[0051] The above decryption method can be executed based on at least one change pattern stored in advance in the decryptor.

[0052] The above decryption method may further include a step of reading change pattern area information for at least one frame belonging to the GDR section from the bit string.

[0053] The above change pattern area information may include information indicating an index of the change pattern selected from a list of change patterns.

[0054] The above change pattern area information may include at least one of a reference point, an angle, a magnification, a tilt degree, an inner / outer transformation, and an affine transformation parameter for applying the above change pattern.

[0055] The above change pattern area information may include information representing at least one image constituting the change pattern.

[0056] The above change pattern area information may be characterized by being entropy decoded.

[0057] The above intra-decoded image area may be characterized in that, for each frame within the GDR section, when the GDR section has a length of m frames, each frame includes a spatial area of ​​at least 1 / m.

[0058] The above intra-decoded image coding unit may be characterized in that it is decoded in a manner that does not refer to an image coding unit that does not belong to the same frame or is not intra-coded.

[0059] According to the present invention, by applying an improved GDR technology with minimal data consumption, it is expected that the advantageous effects of providing performance such as effective random access and subjective image quality while maintaining an appropriate bit rate compared to similar technologies in the video encoding and decoding field can be expected.

[0060] Figure 1 is a conceptual diagram of a video communication system according to one embodiment of the present invention;

[0061] FIG. 2 is a conceptual diagram of the arrangement of an encoder and decoder in a real-time video streaming environment according to one embodiment of the present invention.

[0062] Figure 3 is a functional unit conceptual diagram of a video decoder according to one embodiment of the present invention;

[0063] Figure 5 is a conceptual diagram of a frame type according to one embodiment of the present invention;

[0064] Figure 6 is a conceptual diagram showing the structure of a video encoder according to the H.266 / VVC standard.

[0065] Figure 7 is a conceptual diagram showing the application method of GDR technology.

[0066] Figure 8 is a conceptual graph showing the expected effect on bit rate for the application of GDR technology.

[0067] Figure 9 is a conceptual diagram showing the impact on the encoding / decoding process due to the application of GDR technology.

[0068] FIG. 10 is an example of providing frame information in a GDR section according to one embodiment of the present invention.

[0069] Figure 11 is a conceptual diagram of region division by function according to one embodiment of the present invention.

[0070] Figure 12 is another conceptual diagram of area division by function according to one embodiment of the present invention.

[0071] Figure 13 is an example of a method for dividing a region by a function according to some embodiments of the present invention.

[0072] Figure 14 is a conceptual diagram of area division by pattern according to one embodiment of the present invention.

[0073] Figure 15 is an exemplary diagram showing various forms of patterns according to one embodiment of the present invention.

[0074] FIG. 16 is an exemplary diagram showing a method of projecting an image of a pattern according to various embodiments of the present invention;

[0075] FIG. 17 is an example of GDR implementation by forcing intra coding according to one embodiment of the present invention, and

[0076] Fig. 18 is a flowchart illustrating a method for forcing intra encoding according to one embodiment of the present invention.

[0077] The present invention is susceptible to various modifications and embodiments. Specific embodiments are illustrated and described in detail in the drawings. However, this is not intended to limit the present invention to specific embodiments, but rather to encompass all modifications, equivalents, and alternatives falling within the spirit and technical scope of the present invention.

[0078] While terms such as "first" and "second" may be used to describe various components, these components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present invention, the first component could be referred to as the "second component," and similarly, the second component could also be referred to as the "first component." The term "and / or" includes any combination of multiple related items described herein or any of multiple related items described herein, and is non-exclusive unless otherwise indicated. The listing of items in this application is merely an exemplary description to facilitate the spirit and possible implementation methods of the present invention, and therefore is not intended to limit the scope of embodiments of the present invention.

[0079] As used herein, "A or B" can mean "only A," "only B," or "both A and B." In other words, as used herein, "A or B" can be interpreted as "A and / or B." For example, as used herein, "A, B or C" can mean "only A," "only B," "only C," or "any combination of A, B and C."

[0080] As used herein, a slash ( / ) or a comma can mean "and / or." For example, "A / B" can mean "A and / or B." Accordingly, "A / B" can mean "only A," "only B," or "both A and B." For example, "A, B, C" can mean "A, B, or C."

[0081] In this specification, "at least one of A and B" may mean "only A", "only B" or "both A and B". Additionally, in this specification, the expressions "at least one of A or B" or "at least one of A and / or B" may be interpreted identically to "at least one of A and B".

[0082] Additionally, in this specification, “at least one of A, B and C” can mean “only A,” “only B,” “only C,” or “any combination of A, B and C.” Additionally, “at least one of A, B or C” or “at least one of A, B and / or C” can mean “at least one of A, B and C.”

[0083] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0084] The terminology used in this application is only used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprise" or "have" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0085] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by those of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning within the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0086] In describing the invention in this application, embodiments may be described or illustrated in terms of unit blocks that perform the described function or functions. The blocks may be expressed in this application as one or more devices, units, modules, parts, etc. The blocks may be implemented in hardware by one or more logic gates, integrated circuits, processors, controllers, memories, electronic components, or information processing hardware implementation methods, but not limited thereto. Alternatively, the blocks may be implemented in software by application software, operating system software, firmware, or information processing software implementation methods, but not limited thereto. A single block may be implemented by being separated into multiple blocks that perform the same function, or conversely, a single block may be implemented to perform the functions of multiple blocks simultaneously. The blocks may also be implemented by being physically separated or combined according to any criteria. The blocks may also be implemented to operate in an environment where their physical locations are not specified and are separated from each other by a communication network, the Internet, a cloud service, or a communication method, but not limited thereto. All of the above implementation methods are within the realm of various embodiments that can be taken by a person skilled in the field of information and communication technology to implement the same technical idea, and therefore, any detailed implementation method should be interpreted as being included within the technical idea scope of the invention of the present application.

[0087] Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the attached drawings. To facilitate a comprehensive understanding of the present invention, the same reference numerals will be used for identical components in the drawings, and redundant descriptions of identical components will be omitted. Furthermore, the multiple embodiments are not mutually exclusive, and it is assumed that some embodiments may be combined with one or more other embodiments to form new embodiments.

[0088]

[0089] digital video codec

[0090] Figure 1 is a conceptual diagram of a video communication system according to one embodiment of the present invention. The video communication system (100) may be configured to include at least two terminals (110, 120) connected to each other via a network (105).

[0091] In one embodiment of the present invention, the above-described FIG. 1 may refer to a block diagram for configuring a unidirectional video communication network. A first terminal (110) among the terminals may encode video data in order to transmit (111) the video data via a network (105). A second terminal (120) among the terminals may be configured to receive (121) the encoded video data via a network and decode and display the same.

[0092] In another embodiment of the present invention, the above-described FIG. 1 may refer to a block diagram for configuring a two-way video communication network. For the two-way video communication, each terminal (110, 120) may be configured to encode video data acquired by itself for video transmission (112, 122) to each other terminal via the network. Each terminal may also be configured to receive (113, 123) video data transmitted by another terminal via the network, decode the same, and display the decoded video data.

[0093] The terminals (110, 120) shown in Fig. 1 may be exemplified as devices such as server computers, personal computers, portable computers, and smartphones, depending on the embodiment, but are not limited thereto. Any device corresponding to a commonly used computing device may be used. For example, according to an embodiment of the present invention, each terminal (110, 120) may mean a desktop computer, a laptop computer, a tablet PC, a mobile phone, a smart phone, a personal digital assistant (PDA), a workstation, an electronic calculator, a server computer, a cloud computer, a virtual computer, a quantum computer, or any other electronic, electrical, or quantum computing device implemented in a movable or non-movable form, and in particular, it may be interpreted as any device that is designed to operate as a terminal device according to an embodiment of the present invention among such devices, is capable of operating as a terminal device according to an embodiment of the present invention, or is capable of installing and / or executing a computer program that enables the terminal device to operate as an embodiment of the present invention or perform a method corresponding to such an operation.

[0094] Each of the above terminals (110, 120) may be implemented by a plurality of functional units that are interconnected in various forms, such as a bus, a circuit, or a relationship between a routine and a subroutine, and configured to exchange information within each of the above terminals (110, 120). In addition, through the interconnection, the terminals may be configured to include a processor (130) having a calculation function and a memory (140) connected to the processor for the purpose of executing or supporting the operation of a functional unit that primarily requires calculation among the above functional units.

[0095] The processor (130) described in this specification may mean one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding.

[0096] Even if the processor (130) is expressed singularly for the sake of ease of understanding, a person of ordinary skill in the art will recognize that the processor (130) may include multiple processing elements and / or multiple types of processing elements. For example, a device according to one embodiment of the present invention may include multiple processors or one processor and one controller as the processor (130). In addition, the processor (130) may be implemented using various processing configurations, such as a parallel processor or a multi-core processor.

[0097] The processor (130) may be configured to execute an operating system (OS) and one or more software programs running on the OS. In addition, the processor may access, store, manipulate, process, and generate data in response to the execution of the software. The software program may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing device to operate as desired or independently or collectively command a processing device. The software may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave to be interpreted by the processor (130) or to provide instructions or data to the processor. The software may be distributed among multiple computer systems connected to the network (105), such as the terminals (110, 120), and stored or executed in a distributed manner.

[0098] The software may also be implemented in the form of program commands that can be executed through various computer means and recorded or stored in the memory (140). The memory (140) may be a computer-readable recording medium, and program commands, data files, data structures, etc. may be recorded singly or in combination in the computer-readable recording medium. The program commands stored in the memory (140) may be based on a command system specifically designed and configured for the embodiment of the present invention, or may follow a command system known and available to those skilled in the art of computer software, for example, a command system exemplified by the assembly language, C, C++, Java, Python, etc. It should be understood that the command system and the program commands therefrom include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by the device and / or the processor (130) according to an embodiment of the present invention using an interpreter, etc.

[0099] The computer-readable recording medium constituting the device according to one embodiment of the present invention, including the memory (140) described in this specification, may include a temporary or volatile recording medium that is maintained only while the processor is operating, such as a processor cache, a RAM, a flash memory, or a relatively non-volatile or long-term recording medium, such as a magnetic media such as a hard disk, a floppy disk, and a magnetic tape, an optical media such as a CD-ROM, a DVD, a magneto-optical media such as a floptical disk, or a solid state memory, or may include a read-only recording medium, such as a ROM arranged on hardware, and furthermore, the hardware itself configured to perform an operation equivalent to a series of program commands by a hard-wired structure by circuit wiring, and each step for performing the operation for implementing the embodiment of the present invention can be viewed as being recorded by the connection and arrangement of the hardware components, and therefore, the connection and arrangement method It is obvious to those skilled in the art that this can be seen as equivalent to the above memory (140).

[0100] The embodiments described above with respect to the processor (130) and the memory (140) are not mutually exclusive, and may be selected or combined and implemented as needed. For example, one hardware device may be configured to operate as a module composed of one or more of the software to perform the operations of an embodiment of the present invention, and vice versa. As another example, in the present specification, all or part of the operations assigned to a certain functional unit may be implemented by one or more of the software stored in a device according to an embodiment of the present invention (preferably, in any one of the recording media belonging to the category of the memory) and configured to be executed by the processor, and in this case, such a functional unit may be referred to as a functional unit "included" in the processor.

[0101] The present invention is applicable to all environments for creating a one-way or two-way video communication network, and it should be understood that the network (105) can be created by any means for transporting encoded video data between the terminals (110, 120).

[0102] In one embodiment of the present invention, the network (105) may refer to a wired or wireless communication network. In this case, depending on the embodiment, the network may be configured to communicate information using any communication standard, and the communication standard may include packet-based communication. The packet communication may be understood to mean including packets known as TCP or UDP, for example. The wired communication method of the network (105) may be a method of connecting to an external communication network by means of a telephone line of the RJ-11 standard, an Ethernet cable belonging to various categories of the RJ-45 standard, other coaxial cables, metal cables, optical cables, and various other wired media. The wireless communication method of the above network (105) may include, depending on the embodiment, a short-range wireless communication method including Bluetooth, Wi-Fi, Zigbee, and NFC (near field communication), or may include a long-range wireless communication method that may be referred to as a name of a wireless communication technology collectively called a generation name of a communication standard such as Wibro, WiMax, Global Systems for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long-Term Evolution (LTE), New Radio (NR), and other 2G, 3G, 4G, 5G, and 6G, and a name of an international standard wireless communication standard such as IMT-2000, IMT-Adcanced, IMT-2020, and IMT-2030. Of course, even if any conventional or newly developed wired or wireless communication means, method, standard, and protocol are applied to the implementation of the network (105), there is no problem in achieving the purpose of the present invention as long as it is a means configured to perform transmission and reception in an information and communication device such as the terminal (110, 120).It is also obvious that one network (105) can be configured in a mixed manner by one or more wired and / or wireless standards.

[0103] However, in another embodiment of the present invention, the network (105) may be understood to include a process of transmitting information using a computer-readable recording medium. In this case, the configuration of the network is not limited to a communication medium, and should be understood to include a process of temporarily storing and physically transporting information in a computer-readable memory and / or recording medium. The computer-readable recording medium used for transmitting information may be understood to mean a relatively non-volatile or long-term recording medium such as a magnetic media such as a hard disk, a floppy disk, and a magnetic tape, an optical media such as a CD-ROM or DVD, a magneto-optical media such as a floptical disk, or a solid state memory, which is mainly used as a means of transporting data between computing devices.

[0104] Any other means of information communication or transport, regardless of the method employed, can be considered within the scope of the present invention as long as it has a structure that supports the transmission and decoding of video data in an encoded state. Therefore, in addition to the examples listed above, any means of information communication or transport, whether known in the past or newly available, can fall within the scope of application of the present invention.

[0105] FIG. 2 is a conceptual diagram illustrating the arrangement of an encoder and decoder in a real-time video streaming environment according to one embodiment of the present invention. The streaming system (200) illustrated in FIG. 2 can be applied to video data communication networks, including, for example, digital broadcasting, video telephony, and video conferencing. However, it should be noted that technical structures identical or similar to the streaming system can be equally applied even when information is transmitted via a recording medium, as described above.

[0106] According to one embodiment of the present invention, the streaming system may include a video source (210) that generates a video stream. The video source may include a digital video acquisition means (212), which may be, for example, a digital camera or other device, for acquiring uncompressed raw video. The raw video stream (215) may have a large capacity and may therefore be compressed by a video encoder (217) coupled or connected to the video source.

[0107] The above encoder (217) may be configured as a means including hardware, software, or a combination of the two, configured to implement an image encoding method and / or an implementation method thereof according to one embodiment of the present invention.

[0108] Through the encoder (217), an encoded bitstream (219) having a reduced capacity compared to the original video stream can be output. The bitstream (219) can be provided in real time via a relay device, which may be referred to as a streaming server (220), and / or can be stored in a recording medium (225) of the streaming server (220) for subsequent use.

[0109] The streaming system (200) may include at least one streaming client (230, 240) that connects to the streaming server (220) to receive the encoded bit string (229) in real time or obtain it later. The streaming client may include a video decoder (232) that obtains the encoded bit string (229) (which may also be considered as a copy of the bit string (219) received by the streaming server), decodes the bit string (229), and outputs the resulting video data as video data in a form that can be displayed by a display (235) or other visual, auditory, or other sensory display means.

[0110] As described above, the functions for encoding and decoding video data are collectively called a coder-and-decoder, or video codec.

[0111] FIG. 3 is a conceptual diagram of a functional unit of a video decoder according to an embodiment of the present invention. As shown in FIG. 3, a receiving unit (310) can receive at least one encoded video data to be decoded by a decoder (305). In an embodiment of the present invention, the encoded video data may be independent for each reception, and the decoding procedure of each independent video data may be independent from the decoding procedure of other video data. The encoded video data may be received by the receiving unit (310) through a hardware or software connection (315) to a device storing the same, and as described above, the storing device may be a type of streaming server located at the other end of a communication network, or may mean a physical recording medium, but is not limited thereto.

[0112] The above-described receiving unit (310) can receive the encoded video data together with other data accompanying it, such as encoded audio data or other auxiliary data, and each of the data can be separated from the video data and provided to an appropriate processing function unit (312) other than the video decoder.

[0113] When the video data is provided through a communication network, a buffer memory (320) may be coupled between the receiving unit (310) and the decoder (305) to minimize delay and disconnection according to the network environment. The buffer memory (320) may refer to a computer-readable recording medium that temporarily stores the received video data and stably supplies it to a parser (330) corresponding to the input terminal of the decoder (305). However, if the bandwidth of the communication network is sufficient, if video data is read from a recording medium in a local location that is not physically separated, or if the possibility of communication delay is not predicted in other environments, the buffer memory may be unnecessary.

[0114] The video decoder (305) may include the parser (330) as its input terminal to interpret the encoded video data. The parser may perform a function of separating (parsing) a plurality of pieces of information stored in the form of a bit string in the encoded video data according to a predetermined rule, and, if necessary, performing an entropy decoding (335) of entropy-coded video data, thereby performing a function of reconstructing symbols (338), which are paragraphs of video encoding information. The symbols (338) may include all information for controlling the operation of the decoder (305), and / or may further include information for controlling a device that is attached to and operable with the decoder (305), such as a display device. Control information for controlling the above display device may include information in a format called supplementary enhancement information (SEI) or video usability information (VUI).

[0115] As described above, the parser (330) may be configured to perform entropy decoding (335) of the encoded video data. The entropy encoding method of the encoded video data may vary depending on the encoding standard, and decoding may be performed accordingly. Representative examples of the entropy encoding standard may include variable length coding, Huffman coding, and arithmetic coding, and each of the encoding methods may be a context-adaptive or context-sensitive method depending on the standard, and may also be based on principles widely known to those skilled in the art.

[0116] The parser (330) may be configured to extract at least one partial image from the encoded video data. The definition of the partial image may vary depending on the encoding standard, and depending on the standard, one or more of the examples listed below may correspond simultaneously and overlappingly. The partial image may be defined in units such as, for example, a group of pictures (GOPs), pictures / frames, tiles, slices, macroblocks, blocks, subblocks, transform units (TUs), and prediction units (PUs).

[0117] The parser (330) may be configured to extract encoding information, such as transform coefficients, quantization parameters (QPs), and / or motion vectors, from the encoded video data. The parser (330) may be configured to perform entropy decoding (335) and parsing operations on the video data received from the buffer memory, and to selectively decode symbols (338) representing the encoding information. In addition, the parser (330) may be configured to selectively supply a specific symbol (338) to a specific decoding function unit within the decoder (305), such as an inverse quantization and inverse transform unit (340), an intra prediction unit (350), an inter prediction unit (355), or a loop filter unit (360). Control of such information supply can be determined by the information sequence contained in the encoded video, and may vary depending on the encoding standard, and is not limited within the scope of the embodiments of the present invention, and is not described in detail in this conceptual diagram.

[0118] The decoder (305) may be comprised of a number of conceptual functional units that receive and process the encoded information provided by the parser (330). It should be readily apparent that these conceptual functional units may be combined or further subdivided, depending on implementation needs. For example, they may be further separated for ease of implementation, or integrated into one for operational efficiency. In any case, each functional unit may be configured to perform close interaction with each other. However, despite the possibility of such integration or separation, the following description will be given as a combination of conceptual functional units to illustrate the video data decoding procedure applied as an embodiment of the present invention.

[0119] The decoder may include an inverse quantization and inverse transformation unit (340). The inverse quantization and inverse transformation unit (340) may be configured to receive encoding information including a method to be used for numerical transformation (transform), a block size, quantization coefficients for recovering quantized information, and distinction information of a quantization matrix that simplifies and represents the quantized coefficients from the parser (330), and may be configured to output block values ​​(341) that can be input to an aggregator (370) as a result of processing the encoding information.

[0120] In one embodiment of the present invention, the output values ​​of the inverse quantization and inverse transformation unit (340) may include intra-prediction encoded block values. The intra-predicted block values ​​may refer to values ​​that can be decoded without using prediction information from a previously decoded partial image, for example, a previous frame, but using prediction information within a partial image currently being decoded, for example, a current frame.

[0121] Prediction information within the current partial image may be provided by the intra prediction unit (350). According to an embodiment of the present invention, the intra prediction unit (350) generates a block value of the same form as the block being decoded as prediction information by using image information of a spatially adjacent area derived from a partial image currently being decoded and of which decoding has been partially completed. The partial image information may be provided (381) from a current image buffer, a so-called line buffer (380). The merging unit (370), according to an embodiment, may be configured to merge the prediction information (351) generated by the intra prediction unit (350) with the block values ​​(341) provided by the inverse quantization and inverse transformation unit (340).

[0122] In another embodiment, the output values ​​of the inverse quantization and inverse transformation unit (340) may include block values ​​subjected to motion compensation as inter-prediction encoded block values, and in some cases, block values ​​subjected to motion compensation. In this case, the inter-prediction unit (355) may extract and use sample information (386) used for motion-based prediction from a reference picture buffer (385). The information (356) derived by performing motion compensation on the sample information based on the symbols (338) included in the block values ​​as the output values ​​may be configured to be merged with the block values ​​(341) provided by the inverse quantization and inverse transformation unit (340) by the merger unit (370). In this case, the block values ​​(341) may be referred to as so-called differential or residual values.

[0123] The position information within the memory used by the inter prediction unit (355) to extract the sample information from the reference image may be determined by a motion vector provided to the inter prediction unit (355) which is composed of a combination of symbols (338) for representing, for example, X, Y, and other specific points of the reference image. The inter prediction unit (355) may also include a function for interpolating and using the sample values ​​when a so-called 'subsampling' capable motion vector is provided, and may further include a function for predicting and reinforcing the value of the motion vector.

[0124] The output values ​​(371) of the above merging unit (370) may be provided to the loop filter unit (360) and processed by various loop filtering methods. The loop filter unit (360) may be configured to receive not only the block unit output (371) of the merging unit (370) but also the symbol (338) provided from the parser (330) to control its operation. The output of the loop filter unit (360) may be output to an external display means such as the display device through an output connection (390), but may be stored (361) in a line buffer (380) for use in prediction for interpreting a subsequent intra or inter coded block value, and may also be stored in a reference image buffer (385) through this.

[0125] Certain partial images, such as frames, after their decoding is completed, can be utilized as reference images for performing predictive decoding in a subsequent decoding process. One partial image, such as a frame, can be gradually accumulated in a line buffer (380) and decoded, and when one frame is decoded, the contents of the line buffer (380) are transferred (383) to the reference image buffer (385), and a new line buffer (380) can be allocated for decoding the new frame.

[0126] The above video decoder (305) may be configured to perform a decoding operation according to a predetermined video compression technique that may be documented by various international standards or commercial standards. The standards may include, for example, international standard recommendations such as H.264, H.265, and H.266 defined by the International Telecommunication Union Standardization Sub-Division (ITU-T). Those skilled in the art will understand that each of the above recommendations is equivalent to an international standard jointly defined by the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC). The encoded video data may comply with a specific bitstream syntax defined by the relevant standards, as defined and required by the video compression standard documents and standard documents, and specifically by the profiles and levels specified within such documents. In addition, the complexity of the encoded video data may be limited to a certain level to comply with the profiles and levels. For example, a profile or level may be configured to limit a maximum picture size, a maximum decoding speed, and a maximum reference picture size. These limitations may, in some embodiments, also be further constrained via metadata signals for a hypothetical reference decoder (HRD) and HRD buffer management included in the encoded video data.

[0127] According to one embodiment of the present invention, the receiver (310) may receive additional redundant data along with the encoded video. The additional data may be considered as part of the encoded video data. The additional data may include information that may be used by the decoder (305) to properly decode the data or to more accurately reconstruct an image that approximates the original image. The additional data may be provided in the form of, for example, layers for temporal, spatial, or signal-to-noise ratio (SNR) enhancement, redundant slices, redundant images, and forward error correction codes.

[0128] FIG. 4 is a functional unit conceptual diagram of a video encoder according to one embodiment of the present invention. The encoder (405) may be configured to receive original video information (402) from a video source (401) and perform encoding.

[0129] The original video information (402) may have any suitable bit depth, for example, 8 bits, 10 bits, 12 bits, etc. In addition, the original video information (402) may have any suitable color space, for example, R / G / B, Y / U / V, Y / Cb / Cr, etc. In addition, the original video source may have any suitable sampling structure corresponding to the color space, for example, Y / Cb / Cr 4:2:0, Y / Cb / Cr 4:4:4, etc. The original video source having such a predetermined format may be provided to the encoder in the form of a digital video stream.

[0130] In a one-way video communication network, the original video information (402) can be obtained from a recording medium storing a previously prepared video source. In a two-way video communication network, the original video information (402) can be obtained from an image acquisition device, such as a camera, that generates at least one video transmission stream included in the two-way video communication.

[0131] The video data including the above original video information (402) may be configured as a plurality of partial images configured to simulate motion by being played back in chronological order. The partial images may be expressed as concepts such as pictures or frames, for example. The partial images may include one or more samples depending on the type of sampling structure, color space, etc. being used. Those skilled in the art will understand that the terms "samples" and "pixels" in digital images are closely related. The operation of the encoder will be described below with reference to such samples.

[0132] According to one embodiment of the present invention, the encoder (405) may be configured to encode and compress (partial) images constituting the original video information (402) in real time (or according to other temporal requirements required according to the implementation method) into the form of encoded video information.

[0133] In the encoder (405), the control unit (450) may be a functional unit configured to control an appropriate encoding speed. The control unit (450) may be configured to control other functional units and be functionally coupled to the following functional units as described below. The parameters set by the control unit (450) may include parameters related to bitrate control, such as skip of the image, quantizer, variable values ​​for applying a picture quality optimization technique, and may also include values ​​such as the size of the image, the structure of a group of pictures (GOP), and the maximum search range of a motion vector. A person skilled in the art will be able to understand various other functions that the control unit (450) may have, and such other functions may be added or removed according to the design of a video encoder optimized for an individual system design.

[0134] According to an embodiment of the present invention, the encoder (405) may be configured to operate in a structure such as a "coding loop" well known to those skilled in the art. To simplify the description by way of example, the coding loop may be configured with an internal encoder (so-called "source coder") (410) responsible for receiving an image to be encoded and generating symbols based on at least one reference image that has been encoded in the past, and a local decoder (420) configured to be connected to the internal encoder. The local decoder (420) may be configured to perform an operation to reproduce sample data to be generated by a decoder (490) located at an actual remote location that will receive encoded video information from the encoder (405) by receiving an output of the internal encoder (410).

[0135] Video data composed of sample data reconstructed by the internal decoder (420) may be configured to be input to the reference picture buffer of the encoder (405). As described above, the internal decoder (420) is implemented to reproduce the result output by the encoder (405) and to be decoded by a remote decoder, so the video data recorded in the reference picture buffer may also be identical in bit units to the information of the reference picture buffer of the remote decoder. That is, the prediction function unit that may be included in the encoder (405) may read the same values ​​as the sample values ​​of the previous frame that the decoder will later refer to in the decoding process from the reference picture buffer of the encoder (405).

[0136] As described above, the principle of achieving matching of the reference image buffer between the encoder (405) and the decoder (490) by means of the internal decoder (420) on the encoder (405) side is well known to those skilled in the art, and a method of responding to an environment in which such an environment is not guaranteed (e.g., information loss due to communication failure, etc.) can also follow what is known to those skilled in the art.

[0137] An embodiment of the operation method of the internal decoder (420) has been described in detail above with reference to FIG. 3. The decoder of FIG. 3 may be regarded as the aforementioned "remote" decoder (490). The internal decoder (420) may be implemented excluding lossless encoding and decoding sections such as the parser (330) or entropy decoding (335). This is because the internal encoder (405) is implemented to simply reproduce the operation of a decoder located at a remote location, and thus may directly decode symbols without requiring a process of compressing and then decompressing symbols. Accordingly, the functional units preceding the parser and entropy decoder as shown in FIG. 3 may not be provided or may be implemented at least partially.

[0138] As described above, according to a preferred embodiment of the present invention, any decoder function (excluding a parser and an entropy decoder) present in the decoder can naturally exist as a substantially identical function in the corresponding encoder (405).

[0139] The operation of the encoding function unit that may be included in the above encoder (405) can be considered as the inverse of the decoder function unit. Therefore, the embodiment can be explained by performing the operation of the decoder function unit in reverse. For example, a quantization and transform function unit corresponding to the inverse quantization and inverse transform unit may be provided, and an inter prediction encoding unit corresponding to the inter prediction unit may be provided. In addition, some additional explanations will be added.

[0140] The internal encoder (410) may be configured to perform encoding on input image information, for example, an input frame, by a predictive encoding method executed by a predictive encoding unit (440) that operates by referencing at least one temporally previous encoded partial image, for example, frames, from a reference image buffer (430) from at least one reference image information, for example, video data designated as a reference frame. In this case, the encoder (405) may be configured to encode a differential between blocks of samples constituting the input image and blocks of samples constituting the reference image.

[0141] The internal decoder (420) can decode video data that can be designated as the reference picture from symbols generated by the internal encoder (410). As described above, since the video data is subject to the same decoding operation as that performed by a remote decoder, the video data used as the reference picture may be provided to the encoder (405) in a form that has undergone lossy compression and has suffered some damage, and this operation may be intended to ensure operational consistency with the decoder.

[0142] The prediction encoding unit (440) may be configured to perform a prediction search operation within the encoder (405). The prediction search operation may refer to an operation corresponding to the inter prediction or intra prediction described in the description of the decoder. For image information that is input and scheduled to be newly encoded, the prediction unit may access the reference image buffer (430) to retrieve information such as a motion vector, a block shape, and metadata that may include the same, which are information indicating points of a reference image that can function as prediction reference information suitable for the new image information, and a sample block to be actually referenced. The prediction encoding unit (440) may operate on the basis of a so-called "sample block by pixel block" to obtain appropriate prediction reference information. According to one embodiment of the present invention, at least one prediction reference information may be designated for the input image, which designates at least one reference image information stored in the reference image buffer (430), as determined based on the search results obtained by the prediction encoding unit (440).

[0143] In one embodiment of the present invention, the control unit (450) may be configured to manage the overall encoding operation of the internal encoder (410), including setting parameters used to encode video data.

[0144] All outputs of the above-described functional units may be subjected to entropy encoding (460) in order to be finally output. The entropy encoding (460) may include various entropy coding techniques, such as variable length coding, Huffman coding, and arithmetic coding, for the symbols generated by the various functional units as described above, and each encoding method may be a context-adaptive or context-sensitive method according to the standard, or may be based on principles widely known to those skilled in the art. Such entropy encoding (460) can typically achieve lossless compression, and thus can be configured to convert at least one symbol generated by the functional units into encoded video data.

[0145] The above control unit (450) may, when controlling the operation of the encoder (405), apply the type of encoding of a specific partial image during the encoding period to each partial image, such as a picture or frame. Depending on the type, the method by which the partial image is encoded may be affected. Depending on the embodiment, the type may include what is categorized as the following "frame type."

[0146] Fig. 5 is a conceptual diagram of a frame type according to one embodiment of the present invention. The following description will be made with reference to Fig. 5.

[0147] An intra (“I”) picture (510) may refer to a picture that can be encoded and decoded using only its own information without referring to other picture information in the video data through predictive encoding. The “I” picture may be designated by names such as a key frame, an independent / instantaneous decoder referh (IDR) frame, and a clean random-access (CRA) frame, depending on the video encoding standard, and the “I” pictures designated by the various names as described above may have various modifications and application methods as permitted by each standard and may be partially different from each other. In addition to those listed above, various application methods for implementing the “I” picture may be by various methods that are already known to those skilled in the art or may be newly provided.

[0148] A prediction ("P") picture (520) may refer to a picture that can be encoded and decoded through intra or inter prediction based on at least one prediction information and / or a motion vector that designates at least one reference picture to predict sample values ​​of a block constituting the picture. The "P" picture may be configured to refer to only one reference frame, or may be configured to refer to one or more reference frames, according to a video encoding standard. When referring to more than one reference frame, sample information and / or associated metadata derived from multiple reference pictures may be used to reconstruct a single block. However, in common cases, a picture designated as a "P" picture may be understood as a picture that performs reference only to a temporally preceding picture.

[0149] A bidirectional prediction ("B") picture (530) may refer to a picture that can be encoded and decoded through intra or inter prediction based on at least one prediction information and / or motion vector that designates at least two reference pictures to predict sample values ​​of blocks constituting the picture. In a common case, a picture designated as the "B" picture is distinct from a picture designated as the "P" picture, and may be understood as a picture that performs a reference without being limited to a temporally preceding picture.

[0150] Video data may be spatially divided into a plurality of sample blocks during the encoding and decoding process, and encoding may be performed in units of the blocks. The block units may include, but are not limited to, sizes such as 4x4, 8x8, 4x8, or 16x16 in units of horizontal / vertical pixels, as is widely known. The block may be encoded using a predictive encoding method with reference to any other (already encoded) blocks, as permitted and / or restricted by the type specified for each partial picture including the block. For example, the blocks of the "I" picture (510) may be encoded without using a predictive encoding method, or with reference to blocks already encoded within the same partial picture. That is, only the so-called intra prediction method may be used. In contrast, the "P" picture (520) may further reference a reference picture encoded in at least one previous time unit, and thus, inter prediction may also be used for encoding along with intra prediction. In the case of a "B" picture (530), reference can be made not only to a previously encoded picture in the encoding order but also to a later reference picture in terms of time unit. However, it is widely known that there may be blocks within a "P" picture or a "B" picture that are encoded without relying on predictive encoding.

[0151] The above video encoder (405) may be configured to perform encoding operations according to a predetermined video compression technique that may be documented by various international standards or commercial standards. Examples of the above standards may include all of those described in the above decoder.

[0152] According to one embodiment of the present invention, the transmitter (470) may buffer the encoded video data generated by the entropy encoding in order to provide / transmit the video data (ultimately to a remote decoder (490)) to a device storing the encoded video data via a hardware or software connection (495). According to an embodiment, when providing / transmitting the encoded video data from the video encoder (405), the transmitter (470) may receive and merge other data accompanying the encoded video data, for example, encoded audio data or other auxiliary data, from a separate source (480).

[0153] According to one embodiment of the present invention, the transmitter (470) may be configured to transmit additional data along with the encoded video. The additional data may be considered part of the encoded video data. The additional data may include information that can be used by a decoder to properly decode the data or to more accurately reconstruct an image that approximates the original image. Examples of the additional data may include all of the examples previously presented with respect to the receiver (310) of the decoder.

[0154] The present invention can be implemented by a digital video compression standard that is widely used and understood by those skilled in the art as described above. The digital video compression standard may include at least one of compression standards known by the standard name such as MPEG-2, MPEG-4 Video, H.263, H.264 / AVC, H.265 / HEVC, H.266 / VVC, VC-1, AV1, QuickTime, VP-9, VP-10, and Motion JPEG.

[0155] Fig. 6 is a conceptual diagram illustrating the structure of a video encoder according to the H.266 / VVC standard. What is depicted in Fig. 6 corresponds to the rough structure of a video encoder widely known by standard codes such as ITU-T H.266 and ISO / IEC 23090-3, and also by the name MPEG-I Part 3 or the general name versatile video coding (VVC).

[0156] According to FIG. 6, a video encoder (605) may be configured to receive uncompressed and unencoded original video data (601) as input and output an encoded bit stream (602). The video data (601) may be supplied directly to a luma mapping unit (610a) when intra-encoded, or may be supplied to a luma mapping unit (610b) via an inter-prediction unit (620) including motion vector extraction. In the intra-encoded case, the mapped luma signal may be supplied to an output merger (606) by selecting (608) at least one of an intra-prediction encoded signal via an intra-prediction unit (625) or an inter-prediction encoded signal output from the luma mapping unit (610b) via the inter-prediction unit (620). The result of the above output merger can be applied to a chroma scaling unit (615). (The operation of the luminance signal mapping unit (610) and the operation of the chroma scaling unit (615) are collectively referred to as a luma mapping / chroma scalaing (LMCS) process.) The reduced chroma signal can be provided to a transform unit (630), and the transform unit (630) can perform an adaptive color transform, particularly on the chroma signal. The coefficients derived as a result of the transform are applied to a quantization unit (640) and quantized. As a result, lossy compression is achieved, and the result of the lossy compression can be output as a bit string (602) through a multi-hypothesis CABAC (650), which is a lossless compression method.

[0157] Meanwhile, the result of the lossy compression may actually enter the decoding process by going through the processes of inverse quantization (645), inverse transform (635), and luminance signal expansion (617) to generate a coding loop. The result of the luminance signal expansion may be supplied to the internal merger (607) together with the result of selecting (608) at least one of the previously generated intra prediction encoding signal or inter prediction encoding signal. The result of the internal merger may go through inverse luma mapping (617), and then may go through processing such as a deblocking filter (660), sample adaptive offset (SAO) (670), and an adaptive loop filter (ALF) to reproduce the image quality improvement process in the decoder. The result of reproducing the operation in the decoder as described above is applied to the reference image buffer (690) and can be reused for prediction encoding by the inter prediction unit (620).

[0158] The present invention can also be used by or in combination with the enhanced compression model (ECM), which is an implementation of a next-generation video codec being developed by the joint video experts team (JVET), an international standardization expert group, to improve H.266 / VVC. According to the standardization progress document of the above JVET, document number ISO / IEC JTC 1 / SC 29 / WG 5 N 190 (also document number JVET AC2025-v1), the improved compression model may include an improved intra prediction coding method, an improved inter prediction coding method, an improved transform and transform coefficient coding method, an improved adaptive loop filtering method, a bilateral filtering method, a new sample adaptive offset (SAO) method for image quality improvement, an extended entropy coding method, and an improved gradual decoding refresh (GDR) technique, and such techniques may be included in the present invention as background techniques for implementing the present invention.

[0159]

[0160] General method of applying GDR

[0161] The present invention provides a method for applying an improved gradual decoder refresh (GDR) technique that can be used in the video encoding and decoding field including the above-described embodiments, and a device to which such a method is applied.

[0162] Figure 7 is a conceptual diagram illustrating a method of applying GDR technology. Video information, according to conventional standards, is composed of multiple still image frames. A collection of such frames may be referred to as a video sequence. The types of frames included in the sequence may include intra-frames and inter-frames, as described above, and may further include frame types that perform functions not described or not described in the present invention.

[0163] Referring to FIG. 7, in the past, updated information for the entire image frame was provided through one intra frame (10) appearing during a sequence, and the remaining inter frames (11) were configured to only refer to the updated information. However, according to the GDR method, it is provided across multiple frames by a GDR frame section (20) defined within the sequence, and the GDR frame section (20) may be composed of one GDR frame (21), one recovery point frame (23), and intermediate frames (22) that can be added therebetween with an arbitrary length of 0 or more depending on the length of the GDR frame section (20).

[0164] The above GDR frame section (20) has a length of at least two frames, and in such a case, may be composed of one GDR frame (21) and one recovery point frame (22). In the case where it has a length of three or more frames, it may be composed of one GDR frame (21), one recovery point frame (22), and at least one intermediate frame (22). In the case where the length of the GDR frame section is one frame, it is not distinguished from a typical intra frame.

[0165] The above GDR frame section (20) has been disclosed as having a structure that sequentially updates in the horizontal direction (21) or sequentially updates in the vertical direction (22) in its update direction. In addition, according to the JVET-AE0145 document among the technical contributions contributed to the above JVET, a method of designating the update direction in the GDR frame in both directions in the horizontal and / or vertical directions has been disclosed.

[0166] Fig. 8 is a conceptual graph illustrating the expected effects in bit rate for the application of GDR technology. By applying the GDR technology, unlike the sharp bit rate gap (32) that occurred between conventional intra-frames and inter-frames, as shown in the exemplary graph (30) of Fig. 8, the effect of stabilizing the bit rate gap (34) by distributing the burden of this bit rate among sequential GDR frames can be expected.

[0167] Fig. 9 is a conceptual diagram illustrating the effect of the application of GDR technology on the encoding / decoding process. When encoding a frame of a GDR section, the current frame (900) can be divided into at least two regions. The first section (910) can be a region commonly referred to as a refreshed region or a clean region. The first section can preferably be interpreted separately as a refreshed region (911) and a clean region (911). In addition, the second section (920) can be a region commonly referred to as an unrefreshed region or a dirty region.

[0168] The above-mentioned updated area (911) may refer to an area that is encoded so that only the current GDR section frame (900) can be read and decoded. Accordingly, the image coding unit included in the above-mentioned updated area (911) may be, for example, a macroblock, a block, a subblock, a coding tree unit (CTU), a coding unit, a prediction unit, a transform unit, a partitioned area, and at least one of various other coding units, and may be encoded by a constrained intra coding method.

[0169] The above clean area (912) may refer to an area that can be encoded based on information that could be derived from an updated area (911) that appeared in a preceding frame (950) (which may be a GDR frame or an intermediate frame). It may be expected that complete image information for the clean area (912) can be derived at least when the corresponding GDR section is decoded from the beginning. Accordingly, a video coding unit included in the clean area (912) may be intra-coded, but may also be configured to enable inter-coding that refers to the clean area (912) of the preceding frame, and thus, as a result, may be encoded by a constrained intra / inter coding method.

[0170] The above dirty area (920) corresponds to an area that can be encoded without reference to information provided through a GDR frame or intermediate frame. From the perspective of a decoder, in order to successfully decode information in the dirty area (920), information that can only be obtained by performing decoding on an encoded video bitstream prior to the corresponding GDR section may be required. The video encoding unit included in the dirty area (920) may be encoded using an intra or inter method without any restrictions in encoding.

[0171] The limited intra coding may mean that complete image information for a corresponding coding unit can be derived only from information contained within the updated region (911). Accordingly, the limited intra coding may include, for example, a complete prohibition of inter prediction during the coding process, and may also include, for example, a prohibition of intra prediction or a spatial range limitation.

[0172] The above-mentioned limited intra / inter coding may mean that complete image information for a corresponding coding unit can be derived from information of a first section (910) including the updated section (911) and the clean section (912), and information provided in a frame (950) within a preceding GDR section. Accordingly, the limited intra / inter coding may further include, in addition to the features of the limited intra coding, temporal or spatial range restrictions for, for example, selection of a reference picture designated for inter prediction and designation of a motion vector.

[0173] In the execution of the above-described restricted intra-coding and restricted intra / inter-coding methods, according to embodiments of the present invention, even if a prohibited or restricted encoding method is executed, the utilized information may be replaced with temporarily generated information. For example, it may be used as an unusable value that is not reflected in the operation, or as a value reproduced from available values ​​through methods such as padding, filtering, or prediction.

[0174] In one embodiment of the present invention, the encoding units belonging to the updated region (911) can be intra-predictively encoded from information appearing in the encoding units belonging to the updated region (911) (e.g., as in (931) of FIG. 9). This is because the predictive encoding performed within the updated region (911) may not hinder the decoder from acquiring complete image information for the updated region (911).

[0175] In one embodiment of the present invention, the coding units belonging to the updated region (911) can be intra-predictively encoded from information appearing in the coding units included in the clean region (912) (e.g., as in (932) of FIG. 9), but in another embodiment, such predictive encoding may be prohibited. Allowing predictive encoding for the clean region (912) is advantageous for improving encoding efficiency when the decoder is configured to decode by regarding the time of appearance of a GDR frame as a point of random access. On the other hand, prohibiting predictive encoding for the clean region (912) is advantageous when providing complete random access to frames during the GDR period even at the cost of allowing only partial images to be confirmed for a certain period of time.

[0176] In a preferred embodiment of the present invention, encoding units belonging to the updated region (911) may be prohibited from being intra-predicted encoded from information appearing in encoding units included in the dirty region (920) (for example, cases such as (933) in FIG. 9 are prohibited). Since the information included in the dirty region (920) is likely to be unavailable from the decoder's perspective or is an area being encoded based on outdated information, it may be desirable to prohibit references to such areas.

[0177] Referring to the 2nd edition of the H.266 / VVC standard, a bit string syntax specifying a virtual boundary may be utilized to distinguish between the first section (910) and the second section (920) and to smoothly support the above-described limited intra and intra / inter encoding methods. The syntax may include syntax signals such as "ph_virtual_boundaries_present_flag", "ph_num_ver_virtual_boundaries", "ph_virtual_boundary_pos_x_minus1", "ph_num_hor_virtual_boundaries", and "ph_num_hor_virtual_boundaries" included in the picture header structure syntax of Section 7.3.2.8, for example, in the 2nd edition of the H.266 / VVC standard. For example, the "ph_virtual_boundaries_present_flag" value may be a flag signal indicating that a value for a virtual boundary appears in the bit string, and the "ph_num_ver_virtual_boundaries" and the "ph_virtual_boundary_pos_x_minus1", and the "ph_num_hor_virtual_boundaries" and the "ph_num_hor_virtual_boundaries" signals may be signal values ​​designated to indicate the number of virtual boundaries in the vertical direction and their positions, and the number of virtual boundaries in the horizontal direction and their positions, respectively.

[0178] However, in embodiments of the present invention, the "ph_virtual_boundaries_present_flag" signal value may be maintained as 0, so that information about the virtual boundaries may not be provided. Instead, in various embodiments of the present invention described below, an improved GDR technique implementation method that operates without requiring the signal or using another signal method that replaces the signal may be provided.

[0179] Figure 10 is an exemplary diagram illustrating the provision of frame information in a GDR section according to one embodiment of the present invention. Information regarding the existence of a GDR frame, information regarding the GDR frame, and at least a portion of the length of the GDR section may be included in an encoded video bitstream.

[0180] In one embodiment of the present invention, a flag indicating the presence of a GDR section (1010) may be included in the encoded bit stream. The flag may be, for example, a signal encoded with 1 bit, such as "sps_gdr_enabled_flag" included in the sequence parameter RBSP syntax of section 7.3.2.4 of the 2nd edition of the H.266 / VVC standard. However, it is obvious that such a signal may be provided with an information capacity of at least 1 bit (or less than 1 bit in case of entropy coding) at any position in the bit stream.

[0181] In one embodiment of the present invention, information indicating a frame order may be included in an encoded bit stream. The information may be a variable-length signal such as "ph_pic_order_cnt_lsb" included in the picture header structure syntax of Section 7.3.2.8 of the H.266 / VVC standard, Edition 2. The variable length of the "ph_pic_order_cnt_lsb" signal may be derived from a signal, preferably encoded in 4 bits, such as "sps_log2_max_pic_order_cnt_lsb_minus4" included in the sequence parameter RBSP syntax of Section 7.3.2.4 of the same standard. For example, the bit length may be a value obtained by adding 4 to the value of "sps_log2_max_pic_order_cnt_lsb_minus4". However, it is self-evident that the above signal can be provided with an information capacity of at least 1 bit (or less than 1 bit in case of entropy coding) at any position in the bit string. Referring to Fig. 10, an example of designation (1030) of "ph_pic_order_cnt_lsb" is shown.

[0182] The value of the above "ph_pic_order_cnt_lsb" may mean a sequence number indicating the display order of the current frame, and its range may be 0 or more, 2 or less than the "MaxPicOrderCntLsb" value of the H.266 / VVC standard, which is determined by adding 4 to the value of the above "sps_log2_max_pic_order_cnt_lsb_minus4". Taking the second edition of the H.266 / VVC standard as an example, the "sps_log2_max_pic_order_cnt_lsb_minus4" value is stipulated to have a value of 0 or more and 12 or less, and therefore the "MaxPicOrderCntLsb" value may indicate a value of 16 or more and 65,536 or less. The above "ph_pic_order_cnt_lsb" value can be encoded by modulo-operating with the above "MaxPicOrderCntLsb" value when the actual order of the above frame is greater than or equal to the above "MaxPicOrderCntLsb" value.

[0183] In one embodiment of the present invention, information indicating the temporal frame distance to the end point of a GDR frame, i.e., the recovery point frame, may be included in the encoded bit stream. The information may be a signal such as "ph_recovery_poc_cnt" included in the picture header structure syntax of Section 7.3.2.8 with reference to the second edition of the H.266 / VVC standard. For example, the value of the "ph_recovery_poc_cnt" signal is utilized as a value that determines the frame number in the future at which the recovery point frame will appear, and may indicate a value designated as 0 in the recovery point frame. Referring to FIG. 10, an example of designation (1035) of the "ph_recovery_poc_cnt" signal is shown.

[0184] In one embodiment of the present invention, the entire length of the GDR section (1020) can be derived through information of the encoded bit string. For example, the encoder can specify that any GDR section (1020) must start when the value of the "ph_pic_order_cnt_lsb" signal is 0, so that the encoder and / or decoder can obtain both the frame distance from the start point of the GDR section (1020) and the frame distance to the end point of the GDR section through the current value of the "ph_pic_order_cnt_lsb" signal and the value of the "ph_recovery_poc_cnt" signal. For another example, the decoder may be configured to recognize the start point of the GDR section (1020) by storing the value of the "ph_pic_order_cnt_lsb" signal for the current frame in the start frame (1011) of the GDR section (1020), and the frame distance to the end point of the GDR section (1020) may be obtained through the value of the "ph_recovery_poc_cnt" signal. For another example, the encoder may be configured to provide information about the length of the GDR section (1020) to the decoder by adding an arbitrary signal specifying the total length of the GDR section (1020). The information may be applied differently for each GDR section (1020), or may be applied identically but newly included for every GDR section (1020), or may be applied uniformly for the entire video sequence. Preferably, the arbitrary signal may be implemented to be newly included in the picture header structure syntax of Section 7.3.2.8 in the H.266 / VVC standard specification syntax, but it is self-evident that the signal may be provided with an information capacity of at least 1 bit (or less than 1 bit in case of entropy coding) at any position in the bit string.

[0185]

[0186] First embodiment and encoding and decoding according thereto

[0187] As described above, according to the implementation method of conventional GDR technology, such as the technology presented in the second edition of the H.266 / VVC standard, it is shown that a syntax signal related to the existence and position of a virtual boundary can be used to distinguish a "clean" area from a "dirty" area updated by the GDR technology. In contrast, in the first embodiment of the present invention, an embodiment is described that provides an improved GDR method and device by presenting a method of deriving a virtual boundary by a predetermined function instead of specifying the virtual boundary as described above.

[0188] According to a first embodiment of the present invention, the distinction between a "clean" area and a "dirty" area in a GDR can be determined by a predetermined function that uses a frame number as a variable. The function can be any function that forms a boundary line for spatially dividing an area within an image frame. For example, the function can be determined as a function that draws any straight line, curve, or closed curve. The function, according to one embodiment of the present invention, is a normalized function that can be scaled and applied according to the resolution of the frame to which it is applied. For example, a function representing the boundary line of a GDR is configured to derive a real number between 0 and 1 as a result of a change in the frame number n, and the result value is multiplied by the horizontal resolution of the frame, and as a result, a position at which the boundary line is developed on the frame can be designated by the real number.

[0189] Figure 11 is a conceptual diagram of region division by function according to one embodiment of the present invention. The example of Figure 11 reproduces the method of specifying a virtual boundary in a conventional vertical GDR using a function. The function used in the example of Figure 11 can be expressed as follows.

[0190]

[0191] The above mathematical expression 1 can be applied to all pixels of the frame. In the above mathematical expression 1, R p may mean a discrete (Boolean) value indicating whether the pixel is included in a specific area. The R p can be determined by the function f(x,y). x can be the horizontal coordinate of the pixel in the image normalized by the resolution. For example, if the horizontal resolution of the image is 1280, 0 can be normalized to 0, 640 can be normalized to 0.5, and 1280 can be normalized to 1. Y can be the vertical coordinate of the pixel in the image normalized by the resolution. For example, if the vertical resolution of the image is 720, 0 can be normalized to 0, 360 can be normalized to 0.5, and 720 can be normalized to 1. n gdr can indicate the sequence number of the above frame within the GDR section. l gdr can represent the entire length of the GDR section.

[0192] That is, the above mathematical expression 1 is the sequence n of the current frame within the GDR section. gdr As this increases, the point at which a particular pixel is included in the "clean" area in the vertical direction can be understood as a function that advances from 0 to 1 (1151, 1152, 1153, 1154). Referring to FIG. 11, it can be seen that the "clean" area (1120, 1121, 1122, 1123) corresponding to the first section (910) of FIG. 9 is gradually expanded in the vertical direction according to the order of the frames (1110, 1111, 1112, 1113).

[0193] An example of the above mathematical expression 1 is as follows. According to the example of Fig. 11, the length of the GDR section is 4 frames in total, and therefore l gdr is set to 4. In the first frame (1110), n gdrcan be determined as 1. If all pixels of the first frame (1110) are substituted into the above mathematical expression 1, the vertical position y is n of the vertical resolution. gdr / l gdr , that is, pixels that are less than or equal to 1 / 4 multiple (1151) are included in the “clean” area (1120), and pixels that are not are classified as the “dirty” area (1140). If the vertical resolution of the first frame (1110) is 720 pixels and the coordinate axis of the image is based on the upper left corner of the image, the above mathematical expression 1 produces the same result as setting the virtual boundary at a vertical position of 180 pixels in the first frame.

[0194] Likewise, in the second frame (1111), pixels that are less than or equal to a 2 / 4 multiple (1152), i.e., pixels with a vertical position of 360 or less (1121), in the third frame (1112), pixels that are less than or equal to a 3 / 4 multiple (1153), i.e., pixels with a vertical position of 540 or less (1122), and in the fourth frame (1113), pixels that are less than or equal to a 4 / 4 multiple (1154), i.e., pixels with a vertical position of 720 or less (1123), i.e., the entire frame is included in the "clean" area.

[0195] The above-described embodiment has the advantage of being able to operate in the same manner even if the length of the GDR section is arbitrarily shortened or extended. For example, in FIG. 1, the function of the above mathematical expression 1 operates as if the virtual boundary is specified at the y position {180, 360, 540, 720}, but the length of the GDR section l gdr When increasing to 5 frames, the virtual boundary will behave as if it is specified at the y position {144, 288, 432, 576, 720} over the first to fifth frames (not shown), and l gdrWhen increasing to 6 frames, it can be seen that the virtual boundary is specified at the y position {120, 240, 360, 480, 600, 720} over the first to sixth frames (not shown). In addition, l gdr In the case where the virtual boundary of the first frame is set to 7 frames, the virtual boundary of the first frame is designated at 102.857142… pixels, but since the function of the mathematical expression 1 indicates that the vertical position of the pixel, y, is less than or equal to the boundary line, the same effect as designating it at a point of 102 pixels can be achieved. By the same principle, the virtual boundary of the second frame can be designated at 205 pixels, which is less than or equal to 205.714285… pixels, to achieve the same effect.

[0196] The above-described embodiment has the advantage of being able to operate identically even when the image resolution is arbitrarily changed. This can be understood because the input of the function is normalized to the range 0 to 1. Of course, it is self-evident that those skilled in the art can use various mathematical and algorithmic means other than normalizing the input value and normalizing it to the range 0 to 1 to apply the function to the changing resolution, and it is also self-evident that all examples of such modified implementations fall within the scope of the present invention.

[0197] Fig. 12 is another conceptual diagram of region division by function according to one embodiment of the present invention. The example of Fig. 12 reproduces the method of specifying virtual boundaries in conventional vertical GDR by function, and also shows a case where a "clean" area is re-divided into an update area (911) and a clean area (911) of Fig. 9. As described above, this embodiment is more advantageous in enhancing the performance of random access by allowing only limited intra-coding methods for the update area, and it is necessary to set two virtual boundaries.

[0198] An example using the same function as Equation 1 is shown in Fig. 12. However, Equation 1 can be applied twice per frame to generate two virtual boundaries. That is, n per frame gdr First application in case of and n gdr -The second application in the case of -1 can be executed, and the exclusive section included in the first application but not included in the second application can be considered as an update area, the section included in both the first application and the second application can be considered as a clean area, and the section not included in the first application and the second application can be considered as a dirty area.

[0199] By substituting all pixels of the first frame (1210) into the above mathematical expression 1 using the above method, the vertical position y is n of the vertical resolution. gdr / l gdr , that is, pixels less than 1 / 4 multiple are included in the first application, and the vertical position y is n of the vertical resolution. gdr -1 / l gdr , that is, pixels less than or equal to 0 / 4 are included in the second application. Accordingly, if the vertical resolution of the first frame (1210) is 720 pixels and the coordinate axis of the image is based on the upper left corner of the image, the same result is obtained as setting virtual boundaries at the vertical position 0 pixel point (1250) and 180 pixel point (1251) in the first frame and designating the area between them as the update area (1230). Accordingly, the update area (1230) can be determined as a section from the vertical position 0 pixel to 180 pixels.

[0200] Likewise, in the second frame (1211), it can be seen that the pixels in the section from 1 / 4 multiple (1251) to 2 / 4 multiple (1252), that is, the pixels with a vertical position greater than 180 and less than 360 (1231), in the third frame, the pixels in the section from 2 / 4 multiple (1252) to 3 / 4 multiple (1253), that is, the pixels with a vertical position greater than 360 and less than 540 (1232), and in the fourth frame, the pixels in the section from 3 / 4 multiple (1153) to 4 / 4 multiple (1154), that is, the pixels with a vertical position greater than 540 and less than 720 (1233) are designated as the update area.

[0201] Figure 13 is an exemplary diagram of a method for segmenting regions by a function according to some embodiments of the present invention. Through Figures 13 (a) to (g), various embodiments of a method for segmenting regions by a function, which are not limited to those illustrated, are disclosed. Those skilled in the art will understand that various implementation methods can be applied and implemented by using functions and shapes of figures not illustrated in the present specification through Figure 13, and that all such applied implementation methods fall within the scope of the present invention.

[0202] The function used in the example of (a) in Fig. 13 can be expressed as follows.

[0203]

[0204] By the above mathematical expression 2, it can be understood that GDR update proceeding in a diagonal direction as in (a) of Fig. 13 can be easily implemented.

[0205] Referring to (b) of Fig. 13, it can be understood that a GDR update area and / or a clean area that spreads outward from the center can be easily implemented by modifying a similar function. The implementation of the above embodiment can derive an implementation similar to the method of specifying the update direction in the GDR frame in both horizontal and / or vertical directions as shown in the JVET-AE0145 document among the technical contributions contributed to the JVET, but unlike the method of the above contribution that requires specifying multiple virtual boundaries for each frame, there is an advantage in that it can be implemented with a smaller amount of information by specifying a single area function.

[0206] Referring to (c) of Figure 13, a domain function can be specified as a shape that increases in size by specifying an arbitrary closed curve, for example, a rectangular shape. It can be seen that by such functional transformation, the shape of the GDR update region and / or the clean region can be easily diversified.

[0207] The above closed curve area function can have a changed reference point. For example, instead of the upper left, which is usually taken as the origin of the image coordinate system, any point that can be designated as a normalized or non-normalized position value by any x and y coordinate system, such as the left center, the lower left, the upper center, the center, the lower center, the upper right, the right center, the lower right, or any other point can be applied as the reference point for the function expansion. In addition, the above function can be applied with any aspect ratio when applied to the frame. For example, the image information of the frame has an aspect ratio of 16:9, but the above function can be applied with an aspect ratio of 1:1. According to an embodiment, the encoder and / or decoder according to the present invention can utilize the function by projecting it onto a GDR frame in various ways, such as by stretching the function to be the same as the resolution of the frame, by stretching it while maintaining the aspect ratio of the original function (in the so-called filling manner), or by repeatedly applying an area operation by the function within the resolution of the image to perform tiling.

[0208] Referring to (d) of Fig. 13, an example is shown in which the reference point of the area function expansion drawing a rectangular shape, as in (c) of Fig. 13, is changed to the exact center of the frame, and the applied aspect ratio is changed to 1:1. It can be seen that by such functional modifications, the shapes of the GDR update area and / or the clean area can be more easily diversified.

[0209] The above function can be applied in the reverse direction. That is, it can be applied by reversing the inner and outer sides of the boundary line, or by reversing the inner and outer sides of the closed curve. In this case, the input value of the function is also n. gdr Instead l gdr -n gdr By changing to , the order of sequential application can also be reversed.

[0210] Referring to Fig. 13(e), an example is shown that uses the same domain function expansion method as Fig. 13(d), but with the inner / outer application and the application order reversed. By this functional transformation, it can be seen that various forms of GDR update areas and / or clean areas can be easily realized, not only GDRs that expand from a single origin to the other side or from the inner side to the outer side, but also GDRs that expand from the other side to the origin or from the outer side to the inner side.

[0211] In addition to the embodiments described above, any mathematical expression and / or conditional expression that can designate a boundary or region may be used as the function within the scope of the present invention. For example, referring to (f) of FIG. 13, an example using an elliptic function is shown. For another example, referring to (g) of FIG. 13, an example of obtaining an indeterminate updated region and / or a clean region through the intersection of any two straight lines is shown. Furthermore, the function may also be applied after rotation, skew, scaling, and other affine transformations, depending on the embodiment.

[0212] By adopting a method according to the application of the first embodiment of the present invention described above, it can be seen that, according to the implementation method of the conventional GDR technology, for example, the technology presented in the second edition of the H.266 / VVC standard, syntax signals related to the existence and position of a virtual boundary to distinguish between a "clean" area and a "dirty" area updated by the GDR technology may be unnecessary.

[0213] In any codec standard specification, if the encoder and decoder decide to use a single function, information about the function need not be conveyed via the bitstream.

[0214] In a certain codec standard specification, if an encoder and a decoder use a single function but allow rotation, tilt, scaling, and other affine transformations of the function, an encoder according to the standard can be designed to insert parameter values ​​for the affine transformation, such as angles, reference points, and magnifications, into an image bitstream, and a decoder according to the standard can be designed to read the parameter values ​​and decode a GDR frame.

[0215] In a certain codec standard specification, if an encoder and a decoder decide to use multiple functions that may or may not fall within the above-described embodiments, information indicating that one of the multiple functions has been selected may be provided in the video bitstream. In this case, an encoder according to the standard may insert information regarding the selection of the function into the video bitstream, and a decoder according to the standard may be designed to read the type of the function and decode the GDR frame.

[0216] If a codec standard determines to transmit a mathematical expression of a function between an encoder and a decoder, a binarized value of the mathematical expression of the function used in GDR may need to be provided in the video bitstream. In this case, an encoder according to the standard may be designed to binarize the mathematical expression of the function and insert it into the video bitstream, and a decoder according to the standard may be designed to read the expression of the function and decode the GDR frame.

[0217] Furthermore, in any codec standard specification, if the decoder does not need information about the order in which the GDR frames are updated when the encoder encodes the video bitstream containing the GDR frames by an appropriate method, the information about the function need not be transmitted from the encoder to the decoder through the bitstream, regardless of which function is used. For example, if the encoder encodes a predetermined coding unit by a predetermined GDR method with a limited intra and / or intra / inter coding method, the decoder can perform a decoding procedure corresponding to GDR without transmitting additional signals.

[0218] In the various cases described above, when an encoder inserts some information into a bit string, various types of encoding methods can be applied depending on the specifications and number of cases of the information. The encoding methods can include both fixed-length coding and variable-length coding. As for the variable-length coding, as is widely known, Huffman coding, exponential Golomb coding, context adaptive variable length coding, (context adaptive) arithmetic coding, and other coding methods, particularly, entropy coding methods can be used. It can be seen that, depending on the use of the encoding method, the inserted information can be expressed as at least 1 bit of related information at any position of the bit string.

[0219] It is self-evident that if any encoding method among the types illustrated or not illustrated above is selected, the decoder should be designed to decode the information by the inverse of the same method.

[0220]

[0221] Second embodiment and encoding and decoding according thereto

[0222] According to a second embodiment of the present invention, the distinction between "clean" and "dirty" areas in a GDR may be determined by a predetermined pattern that changes as the frame progresses. The pattern may be any shape that forms a boundary for spatially dividing an area within an image frame.

[0223] Fig. 14 is a conceptual diagram of region division by pattern according to one embodiment of the present invention. In Fig. 14, patterns (1415, 1425, 1435, 1445) that gradually change over a length of 4 frames are provided. The patterns may, for example, store divisions into an updated region, a clean region, and a dirty region. In the example of Fig. 14, the length of the GDR section is fixed to 4 frames, and in each GDR frame (1410), intermediate frames (1420, 1430), and recovery point frame (1440) belonging to the GDR section, patterns (1415, 1425, 1435, 1445) corresponding to each order may be projected according to the change in the pattern that matches the length of the GDR section, corresponding to the resolution thereof.

[0224] By the above projected pattern, the image coding unit may correspond to at least one of a macroblock, a block, a subblock, a coding tree unit (CTU), a coding unit, a prediction unit, a transform unit, a partitioned area, and various other coding units, and it may be determined to which area each coding unit belongs. In one embodiment, all coding units overlapping with an area designated as an updated area in the projected pattern may be coded as belonging to the updated area by a constrained intra coding method. However, in addition to this, there are various possibilities for implementing variations, such as designating an area only when the area overlaps by 50% or more.

[0225] Matters related to techniques and restrictions imposed on encoding in relation to renewal areas, clean areas, dirty areas, etc. may be applied in accordance with the contents of the first embodiment described above.

[0226] The above pattern may be stored in the form of bitmap pixels, or may be stored in the form of vector graphics. According to an embodiment, the pattern may have a fixed resolution and may be arranged to be upscaled or downscaled from the fixed resolution to the resolution of an image frame to which it is actually applied. According to an embodiment, the pattern may be stored independently in a temporal order in which the shape to be changed for each frame is to be stored. According to an embodiment, the pattern may be stored as a residual from a temporally preceding pattern, or as a vector shift, in the shape of the pattern to be changed for each frame, except for the pattern to be applied to the first frame.

[0227] FIG. 15 is an exemplary diagram showing various forms of a pattern according to an embodiment of the present invention. The pattern according to the second embodiment of the present invention is not limited to a simple geometric shape as illustrated in FIG. 14, and may have any arbitrary pattern shape and any arbitrary frame length. As an example, FIG. 15 (a) illustrates a method of changing a pattern that provides a kind of dithering effect as a pattern change of 6 frames in length. Providing a signal of a GDR update area and / or a clean area having a complex structure as illustrated in FIG. 15 (a) may have been impossible or required a very large amount of information in conventional video codec standards. However, according to the present invention, by enabling encoding and / or decoding to be performed according to a predetermined GDR update pattern, it is possible to enhance the GDR process while reducing the amount of information that must be included in the bit rate in relation to the method of performing GDR.

[0228] The above pattern may also be applied after undergoing rotation, skew, scaling, and other affine transformations, depending on the embodiment. For example, Fig. 15(b) illustrates the same pattern as Fig. 15(a) rotated 90 degrees to the right. Those skilled in the art will readily understand that other angles and transformations are possible.

[0229] When applying the above pattern by projecting it onto an image, various methods can be utilized. Fig. 16 is an exemplary diagram showing a method of projecting a pattern onto an image according to various embodiments of the present invention. Even if the same pattern (1600) is given, depending on the embodiment, the encoder and / or decoder according to the present invention can utilize the pattern by projecting it onto a GDR frame in various ways, such as by stretching the pattern to be the same as the resolution of the frame (1610), by stretching it while maintaining the aspect ratio of the original pattern (so-called filling method) (1620), or by repeatedly applying the pattern within the resolution of the image to tile it (1630). In addition, it is obvious that there are various modifications that can be implemented to derive each region required for a GDR image using a given pattern.

[0230] By adopting a method according to the application of the second embodiment of the present invention described above, it can be seen that, according to the implementation method of the conventional GDR technology, for example, the technology presented in the second edition of the H.266 / VVC standard, syntax signals related to the existence and position of a virtual boundary to distinguish between a "clean" area and a "dirty" area updated by the GDR technology may be unnecessary.

[0231] In any codec standard specification, if the encoder and decoder decide to use a single pattern, information about the pattern need not be conveyed via the bitstream.

[0232] In a certain codec standard specification, if an encoder and a decoder use a single pattern, but specify an aspect ratio of the pattern, or allow rotation, tilt, scaling, and other affine transformations, an encoder according to the standard can be designed to insert parameter values ​​for the aspect ratio and / or affine transformation, such as angles, reference points, and scaling factors, into an image bitstream, and a decoder according to the standard can be designed to read the parameter values ​​and decode a GDR frame.

[0233] In a certain codec standard specification, if an encoder and a decoder decide to use multiple patterns that may or may not fall within the above-described embodiments, information indicating that one of the multiple patterns has been selected may be provided in the video bitstream. In this case, an encoder according to the standard may insert information regarding the selection of the pattern into the video bitstream, and a decoder according to the standard may be designed to read the type of the pattern and decode a GDR frame.

[0234] In a certain codec standard specification, if it is decided to transmit the shape of a pattern between an encoder and a decoder, a binarized value of the shape of the pattern to be used in GDR may need to be provided in the video bitstream. In this case, an encoder according to the standard may be designed to binarize the pattern shape and insert it into an video bitstream, and a decoder according to the standard may be designed to read the shape of the pattern and decode a GDR frame. Depending on the embodiment, the binarized form of the pattern may be a binary bitstream in the form of 1 indicating a first section and 0 indicating a second section, a binary bitstream consisting of a 2-bit bitmap in the form of 0 indicating a dirty area, 1 indicating a clean area, and 2 indicating an update area, or a binary bitstream in the form of another application of such expressions.

[0235] Furthermore, in any codec standard specification, if the decoder does not need information about the order in which the GDR frames are updated when the encoder encodes the video bitstream containing the GDR frames by an appropriate method, the information about the pattern need not be transmitted from the encoder to the decoder through the bitstream, regardless of which pattern is used. For example, this is because the decoder can perform a decoding procedure corresponding to GDR without transmitting additional signals simply by encoding a predetermined coding unit by a predetermined GDR method with a limited intra and / or intra / inter coding method.

[0236] In the various cases described above, when an encoder inserts some information into a bit string, various types of encoding methods can be applied depending on the specifications and number of cases of the information. The encoding methods can include both fixed-length coding and variable-length coding. As for the variable-length coding, as is widely known, Huffman coding, exponential Golomb coding, context adaptive variable length coding, (context adaptive) arithmetic coding, and other coding methods, particularly, entropy coding methods can be used. It can be seen that, depending on the use of the encoding method, the inserted information can be expressed as at least 1 bit of related information at any position of the bit string.

[0237] It is self-evident that if any encoding method among the types illustrated or not illustrated above is selected, the decoder should be designed to decode the information by the inverse of the same method.

[0238]

[0239] Third embodiment and encoding and decoding according thereto

[0240] According to a third embodiment of the present invention, when performing GDR, "clean" and "dirty" regions are not separately distinguished, but instead, by forcing the number of intra-coding blocks that are encoded using a conventional method during the GDR period, the actual update region can be input into the bitstream in a form that appears over several frames. The third embodiment of the present invention allows for implementation in the form of progressive intra recovery (PIR), which is a special embodiment of GDR.

[0241] Figure 17 is an exemplary diagram of a GDR implementation by forcing intra coding according to one embodiment of the present invention. Figure 17 illustrates four frames (1720, 1721, 1722, 1723) contained in a GDR section (1710) spanning four frames in length. In the third embodiment of the present invention, a GDR section can be considered as a section in which intra coding is performed at least once for all spatial regions of the image within the section. This method can be implemented by using a conventional encoder, but forcing intra encoding for coding units corresponding to an update threshold among image coding units that may correspond to at least one of various coding units, such as a network abstraction layer unit (NAL unit), a macroblock, a block, a subblock, a coding tree unit (CTU), a coding unit, a prediction unit, a transform unit, a partitioned area, and other various coding units belonging to a frame. For example, as in the example of FIG. 17, if a GDR section (1710) spans a length of 4 frames, it can be easily inferred that the update threshold, which represents a spatial area in which intra encoding is forced to be performed in each frame, can be designated as 25%. That is, when any GDR section has a length of m frames, it can be seen that at least 1 / m spatial regions must be intra-coded for each frame.

[0242] In each frame belonging to a GDR frame and an intermediate frame, information on a spatial region that has been intra-coded at least once can be stored in the encoder. When calculating the update threshold, a region that has already been updated at least once can be excluded. That is, in an intermediate frame (1721, 1722) belonging to a GDR section (1710), all frames can be implemented to be given a goal of intra-coding at least 25% of the image area from a spatial region that has not been intra-coded in preceding frames belonging to the same GDR section (1710) and to perform encoding according to this goal. In addition, in a recovery frame indicating the end of a GDR section, intra-coding can be enforced for all spatial regions that have not been updated previously, thereby ensuring that the entire region of the image can be updated by being intra-coded at least once throughout the GDR section.

[0243] In order to enforce intra-coding within a conventional encoding procedure, a method may be needed to instruct a conventional encoder to perform intra-coding on blocks that should have been subjected to inter-coding. According to one embodiment of the present invention, a method may be used in which blocks scheduled for inter-coding are sorted in descending order of encoding efficiency, and the corresponding inter-coded blocks are changed to intra-coded blocks and instructed to be encoded.

[0244] FIG. 18 is a flowchart illustrating a method for forcing intra-coding according to one embodiment of the present invention. Before proceeding with the explanation, it should be noted that the flowchart illustrated in FIG. 18 is merely an example illustrating the spirit of the present invention and a representative implementation method thereof. It will be apparent that any implementation of the spirit of the present invention by a person skilled in the art using a procedure modified from the flowchart illustrated in FIG. 18 or other procedures falls within the scope of the present invention.

[0245] Referring to FIG. 18, an encoding loop (S1810) for all image encoding units belonging to one frame is disclosed. For each encoding unit, the encoder predicts the encoding efficiency at the time of intra encoding (S1820) and also predicts the encoding efficiency at the time of inter encoding (S1825). This prediction of encoding efficiency may be implemented by any method that is used in a typical encoder implementation or may be newly provided. As a most basic embodiment, a method of actually performing intra encoding and inter encoding processes separately for the same image encoding unit and then evaluating the bit rate thereof may be used. However, a method of predicting this efficiency without performing the entire encoding may also be used by using various indices. For example, it can be predicted by the division level of a coding tree unit (CTU), by the distribution of DCT coefficient values, by the encoding information including the encoding mode set for the surrounding image encoding units, by the encoding result of an already encoded area, or by other methods.

[0246] However, when evaluating the intra encoding efficiency, it should be noted that the intra encoding is used to implement GDR and must be able to be completely decoded from the internal information provided for each frame. Therefore, the intra encoding may be evaluated using the limited intra encoding method described above. As described above, the limited intra encoding may include prohibiting intra prediction or limiting the spatial range within the intra encoding process.

[0247] Referring again to FIG. 18, after predicting the efficiency for each encoding method, an image encoding unit can be encoded using either intra encoding or inter encoding according to predetermined conditions, and its contents can be recorded in bit string syntax.

[0248] As a first condition, it can be confirmed (S1830) whether the current frame is a recovery point frame and whether the corresponding image coding unit includes a spatial region that has not been intra-coded even once within the current GDR interval, i.e., an unupdated unit. If the above condition is satisfied, the corresponding coding unit may be intra-coded (S1860) because it must be used to achieve the purpose of the GDR interval by being intra-coded.

[0249] As a second condition, it can be verified (S1840) whether the intra-coding efficiency is superior to the inter-coding efficiency, or whether they compete within a certain threshold range. This verification procedure is widely used in conventional encoders, and any method already applied or newly applicable to such encoders may be used. If the above condition is satisfied, the corresponding encoding unit can be intra-coded (S1860).

[0250] As a third condition, it can be confirmed (S1850) whether the inter encoding efficiency is within the lower threshold rank within the corresponding frame. This condition can be determined when, as in the above-described embodiment, a method is used in which blocks scheduled for inter encoding are sorted in descending order of encoding efficiency and the corresponding inter-coded block is changed to intra and encoded. The determination of whether it is within the lower threshold rank can be determined after actually performing encoding once for the entire frame, but can also be predicted without performing the entire encoding based on various indices as described above in relation to the prediction of encoding efficiency. If it is determined that the inter encoding unit falls within the lower threshold rank, as an auxiliary condition, it can be determined (S1855) whether the threshold area that must be intra-coded within the corresponding GDR frame has already been achieved. In one implementation method, if the threshold area has already been achieved, it can be configured not to perform any further forced intra encoding in order to ensure the maintenance of a stable bit rate provided by the GDR function. In another implementation, this judgment can also be determined by the sorting order of encoding efficiency, and can be implemented in the form of selecting the number of inter-encoding target units necessary to achieve the critical area in descending order of inter-encoding efficiency. If the above conditions are satisfied, the corresponding encoding unit can be intra-encoded (S1860).

[0251] If none of the above conditions are met, the encoding unit may be inter-encoded (S1865).

[0252] If intra encoding (S1860) is performed, the encoder may be configured to store (S1870) information indicating that the spatial region on the frame where the intra encoding was performed is an area where information has been updated by intra encoding. The stored information may be used as a judgment criterion in the above-described judgment conditions, for example, the first condition (S1830) or the auxiliary condition (S1855) of the third condition.

[0253] When a method according to the application of the third embodiment of the present invention described above is adopted, it can be seen that, according to the implementation method of the conventional GDR technology, such as the technology presented in the second edition of the H.266 / VVC standard, a syntax signal related to the existence and position of a virtual boundary to distinguish between a "clean" area and a "dirty" area updated by the GDR technology may be unnecessary. This may be because the conventional intra-coding syntax is equally utilized to implement the effect of GDR. Furthermore, when a method according to the application of the third embodiment of the present invention is adopted, depending on the implementation method, a separate signal for separately identifying a GDR section or indicating that it is a frame related to GDR may not be provided, and the corresponding frames may be regarded as inter-frames having conventional unidirectional or bidirectional references.

[0254] That is, according to a preferred embodiment of the present invention, in any codec standard specification, when an encoder encodes an image bitstream including a GDR section according to the third embodiment of the present invention by an appropriate method, a decoder can be configured to perform decoding by a normal procedure without requiring separate information related to GDR in order to decode such a frame. In this case, in the encoder, information necessary to force intra encoding is internally used during the encoding process and is erased without being recorded in the bitstream, and such information is not transmitted to the decoder, and the decoder can be configured to decode the provided information by a normal method. By applying the above-described limited intra encoding, even if the decoder interprets the bitstream syntax by a normal method, only limited intra prediction is executed during the decoding process of image information intended as a GDR section, so that image information can be gradually restored in the same manner as when GDR is implemented.

[0255] Of course, if necessary, additional information may be inserted into the bit string for easy implementation of the third embodiment. In this case, when the encoder inserts some information into the bit string, various encoding methods may be applied depending on the specifications and number of cases of the information. The encoding methods may include both fixed-length coding and variable-length coding. As for the variable-length coding, as is widely known, Huffman coding, exponential Golomb coding, context adaptive variable length coding, (context adaptive) arithmetic coding, and other coding methods, particularly, entropy coding methods, may be used. It can be seen that, depending on the use of the encoding method, the inserted information can be expressed as at least one bit of related information at any position of the bit string.

[0256] It is self-evident that if any encoding method among the types illustrated or not illustrated above is selected, the decoder should be designed to decode the information by the inverse of the same method.

Claims

1. A video encoding method including a gradual decoder refresh (GDR) function, A step for setting the frame length of a GDR section in which a stepwise decoder update is to be performed; A step of designating an update area including at least one image encoding unit in at least one frame belonging to the above GDR section; A step of performing intra encoding on at least one image encoding unit included in the above update area; and A step of recording the result of the above encoding in a bit string; including, An encoding method characterized in that all spatial regions within the video resolution within the above GDR section are intra-coded at least once.

2. In paragraph 1, The step of specifying the above renewal area is: An encoding method comprising a step of designating the update area from a spatial area derived by a function that uses a frame number as a variable.

3. In paragraph 2, The step of specifying the above renewal area is: An encoding method comprising a step of designating the update area from a spatial area excluding an area derived from a function resulting from a function inputting the n-1 frame number from an area derived from a function resulting from an n-1 frame number.

4. In paragraph 2, The above function is, An encoding method characterized in that it is a function of a line whose area can be derived by integration or a function of a closed curve whose area can be calculated.

5. In paragraph 2, An encoding method characterized in that the above function is applied by being transformed by at least one operator among an operator for inverting an area to be judged by area, a rotation operator, a skew operator, a scaling operator, and an affine transform operator.

6. In paragraph 2, An encoding method further comprising a step of recording information related to the type of the above function in the encoded bit string.

7. In paragraph 1, The step of specifying the above renewal area is: An encoding method comprising a step of designating the update area from a spatial area specified by a change pattern whose order of change is determined according to the progression of the frame.

8. In paragraph 7, The above change pattern is an encoding method characterized in that the shape of the pattern to be changed for each frame is defined as a residual from a temporally preceding pattern.

9. In paragraph 7, An encoding method characterized in that the above change pattern is applied by being transformed by at least one operator among an operator for inverting an area to be judged by area, a rotation operator, a skew operator, a scaling operator, and an affine transform operator.

10. In paragraph 7, An encoding method further comprising a step of recording information related to the type of the change pattern in the encoded bit string.

11. In paragraph 1, The step of specifying the above renewal area is: An encoding method comprising a step of designating a spatial region so that all spatial regions within the video resolution within the above GDR section can be intra-coded at least once.

12. In paragraph 11, The step of specifying the above renewal area is: An encoding method characterized in that when the above GDR section has a length of m frames, at least 1 / m of a spatial region is designated to be intra-coded for each frame.

13. In paragraph 11, An encoding method further comprising the step of storing, in an encoder, cumulative update region information representing a spatial region that has been intra-coded at least once within the GDR section.

14. In paragraph 11, The step of specifying the above renewal area is: An encoding method comprising a step of specifying that at least one inter encoding target image encoding unit be encoded by changing it into an intra encoding target image encoding unit.

15. In paragraph 14, An encoding method in which the step of changing and designating operates based on the encoding efficiency of the inter-encoding target image encoding unit, and is configured to change and designate when the encoding efficiency of the inter-encoding target image encoding unit is low.

16. A video decoding method including a gradual decoder refresh (GDR) function, A step of receiving an encoded bit string; A step of reading at least one frame constituting a GDR section from the bit string; A step of intra-decoding at least one image coding unit included in at least one frame belonging to the above GDR section; and A step of writing the intra-decrypted result to an output buffer; including, A decoding method characterized in that all image areas within the video resolution within the above GDR section are intra-decoded at least once.

17. In paragraph 16, A decoding method characterized in that the intra-decoded image area is included in a function area corresponding to a spatial area derived by a function that uses the frame number as a variable for each frame within the GDR section.

18. In paragraph 16, A decoding method characterized in that the intra-decoded image area is included in a change pattern area corresponding to a spatial area specified by a change pattern whose order of change is determined according to the progression of frames in each frame within the GDR section.

19. In paragraph 16, A decoding method, characterized in that the intra-decoded image area includes a spatial area of ​​at least 1 / m for each frame in the GDR section when the GDR section has a length of m frames.

20. In paragraph 16, A decoding method, characterized in that the image coding unit to be intra-decoded is decoded in a manner that does not refer to an image coding unit that does not belong to the same frame or is not intra-coded.

Images