Encoding / decoding method, recording medium having bitstream stored therein, and method for transmitting bitstream

Abstract

Provided are an encoding / decoding method, a bitstream transmission method, and a computer-readable recording medium storing a bitstream. The method according to the present disclosure comprises the steps of: acquiring, from a bitstream, information regarding optimization applied to an image; and determining the purpose and method of the optimization on the basis of the information regarding the optimization, wherein the information regarding the optimization may include first information indicating an optimization purpose and second information indicating a type of an optimization method.

Description

Encoding / decoding method, recording medium storing bitstream, and method for transmitting bitstream

[0001] The present disclosure relates to an encoding / decoding method, a recording medium storing a bitstream, and a method for transmitting a bitstream, and relates to a method for encoding / decoding information about optimization applied to a video.

[0002] With the advancement of machine learning technology, demand for image processing-based AI services is increasing. To effectively process the massive amounts of image data required by AI services within limited resources, image compression technologies optimized for machine tasks are essential. However, existing image compression technologies have been developed with the goal of high-resolution, high-quality image processing for human vision, making them unsuitable for AI services. Consequently, active research and development is underway on new machine-oriented image compression technologies suitable for AI services.

[0003] The present disclosure aims to provide an encoding / decoding method with improved encoding / decoding efficiency.

[0004] Additionally, the present disclosure aims to provide a method for efficiently representing information about optimizations applied to a video.

[0005] Additionally, the present disclosure aims to express an encoding optimization method applied according to a purpose related to a user or a machine and suitable uses and properties of an encoded bitstream.

[0006] In addition, the present disclosure aims to provide a non-transitory computer-readable recording medium that stores a bitstream generated by an encoding method according to the present disclosure.

[0007] In addition, the present disclosure aims to provide a method for transmitting a bitstream generated by an encoding method according to the present disclosure.

[0008] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present disclosure belongs from the description below.

[0009] A method according to one aspect of the present disclosure comprises the steps of: obtaining information about optimization applied to a video from a bitstream; and determining a purpose and method of the optimization based on the information about the optimization, wherein the information about the optimization may include first information indicating a purpose of the optimization and second information indicating a type of the optimization method.

[0010] A method according to another aspect of the present disclosure comprises the steps of: performing optimization on a video; generating information about the optimization; and encoding the information about the optimization, wherein the information about the optimization may include first information indicating an optimization purpose and second information indicating an optimization method.

[0011] A computer-readable recording medium according to another aspect of the present disclosure can store a bitstream generated by an encoding method or device of the present disclosure.

[0012] A transmission method according to another aspect of the present disclosure can transmit a bitstream generated by an encoding method or device of the present disclosure.

[0013] The features briefly summarized above regarding the present disclosure are merely exemplary aspects of the detailed description of the present disclosure that follows and do not limit the scope of the present disclosure.

[0014] According to the present disclosure, an encoding / decoding method with improved encoding / decoding efficiency can be provided.

[0015] In addition, according to the present disclosure, by providing a method for efficiently expressing information about optimization applied to a video (whether optimization has been applied, scope of optimization application, and optimization method), the suitability of the optimization method can be determined, and thereby an optimization method suitable for the purpose can be requested and used, thereby improving system efficiency.

[0016] In addition, according to the present disclosure, a non-transitory computer-readable recording medium for storing a bitstream generated by an encoding method according to the present disclosure can be provided.

[0017] Additionally, according to the present disclosure, a method for transmitting a bitstream generated by an encoding method can be provided.

[0018] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the description below.

[0019] FIG. 1 is a schematic diagram of a VCM system to which embodiments of the present disclosure can be applied.

[0020] FIG. 2 is a schematic diagram illustrating a VCM pipeline structure to which embodiments of the present disclosure can be applied.

[0021] FIG. 3 is a schematic diagram of an image / video encoder to which embodiments of the present disclosure can be applied.

[0022] FIG. 4 is a schematic diagram of an image / video decoder to which embodiments of the present disclosure can be applied.

[0023] FIG. 5 is a flowchart schematically illustrating a feature / feature map encoding procedure to which embodiments of the present disclosure may be applied.

[0024] FIG. 6 is a flowchart schematically illustrating a feature / feature map decoding procedure to which embodiments of the present disclosure may be applied.

[0025] Figure 7 is a diagram showing an example of a feature extraction method using a feature extraction network.

[0026] Fig. 8a is a diagram showing data distribution characteristics of a video source, and Fig. 8b is a diagram showing data distribution characteristics of a feature set.

[0027] FIG. 9 is a diagram for explaining a transformation and inverse transformation process to which embodiments of the present disclosure can be applied.

[0028] FIG. 10 is a diagram illustrating an example of a quantization group according to one embodiment of the present disclosure.

[0029] Figure 11 is a block diagram of CABAC for encoding one syntax element.

[0030] Figures 12 and 13 are drawings for explaining the entropy encoding procedure.

[0031] Figures 14 and 15 are drawings for explaining the entropy decoding procedure.

[0032] Figure 16 illustrates an example of a VCM hierarchy.

[0033] Figure 17 is a diagram showing an example of a bitstream composed of encoded abstract features and NNAL information.

[0034] FIG. 18 is a diagram for exemplarily explaining encoding optimization and encoding bitstream usage according to a purpose to which an embodiment according to the present disclosure can be applied.

[0035] Figure 19 is a drawing showing an example of a profile definition structure.

[0036] FIG. 20 is a diagram illustrating an encoding method according to one embodiment of the present disclosure.

[0037] FIG. 21 is a diagram illustrating a decoding method according to one embodiment of the present disclosure.

[0038] FIG. 22 is a diagram illustrating an example of a content streaming system to which embodiments of the present disclosure can be applied.

[0039] FIG. 23 is a diagram showing another example of a content streaming system to which embodiments of the present disclosure can be applied.

[0040] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0041] In describing embodiments of the present disclosure, detailed descriptions of known configurations or functions will be omitted if they are deemed to obscure the gist of the present disclosure. Furthermore, portions unrelated to the description of the present disclosure in the drawings have been omitted, and similar portions have been designated with similar reference numerals.

[0042] In the present disclosure, when a component is said to be "connected," "coupled," or "connected" to another component, this may include not only a direct connection, but also an indirect connection in which another component exists in between. Furthermore, when a component is said to "include" or "have" another component, unless otherwise specifically stated, this does not exclude the other component, but rather implies that the other component may be included.

[0043] In this disclosure, terms such as first, second, etc. are used solely to distinguish one component from another, and do not limit the order or importance of components unless specifically stated otherwise. Accordingly, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.

[0044] In this disclosure, distinct components are used to clearly illustrate their respective characteristics, and do not necessarily imply that the components are separated. That is, multiple components may be integrated into a single hardware or software unit, or a single component may be distributed into multiple hardware or software units. Therefore, even if not specifically mentioned, such integrated or distributed embodiments are also included within the scope of this disclosure.

[0045] In the present disclosure, the components described in various embodiments are not necessarily essential components, and some may be optional components. Therefore, embodiments comprising a subset of the components described in one embodiment are also within the scope of the present disclosure. Furthermore, embodiments including other components in addition to the components described in various embodiments are also within the scope of the present disclosure.

[0046] The present disclosure relates to encoding and decoding of images, and terms used in the present disclosure may have their usual meanings commonly used in the technical field to which the present disclosure belongs, unless newly defined in the present disclosure.

[0047] The present disclosure may be applied to a method disclosed in the Versatile Video Coding (VVC) standard and / or the Video Coding for Machines (VCM) standard. In addition, the present disclosure may be applied to a method disclosed in the essential video coding (EVC) standard, the AOMedia Video 1 (AV1) standard, the 2nd generation of audio video coding standard (AVS2), or the next generation video / image coding standard (e.g., H.267 or H.268, etc.).

[0048] The present disclosure presents various embodiments related to video / image coding, and unless otherwise stated, the embodiments may be performed in combination with each other. In the present disclosure, "video" may mean a set of a series of images over time. An "image" may be information generated by artificial intelligence (AI). Input information used in the process of AI performing a series of tasks, information generated during the information processing process, and output information may be used as images. A "picture" generally means a unit representing one image at a specific time point, and a slice / tile is a coding unit that constitutes a part of a picture in encoding. A single picture may be composed of one or more slices / tiles. In addition, a slice / tile may include one or more CTUs (coding tree units). The CTUs may be divided into one or more CUs. A tile is a rectangular area within a specific tile row and a specific tile column within a picture, and may be composed of multiple CTUs. A tile row may be defined as a rectangular area of ​​CTUs, may have the same height as the height of the picture, and may have a width specified by a syntax element signaled from a bitstream portion such as a picture parameter set. A tile row may be defined as a rectangular area of ​​CTUs, may have the same width as the width of the picture, and may have a height specified by a syntax element signaled from a bitstream portion such as a picture parameter set. A tile scan is a predetermined sequential ordering method of CTUs that divide a picture. Here, CTUs may be sequentially ordered within a tile according to a CTU raster scan, and tiles within a picture may be sequentially ordered according to a raster scan order of tiles in the picture.A slice may contain an integer number of complete tiles, or a contiguous integer number of complete CTU rows within a tile of a picture. A slice may be exclusively contained in a single NAL unit. A picture may consist of one or more tile groups. A tile group may contain one or more tiles. A brick may represent a rectangular region of CTU rows within a tile within a picture. A tile may contain one or more bricks. A brick may represent a rectangular region of CTU rows within a tile. A tile may be divided into multiple bricks, and each brick may contain one or more CTU rows belonging to the tile. A tile that is not divided into multiple bricks may also be treated as a brick.

[0049] In the present disclosure, "pixel" or "pel" may refer to the smallest unit that constitutes a picture (or image). Additionally, "sample" may be used as a term corresponding to a pixel. A sample may generally represent a pixel or a pixel value, and may represent only a pixel / pixel value of a luma component or only a pixel / pixel value of a chroma component.

[0050] In one embodiment, particularly when applied to VCM, a pixel / pixel value may represent independent information of each component or a pixel / pixel value of a component generated through combination, synthesis, or analysis when there is a picture composed of a set of components with different characteristics and meanings. For example, in an RGB input, only the pixel / pixel value of R may be represented, only the pixel / pixel value of G may be represented, or only the pixel / pixel value of B may be represented. For example, only the pixel / pixel value of the Luma component synthesized using the R, G, and B components may be represented. For example, only the pixel / pixel value of the image information extracted through analysis of the R, G, and B components may be represented.

[0051] In the present disclosure, a "unit" may represent a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to the region. One unit may include one luma block and two chroma (e.g., Cb, Cr) blocks. In some cases, the term "unit" may be used interchangeably with terms such as "sample array", "block", or "area". In general, an M×N block may include a set (or array) of samples (or sample array) or transform coefficients consisting of M columns and N rows. In one embodiment, particularly when applied to VCM, a unit may represent a basic unit containing information for performing a specific task.

[0052] In the present disclosure, the "current block" may mean one of the following: a "current coding block," a "current coding unit," a "block to be encoded," a "block to be decoded," or a "block to be processed." When prediction is performed, the "current block" may mean a "current prediction block" or a "block to be predicted." When transformation (inverse transformation) / quantization (inverse quantization) is performed, the "current block" may mean a "current transformation block" or a "block to be transformed." When filtering is performed, the "current block" may mean a "block to be filtered."

[0053] Additionally, in the present disclosure, "current block" may mean "luma block of the current block" unless explicitly described as a chroma block. "Chroma block of the current block" may be expressed by explicitly including explicit description of chroma block, such as "chroma block" or "current chroma block."

[0054] In this disclosure, " / " and "," can be interpreted as "and / or". For example, "A / B" and "A, B" can be interpreted as "A and / or B". Additionally, "A / B / C" and "A, B, C" can mean "at least one of A, B, and / or C."

[0055] In this disclosure, "or" may be interpreted as "and / or." For example, "A or B" may mean 1) "A" only, 2) "B" only, or 3) "A and B." Alternatively, "or" in this disclosure may mean "additionally or alternatively."

[0056] The present disclosure relates to VCM (Video / image coding for machines).

[0057] VCM (Virtual Compression Modeling) is a compression technology that encodes / decodes a portion of a source image / video or information obtained from the source image / video for machine vision purposes. In VCM, the target of encoding / decoding can be referred to as a feature. A feature can refer to information extracted from the source image / video based on the task purpose, requirements, surrounding environment, etc. Features may have a different information format than the source image / video, and accordingly, the compression method and representation format of the feature may also differ from those of the video source.

[0058] VCMs can be applied to a variety of applications. For example, in surveillance systems that recognize and track objects or people, VCMs can be used to store or transmit object recognition information. Furthermore, in intelligent transportation or smart traffic systems, VCMs can be used to transmit vehicle location information collected from GPS, sensing information collected from LIDAR and radar, and various vehicle control information to other vehicles or infrastructure. Furthermore, in the smart city field, VCMs can be used to perform individual tasks for interconnected sensor nodes or devices.

[0059] The present disclosure provides various embodiments related to feature / feature map coding. Unless otherwise specified, the embodiments of the present disclosure may be implemented individually or in combination of two or more.

[0060] VCM System Overview

[0061] FIG. 1 is a schematic diagram of a VCM system to which embodiments of the present disclosure can be applied.

[0062] Referring to FIG. 1, the VCM system may include an encoding device (10) and a decoding device (20).

[0063] The encoding device (10) can compress / encode features / feature maps extracted from source images / videos to generate a bitstream, and transmit the generated bitstream to a decoding device (20) via a storage medium or a network. The encoding device (10) may also be referred to as a feature encoding device. In a VCM system, features / feature maps can be generated in each hidden layer of a neural network. The size and number of channels of the generated feature maps can vary depending on the type of neural network or the location of the hidden layer. In the present disclosure, the feature maps can be referred to as a feature set.

[0064] The encoding device (10) may include a feature acquisition unit (11), an encoding unit (12), and a transmission unit (13).

[0065] The feature acquisition unit (11) can acquire features / feature maps for source images / videos. In some embodiments, the feature acquisition unit (11) can acquire features / feature maps from an external device, for example, a feature extraction network. In this case, the feature acquisition unit (11) performs a feature reception interface function. Alternatively, the feature acquisition unit (11) can acquire features / feature maps by executing a neural network (e.g., CNN, DNN, etc.) using the source images / videos as input. In this case, the feature acquisition unit (11) performs a feature extraction network function.

[0066] According to an embodiment, the encoding device (10) may further include a source image generation unit (not shown) for obtaining a source image / video. The source image generation unit may be implemented as an image sensor, a camera module, or the like, and may obtain the source image / video through a process of capturing, synthesizing, or generating an image / video. In this case, the generated source image / video may be transmitted to a feature extraction network and used as input data for extracting a feature / feature map.

[0067] The encoding unit (12) can encode the feature / feature map acquired by the feature acquisition unit (11). The encoding unit (12) can perform a series of procedures such as prediction, transformation, and quantization to increase encoding efficiency. The encoded data (encoded feature / feature map information) can be output in the form of a bitstream. A bitstream including the encoded feature / feature map information can be referred to as a VCM bitstream.

[0068] The transmission unit (13) can transmit feature / feature map information or data output in the form of a bitstream to a decoding device (20) via a digital storage medium or network in the form of a file or streaming. Here, the digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit (13) can include elements for generating a media file having a predetermined file format or elements for transmitting data via a broadcasting / communication network.

[0069] The decoding device (20) can obtain feature / feature map information from the encoding device (10) and restore the feature / feature map based on the obtained information.

[0070] The decryption device (20) may include a receiving unit (21) and a decryption unit (22).

[0071] The receiving unit (21) can receive a bitstream from the encoding device (10), obtain feature / feature map information from the received bitstream, and transmit it to the decoding unit (22).

[0072] The decoding unit (22) can decode the feature / feature map based on the acquired feature / feature map information. The decoding unit (22) can perform a series of procedures, such as inverse quantization, inverse transformation, and prediction, corresponding to the operation of the encoding unit (14), to increase decoding efficiency.

[0073] According to an embodiment, the decryption device (20) may further include a task analysis / rendering unit (23).

[0074] The task analysis / rendering unit (23) can perform task analysis based on the decrypted feature / feature map. In addition, the task analysis / rendering unit (23) can render the decrypted feature / feature map into a form suitable for task execution. Various machine-oriented tasks can be performed based on the task analysis results and the rendered feature / feature map.

[0075] Above, the VCM system can encode / decode features extracted from source images / videos according to user and / or machine requests, task objectives, and surrounding environments, and perform various machine-directed tasks based on the decoded features. The VCM system can also be implemented by extending / redesigning a video / image coding system, and can perform various encoding / decoding methods defined in the VCM standard.

[0076] VCM pipeline

[0077] FIG. 2 is a schematic diagram illustrating a VCM pipeline structure to which embodiments of the present disclosure can be applied.

[0078] Referring to FIG. 2, the VCM pipeline (200) may include a first pipeline (210) for encoding / decoding images / videos and a second pipeline (220) for encoding / decoding features / feature maps. In the present disclosure, the first pipeline (210) may be referred to as a video codec pipeline, and the second pipeline (220) may be referred to as a feature codec pipeline.

[0079] The first pipeline (210) may include a first stage (image / video encoder) (211) that encodes an input image / video and a second stage (image / video decoder) (212) that decodes the encoded image / video to generate a restored image / video. The restored image / video may be used for human viewing, i.e., for human vision.

[0080] The second pipeline (220) may include a third stage (feature extraction network) (221) for extracting features / feature maps from input images / videos, a fourth stage (VCM encoder) (222) for encoding the extracted features / feature maps, and a fifth stage (VCM decoder) (223) for decoding the encoded features / feature maps to generate restored features / feature maps. The restored features / feature maps may be used for machine (vision) tasks. Here, the machine (vision) task may refer to a task in which images / videos are consumed by a machine. The machine (vision) task may be applied to service scenarios such as, for example, surveillance, intelligent transportation, smart city, intelligent industry, and intelligent content. In some embodiments, the restored features / feature maps may also be used for human vision.

[0081] In some embodiments, the encoded feature / feature map in the fourth stage (222) may be transferred to the first stage (221) and used to encode an image / video. In this case, an additional bitstream may be generated based on the encoded feature / feature map, and the generated additional bitstream may be transferred to the second stage (222) and used to decode the image / video.

[0082] In some embodiments, the decrypted feature / feature map in the fifth stage (223) may be transferred to the second stage (222) and used to decrypt the image / video.

[0083] Although FIG. 2 illustrates a case where the VCM pipeline (200) includes a first pipeline (210) and a second pipeline (220), this is merely exemplary and embodiments of the present disclosure are not limited thereto. For example, the VCM pipeline (200) may include only the second pipeline (220), or the second pipeline (220) may be extended into multiple feature codec pipelines.

[0084] Meanwhile, in the first pipeline (210), the first stage (211) may be performed by an image / video encoder, and the second stage (212) may be performed by an image / video decoder. In addition, in the second pipeline (220), the third stage (221) may be performed by a VCM encoder (or a feature / feature map encoder), and the fourth stage (222) may be performed by a VCM decoder (or a feature / feature map decoder). The encoder / decoder structure will be described in detail below.

[0085] Encoder

[0086] FIG. 3 is a schematic diagram of an image / video encoder to which embodiments of the present disclosure can be applied.

[0087] Referring to FIG. 3, the image / video encoder (300) may include an image partitioner (310), a prediction unit (predictor) 320, a residual processor (residual processor) 330, an entropy encoder (entropy encoder) 340, an adder (adder) 350, a filter (filter) 360, and a memory (memory) 370. The prediction unit (320) may include an inter prediction unit (321) and an intra prediction unit (322). The residual processor (330) may include a transformer (transformer) 332, a quantizer (quantizer) 333, a dequantizer (dequantizer) 334, and an inverse transformer (inverse transformer) 335. The residual processing unit (330) may further include a subtractor (331). The addition unit (350) may be referred to as a reconstructor or a recontructed block generator. The image segmentation unit (310), the prediction unit (320), the residual processing unit (330), the entropy encoding unit (340), the addition unit (350), and the filtering unit (360) described above may be configured by one or more hardware components (e.g., an encoder chipset or a processor) depending on the embodiment. In addition, the memory (370) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component described above may further include the memory (370) as an internal / external component.

[0088] The image segmentation unit (310) can segment an input image (or picture, frame) input to the image / video encoder (300) into one or more processing units. For example, the processing unit may be referred to as a coding unit (CU). The coding unit may be recursively segmented from a coding tree unit (CTU) or a largest coding unit (LCU) according to a Quad-tree binary-tree ternary-tree (QTBTTT) structure. For example, one coding unit may be segmented into a plurality of coding units of deeper depth based on a quad-tree structure, a binary tree structure, and / or a ternary structure. In this case, for example, the quad-tree structure may be applied first, and the binary tree structure and / or the ternary structure may be applied later. Alternatively, the binary tree structure may be applied first. The image / video coding procedure according to the present disclosure may be performed based on a final coding unit that is no longer divided. In this case, the largest coding unit may be used as the final coding unit based on coding efficiency according to image characteristics, etc., or, if necessary, the coding unit may be recursively divided into coding units of lower depths, and the coding unit with the optimal size may be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration, which will be described later. As another example, the processing unit may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may each be divided or partitioned from the final coding unit described above.The prediction unit may be a unit of sample prediction, and the transform unit may be a unit for deriving a transform coefficient and / or a unit for deriving a residual signal from a transform coefficient.

[0089] The term "unit" may be used interchangeably with terms such as "block" or "area" in some cases. In general, an M×N block can represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luma component or only the pixel / pixel value of the chroma component. A sample can be used as a term corresponding to a pixel or a pel.

[0090] The image / video encoder (300) can generate a residual signal (residual block, residual sample array) by subtracting a prediction signal (predicted block, prediction sample array) output from an inter-prediction unit (321) or an intra-prediction unit (322) from an input image signal (original block, original sample array), and the generated residual signal is transmitted to a conversion unit (332). In this case, as illustrated, a unit that subtracts a prediction signal (prediction block, prediction sample array) from an input image signal (original block, original sample array) within the image / video encoder (300) may be referred to as a subtraction unit (331). The prediction unit can perform prediction on a block to be processed (hereinafter, referred to as a current block) and generate a predicted block including prediction samples for the current block. The prediction unit can determine whether intra-prediction or inter-prediction is applied to the current block or CU unit. The prediction unit can generate various information regarding prediction, such as prediction mode information, and transmit it to the entropy encoding unit (340). The information regarding prediction can be encoded in the entropy encoding unit (340) and output in the form of a bitstream.

[0091] The intra prediction unit (322) can predict the current block by referring to samples within the current picture. At this time, the referenced samples may be located in the neighborhood of the current block or may be located away from it depending on the prediction mode. In intra prediction, the prediction modes may include multiple non-directional modes and multiple directional modes. The non-directional modes may include, for example, a DC mode and a planar mode. The directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes depending on the degree of detail in the prediction direction. However, this is only an example, and a greater or lesser number of directional prediction modes may be used depending on the settings. The intra prediction unit (322) may also determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.

[0092] The inter prediction unit (321) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between the neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on the inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring block can include a spatial neighboring block existing in the current picture and a temporal neighboring block existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. Temporal neighboring blocks may be referred to as collocated reference blocks, collocated CUs (colCUs), etc., and reference pictures including temporal neighboring blocks may be referred to as collocated pictures (colPic). For example, the inter prediction unit (321) may construct a motion information candidate list based on neighboring blocks, and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction may be performed based on various prediction modes, and for example, in the case of skip mode and merge mode, the inter prediction unit (321) may use the motion information of neighboring blocks as the motion information of the current block. In the case of skip mode, unlike the merge mode, a residual signal may not be transmitted.In the motion vector prediction (MVP) mode, the motion vector of the surrounding blocks is used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0093] The prediction unit (320) can generate a prediction signal based on various prediction methods. For example, the prediction unit can apply intra prediction or inter prediction for prediction of a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). In addition, the prediction unit can be based on an intra block copy (IBC) prediction mode or a palette mode for prediction of a block. The IBC prediction mode or palette mode can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in the present disclosure. The palette mode can be viewed as an example of intra coding or intra prediction. When palette mode is applied, sample values ​​within a picture can be signaled based on information about the palette table and palette index.

[0094] The prediction signal generated by the prediction unit (320) can be used to generate a reconstructed signal or a residual signal. The transformation unit (332) can apply a transformation technique to the residual signal to generate transform coefficients. For example, the transformation technique can include at least one of a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), a Karhunen-Loeve Transform (KLT), a Graph-Based Transform (GBT), or a Conditionally Non-linear Transform (CNT). Here, GBT refers to a transformation obtained from a graph when the relationship information between pixels is expressed as a graph. CNT refers to a transformation obtained based on generating a prediction signal using all previously reconstructed pixels. In addition, the transformation process can be applied to a pixel block having a square equal size, or can be applied to a block of a non-square variable size.

[0095] The quantization unit (333) quantizes the transform coefficients and transmits them to the entropy encoding unit (340), and the entropy encoding unit (340) can encode the quantized signal (information about the quantized transform coefficients) and output it as a bitstream. The information about the quantized transform coefficients may be called residual information. The quantization unit (333) can rearrange the quantized transform coefficients in the form of a block into the form of a one-dimensional vector based on the coefficient scan order, and can also generate information about the quantized transform coefficients based on the quantized transform coefficients in the form of a one-dimensional vector. The entropy encoding unit (340) can perform various encoding methods, such as, for example, exponential Golomb, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), etc. The entropy encoding unit (340) may encode, together or separately, information necessary for image / video restoration (e.g., values ​​of syntax elements, etc.) in addition to quantized transform coefficients. The encoded information (e.g., encoded image / video information) may be transmitted or stored in the form of a bitstream in units of NAL (network abstraction layer) units. The image / video information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the image / video information may further include general constraint information. In addition, the image / video information may further include a method of generating and using encoded information, a purpose, etc. In the present disclosure, information and / or syntax elements transmitted / signaled from an image / video encoder to an image / video decoder may be included in the image / video information.Image / video information can be encoded through the encoding process described above and included in a bitstream. The bitstream can be transmitted through a network or stored in a digital storage medium. Here, the network can include a broadcasting network and / or a communication network, and the digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmission unit (not shown) that transmits a signal output from an entropy encoding unit (340) and / or a storage unit (not shown) that stores the signal can be configured as an internal / external element of the image / video encoder (300), or the transmission unit can be included in the entropy encoding unit (340).

[0096] The quantized transform coefficients output from the quantization unit (333) can be used to generate a prediction signal. For example, by applying inverse quantization and inverse transformation to the quantized transform coefficients through the inverse quantization unit (334) and the inverse transform unit (335), a residual signal (residual block or residual samples) can be reconstructed. The addition unit (350) can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from the inter prediction unit (321) or the intra prediction unit (322). When there is no residual for the block to be processed, such as when skip mode is applied, the predicted block can be used as a reconstructed block. The addition unit (350) may be called a reconstructor or a reconstructed block generation unit. The generated restoration signal can be used for intra prediction of the next processing target block within the current picture, and can also be used for inter prediction of the next picture after filtering as described below.

[0097] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture encoding and / or restoration process.

[0098] The filtering unit (360) can improve subjective / objective picture quality by applying filtering to the restoration signal. For example, the filtering unit (360) can apply various filtering methods to the restoration picture to generate a modified restoration picture and store the modified restoration picture in the memory (370), specifically, in the DPB of the memory (370). The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filtering unit (360) can generate various information regarding filtering and transmit it to the entropy encoding unit (340). The information regarding filtering can be encoded by the entropy encoding unit (340) and output in the form of a bitstream.

[0099] The modified restored picture transmitted to the memory (370) can be used as a reference picture in the inter prediction unit (321). Through this, prediction mismatch in the encoder and decoder stages can be avoided, and encoding efficiency can also be improved.

[0100] The DPB of the memory (370) can store the modified restored picture to be used as a reference picture in the inter prediction unit (321). The memory (370) can store motion information of a block from which motion information in the current picture is derived (or encoded) and / or motion information of blocks in a picture that has already been restored. The stored motion information can be transferred to the inter prediction unit (321) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (370) can store restored samples of restored blocks in the current picture and can transfer the stored restored samples to the intra prediction unit (322).

[0101] Meanwhile, the VCM encoder (or feature / feature map encoder) may have a structure basically identical / similar to the image / video encoder (300) described with reference to FIG. 3 in that it performs a series of procedures such as prediction, transformation, and quantization to encode a feature / feature map. However, the VCM encoder differs from the image / video encoder (300) in that it targets a feature / feature map for encoding, and accordingly, the name of each unit (or component) (e.g., image segmentation unit (310), etc.) and its specific operation contents may be different from the image / video encoder (300). The specific operation contents of the VCM encoder will be described in detail later.

[0102] Decoder

[0103] FIG. 4 is a schematic diagram of an image / video decoder to which embodiments of the present disclosure can be applied.

[0104] Referring to FIG. 4, the image / video decoder (400) may include an entropy decoder (410), a residual processor (420), a predictor (430), an adder (440), a filter (450), and a memory (460). The predictor (430) may include an inter-prediction unit (431) and an intra-prediction unit (432). The residual processor (420) may include a dequantizer (421) and an inverse transformer (422). The entropy decoding unit (410), residual processing unit (420), prediction unit (430), addition unit (440), and filtering unit (450) described above may be configured by a single hardware component (e.g., a decoder chipset or processor) depending on the embodiment. In addition, the memory (460) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (460) as an internal / external component.

[0105] When a bitstream including video / image information is input, the video / video decoder (400) can restore the video / image in accordance with the process in which the video / image information is processed in the video / video encoder (300) of FIG. 3. For example, the video / video decoder (400) can derive units / blocks based on block division-related information obtained from the bitstream. The video / video decoder (400) can perform decoding using a processing unit applied in the video / video encoder. Accordingly, the processing unit of decoding may be, for example, a coding unit, and the coding unit may be divided from a coding tree unit or a maximum coding unit according to a quad tree structure, a binary tree structure, and / or a ternary tree structure. One or more transform units may be derived from the coding unit. Then, the restored video signal decoded and output by the video / video decoder (400) can be played back through a playback device.

[0106] The image / video decoder (400) can receive a signal output from the encoder (300) of FIG. 3 in the form of a bitstream, and the received signal can be decoded through the entropy decoding unit (410). For example, the entropy decoding unit (410) can parse the bitstream to derive information (e.g., image / video information) necessary for image restoration (or picture restoration). The image / video information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the image / video information may further include general constraint information. In addition, the image / video information may include a method of generating and using decoded information, a purpose, etc. The image / video decoder (400) can decode a picture further based on information on the parameter set and / or the general constraint information. Signaling / received information and / or syntax elements can be decoded through a decoding procedure and obtained from the bitstream. For example, the entropy decoding unit (410) can decode information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output the values ​​of syntax elements necessary for image restoration and the quantized values ​​of transform coefficients for the residual. More specifically, the CABAC entropy decoding method receives a bin corresponding to each syntax element in the bitstream, determines a context model using information of the syntax element to be decoded and decoding information of the surrounding and decoding target blocks or information of symbols / bins decoded in the previous step, and predicts the occurrence probability of the bin according to the determined context model to perform arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element.At this time, the CABAC entropy decoding method can update the context model using the information of the decoded symbol / bin for the context model of the next symbol / bin after determining the context model. Information regarding prediction among the information decoded by the entropy decoding unit (410) is provided to the prediction unit (inter prediction unit (432) and intra prediction unit (431)), and residual values ​​on which entropy decoding is performed by the entropy decoding unit (410), i.e., quantized transform coefficients and related parameter information, can be input to the residual processing unit (420). The residual processing unit (420) can derive a residual signal (residual block, residual samples, residual sample array). In addition, information regarding filtering among the information decoded by the entropy decoding unit (410) can be provided to the filtering unit (450). Meanwhile, a receiving unit (not shown) that receives a signal output from an image / video encoder may be further configured as an internal / external element of the image / video decoder (400), or the receiving unit may be a component of an entropy decoding unit (410). Meanwhile, the image / video decoder according to the present disclosure may be called an image / video decoding device, and the image / video decoder may be divided into an information decoder (image / video information decoder) and a sample decoder (image / video sample decoder). In this case, the information decoder may include an entropy decoding unit (410), and the sample decoder may include at least one of an inverse quantization unit (321), an inverse transformation unit (322), an addition unit (440), a filtering unit (450), a memory (460), an inter prediction unit (432), and an intra prediction unit (431).

[0107] The inverse quantization unit (421) can inverse quantize the quantized transform coefficients and output the transform coefficients. The inverse quantization unit (421) can rearrange the quantized transform coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scanning order performed in the image / video encoder. The inverse quantization unit (321) can perform inverse quantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain transform coefficients.

[0108] In the inverse transform unit (422), the transform coefficients are inversely transformed to obtain a residual signal (residual block, residual sample array).

[0109] The prediction unit (430) can perform a prediction on the current block and generate a predicted block containing prediction samples for the current block. Based on the prediction information output from the entropy decoding unit (410), the prediction unit can determine whether intra-prediction or inter-prediction is applied to the current block, and can determine a specific intra / inter-prediction mode.

[0110] The prediction unit (420) can generate a prediction signal based on various prediction methods. For example, the prediction unit can apply intra prediction or inter prediction for prediction of a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). In addition, the prediction unit can be based on an intra block copy (IBC) prediction mode or a palette mode for prediction of a block. The IBC prediction mode or palette mode can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in this document. The palette mode can be viewed as an example of intra coding or intra prediction. When palette mode is applied, information about the palette table and palette index can be signaled by including it in the image / video information.

[0111] The intra prediction unit (431) can predict the current block by referring to samples within the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from it, depending on the prediction mode. In intra prediction, the prediction modes may include multiple non-directional modes and multiple directional modes. The intra prediction unit (431) can also determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.

[0112] The inter prediction unit (432) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between the neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on the inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring blocks can include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit (432) can construct a motion information candidate list based on the neighboring blocks, and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter prediction can be performed based on various prediction modes, and the information about the prediction can include information indicating the mode of inter prediction for the current block.

[0113] The addition unit (440) can generate a restoration signal (restored picture, restoration block, restoration sample array) by adding the acquired residual signal to the prediction signal (predicted block, prediction sample array) output from the prediction unit (including the inter-prediction unit (432) and / or intra-prediction unit (431)). When there is no residual for the block to be processed, such as when skip mode is applied, the predicted block can be used as the restoration block.

[0114] The addition unit (440) may be referred to as a restoration unit or restoration block generation unit. The generated restoration signal may be used for intra prediction of the next processing target block within the current picture, may be output after filtering as described below, or may be used for inter prediction of the next picture.

[0115] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture decoding process.

[0116] The filtering unit (450) can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit (450) can apply various filtering methods to the restored picture to generate a modified restored picture, and transmit the modified restored picture to the memory (460), specifically, the DPB of the memory (460). The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0117] The (modified) reconstructed picture stored in the DPB of the memory (460) can be used as a reference picture in the inter prediction unit (432). The memory (460) can store motion information of a block from which motion information is derived (or decoded) within the current picture and / or motion information of blocks within a picture that has already been reconstructed. The stored motion information can be transmitted to the inter prediction unit (432) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (460) can store reconstructed samples of reconstructed blocks within the current picture and transmit them to the intra prediction unit (431).

[0118] Meanwhile, the VCM decoder (or feature / feature map decoder) may have a structure basically identical / similar to the image / video decoder (400) described above with reference to FIG. 4 in that it performs a series of procedures such as prediction, inverse transformation, and inverse quantization to decode a feature / feature map. However, the VCM decoder is different from the image / video decoder (400) in that it targets a feature / feature map for decoding, and accordingly, the name of each unit (or component) (e.g., DPB, etc.) and its specific operation contents may be different from the image / video decoder (400). The operation of the VCM decoder may correspond to the operation of the VCM encoder, and the specific operation contents will be described in detail later.

[0119] Feature / feature map encoding procedure

[0120] FIG. 5 is a flowchart schematically illustrating a feature / feature map encoding procedure to which embodiments of the present disclosure may be applied.

[0121] Referring to FIG. 5, the feature / feature map encoding procedure may include a prediction procedure (S510), a residual processing procedure (S520), and an information encoding procedure (S530).

[0122] The prediction procedure (S510) can be performed by the prediction unit (320) described above with reference to FIG. 3.

[0123] Specifically, the intra prediction unit (322) can predict the current block (i.e., the set of feature elements to be currently encoded) by referring to feature elements within the current feature / feature map. Intra prediction can be performed based on the spatial similarity of feature elements constituting the feature / feature map. For example, feature elements included in the same region of interest (RoI) within an image / video can be estimated to have similar data distribution characteristics. Accordingly, the intra prediction unit (322) can predict the current block by referring to the restored feature elements within the region of interest including the current block. At this time, the referenced feature elements may be positioned adjacent to the current block or spaced apart from the current block depending on the prediction mode. Intra prediction modes for feature / feature map encoding can include a plurality of non-directional prediction modes and a plurality of directional prediction modes. Non-directional prediction modes may include prediction modes corresponding to, for example, the DC mode and the planar mode of an image / video encoding procedure. Furthermore, directional modes may include prediction modes corresponding to, for example, the 33 directional modes or the 65 directional modes of an image / video encoding procedure. However, this is merely an example, and the types and number of intra-prediction modes may be set / changed in various ways depending on the embodiment.

[0124] The inter prediction unit (321) can predict the current block based on a reference block (i.e., a set of referenced feature elements) specified by motion information on a reference feature / feature map. Inter prediction can be performed based on the temporal similarity of feature elements constituting the feature / feature map. For example, temporally consecutive features can have similar data distribution characteristics. Therefore, the inter prediction unit (321) can predict the current block by referring to the restored feature elements of features that are temporally adjacent to the current feature. At this time, the motion information for specifying the referenced feature elements can include a motion vector and a reference feature / feature map index. The motion information can further include information regarding the inter prediction direction (e.g., L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, neighboring blocks may include spatial neighboring blocks existing in the current feature / feature map and temporal neighboring blocks existing in the reference feature / feature map. The reference feature / feature map including the reference block and the reference feature / feature map including the temporal neighboring blocks may be the same or different. The temporal neighboring blocks may be referred to as collocated reference blocks, and the reference feature / feature map including the temporal neighboring blocks may be referred to as collocated feature / feature maps. The inter prediction unit (321) may construct a motion information candidate list based on the neighboring blocks and generate information indicating which candidate is used to derive the motion vector and / or reference feature / feature map index of the current block.Inter prediction can be performed based on various prediction modes. For example, in the case of skip mode and merge mode, the inter prediction unit (321) can use the motion information of the surrounding blocks as the motion information of the current block. In the case of skip mode, unlike the merge mode, a residual signal may not be transmitted. In the case of motion vector prediction (MVP) mode, the motion vector of the surrounding blocks can be used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference. In addition to the intra prediction and inter prediction described above, the prediction unit (320) can generate a prediction signal based on various prediction methods.

[0125] The prediction signal generated by the prediction unit (320) can be used to generate a residual signal (residual block, residual feature elements) (S520). The residual processing procedure (S520) can be performed by the residual processing unit (330) described above with reference to FIG. 3. In addition, (quantized) transform coefficients can be generated through a transformation and / or quantization procedure for the residual signal, and the entropy encoding unit (340) can encode information about the (quantized) transform coefficients as residual information in the bitstream (S530). In addition to the residual information, the entropy encoding unit (340) can encode information necessary for feature / feature map restoration, such as prediction information (e.g., prediction mode information, motion information, etc.), in the bitstream.

[0126] Meanwhile, the feature / feature map encoding procedure may further include a procedure (S530) for encoding information for feature / feature map restoration (e.g., prediction information, residual information, partitioning information, etc.) and outputting it in the form of a bitstream, as well as a procedure for generating a restored feature / feature map for the current feature / feature map and a procedure (optional) for applying in-loop filtering to the restored feature / feature map.

[0127] The VCM encoder can derive (modified) residual feature(s) from the quantized transform coefficient(s) through inverse quantization and inverse transformation, and can generate a restored feature / feature map based on the predicted feature(s) and the (modified) residual feature(s) which are the outputs of step S510. The restored feature / feature map generated in this way can be identical to the restored feature / feature map generated by the VCM decoder. If an in-loop filtering procedure is performed on the restored feature / feature map, a modified restored feature / feature map can be generated through the in-loop filtering procedure on the restored feature / feature map. The modified restored feature / feature map can be stored in a decoded feature buffer (DFB) or memory, and can be used as a reference feature / feature map in a subsequent feature / feature map prediction procedure. In addition, (in-loop) filtering-related information (parameters) can be encoded and output in the form of a bitstream. Through the in-loop filtering procedure, noise that may occur during feature / feature map coding can be removed, and the performance of feature / feature map-based tasks can be improved. Furthermore, by performing the in-loop filtering procedure in both the encoder and decoder stages, the consistency of prediction results can be guaranteed, the reliability of feature / feature map coding can be improved, and the amount of data transmission for feature / feature map coding can be reduced.

[0128] Feature / feature map decoding procedure

[0129] FIG. 6 is a flowchart schematically illustrating a feature / feature map decoding procedure to which embodiments of the present disclosure may be applied.

[0130] Referring to FIG. 6, the feature / feature map decoding procedure may include an image / video information acquisition procedure (S610), a feature / feature map restoration procedure (S620 to S640), and an in-loop filtering procedure (S650) for the restored feature / feature map. The feature / feature map restoration procedure may be performed based on a prediction signal and a residual signal obtained through the inter / intra prediction (S620) and residual processing (S630), inverse quantization for quantized transform coefficients, and inverse transformation) processes described in the present disclosure. A modified restored feature / feature map may be generated through the in-loop filtering procedure for the restored feature / feature map, and the modified restored feature / feature map may be output as a decoded feature / feature map. The decoded feature / feature map is stored in the decoded feature buffer (DFB) or memory, and can be used as a reference feature / feature map in the inter prediction procedure during subsequent decoding of the feature / feature map. In some cases, the in-loop filtering procedure described above may be omitted. In this case, the reconstructed feature / feature map can be output as a decoded feature / feature map, and can be stored in the decoded feature buffer (DFB) or memory, and can be used as a reference feature / feature map in the inter prediction procedure during subsequent decoding of the feature / feature map.

[0131] Feature extraction methods and data distribution characteristics

[0132] Figure 7 is a diagram showing an example of a feature extraction method using a feature extraction network.

[0133] Referring to FIG. 7, a feature extraction network (700) can output a feature set (720) of a video source (710) by receiving a video source (710) as input and performing a feature extraction operation. The feature set (720) includes a plurality of features (C0, C1, ..., C) extracted from the video source (710). n ) can be included and can be expressed as a feature map. Each feature (C0, C1, ..., C n) contains multiple feature elements and may have different data distribution characteristics.

[0134] In FIG. 7, W, H, and C may represent the width, height, and number of channels of the video source (710), respectively. Here, the number of channels (C) of the video source (710) may be determined based on the image format of the video source (710). For example, if the video source (710) has an RGB image format, the number of channels (C) of the video source (710) may be 3.

[0135] In addition, W', H' and C' may represent the width, height and number of channels of the feature set (720), respectively. The number of channels (C') of the feature set (720) is the number of features (C0, C1, ..., C) extracted from the video source (710). n ) may be equal to the total number (n+1). In one example, the number of channels (C') of the feature set (720) may be greater than the number of channels (C) of the video source (710).

[0136] The properties (W', H', C') of the feature set (720) may vary depending on the properties (W, H, C) of the video source (710). For example, as the number of channels (C) of the video source (710) increases, the number of channels (C') of the feature set (720) may also increase. In addition, the properties (W', H', C') of the feature set (720) may vary depending on the type and properties of the feature extraction network (700). For example, when the feature extraction network (700) is implemented as an artificial neural network (e.g., CNN, DNN, etc.), each feature (C0, C1, ..., C n ) may also vary depending on the location of the layer outputting the feature set (720) properties (W', H', C').

[0137] The video source (710) and the feature set (720) may have different data distribution characteristics. For example, the video source (710) may generally consist of one (grayscale image) channel or three (RGB image) channels. The pixels included in the video source (710) may have the same integer value range for all channels and may have non-negative values. In addition, each pixel value may be evenly distributed within a predetermined integer value range. In contrast, the feature set (720) may consist of a variety of channels (e.g., 32, 64, 128, 256, 512, etc.) depending on the type of the feature extraction network (700) (e.g., CNN, DNN, etc.) and the layer location. The feature elements included in the feature set (720) may have different real value ranges for each channel and may have negative values. In addition, each feature element value may be concentrated in a specific area within a predetermined real value range.

[0138] Fig. 8a is a diagram showing data distribution characteristics of a video source, and Fig. 8b is a diagram showing data distribution characteristics of a feature set.

[0139] First, referring to Fig. 8a, the video source is composed of three channels: R, G, and B, and each pixel value can have an integer value range from 0 to 255. In this case, the data type of the video source can be expressed as an 8-bit integer type.

[0140] In contrast, referring to Fig. 8b, the feature set consists of 64 channels (features), and each feature element value can have a real value range from -∞ to +∞. In this case, the data type of the feature set can be expressed as a 32-bit floating point type.

[0141] A feature set can have floating-point feature element values, and each channel (or feature) can have different data distribution characteristics. An example of the data distribution characteristics for each channel of a feature set is shown in Table 1.

[0142]

[0143] Referring to Table 1, the feature set has a total of n+1 channels (C0, C1, ..., C n ) can be composed of. The mean (μ), standard deviation (σ), maximum (Max) and minimum (Min) values ​​of the feature elements are provided for each channel (C0, C1, ..., C n ) may be different for each channel. For example, the mean (μ) of the feature elements included in channel 0 (C0) may be 10, the standard deviation (σ) may be 20, the maximum (Max) may be 90, and the minimum (Min) may be 60. In addition, the mean (μ) of the feature elements included in channel 1 (C1) may be 30, the standard deviation (σ) may be 10, the maximum (Max) may be 70.5, and the minimum (Min) may be -70.2. In addition, the mean (μ) of the feature elements included in channel n (Cn) may be 100, the standard deviation (σ) may be 5, the maximum (Max) may be 115.8, and the minimum (Min) may be 80.2.

[0144] Quantization of features / feature maps can be performed, for example, based on the different data distribution characteristics for each channel described above. Floating-point type feature / feature map data can be converted to integer type through quantization.

[0145] Meanwhile, due to spatiotemporal similarities between consecutive frames, feature sets and / or channels sequentially extracted from a video source may have identical / similar data distribution characteristics. An example of the data distribution characteristics of consecutive feature sets is shown in Table 2.

[0146]

[0147] In Table 2, f F0refers to the first feature set extracted from frame 0 (F0), and f F1 refers to the second feature set extracted from frame 1 (F1), and f F2 refers to the third feature set extracted from frame 2 (F2).

[0148] Referring to Table 2, the first to third consecutive feature sets (f F0 , f F1 , f F2 ) can have the same / similar mean (μ), standard deviation (σ), maximum (Max) and minimum (Min) values.

[0149] Additionally, due to spatiotemporal similarities between consecutive frames, corresponding channels within feature sets sequentially extracted from a video source may have identical or similar data distribution characteristics. An example of the data distribution characteristics of corresponding channels within consecutive feature sets is shown in Table 3.

[0150]

[0151] In Table 3, f F0C0 refers to the first channel in the first feature set extracted from frame 0 (F0), and f F1C0 refers to the first channel in the second feature set extracted from frame 1 (F1).

[0152] Referring to Table 3, the first channel (f) within the first feature set F0C0 ) and the first channel (f) in the second feature set corresponding to it F1C0 ) can have the same / similar mean (μ), standard deviation (σ), maximum value (Max), and minimum value (Min).

[0153] Prediction for features / feature maps can be performed based on, for example, the similarity of data distribution characteristics between the feature sets or channels described above.

[0154] Transformation / Inverse transformation

[0155] As described above, the image / video encoder can derive a residual block based on a predicted block, and can apply transformation and quantization to the derived residual block to derive quantized transform coefficients. Similarly, the VCM encoder can derive a residual block based on a predicted block (or feature elements), and can apply transformation and quantization to the derived residual block to derive quantized transform coefficients for a feature / feature map. Information about the quantized transform coefficients (residual information) can be included in the residual coding syntax, encoded, and then output in the form of a bitstream.

[0156] A decoder (video decoder and VCM decoder) can obtain information about quantized transform coefficients (residual information) from a bitstream and decode the information to derive quantized transform coefficients. In addition, the decoder can derive a residual block through inverse quantization / inverse transformation based on the quantized transform coefficients. As described above, at least one of quantization / inverse quantization and / or transform / inverse transformation can be omitted. If transform / inverse transformation is omitted, the transform coefficient can be referred to as a coefficient or a residual coefficient, or can still be referred to as a transform coefficient for consistency of expression. Whether transform / inverse transformation is omitted can be signaled based on a predetermined flag, for example, transform_skip_flag. Meanwhile, in VCM, whether transform / inverse transformation is omitted can be determined on a feature / feature map basis or on a channel basis.

[0157] Transformation / inverse transformation can be performed based on transformation kernel(s). For example, according to embodiments, a multiple transform selection (MTS) scheme can be applied. In this case, some of a plurality of transformation kernel sets can be selected and applied to the current block. The transformation kernels can be referred to by various terms such as transformation matrix, transformation type, etc. For example, a transformation kernel set can represent a combination of vertical transformation kernels (vertical transformation kernels) and horizontal transformation kernels (horizontal transformation kernels).

[0158] MTS index information, e.g., mts_idx, may be signaled to indicate a set of transformation kernels for the current block or feature / feature map. An example of MTS index information is shown in Table 4.

[0159]

[0160] In Table 4, trTypeHor can represent a horizontal transform kernel, and trTypeVer can represent a vertical transform kernel. A trTypeHor / trTypeVer value of 0 can represent DCT2, a trTypeHor / trTypeVer value of 1 can represent DST7, and a trTypeHor / trTypeVer value of 2 can represent DCT8. However, this is just an example, and other values ​​may be mapped to other DCT / DST or transform kernels by convention.

[0161] In the present disclosure, the MTS-based transform / inverse transform is applied as a primary transform / inverse-transform, and a secondary transform / inverse-transform may be further applied. In this case, the transform at the encoder stage may proceed in the order of the primary transform and the secondary transform, and the transform at the decoder stage may proceed in the order of the secondary inverse transform and the primary inverse transform. In some embodiments, the secondary transform may be applied only to coefficients in the upper left low-frequency region of the coefficient block to which the primary transform is applied. Such a secondary transform may be referred to as a low frequency non-separable transform (LFNST). In addition, in some embodiments, the number of coefficients may be reduced as a result of performing the secondary transform. Such a secondary transform may be referred to as a reduced secondary transform (RST).

[0162] FIG. 9 is a diagram for explaining a transformation and inverse transformation process to which embodiments of the present disclosure may be applied. In FIG. 9, the transformation unit (910) may correspond to the transformation unit (332) of FIG. 3 , and the inverse transformation unit (920) may correspond to the inverse transformation unit (335) of FIG. 3 or the inverse transformation unit (422) of FIG. 4 .

[0163] Referring to FIG. 9, the conversion unit (910) may include a first conversion unit (911) and a second conversion unit (912).

[0164] The primary transform unit (911) can apply a primary transform to the residual data (A) (or residual feature elements) to generate (primary) transform coefficients (B). In the present disclosure, the primary transform may be referred to as a core transform. The primary transform may be performed based on an MTS scheme. When the existing MTS is applied, a transformation from the spatial domain to the frequency domain may be applied to the residual signal (or residual block) based on DCT type 2, DST type 7, DCT type 8, etc. to generate transform coefficients (or primary transform coefficients). Here, DCT type 2, DST type 7, DCT type 8, etc. may be referred to as a transform type, a transform kernel, or a transform core. However, this is merely an example, and the primary transform may also be applied to MTS kernels having different configurations from the above-described MTS kernels.

[0165] The secondary transform unit (912) can apply a secondary transform to the (primary) transform coefficients (B) to generate (secondary) transform coefficients (C). The secondary transform may be a non-separable transform, such as the aforementioned LFNST or RST. For example, when a 4×4 block is used, the non-separable secondary transform may be performed as follows.

[0166] A 4×4 input block X can be expressed as in Equation 1 below.

[0167] [Formula 1]

[0168]

[0169] If the input block X is represented in vector form, can be expressed as the following equation 2.

[0170] [Formula 2]

[0171]

[0172] In this case, the non-separable second-order transform can be calculated as shown in Equation 3 below.

[0173] [Formula 3]

[0174]

[0175] Here, represents a transformation coefficient vector, T represents a 16×16 non-separable transformation matrix, and · represents the multiplication of a matrix and a vector.

[0176] 16×1 transformation coefficient vector through the above mathematical formula can be derived, can be reconstructed into 4×4 blocks according to the scan order (e.g., horizontal, vertical, diagonal, or predefined / stored scan order).

[0177] Next, the inverse transformation unit (820) may include a (inverse) secondary transformation unit (821) and a (inverse) primary transformation unit (822).

[0178] The (inverse) secondary transform unit (921) can apply an (inverse) secondary transform to the inverse quantized (secondary) transform coefficients (C') to generate (first) (inverse) transform coefficients (B'). Here, the (inverse) secondary transform may correspond to the reverse process of the secondary transform performed by the transform unit (910). The (inverse) secondary transform may be applied to the upper left corner of the coefficient block according to an embodiment.

[0179] The (inverse) first transformation unit (922) can generate residual samples (A') by applying the (inverse) first transformation to the (inverse) transformation coefficients (B').

[0180] Quantization / dequantization

[0181] As described above, the quantization unit (333) of the image / video encoder (300) can apply quantization to transform coefficients to derive quantized transform coefficients, and the inverse quantization unit (334) of the image / video encoder (300) or the inverse quantization unit (421) of the image / video decoder (400) can apply inverse quantization to the quantized transform coefficients to derive transform coefficients. Similarly, the VCM encoder can apply quantization to transform coefficients to derive quantized transform coefficients, and the inverse quantization unit of the VCM encoder or the inverse quantization unit of the VCM decoder can apply inverse quantization to the quantized transform coefficients to derive transform coefficients. In general, in feature / feature map coding, the quantization rate can be changed, and the compression rate can be adjusted using the changed quantization rate. From an implementation perspective, a quantization parameter (QP) can be used instead of directly using the quantization rate in consideration of complexity. For example, quantization parameters with integer values ​​from 0 to 63 can be used, and each quantization parameter value can correspond to an actual quantization rate. The quantization parameter (QPY) for the luma component (luma sample) and the quantization parameter (QPC) for the chroma component (chroma sample) can be set differently.

[0182] The quantization process takes a transform coefficient (C) as input, divides it by a quantization rate (Qstep), and obtains a quantized transform coefficient (C') based on this. In this case, considering the computational complexity, the quantization rate can be multiplied by a scale to make it into an integer, and a shift operation can be performed by a value corresponding to the scale value. The quantization scale can be derived based on the product of the quantization rate and the scale value. In other words, the quantization scale can be derived according to the QP. The quantized transform coefficient (C') can also be derived based on this by applying the quantization scale to the transform coefficient (C).

[0183] The inverse quantization process is the reverse process of the quantization process. By multiplying the quantized transform coefficient (C') by the quantization rate (Qstep), a restored transform coefficient (C'') can be obtained based on this. In this case, a level scale can be derived according to the quantization parameter, and by applying the level scale to the quantized transform coefficient (C'), a restored transform coefficient (C'') can be derived based on this. The restored transform coefficient (C'') may be somewhat different from the original transform coefficient (C) due to loss in the transformation and / or quantization process. Therefore, inverse quantization is performed in the encoder in the same way as in the decoder.

[0184] Meanwhile, an adaptive frequency-based weighted quantization technique that adjusts the quantization strength according to frequency may be applied. The adaptive frequency-based weighted quantization technique is a method of applying different quantization strengths to different frequencies. The adaptive frequency-based weighted quantization can apply different quantization strengths to each frequency using a predefined quantization scaling matrix. That is, the quantization / dequantization process described above may be performed further based on the quantization scaling matrix. For example, a different quantization scaling matrix may be used depending on the size of the current block and / or whether the prediction mode applied to the current block to generate the residual signal of the current block is inter-prediction or intra-prediction. The quantization scaling matrix may be referred to as a quantization matrix or a scaling matrix. The quantization scaling matrix may be predefined. In addition, for frequency-adaptive scaling, frequency-based quantization scale information for the quantization scaling matrix may be configured / encoded in the encoder and signaled to the decoder. The frequency-specific quantization scale information may be referred to as quantization scaling information. The frequency-specific quantization scale information may include scaling list data (scaling_list_data). A (modified) quantization scaling matrix may be derived based on the scaling list data. In addition, the frequency-specific quantization scale information may include a presence flag (present flag) information indicating whether the scaling list data exists. Alternatively, if the scaling list data is signaled at a higher level (e.g., sequence level, feature set group level, etc.), information indicating whether the scaling list data is modified at a lower level (e.g., feature set level, channel level, etc.) may be further included.

[0185] Feature quantization / dequantization can be performed based on a predetermined quantization group. Specifically, multiple quantization intervals can be set based on the data distribution characteristics of the feature set. The set quantization intervals have different data distribution ranges and can be defined as a single quantization group. In addition, feature quantization / dequantization operations can be performed by converting the data distribution of each channel within the feature set into the data distribution range of one of the quantization intervals.

[0186] FIG. 10 is a diagram illustrating an example of a quantization group according to one embodiment of the present disclosure.

[0187] Referring to FIG. 10, a quantization group may include four quantization intervals (A, B, C, D) having different data distribution ranges. Each quantization interval (A, B, C, D) within a quantization group is defined using a minimum value and a maximum value, and may be set based on the data distribution characteristics of the current feature / feature map. For example, quantization interval A may be set to [-1, 3], quantization interval B may be set to [0, 2], quantization interval C may be set to [-2, 4], and quantization interval D may be set to [-2, 1].

[0188] The VCM encoder can quantize each channel (or feature) in a feature set based on preset quantization intervals (A, B, C, D) without having to separately calculate the maximum and minimum values ​​of feature elements. For example, channel 1 can be quantized based on quantization interval A having the most similar data distribution range. In addition, channel 2 can be quantized based on quantization interval B having the most similar data distribution range. In addition, channel 3 can be quantized based on quantization interval C having the most similar data distribution range. In addition, channel 4 can be quantized based on quantization interval D having the most similar data distribution range. In this case, the VCM encoder can encode / signal the number, minimum value, and maximum value of quantization intervals as feature quantization-related information. Additionally, the VCM encoder can signal quantization interval index information, which represents the quantization interval used for encoding the current channel (or feature), as feature quantization-related information. Accordingly, there is no need to signal the number of quantization bits for each channel and the maximum and minimum values ​​of feature elements, which can reduce the amount of transmission bits and further improve encoding / signaling efficiency.

[0189] The VCM decoder can form a quantization group identical to the quantization group formed by the VCM encoder based on the number of quantization intervals received from the VCM encoder and the minimum and maximum values ​​of each interval. Alternatively, the VCM decoder can form a quantization group based on the data distribution characteristics of the restored feature sets. In addition, the VCM decoder can dequantize the current feature based on the quantization interval identified by the quantization interval index information received from the VCM encoder.

[0190] An example of a feature quantization operation based on a quantization group is as shown in Equations 4 and 5.

[0191] [Formula 4]

[0192]

[0193] [Formula 5]

[0194]

[0195] In Equation 4, F n is a feature set ( ) can mean the nth channel (n is an integer greater than or equal to 1). Here, R can mean the width of the feature set, and C can mean the height of the feature set. In addition, max(Fn) can mean the nth channel (F n ) means the maximum value of my feature elements, and min(F n ) is the nth channel (F n ) can mean the minimum value of my feature elements.

[0196] Referring to Equation 4, the nth channel (F n ) can be normalized to have feature element values ​​between 0 and 1 based on the maximum and minimum values ​​of the feature elements within the channel.

[0197] In formula 5, Silver feature set ( ) can mean the normalized nth channel. Here, R can mean the width of the feature set, and C can mean the height of the feature set. Also, max( ) is the nth channel (F n ) applied to the quantization interval ( ) means the maximum value, and min( ) is the nth channel (F n ) applied to the quantization interval ( ) can mean the minimum value.

[0198] Referring to Equation 5, the normalized nth channel ( ) is the nth channel (F n ) applied to the quantization interval ( ) maximum value (max( )) and minimum value (min( )) can be quantized based on.

[0199] Meanwhile, quantization-related information may be encoded in the bitstream for inverse quantization. Depending on the embodiment, the quantization-related information may include global quantization information, activation function information, quantization bit count, and quantization interval index information. The global quantization information may indicate whether the quantization bit count is set for each quantization interval or is set to be the same for all quantization intervals. The activation function information may indicate the type of activation function applied to the current feature. For example, the activation function information may indicate whether the activation function applied to the current feature is a first activation function that must signal both the maximum and minimum values ​​of each quantization interval or a second activation function that only needs to signal the maximum value of each quantization interval. The quantization bit count may be set for each quantization interval based on the global quantization information described above, or may be set to be the same for all quantization intervals. If the quantization bit count is set to be the same for all quantization intervals, the quantization bit count may be signaled only once for all quantization intervals.

[0200] In addition, the quantization-related information may further include the number of quantization intervals within the quantization group and the minimum and maximum values ​​of each quantization interval. In this case, the minimum and maximum values ​​of each quantization interval may be adaptively signaled based on the type of the activation function. For example, if the activation function applied to the current feature is the first activation function (e.g., Leaky ReLU), both the minimum and maximum values ​​of each quantization interval may be signaled. In contrast, if the activation function applied to the current feature is the second activation function (e.g., ReLU), the minimum value of each quantization interval is estimated to be 0, and only the maximum value of each quantization interval may be signaled.

[0201] Entropy coding

[0202] As described above, part or all of the feature / feature map information may be encoded by the entropy encoding unit (340), and part or all of the feature / feature map information may be decoded by the entropy decoding unit (410). In this case, the feature / feature map information may be encoded / decoded in units of syntax elements, similar to image / video information. In the present disclosure, encoding / decoding information may include encoding / decoding by the method described in this paragraph.

[0203] Figure 11 shows a block diagram of CABAC for encoding one syntax element.

[0204] Referring to Fig. 11, the encoding process of CABAC can first convert the input signal into a binary value through binarization if the input signal is a syntax element rather than a binary value. If the input signal is already a binary value, it can be bypassed without going through binarization. Here, each binary 0 or 1 that constitutes the binary value can be called a bin. For example, if the binary string (bin string) after binarization is 110, 1, 1, and 0 can each be called a bin. The bin(s) for one syntax element can represent the value of the corresponding syntax element.

[0205] Binarized bins can be input to a regular coding engine or a bypass coding engine. The regular coding engine can assign a context model that reflects the probability value for each bin and encode the bin based on the assigned context model. The regular coding engine can update the probability model for each bin after coding. Bins coded in this way are called context-coded bins. The bypass coding engine can omit the process of estimating the probability of each input bin and updating the probability model applied to the bin after coding. Instead of assigning a context, the engine can apply a uniform probability distribution (e.g., 50:50) to the input bins, thereby improving coding speed. Bins coded in this way are called bypass bins. The context model can be assigned and updated for each bin subject to context coding (regular coding), and the context model can be indicated based on ctxidx or ctxInc. ctxidx can be derived based on ctxInc. Specifically, for example, the context index (ctxidx) indicating the context model for each of the regular coded bins can be derived as the sum of ctxInc (context index increment) and ctxIdxOffset (context index offset). Here, ctxInc can be derived differently for each bin. ctxIdxOffset can be represented as the lowest value of ctxIdx. ctxIdxOffset can generally be determined according to the slice type, and the context model for one syntax element in the slice can be distinguished / derived based on ctxInc.

[0206] In the entropy encoding process, encoding can be performed via the regular coding engine or the bypass coding engine, and the coding path can be switched. Entropy decoding can be performed in reverse order, similar to the entropy encoding process.

[0207] The entropy coding described above can be performed, for example, as shown in FIGS. 12 and 13.

[0208] Figures 12 and 13 are drawings for explaining the entropy encoding procedure.

[0209] Referring to FIGS. 12 and 13, an encoder (entropy encoding unit) may perform an entropy coding procedure on feature / feature map information. The feature / feature map information may include prediction-related information (e.g., inter / intra prediction distinction information, intra prediction mode information, inter prediction mode information, etc.), residual information, in-loop filtering-related information, etc., or may include various syntax elements related thereto. Such entropy coding may be performed on a syntax element basis. The entropy encoding unit may be the entropy encoding unit (340) of the image / video encoder (300) of FIG. 3 described above.

[0210] The encoder can perform binarization on the target syntax element (S1310). Here, binarization can be based on various binarization methods, such as the Truncated Rice binarization process and the Fixed-length binarization process, and the binarization method for the target syntax element can be predefined. The binarization procedure can be performed by the binarization unit (1301) within the entropy encoding unit (1300).

[0211] The encoder can perform entropy encoding on the target syntax element (S1220). The encoder can encode the empty string of the target syntax element based on a regular coding-based (context-based) or bypass coding-based entropy coding technique such as context-adaptive arithmetic coding (CABAC) or context-adaptive variable length coding (CAVLC), and the output thereof can be included in the bitstream. The entropy encoding procedure can be performed by the entropy encoding processing unit (1402) within the entropy encoding unit (1300). As described above, the output bitstream can be transmitted to the decoder via a (digital) storage medium or a network.

[0212] Figures 14 and 15 are drawings for explaining the entropy decoding procedure.

[0213] Referring to FIGS. 14 and 15, a decoder (entropy decoding unit) can decode encoded feature / feature map information. The feature / feature map information can include prediction-related information (e.g., inter / intra prediction distinction information, intra prediction mode information, inter prediction mode information, etc.), residual information, in-loop filtering-related information, etc., or can include various syntax elements related thereto. Entropy coding can be performed in units of syntax elements. The entropy decoding unit can be the entropy decoding unit (410) of the image / video decoder (400) of FIG. 4 described above.

[0214] The decoder can perform binarization on the target syntax element (S1410). Here, the binarization can be based on various binarization methods such as the Truncated Rice binarization process and the Fixed-length binarization process, and the binarization method for the target syntax element can be predefined. The decoder can derive available empty strings (empty string candidates) for available values ​​of the target syntax element through the binarization process. The binarization process can be performed by the binarization unit (1501) within the entropy decoding unit (1500).

[0215] The decoder can perform entropy decoding on the target syntax element (S1420). The decoder can sequentially decode and parse each bin for the target syntax element from the input bit(s) in the bitstream, and compare the derived bin string with the available bin strings for the corresponding syntax element. If the derived bin string is equal to one of the available bin strings, the value corresponding to the bin string can be derived as the value of the corresponding syntax element. If not, the next bit in the bitstream can be further parsed and the above-described procedure can be performed again. Through this process, it is possible to signal specific information (specific syntax element) using variable-length bits without using start bits or end bits in the bitstream. This allows relatively fewer bits to be allocated to low values, thereby improving overall coding efficiency.

[0216] The decoder can decode each bin within a bin string from a bitstream based on a context-based or bypass-based entropy coding technique such as CABAC or CAVLC. The entropy decoding procedure can be performed by an entropy decoding processing unit (1502) within an entropy decoding unit (1500). As described above, the bitstream can include various information for feature / feature map decoding. As described above, the bitstream can be transmitted to a decoding device via a (digital) storage medium or a network.

[0217] In the present disclosure, a table including syntax elements (a syntax table) may be used to indicate signaling of information from an encoder to a decoder. The order of the syntax elements in the table including the syntax elements used in the present disclosure may indicate a parsing order of the syntax elements from a bitstream. An encoder may construct and encode a syntax table such that the syntax elements can be parsed by a decoder in the corresponding parsing order, and the decoder may parse and decode the syntax elements of the corresponding syntax table from a bitstream in the corresponding parsing order to obtain the values ​​of the syntax elements.

[0218] Coding hierarchy and structure

[0219] Figure 16 illustrates an example of a VCM hierarchy.

[0220] Referring to FIG. 16, the VCM hierarchy may be composed of a feature extraction layer (1610), a neural network (feature) abstraction layer (1620), and a feature coding layer (1630).

[0221] The feature extraction layer (1610) refers to a layer for extracting features from an input source, and may also include the results of the extraction.

[0222] The feature coding layer (1630) refers to a layer for compressing extracted features, and may also include the results of compression.

[0223] The neural network (feature) abstraction layer (1620) can abstract information generated by the feature extraction layer (1610) (e.g., information about extracted features / feature maps) and pass it to the feature coding layer (1630). The neural network abstraction layer (1720) can hide the internals of the feature extraction layer (1610) through information abstraction and provide a consistent feature interface function. Accordingly, even if the compression target changes due to a change in tool (e.g., CNN, DNN, etc.), the feature coding layer (1630) can perform a consistent feature coding procedure. In the present disclosure, the neural network abstraction layer (NNAL) may also be referred to as a feature abstraction layer.

[0224] The interface between the feature extraction layer (1610) and the neural network abstraction layer (1620) and the interface between the feature coding layer (1630) and the neural network abstraction layer (1620) can be defined in advance, and the operation in the neural network abstraction layer (1620) can be set to be changed later.

[0225] Figure 17 is a diagram showing an example of a bitstream composed of encoded abstract features and NNAL information.

[0226] A bitstream configured as illustrated in Fig. 17 may be referred to as an NNAL (Neural Network Abstraction Layer) unit. An NNAL unit may be an independent feature restoration unit. The input features for an NNAL unit may be extracted from the same layer within a neural network. Accordingly, the input features for an NNAL unit may be forced to have the same characteristics. For example, the same feature extraction method may be applied to the input features for an NNAL unit.

[0227] An NNAL unit may include an NNAL unit header and an NNAL unit payload. The NNAL unit header may include all information necessary to utilize encoded features for a task. The NNAL unit payload may include abstracted feature information. The NNAL unit payload may include a group header and group data. The group header may include configuration information of feature group data, such as information about the temporal order, number, and common properties of feature channels constituting the feature group. A feature channel may mean an encoded feature unit. The group data may include a plurality of feature channels and a coding indicator, and each feature channel may include type information, prediction information, side information, and residual information (data). In this case, the type information may indicate an encoding method code, and the prediction information may indicate a prediction method. Additionally, side information may represent additional information required for decoding (e.g., entropy coding, quantization-related information, etc.), and residual information may include information about encoded feature elements (i.e., a set of feature value information).

[0228] The present disclosure describes an encoding method applied according to the purpose of a user or machine, a suitable use of an encoded bitstream, and a method for expressing properties.

[0229] FIG. 18 is a diagram for exemplarily explaining encoding optimization and encoding bitstream usage according to a purpose to which an embodiment according to the present disclosure can be applied.

[0230] Referring to Figure 18, Component A may represent an example of a video provider. The video may include both real-time encoding and pre-encoded bitstreams. The encoder of Component A may include a function for optimizing input data for its intended use. Here, optimization may be achieved through preprocessing, encoding, and postprocessing.

[0231] If the video is used for machine analysis, the goal may be to achieve the desired machine task performance using fewer bits and resources. In this case, optimization methods may vary depending on the properties of the machine task being performed.

[0232] If the video is intended for human viewing, the goal may be to convey the desired information to the user using fewer bits and resources. In this case, optimization methods may vary depending on the user's characteristics, i.e., their visual system.

[0233] If the video's destination or purpose changes, the optimization method may also change. Users or machines must be able to identify whether the bitstream they are currently receiving is optimized, the scope of optimization, and the optimization method. This allows them to determine the suitability of the received bitstream and request an optimization method that best suits their purpose.

[0234] Below, we describe a method for expressing optimization objectives, properties, and scope. Using this method, users or machines can assess the suitability of optimization methods, enabling them to efficiently request and utilize optimization methods that meet their objectives, thereby improving system efficiency.

[0235] The representation method defined in this embodiment may be included in the encoded video bitstream and information required for transmitting the encoded video. For example, such information may be transmitted via a supplemental enhancement information (SEI) message.

[0236] Tables 5 through 7 below illustrate examples of syntax structures representing optimization information encoded in the encoder. First, syntax elements that can be signaled depending on whether optimization has occurred are described with reference to Table 5.

[0237]

[0238] According to Table 5, the encoded optimization information (encoder_optimization_info) may include syntax elements: an optimization persistence cancel flag (optimization_cancel_flag), an optimization persistence flag (optimization_persistence_flag), a first syntax element (optimization_for_human_viewing_flag), a second syntax element (optimization_for_machine_analysis_flag), and optimization attribute information (optimization_type).

[0239] The syntax element optimization_cancel_flag can indicate whether the persistence of the optimization being applied is canceled. If the value of the optimization persistence cancel flag is 1, it can indicate that the persistence of previously applied optimizations is canceled. If the value is 0, it can indicate that optimization_persistence_flag, optimization_for_human_viewing_flag, optimization_for_machine_analysis_flag, and optimization_type are defined subsequently. It can indicate that the optimizations and scopes identified by optimization_persistence_flag, optimization_for_machine_analysis_flag, and optimization_type defined subsequently can be applied.

[0240] The syntax element optimization_persistence_flag can indicate the persistence of the optimization indicated by optimization_type. A value of 1 indicates that the optimization identified by optimization_type can be applied to the current picture and all subsequent pictures. A value of 0 indicates that the optimization identified by optimization_type can only be applied to the current picture.

[0241] The first syntax element, optimization_for_human_viewing_flag, can indicate whether the optimization goal is for human viewing. A value of 1 indicates that it is for human viewing. In other words, it can indicate that the optimization goal and the encoded bitstream are for human viewing (suitable for human viewing). A value of 0 indicates that it is for human viewing or not. In other words, the optimization goal and the encoded bitstream may or may not be for human viewing.

[0242] The second syntax element, optimization_for_machine_analysis_flag, can indicate whether the optimization objective is for machine analysis. If its value is 1, it can indicate that it is for machine analysis. In other words, it can indicate that the optimization objective and the encoded bitstream are for machine analysis (suitable for performing machine tasks). If its value is 0, it can indicate that it is for machine analysis or not. In other words, the optimization objective and the encoded bitstream may or may not be for machine analysis.

[0243] In various embodiments, optimizations may have unintended consequences on encoding quality. For example, optimizations may be performed for machine analysis, but may result in improved quality for human viewing. In this case, optimization_for_machine_analysis_flag may be set to 1, and optimization_for_human_viewing_flag may be set to 0. This allows for a clearer definition of the intended purpose of the optimization. In another example, optimizations may be performed for human viewing, but may result in improved quality for machine analysis. In this case, optimization_for_machine_analysis_flag may be set to 0, and optimization_for_human_viewing_flag may be set to 1. In other words, neither optimization_for_human_viewing_flag nor optimization_for_machine_analysis_flag may be set to 0.

[0244] In the present disclosure, optimization_for_human_viewing_flag and optimization_for_machine_analysis_flag are each referred to as flags, but are not limited thereto and may also be referred to as indicators.

[0245] Referring to Table 5, optimization_cancel_flag is defined, and optimization_persistence_flag, optimization_for_human_viewing_flag, optimization_for_machine_analysis_flag, and optimization_type can be defined through optimization_cancel_flag (i.e., if the value of optimization_cancel_flag is 0 (!optimization_cancel_flag)).

[0246] The definition of the values ​​of optimization_for_human_viewing_flag and optimization_for_machine_analysis_flag may have the following restrictions. For example, if the optimization purpose is limited to human view and machine analysis, the values ​​may not be 0 at the same time. In other words, the values ​​of optimization_for_human_viewing_flag and optimization_for_machine_analysis_flag may not be 0 at the same time. In this case, if the preceding flag value among optimization_for_human_viewing_flag and optimization_for_machine_analysis_flag is 0, the following flag value becomes 1, so there is no need to encode the corresponding value in the bitstream. The decoder or receiver can process the value as 1. Tables 6 and 7 below can show examples.

[0247]

[0248]

[0249] According to Table 6 above, if the value of optimization_for_human_viewing_flag is 1, optimization_for_machine_analysis_flag is defined, and if the value of optimization_for_human_viewing_flag is 0, optimization_for_machine_analysis_flag can be processed as a value of 1. In other words, there is no need to encode the value of optimization_for_machine_analysis_flag in the bitstream on the encoder side, and the decoder side can also infer that the value is 1 without parsing it.

[0250] In another embodiment, according to Table 7 above, if the value of optimization_for_machine_analysis_flag is 1, optimization_for_human_viewing_flag is defined, and if the value of optimization_for_machine_analysis_flag is 0, optimization_for_human_viewing_flag can be processed as a value of 1. In other words, there is no need to encode the value of optimization_for_human_viewing_flag in the bitstream on the encoder side, and the decoder side can also infer that value as 1 without parsing it.

[0251] optimization_type can indicate the properties (or type) of an optimization method. The optimization properties identified through optimization_type can be expressed as shown in Table 8 below.

[0252]

[0253] According to Table 8, the properties of an optimization method can be defined depending on the value of optimization_type. The optimization properties defined in Table 8 are only examples, and new optimization properties can be defined using this structure. The optimization properties and methods can be identified by including only some of the optimization properties defined in Table 8, or they can be configured to include other optimization properties and methods not defined in Table 8.

[0254] Table 9 below illustrates an example that can perform the same function as Table 8, but using a different representation. In other words, Table 9 is another example of representing multiple optimization methods through a bit mask for optimization_type.

[0255]

[0256] According to Table 9, color vision deficiency optimization is for human vision, so its value may be defined only when the value of optimization_for_human_viewing_flag is 1.

[0257] As another embodiment, Table 10 below illustrates an embodiment of a syntax for additionally expressing additional information required according to optimization properties.

[0258]

[0259] According to Table 10, the additional information may include a first optimization objective flag (optimizationForCVDFlag), color vision deficiency type identification information (CVD_type_idc), and color vision deficiency optimization method identification information (CVD_optimization_method_idc). If optimizationForCVDFlag is true (or its value is 1), CVD_type_idc and CVD_optimization_method_idc may be defined.

[0260] The variable optimizationForCVDFlag can indicate whether optimization_type includes color vision optimization. If it does, its value will be 1 and can be derived by "optimizationForCVDFlag = ((optimization_type & 0x20) > 0) ? 1 : 0". That is, it can be derived according to the values ​​defined in Table 9.

[0261] The syntax element CVD_type_idc may indicate identification information for identifying the type of color vision deficiency. Table 11 below may show examples of types of color vision deficiency.

[0262]

[0263] CVD_optimization_method_idc may represent identification information indicating the type of optimization method for color vision deficiency. Table 12 below may show an example of the type of optimization method for color vision deficiency.

[0264]

[0265] Table 13 below illustrates an example that can perform the same function as Table 12, but using a different representation method. In other words, Table 13 is another example of representing the type of optimization method through a bit mask for identifying information on color vision deficiency optimization methods.

[0266]

[0267] Optimization for color vision deficiency may also be included in spatial quality. In this case, unlike in Table 8-9, optimization_type for color vision deficiency optimization may not be defined separately. Table 14 below provides an example of expressing optimization properties and objectives without separately defining optimization property information.

[0268]

[0269] Table 15 below illustrates an example that can perform the same function as Table 14, but using a different representation. In other words, Table 15 illustrates another example of expressing multiple optimization properties and objectives through bit masks.

[0270]

[0271] As another example, Table 16 below shows an example of a syntax structure for additionally expressing additional information required according to optimization properties.

[0272]

[0273] According to Table 16, the additional information may include a second optimization objective flag (spatialQualityOptimization_flag) and spatial quality optimization method identification information (spatiai_quality_method_idc). If spatialQualityOptimization_flag is true (or its value is 1), spatiai_quality_method_idc may be defined.

[0274] The variable spatialQualityOptimization_flag can indicate whether optimization_type includes spatial quality optimization. If so, its value will be 1 and can be derived by "spatialQualityOptimization_flag = ((optimization_type & 0x10 ) > 0 ) ? 1 : 0".

[0275] The syntax element spatiai_quality_method_idc may identify the type of optimization method when optimization_type includes spatial quality optimization. Table 17 below may provide examples of types of spatial quality optimization methods.

[0276]

[0277] Table 17 shows an example of defining one optimization method for each identifier, and the purpose of activating color vision deficiency can be expressed more specifically as in Table 18 below.

[0278]

[0279] According to Table 18, the identifier '100' may indicate that the purpose of color vision deficiency optimization is to reduce the colors of the video to fit a color space that the user can perceive. Furthermore, the identifier '101' may indicate that the purpose of color vision deficiency optimization is to transform the colors of the video so that the user can distinguish them.

[0280] Table 19 below shows an example that can perform the same function as Table 17 but with a different expression method, and Table 20 below shows an example that can perform the same function as Table 18 but with a different expression method. In other words, Table 19 shows another example that expresses multiple optimization methods through a bit mask, and Table 20 shows another example that expresses multiple optimization methods / objectives through a bit mask.

[0281]

[0282]

[0283] In various embodiments, Tables 21 to 23 below illustrate embodiments that define optimization and encoding properties for a set of information defined in an encoded bitstream, such as a sequence parameter set and a picture parameter set. Specifically, Tables 21 and 23 may illustrate examples defined in a sequence parameter set, and Table 22 may illustrate examples defined in a picture parameter set.

[0284]

[0285]

[0286]

[0287] According to Table 21, the color vision deficiency optimization flag (color_vision_deficiency_optimization_flag), which is a syntax element, can be included in the sequence parameter set information (seq_parameter_set_rbsp). According to Table 22, the color_vision_deficiency_optimization_flag can be included in the picture parameter set information (picture_parameter_set_rbsp).

[0288] The syntax element color_vision_deficiency_optimization_flag can indicate whether the encoded bitstream is optimized for color vision deficiencies. A value of 1 indicates that the encoded bitstream is optimized for color vision deficiencies. A value of 0 indicates that the encoded bitstream is not optimized for color vision deficiencies.

[0289] Table 21 above shows an example of defining color_vision_deficiency_optimization_flag in a sequence parameter set, and Table 22 above shows an example of defining color_vision_deficiency_optimization_flag in a picture parameter set.

[0290] The syntax elements described above may be defined in other locations (e.g., profile_tier_level, decoding capability information, etc.) other than those defined in Tables 21 to 23.

[0291] According to Table 23, the sequence parameter set information may include optimization target identification information (optimization_idc), which is a syntax element.

[0292] optimization_idc is identification information indicating the optimization target of the encoded bitstream, and an example of its value can be expressed as shown in Table 24 below.

[0293]

[0294] According to Table 24, a value of 00 for the optimization target identification information indicates a bitstream optimized for machine analysis. A value of 01 indicates a bitstream optimized for people with normal visual systems. Furthermore, a value of 11 indicates a bitstream optimized for people with visual systems different from the normal visual system, such as those with color vision deficiency. In addition to the optimization targets and methods defined above, other values ​​may be defined.

[0295] Figure 19 is a drawing showing an example of a profile definition structure.

[0296] In Fig. 19, a profile can indicate constraints and purposes for a coding tool (algorithm) used for video encoding / decoding. Each profile is defined independently and can be distinguished by an identifier indicating the profile. For example, the profile identifier (gener_profile_idc) of Main 10 can be defined as 10, and the profile identifier (gener_profile_idc) of Multilayer main 10 can be defined as 17.

[0297] Information on optimization and encoding properties may be added to the set of information and profile defined as in Fig. 19, but may also be defined separately for the profile. For example, in the case of abnormal color vision (e.g., color vision deficiency), the color information that a user can perceive may be different from that of a user with normal color vision. Therefore, the encoder may allocate more bits to the color information that a user with abnormal color vision can perceive, or may convert the color information for information transmission. The encoding method and coding tool applied may be defined according to the values ​​of optimization_idc, color_vision_deficiency_optimization_flag, and general_profile_idc defined in this embodiment. For example, if the value of the color vision deficiency optimization flag is 1, the optimization target identification information is 11, or the profile is for abnormal color vision, the encoder may encode by applying differential quality to U and V in the YUV color space depending on the type and degree of the color vision deficiency. As another example, the encoder may reduce or remove information that is imperceptible to users with abnormal color vision, or change it into perceptible information. Since these operations are unnecessary for users with normal color vision, the corresponding coding tool may not be necessary for them. Since the coding tool applied may differ depending on the optimization target, a distinction may be necessary at the profile level. For example, a profile for abnormal color vision (e.g., color vision deficiency) (e.g., Main 10 abnormal color vision) may be defined separately while corresponding to the Main 10 profile. In another embodiment, in addition to the Main 10 profile, other profiles defined in FIG. 21 may also be used to further define profiles for specific optimizations and purposes. In this case, a definition for the added profile may be required, as shown in Table 25 below.

[0298]

[0299] According to Table 25, gener_profile_idc can be defined in the sequence parameter set.

[0300] Depending on the values ​​of optimization_idc and color_vision_deficiency_optimization_flag defined in this embodiment, applicable and usable coding tools and encoding methods may also vary.

[0301] Below, we describe the syntax elements that individually represent the applied optimization method / property.

[0302] Tables 26 and 27 below show examples of syntax structures representing optimization information, including syntax elements that individually express optimization properties and scopes of application.

[0303]

[0304] According to Table 26, the encoded optimization information may include syntax elements indicating optimization_cancel_flag, optimization_persistence_flag, optimization_for_human_viewing_flag, optimization_for_machine_analysis_flag, and optimization_type.

[0305] Syntax elements indicating optimization_type may include an object-based optimization flag (bject_based_flag), a temporal resampling optimization flag (temporal_resampling_flag), a spatial resampling optimization flag (spatial_resampling_flag), a temporal quality optimization flag (temporal_quality_flag), a spatial quality optimization flag (spatial_quality_flag), and a color vision deficiency optimization flag (color_vision_deficiency_flag).

[0306] The object_based_flag flag can indicate whether object-based optimization is applied. A value of 1 indicates that object-based optimization (e.g., preprocessing, encoding) has been applied, while a value of 0 indicates that object-based optimization has not been applied. For example, a value of 1 for the object-based optimization flag indicates that optimization has been performed to improve the recognition performance of the target object.

[0307] The temporal_resampling_flag flag can indicate whether optimization using temporal resampling is applied. A value of 1 indicates that optimization using temporal resampling is applied, while a value of 0 indicates that optimization using temporal resampling is not applied. Temporal resampling can include both downsampling and upsampling.

[0308] The spatial_resampling_flag flag can indicate whether optimization using spatial resampling is applied. A value of 1 indicates that optimization using spatial resampling is applied, while a value of 0 indicates that optimization using spatial resampling is not applied. Spatial resampling can include both downsampling and upsampling.

[0309] The temporal_quality_flag flag can indicate whether temporally differential quality is applied. A value of 1 indicates that temporally differential quality is applied, while a value of 0 indicates that temporally differential quality is not applied. For example, different quantizations can be applied to different temporal layers.

[0310] The spatial_quality_flag flag can indicate whether spatial quality optimization is applied. Spatial quality optimization can mean either removing unnecessary information or enhancing necessary information. A value of 1 indicates that spatial quality optimization is applied, while a value of 0 indicates that spatial quality optimization is not applied.

[0311] The color_vision_deficiency_flag can indicate whether optimizations are applied for people with abnormal color vision (or visual acuity) or visual perception problems. For example, it can indicate whether optimizations are applied for color vision deficiency. A value of 1 indicates that optimizations for color vision deficiency are applied, while a value of 0 indicates that optimizations for color vision deficiency are not applied.

[0312] Referring to Table 26, if color_vision_deficiency_flag is true (or its value is 1), color vision deficiency type identification information (CVD_type_idc) and color vision deficiency optimization method identification information (CVD_optimization_method_idc) can be defined. For example, CVD_type_idc can be represented as in Table 11 described above, and CVD_optimization_method_idc can be represented as in Table 12 described above. In addition to CVD_type_idc and CVD_optimization_method_idc, information indicating the type and degree of the user's color vision deficiency can be defined, and detailed information on the applied optimization method (e.g., coefficient of variation, etc.) can also be defined.

[0313] Additionally, if temporal_resampling_flag is true (or its value is 1), resampling information (resampling_info) may be defined, which indicates the temporal resampling optimization method and details. This resampling_info may indicate the original temporal resolution and the inter-frame spacing in the case of the original temporal resolution. For example, in the case of downsampling, resampling_info may indicate the number of frames omitted between each encoded frame. As another example, in the case of upsampling, resampling_info may indicate how many frames are added to the original inter-frame spacing. In other words, resampling_inf may indicate the number of frames added to the original inter-frame spacing.

[0314] Additionally, if spatial_quality_flag is true (or its value is 1), spatiai_quality_method_idc can be defined. In the case of optimization for color vision deficiency, this information can be included in the spatial quality. This can be expressed as in Tables 17 to 20 described above. In this case, the applied optimization properties and objectives can be expressed through encoding of detailed optimization information.

[0315] According to various embodiments, Table 27 may represent an example syntax structure representing an embodiment that defines color_vision_deficiency_flag when optimization_for_human_viewing_flag is true.

[0316]

[0317] According to Table 27, if optimization_for_human_viewing_flag is true, i.e. the optimization goal is for human vision, color_vision_deficiency_flag can be defined.

[0318] Below, an embodiment is described in which information related to optimization for color vision deficiency is expressed as an independent message.

[0319] Table 28 below provides an example of a syntax structure representing color vision deficiency optimization information (color_vision_deficiency_optimization_info). Other optimization methods defined above can also be defined independently in this manner.

[0320]

[0321] According to Table 28, if optimization_cancel_flag is false (or its value is 0), optimization_persistence_flag, CVD_type_idc, and CVD_optimization_method_idc may be defined. The definitions of these syntax elements may be as described above in Tables 5 through 15.

[0322] FIG. 20 is a diagram illustrating an encoding method according to one embodiment of the present disclosure. As an example, the encoding method of FIG. 20 may be performed by an encoding device (or encoder).

[0323] Referring to FIG. 20, the encoder can perform optimization on the video (S2000). Specifically, the encoder can perform optimizations, such as preprocessing, encoding, and / or postprocessing, on the input data (i.e., the video). This optimization can be performed to suit the intended use of the video.

[0324] The encoder can generate information about the performed optimization (S2010). The information about the optimization can include first information indicating the optimization purpose and second information indicating the type of optimization method. The first information can include a first syntax element indicating whether the optimization purpose is for machine analysis or a second syntax element indicating whether the optimization purpose is for human viewing. For example, the first syntax element can be optimization_for_machine_analysis_flag, and the second syntax element can be optimization_for_human_viewing_flag.

[0325] For example, the second information may indicate whether the type of optimization method is at least one of object-based optimization, temporal resampling optimization, spatial resampling optimization, temporal quality optimization, spatial quality optimization, or color vision deficiency optimization.

[0326] As another example, the second information may include a first flag (object_based_flag) indicating whether the type of optimization method is object-based optimization, a second flag (temporal_resampling_flag) indicating whether the type of optimization method is temporal resampling optimization, a third flag (spatial_resampling_flag) indicating whether the type of optimization method is spatial resampling optimization, a fourth flag (temporal_quality_flag) indicating whether the type of optimization method is temporal quality optimization, a fifth flag (spatial_quality_flag) indicating whether the type of optimization method is spatial quality optimization, or a sixth flag (color_vision_deficiency_flag) indicating whether the type of optimization method is color vision deficiency optimization.

[0327] In various embodiments, information about the optimization may further include a cancellation flag (optimization_cancel_flag) indicating whether to cancel the persistence of a previously applied optimization, and a persistence flag (optimization_persistence_flag) indicating the persistence of the optimization. If the value of the cancellation flag is 1, it indicates that the persistence of a previously applied optimization is canceled, and if the value of the cancellation flag is 0 (or not true), optimization_persistence_flag, optimization_for_human_viewing_flag, optimization_for_machine_analysis_flag, and optimization_type may be defined.

[0328] If the value of the persistence flag is 1, this indicates that the optimization is applied to the current picture and all subsequent pictures, and if the value of the persistence flag is 0, this indicates that the optimization is applied only to the current picture.

[0329] If the value of optimization_for_human_viewing_flag is set to a value indicating that it is for human viewing (i.e., 1), the value of optimization_for_machine_analysis_flag may be set to a value indicating that it is not for machine analysis (i.e., 0).

[0330] If the value of optimization_for_human_viewing_flag is set to a value indicating that it is not for human viewing (i.e., 0), the value of optimization_for_machine_analysis_flag may be set to a value indicating that it is for machine analysis (i.e., 1).

[0331] That is, the value of the first syntax element and the value of the second syntax element may not both be set to 0. In other words, the value of the first syntax element may be a value indicating that the optimization purpose is not for machine analysis, and the value of the second syntax element may be a value indicating that the optimization purpose is for user perspective. Alternatively, the value of the first syntax element may be a value indicating that the optimization purpose is for machine analysis, and the value of the second syntax element may be a value indicating that the optimization purpose is not for user perspective.

[0332] The value of the first syntax element indicating that the optimization purpose is not for machine analysis and the value of the second syntax element indicating that the optimization purpose is not for user viewing may not be assigned (or set) at the same time (or both). In other words, the value of the first syntax element may not be assigned (or set) to a value indicating that the optimization purpose is not for machine analysis and the value of the second syntax element may not be assigned (or set) to a value indicating that the optimization purpose is not for user viewing at the same time (or both).

[0333] The encoder can encode information about the optimization generated in this manner (S2020). Information about the optimization (encoder_optimization_info) including the first information and the second information can be encoded in the bitstream.

[0334] FIG. 21 is a diagram illustrating a decoding method according to one embodiment of the present disclosure. As an example, the decoding method of FIG. 21 may be performed by a decoding device (or decoder).

[0335] Referring to FIG. 21, the decoder can obtain information about optimization applied to a video from a bitstream (S2100). Information about the optimization can be obtained, including first information indicating the optimization purpose and second information indicating the type of optimization method.

[0336] The decoder can determine the purpose and method of optimization based on information about the optimization (S2110). The decoder can determine the optimization purpose based on first information indicating the optimization purpose, and determine the optimization method based on second information indicating the type of optimization method.

[0337] For example, whether the optimization objective is machine analysis or user perspective may be determined based on the value of the first syntax element and / or the value of the second syntax element. If the optimization objective is determined to be machine analysis, this may indicate that the optimization objective is suitable for performing a machine task. If the optimization objective is determined to be user perspective, this may indicate that the optimization objective is suitable for the user perspective.

[0338] As described above, the value of the first syntax element may be a value indicating that the optimization purpose is not for machine analysis, and the value of the second syntax element may be a value indicating that the optimization purpose is for user perspective. Alternatively, the value of the first syntax element may be a value indicating that the optimization purpose is for machine analysis, and the value of the second syntax element may be a value indicating that the optimization purpose is not for user perspective. The value of the first syntax element indicating that the optimization purpose is not for machine analysis and the value of the second syntax element indicating that the optimization purpose is not for user perspective may not be assigned (or set) at the same time (or both). In other words, the value of the first syntax element may not be assigned (or set) to a value indicating that the optimization purpose is not for machine analysis, and the value of the second syntax element may not be assigned (or set) to a value indicating that the optimization purpose is not for user perspective.

[0339] The optimization method can be determined by the value of the syntax element indicating the type of optimization method, or by the syntax elements expressed for each type of optimization method. For example, as shown in Table 14, if the value is "(optimization_type & 0x10) != 0", the type of optimization method can be determined to be object-based optimization.

[0340] In this disclosure, the names of the syntax elements described above are arbitrarily assigned for clarity of explanation and are not intended to limit the names of the syntax elements. Furthermore, each syntax element may also be referred to as information. Furthermore, while syntax elements may be obtained from a bitstream, they may also be derived from other syntax elements, which may also be included in embodiments of the present disclosure.

[0341] Additionally, the bitstream generated by the image encoding method may be stored in a non-transitory computer-readable recording medium.

[0342] Additionally, as another example, a bitstream generated by a video encoding method may be transmitted to another device (e.g., a video decoding device, etc.). In this case, the method for transmitting the bitstream may include a process for transmitting the bitstream.

[0343] While the exemplary methods of this disclosure are presented as a series of operations for clarity of description, this is not intended to limit the order in which the steps are performed, and individual steps may be performed simultaneously or in different orders, if desired. To implement a method according to this disclosure, additional steps may be included in addition to the steps illustrated, some steps may be excluded and the remaining steps included, or some steps may be excluded and additional steps included.

[0344] In the present disclosure, a video encoding device or video decoding device performing a predetermined operation (step) may perform an operation (step) of checking the conditions or circumstances under which the operation (step) is performed. For example, if it is described that a predetermined operation is performed when a predetermined condition is satisfied, the video encoding device or video decoding device may perform an operation of checking whether the predetermined condition is satisfied and then perform the predetermined operation.

[0345] The various embodiments of the present disclosure are not intended to list all possible combinations but rather to illustrate representative aspects of the present disclosure, and the matters described in the various embodiments may be applied independently or in combination of two or more.

[0346] The embodiments described in this disclosure may be implemented and performed on a processor, microprocessor, controller, or chip. For example, the functional units depicted in each drawing may be implemented and performed on a computer, processor, microprocessor, controller, or chip. In this case, information (e.g., information on instructions) or algorithms for implementation may be stored on a digital storage medium.

[0347] In addition, the decoder (decoding device) and encoder (encoding device) to which the embodiment(s) of the present disclosure are applied may be included in a multimedia broadcasting transmitting and receiving device, a mobile communication terminal, a home cinema video device, a digital cinema video device, a surveillance camera, a video conversation device, a real-time communication device such as a video communication, a mobile streaming device, a storage medium, a camcorder, a video-on-demand (VoD) service providing device, an OTT video (Over the top video) device, an Internet streaming service providing device, a three-dimensional (3D) video device, a VR (virtual reality) device, an AR (argumente reality) device, a video phone video device, a transportation terminal (e.g., a vehicle (including an autonomous vehicle) terminal, a robot terminal, an airplane terminal, a ship terminal, etc.), and a medical video device, and may be used to process a video signal or a data signal. For example, OTT video (Over the top video) devices may include game consoles, Blu-ray players, Internet-connected TVs, home theater systems, smartphones, tablet PCs, and DVRs (Digital Video Recorders).

[0348] In addition, the processing method to which the embodiment(s) of the present disclosure are applied can be produced in the form of a computer-executable program and can be stored in a computer-readable recording medium. Multimedia data having a data structure according to the embodiment(s) of the present document can also be stored in a computer-readable recording medium. The computer-readable recording medium includes all types of storage devices and distributed storage devices in which computer-readable data is stored. The computer-readable recording medium can include, for example, a Blu-ray disc (BD), a universal serial bus (USB), a ROM, a PROM, an EPROM, an EEPROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device. In addition, the computer-readable recording medium includes a medium implemented in the form of a carrier wave (e.g., transmission via the Internet). In addition, a bitstream generated by an encoding method can be stored in a computer-readable recording medium or transmitted via a wired or wireless communication network.

[0349] Additionally, the embodiments of the present disclosure may be implemented as a computer program product by program code, and the program code may be executed on a computer by the embodiments of the present disclosure. The program code may be stored on a computer-readable carrier.

[0350] FIG. 22 is a diagram illustrating an example of a content streaming system to which embodiments of the present disclosure can be applied.

[0351] Referring to FIG. 22, a content streaming system to which an embodiment of the present disclosure is applied may largely include an encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.

[0352] The encoding server compresses content input from multimedia input devices, such as smartphones, cameras, and camcorders, into digital data, generates a bitstream, and transmits it to a streaming server. Alternatively, if multimedia input devices, such as smartphones, cameras, and camcorders, generate the bitstream directly, the encoding server may be omitted.

[0353] A bitstream can be generated by a video encoding method and / or a video encoding device to which an embodiment of the present disclosure is applied, and a streaming server can temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0354] A streaming server transmits multimedia data to a user device based on a user request via a web server, and the web server can act as an intermediary, informing the user of available services. When a user requests a desired service from the web server, the web server transmits the request to the streaming server, which then transmits the multimedia data to the user. The content streaming system may include a separate control server, in which case the control server may control commands and responses between each device within the content streaming system.

[0355] A streaming server can receive content from a media repository and / or encoding server. For example, when receiving content from an encoding server, the content can be received in real time. In this case, the streaming server can store the bitstream for a certain period of time to provide a smooth streaming service.

[0356] Examples of user devices include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, head mounted displays (HMDs)), digital TVs, desktop computers, and digital signage.

[0357] Each server within a content streaming system can be operated as a distributed server, in which case the data received from each server can be processed in a distributed manner.

[0358] FIG. 23 is a diagram showing another example of a content streaming system to which embodiments of the present disclosure can be applied.

[0359] Referring to FIG. 23, in an embodiment such as VCM, depending on the performance of the device, the user's request, the characteristics of the task to be performed, etc., the task may be performed on the user terminal or on an external device (e.g., a streaming server, an analysis server, etc.). In this way, in order to transmit the information required for performing the task to the external device, the user terminal may directly or through an encoding server generate a bitstream containing the information required for performing the task (e.g., information such as the task, neural network, and / or purpose).

[0360] The analysis server can decrypt encoded information received from the user terminal (or encoding server) and then perform a task requested by the user terminal. The analysis server can transmit the results obtained through the task performance back to the user terminal or to another linked service server (e.g., a web server). For example, the analysis server can perform a task to determine fire and transmit the results obtained to a fire-related server. The analysis server may include a separate control server, in which case the control server can control commands / responses between each device associated with the analysis server and the server. In addition, the analysis server can request desired information from the web server based on information about the tasks the user device wishes to perform and can perform. When the analysis server requests a desired service from the web server, the web server can transmit this request to the analysis server, and the analysis server can transmit data corresponding to the request to the user terminal. In this case, the control server of the content streaming system can control commands / responses between each device within the streaming system.

[0361] Embodiments according to the present disclosure can be used to encode / decode video.

Claims

1. A step of obtaining information about optimization applied to a video from a bitstream; and A step of determining the purpose and method of the optimization based on the information about the optimization, A method wherein the information on the above optimization includes first information indicating an optimization purpose and second information indicating a type of optimization method.

2. In paragraph 1, the first information is, A method comprising a first syntax element indicating whether the optimization purpose is for machine analysis or a second syntax element indicating whether the optimization purpose is for human viewing.

3. In paragraph 2, A method wherein the value of the first syntax element indicates that the optimization purpose is for machine analysis.

4. In paragraph 2, A method wherein the value of the second syntax element indicates that the optimization purpose is for the user's view.

5. In paragraph 2, A method wherein the value of the first syntax element indicating that the optimization purpose is not for the machine analysis and the value of the second syntax element indicating that the optimization purpose is not for the user view are not set at the same time.

6. In paragraph 1, the second information is, A method for indicating whether the type of the above optimization method includes at least one of object-based optimization, temporal resampling optimization, spatial resampling optimization, temporal quality optimization, spatial quality optimization, or color vision deficiency optimization.

7. In the 6th paragraph, the spatial resampling optimization method includes upsampling and downsampling.

8. In paragraph 1, the information on the optimization is: A method further comprising a cancellation flag indicating whether to cancel the persistence of a previously applied optimization and a persistence flag indicating the persistence of the optimization.

9. In paragraph 7, The value of the persistence flag above being 1 indicates that the optimization is applied to the current picture and all subsequent pictures, A method in which the value of the persistence flag is 0 indicates that the optimization is applied only to the current picture.

10. In paragraph 1, the second information is: A method comprising a first flag indicating whether the type of the optimization method is object-based optimization, a second flag indicating whether the type of the optimization method is temporal resampling optimization, a third flag indicating whether the type of the optimization method is spatial resampling optimization, a fourth flag indicating whether the type of the optimization method is temporal quality optimization, a fifth flag indicating whether the type of the optimization method is spatial quality optimization, or a sixth flag indicating whether the type of the optimization method is color vision deficiency optimization.

11. In paragraph 6, Resampling information indicating a temporal resampling optimization method and details is defined based on the value of the second flag, A method wherein the above resampling information represents the original temporal resolution and the inter-frame interval in the case of the original temporal resolution.

12. Steps to perform optimization for the video; a step of generating information for the above optimization; and comprising a step of encoding information about the above optimization, A method in which the information on the above optimization includes first information indicating an optimization purpose and second information indicating an optimization method.

13. In paragraph 12, the first information is: A method comprising a first syntax element indicating whether the purpose of the optimization is for machine analysis or a second syntax element indicating whether the purpose of the optimization is for user viewing.

14. In paragraph 13, A method wherein the value of the first syntax element indicates that the optimization purpose is for machine analysis.

15. In paragraph 13, A method wherein the value of the second syntax element indicates that the optimization purpose is for the user's view.

16. In paragraph 13, A method wherein the value of the first syntax element indicating that the optimization purpose is not for the machine analysis and the value of the second syntax element indicating that the optimization purpose is not for the user view are not set at the same time.

17. In paragraph 12, the information on the optimization is: A method further comprising a cancellation flag indicating whether to cancel the persistence of a previously applied optimization and a persistence flag indicating the persistence of the optimization.

18. A computer-readable recording medium storing a bitstream generated by an encoding method, wherein the bitstream includes information about optimization applied to the video, A computer-readable recording medium comprising first information indicating an optimization purpose and second information indicating an optimization method, wherein the information regarding the optimization is a computer-readable recording medium.

19. A step of generating a bitstream for a video, wherein the bitstream is generated by a step of performing optimization for the video; a step of generating information about the optimization; and a step of encoding information about the optimization, and A step of transmitting data including the bitstream, A method in which the information on the above optimization includes first information indicating an optimization purpose and second information indicating an optimization method.

Images