Image data processing method and device, recording medium storing bitstream, and bitstream transmission method

Abstract

Provided are an image data processing method and device, a recording medium on which a bitstream generated by means of an image data processing method is stored, and a bitstream transmission method. An image data processing method, according to the present disclosure, comprises the steps of: decoding image data obtained from a bitstream; obtaining information about an optimization method of the image data; and post-processing the decoded image data on the basis of the information about the optimization method, wherein the information about the optimization method may include optimization attribute information indicating a type of at least one optimization method applied to the image data.

Description

Image data processing method, device, recording medium storing bitstream, and bitstream transmission method

[0001] The present disclosure relates to a method and device for processing image data, and more particularly, to a method and device for processing image data based on image data optimization, a recording medium storing a bitstream generated by the image data processing method / device of the present disclosure, and a method for transmitting a bitstream.

[0002] With the advancement of machine learning technology, demand for image processing-based AI services is increasing. To effectively process the massive amounts of image data required by AI services within limited resources, image compression technologies optimized for machine tasks are essential. However, existing image compression technologies have been developed with the goal of high-resolution, high-quality image processing for human vision, making them unsuitable for AI services. Consequently, active research and development is underway on new machine-oriented image compression technologies suitable for AI services.

[0003] The present disclosure aims to provide an image data processing method and device with improved processing efficiency.

[0004] In addition, the present disclosure aims to provide an image data processing method and device that performs pre- / post-processing on an input source based on a predetermined optimization method.

[0005] In addition, the present disclosure aims to provide an image data processing method and device based on a predetermined interface defining optimization information.

[0006] In addition, the present disclosure aims to provide an image processing method and device capable of removing noise from an input source and enhancing information based on a predetermined optimization method.

[0007] In addition, the present disclosure aims to provide a method for transmitting a bitstream generated by an image processing method or device according to the present disclosure.

[0008] In addition, the present disclosure aims to provide a recording medium storing a bitstream generated by an image processing method or device according to the present disclosure.

[0009] In addition, the present disclosure aims to provide a recording medium storing a bitstream received and decoded by an image processing method according to the present disclosure and used for image restoration.

[0010] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary skill in the technical field to which the present disclosure belongs from the description below.

[0011] A method for processing image data according to one aspect of the present disclosure includes a step of decoding image data obtained from a bitstream, a step of obtaining information on an optimization method for the image data, and a step of post-processing the decoded image data based on the information on the optimization method, wherein the information on the optimization method may include optimization attribute information indicating a type of at least one optimization method applied to the image data.

[0012] According to another aspect of the present disclosure, a receiving device includes a memory and at least one processor, wherein the at least one processor decodes image data obtained from a bitstream, obtains information on an optimization method for the image data, and post-processes the decoded image data based on the information on the optimization method, wherein the information on the optimization method may include optimization attribute information indicating a type of at least one optimization method applied to the image data.

[0013] According to another aspect of the present disclosure, a method for processing image data includes a step of determining an optimization method for image data, a step of pre-processing the image data based on the optimization method, and a step of encoding the pre-processed image data and information about the optimization method, wherein the information about the optimization method may include optimization attribute information indicating a type of at least one optimization method applied to the image data.

[0014] According to another aspect of the present disclosure, a transmission device includes a memory and at least one processor, wherein the at least one processor determines an optimization method for image data, pre-processes the image data based on the optimization method, and encodes the pre-processed image data and information about the optimization method, wherein the information about the optimization method may include optimization attribute information indicating a type of at least one optimization method applied to the image data.

[0015] A recording medium according to another aspect of the present disclosure can store a bitstream generated by an image processing method or transmission device of the present disclosure.

[0016] A bitstream transmission method according to another aspect of the present disclosure can transmit a bitstream generated by an image processing method or transmission device of the present disclosure to a receiving device.

[0017] The features briefly summarized above regarding the present disclosure are merely exemplary aspects of the detailed description of the present disclosure that follows and do not limit the scope of the present disclosure.

[0018] According to the present disclosure, an image data processing method and device with improved processing efficiency can be provided.

[0019] In addition, according to the present disclosure, an image data processing method and device that perform pre / post-processing on an input source based on a predetermined optimization method can be provided.

[0020] Additionally, according to the present disclosure, an image data processing method and device based on a predetermined interface defining optimization information can be provided.

[0021] In addition, according to the present disclosure, an image processing method and device capable of removing noise from an input source and enhancing information based on a predetermined optimization method can be provided.

[0022] Additionally, according to the present disclosure, a method for transmitting a bitstream generated by an image processing method or device according to the present disclosure can be provided.

[0023] In addition, according to the present disclosure, a recording medium storing a bitstream generated by an image processing method or device according to the present disclosure can be provided.

[0024] In addition, according to the present disclosure, a recording medium storing a bitstream received and decoded by a receiving device according to the present disclosure and used for image restoration can be provided.

[0025] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the description below.

[0026] FIG. 1 is a schematic diagram of a VCM system to which embodiments of the present disclosure can be applied.

[0027] FIG. 2 is a schematic diagram illustrating a VCM pipeline structure to which embodiments of the present disclosure can be applied.

[0028] FIG. 3 is a schematic diagram of an image / video encoder to which embodiments of the present disclosure can be applied.

[0029] FIG. 4 is a schematic diagram of an image / video decoder to which embodiments of the present disclosure can be applied.

[0030] FIG. 5 is a flowchart schematically illustrating a feature / feature map encoding procedure to which embodiments of the present disclosure may be applied.

[0031] FIG. 6 is a flowchart schematically illustrating a feature / feature map decoding procedure to which embodiments of the present disclosure may be applied.

[0032] Figure 7 is a diagram showing an example of a VCM hierarchical structure.

[0033] FIG. 8 is a diagram illustrating an example of a VCM bitstream composed of encoded abstract features and NNAL information.

[0034] Figure 9 is a drawing to explain various issues of the VCM system.

[0035] FIGS. 10 and 11 are drawings showing a VCM system according to one embodiment of the present disclosure.

[0036] Figures 12 and 13 are drawings showing a hierarchical interface structure between an optimizer and an encoder.

[0037] Figure 14 is an example of expressing the location and size of the ROI area within the input source in pixels.

[0038] Figure 15 is an example of expressing the location and size of the ROI area within the input source in predetermined units.

[0039] Figure 16 is a diagram showing the process of optimizing an input source and transmitting it to an encoder.

[0040] Figure 17 is a diagram showing the process of encoding the optimized input source of Figure 16.

[0041] FIG. 18 is a diagram showing the application order of an optimization method according to one embodiment of the present disclosure.

[0042] FIG. 19 is a diagram illustrating a VCM system according to one embodiment of the present disclosure.

[0043] FIG. 20 is a diagram for explaining an optimization setting process according to one embodiment of the present disclosure.

[0044] FIG. 21 is a flowchart illustrating an image data processing method according to one embodiment of the present disclosure.

[0045] FIG. 22 is a flowchart illustrating an image data processing method according to one embodiment of the present disclosure.

[0046] FIG. 23 is a diagram illustrating an example of a content streaming system to which embodiments of the present disclosure can be applied.

[0047] FIG. 24 is a diagram showing another example of a content streaming system to which embodiments of the present disclosure can be applied.

[0048] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0049] In describing embodiments of the present disclosure, detailed descriptions of known configurations or functions will be omitted if they are deemed to obscure the gist of the present disclosure. Furthermore, portions unrelated to the description of the present disclosure in the drawings have been omitted, and similar portions have been designated with similar reference numerals.

[0050] In the present disclosure, when a component is said to be "connected," "coupled," or "connected" to another component, this may include not only a direct connection, but also an indirect connection in which another component exists in between. Furthermore, when a component is said to "include" or "have" another component, unless otherwise specifically stated, this does not exclude the other component, but rather implies that the other component may be included.

[0051] In this disclosure, terms such as first, second, etc. are used solely to distinguish one component from another, and do not limit the order or importance of components unless specifically stated otherwise. Accordingly, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.

[0052] In this disclosure, distinct components are used to clearly illustrate their respective characteristics, and do not necessarily imply that the components are separated. That is, multiple components may be integrated into a single hardware or software unit, or a single component may be distributed into multiple hardware or software units. Therefore, even if not specifically mentioned, such integrated or distributed embodiments are also included within the scope of this disclosure.

[0053] In the present disclosure, the components described in various embodiments are not necessarily essential components, and some may be optional components. Therefore, embodiments comprising a subset of the components described in one embodiment are also within the scope of the present disclosure. Furthermore, embodiments including other components in addition to the components described in various embodiments are also within the scope of the present disclosure.

[0054] The present disclosure relates to encoding and decoding of images, and terms used in the present disclosure may have their usual meanings commonly used in the technical field to which the present disclosure belongs, unless newly defined in the present disclosure.

[0055] The present disclosure may be applied to a method disclosed in the Versatile Video Coding (VVC) standard and / or the Video Coding for Machines (VCM) standard. In addition, the present disclosure may be applied to a method disclosed in the essential video coding (EVC) standard, the AOMedia Video 1 (AV1) standard, the 2nd generation of audio video coding standard (AVS2), or the next generation video / image coding standard (e.g., H.267 or H.268, etc.).

[0056] The present disclosure presents various embodiments related to video / image coding, and unless otherwise stated, the embodiments may be performed in combination with each other. In the present disclosure, "video" may mean a set of a series of images over time. An "image" may be information generated by artificial intelligence (AI). Input information used in the process of AI performing a series of tasks, information generated during the information processing process, and output information may be used as images. A "picture" generally means a unit representing one image of a specific time period, and a slice / tile is a coding unit that constitutes a part of a picture in encoding. A single picture may be composed of one or more slices / tiles. In addition, a slice / tile may include one or more CTUs (coding tree units). The CTUs may be divided into one or more CUs. A tile is a rectangular area within a specific tile row and a specific tile column within a picture, and may be composed of multiple CTUs. A tile row may be defined as a rectangular area of ​​CTUs, and may have a height equal to the height of the picture, and a width specified by a syntax element signaled from a bitstream portion such as a picture parameter set. A tile row may be defined as a rectangular area of ​​CTUs, and may have a width equal to the width of the picture, and a height specified by a syntax element signaled from a bitstream portion such as a picture parameter set. A tile scan is a predetermined sequential ordering method of CTUs that divide a picture. Here, CTUs may be sequentially ordered within a tile according to a CTU raster scan, and tiles within a picture may be sequentially ordered according to a raster scan order of tiles in the picture.A slice may contain an integer number of complete tiles, or a contiguous integer number of complete CTU rows within a tile of a picture. A slice may be exclusively contained in a single NAL unit. A picture may consist of one or more tile groups. A tile group may contain one or more tiles. A brick may represent a rectangular region of CTU rows within a tile of a picture. A tile may contain one or more bricks. A brick may represent a rectangular region of CTU rows within a tile. A tile may be divided into multiple bricks, and each brick may contain one or more CTU rows belonging to the tile. A tile that is not divided into multiple bricks may also be treated as a brick.

[0057] In the present disclosure, "pixel" or "pel" may refer to the smallest unit that constitutes a picture (or image). Additionally, "sample" may be used as a term corresponding to a pixel. A sample may generally represent a pixel or a pixel value, and may represent only a pixel / pixel value of a luma component or only a pixel / pixel value of a chroma component.

[0058] In one embodiment, particularly when applied to VCM, a pixel / pixel value may represent independent information of each component or a pixel / pixel value of a component generated through combination, synthesis, or analysis when there is a picture composed of a set of components with different characteristics and meanings. For example, in an RGB input, only the pixel / pixel value of R may be represented, only the pixel / pixel value of G may be represented, or only the pixel / pixel value of B may be represented. For example, only the pixel / pixel value of the Luma component synthesized using the R, G, and B components may be represented. For example, only the pixel / pixel value of the image information extracted through analysis of the R, G, and B components may be represented.

[0059] In the present disclosure, a "unit" may represent a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to the region. One unit may include one luma block and two chroma (e.g., Cb, Cr) blocks. In some cases, the term "unit" may be used interchangeably with terms such as "sample array," "block," or "area." In general, an MxN block may include a set (or array) of samples (or sample array) or transform coefficients consisting of M columns and N rows. In one embodiment, particularly when applied to VCM, a unit may represent a basic unit containing information for performing a specific task.

[0060] In the present disclosure, the "current block" may mean one of the following: a "current coding block," a "current coding unit," a "block to be encoded," a "block to be decoded," or a "block to be processed." When prediction is performed, the "current block" may mean a "current prediction block" or a "block to be predicted." When transformation (inverse transformation) / quantization (inverse quantization) is performed, the "current block" may mean a "current transformation block" or a "block to be transformed." When filtering is performed, the "current block" may mean a "block to be filtered."

[0061] Additionally, in the present disclosure, "current block" may mean "luma block of the current block" unless explicitly described as a chroma block. "Chroma block of the current block" may be expressed by explicitly including explicit description of chroma block, such as "chroma block" or "current chroma block."

[0062] In this disclosure, " / " and "," can be interpreted as "and / or". For example, "A / B" and "A, B" can be interpreted as "A and / or B". Additionally, "A / B / C" and "A, B, C" can mean "at least one of A, B, and / or C."

[0063] In this disclosure, "or" may be interpreted as "and / or." For example, "A or B" may mean 1) "A" only, 2) "B" only, or 3) "A and B." Alternatively, "or" in this disclosure may mean "additionally or alternatively."

[0064] The present disclosure relates to VCM (Video / image coding for machines).

[0065] VCM (Virtual Compression Modeling) is a compression technology that encodes / decodes a portion of a source image / video or information obtained from the source image / video for machine vision purposes. In VCM, the target of encoding / decoding can be referred to as a feature. A feature can refer to information extracted from the source image / video based on the task purpose, requirements, surrounding environment, etc. Features may have a different information format than the source image / video, and accordingly, the compression method and representation format of the feature may also differ from those of the video source.

[0066] VCMs can be applied to a variety of applications. For example, in surveillance systems that recognize and track objects or people, VCMs can be used to store or transmit object recognition information. Furthermore, in intelligent transportation or smart traffic systems, VCMs can be used to transmit vehicle location information collected from GPS, sensing information collected from LIDAR and radar, and various vehicle control information to other vehicles or infrastructure. Furthermore, in the smart city field, VCMs can be used to perform individual tasks for interconnected sensor nodes or devices.

[0067] The present disclosure provides various embodiments related to feature / feature map coding. Unless otherwise specified, the embodiments of the present disclosure may be implemented individually or in combination of two or more.

[0068] VCM System Overview

[0069] FIG. 1 is a schematic diagram of a VCM system to which embodiments of the present disclosure can be applied.

[0070] Referring to FIG. 1, the VCM system may include an encoding device (10) and a decoding device (20).

[0071] The encoding device (10) can compress / encode features / feature maps extracted from source images / videos to generate a bitstream, and transmit the generated bitstream to a decoding device (20) via a storage medium or a network. The encoding device (10) may also be referred to as a feature encoding device. In a VCM system, features / feature maps can be generated in each hidden layer of a neural network. The size and number of channels of the generated feature maps can vary depending on the type of neural network or the location of the hidden layer. In the present disclosure, the feature maps can be referred to as a feature set.

[0072] The encoding device (10) may include a feature acquisition unit (9), an encoding unit (10), and a transmission unit (13).

[0073] The feature acquisition unit (9) can acquire features / feature maps for source images / videos. In some embodiments, the feature acquisition unit (9) can acquire features / feature maps from an external device, for example, a feature extraction network. In this case, the feature acquisition unit (9) performs a feature reception interface function. Alternatively, the feature acquisition unit (9) can acquire features / feature maps by executing a neural network (e.g., CNN, DNN, etc.) using the source images / videos as input. In this case, the feature acquisition unit (9) performs a feature extraction network function.

[0074] According to an embodiment, the encoding device (10) may further include a source image generation unit (not shown) for obtaining a source image / video. The source image generation unit may be implemented as an image sensor, a camera module, or the like, and may obtain the source image / video through a process of capturing, synthesizing, or generating an image / video. In this case, the generated source image / video may be transmitted to a feature extraction network and used as input data for extracting a feature / feature map.

[0075] The encoding unit (10) can encode the feature / feature map acquired by the feature acquisition unit (9). The encoding unit (10) can perform a series of procedures such as prediction, transformation, and quantization to increase encoding efficiency. The encoded data (encoded feature / feature map information) can be output in the form of a bitstream. A bitstream including the encoded feature / feature map information can be referred to as a VCM bitstream.

[0076] The transmission unit (13) can transmit feature / feature map information or data output in the form of a bitstream to a decoding device (20) via a digital storage medium or network in the form of a file or streaming. Here, the digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit (13) can include elements for generating a media file having a predetermined file format or elements for transmitting data via a broadcasting / communication network.

[0077] The decoding device (20) can obtain feature / feature map information from the encoding device (10) and restore the feature / feature map based on the obtained information.

[0078] The decryption device (20) may include a receiving unit (21) and a decryption unit (22).

[0079] The receiving unit (21) can receive a bitstream from the encoding device (10), obtain feature / feature map information from the received bitstream, and transmit it to the decoding unit (22).

[0080] The decoding unit (22) can decode the feature / feature map based on the acquired feature / feature map information. The decoding unit (22) can perform a series of procedures, such as inverse quantization, inverse transformation, and prediction, corresponding to the operation of the encoding unit (14), to increase decoding efficiency.

[0081] According to an embodiment, the decryption device (20) may further include a task analysis / rendering unit (23).

[0082] The task analysis / rendering unit (23) can perform task analysis based on the decrypted feature / feature map. In addition, the task analysis / rendering unit (23) can render the decrypted feature / feature map into a form suitable for task execution. Various machine-oriented tasks can be performed based on the task analysis results and the rendered feature / feature map.

[0083] Above, the VCM system can encode / decode features extracted from source images / videos according to user and / or machine requests, task objectives, and surrounding environments, and perform various machine-directed tasks based on the decoded features. The VCM system can also be implemented by extending / redesigning a video / image coding system, and can perform various encoding / decoding methods defined in the VCM standard.

[0084] VCM pipeline

[0085] FIG. 2 is a schematic diagram illustrating a VCM pipeline structure to which embodiments of the present disclosure can be applied.

[0086] Referring to FIG. 2, the VCM pipeline (200) may include a first pipeline (210) for encoding / decoding images / videos and a second pipeline (220) for encoding / decoding features / feature maps. In the present disclosure, the first pipeline (210) may be referred to as a video codec pipeline, and the second pipeline (220) may be referred to as a feature codec pipeline.

[0087] The first pipeline (210) may include a first stage (211) for encoding input images / videos and a second stage (212) for decoding the encoded images / videos to generate restored images / videos. The restored images / videos may be used for human viewing, i.e., for human vision.

[0088] The second pipeline (220) may include a third stage (221) for extracting features / feature maps from input images / videos, a fourth stage (222) for encoding the extracted features / feature maps, and a fifth stage (223) for decoding the encoded features / feature maps to generate restored features / feature maps. The restored features / feature maps may be used for machine (vision) tasks. Here, the machine (vision) task may refer to a task in which images / videos are consumed by a machine. The machine (vision) task may be applied to service scenarios such as, for example, surveillance, intelligent transportation, smart city, intelligent industry, and intelligent content. In some embodiments, the restored features / feature maps may also be used for human vision.

[0089] In some embodiments, the encoded feature / feature map in the fourth stage (222) may be transferred to the first stage (221) and used to encode an image / video. In this case, an additional bitstream may be generated based on the encoded feature / feature map, and the generated additional bitstream may be transferred to the second stage (222) and used to decode the image / video.

[0090] In some embodiments, the decrypted feature / feature map in the fifth stage (223) may be transferred to the second stage (222) and used to decrypt an image / video.

[0091] Although FIG. 2 illustrates a case where the VCM pipeline (200) includes a first pipeline (210) and a second pipeline (220), this is merely exemplary and embodiments of the present disclosure are not limited thereto. For example, the VCM pipeline (200) may include only the second pipeline (220), or the second pipeline (220) may be extended into multiple feature codec pipelines.

[0092] Meanwhile, in the first pipeline (210), the first stage (211) may be performed by an image / video encoder, and the second stage (212) may be performed by an image / video decoder. In addition, in the second pipeline (220), the third stage (221) may be performed by a VCM encoder (or a feature / feature map encoder), and the fourth stage (222) may be performed by a VCM decoder (or a feature / feature map decoder). The encoder / decoder structure will be described in detail below.

[0093] Encoder

[0094] FIG. 3 is a schematic diagram of an image / video encoder to which embodiments of the present disclosure can be applied.

[0095] Referring to FIG. 3, the image / video encoder (300) may include an image partitioner (310), a prediction unit (predictor) 320, a residual processor (residual processor) 330, an entropy encoder (entropy encoder) 340, an adder (adder) 350, a filter (filter) 360, and a memory (memory) 370. The prediction unit (320) may include an inter prediction unit (321) and an intra prediction unit (322). The residual processor (330) may include a transformer (transformer) 332, a quantizer (quantizer) 333, a dequantizer (dequantizer) 334, and an inverse transformer (inverse transformer) 335. The residual processing unit (330) may further include a subtractor (331). The addition unit (350) may be referred to as a reconstructor or a recontructed block generator. The image segmentation unit (310), the prediction unit (320), the residual processing unit (330), the entropy encoding unit (340), the addition unit (350), and the filtering unit (360) described above may be configured by one or more hardware components (e.g., an encoder chipset or a processor) depending on the embodiment. In addition, the memory (370) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component described above may further include the memory (370) as an internal / external component.

[0096] The image segmentation unit (310) can segment an input image (or picture, frame) input to the image / video encoder (300) into one or more processing units. For example, the processing unit may be referred to as a coding unit (CU). The coding unit may be recursively segmented from a coding tree unit (CTU) or a largest coding unit (LCU) according to a Quad-tree binary-tree ternary-tree (QTBTTT) structure. For example, one coding unit may be segmented into a plurality of coding units of deeper depth based on a quad-tree structure, a binary tree structure, and / or a ternary structure. In this case, for example, the quad-tree structure may be applied first, and the binary tree structure and / or the ternary structure may be applied later. Alternatively, the binary tree structure may be applied first. The image / video coding procedure according to the present disclosure may be performed based on a final coding unit that is no longer divided. In this case, the largest coding unit may be used as the final coding unit based on coding efficiency according to image characteristics, etc., or, if necessary, the coding unit may be recursively divided into coding units of lower depths, and the coding unit with the optimal size may be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration, which will be described later. As another example, the processing unit may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may each be divided or partitioned from the final coding unit described above.The prediction unit may be a unit of sample prediction, and the transform unit may be a unit for deriving a transform coefficient and / or a unit for deriving a residual signal from a transform coefficient.

[0097] The term "unit" may be used interchangeably with terms such as "block" or "area" in some cases. In general, an MxN block can represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luma component or only the pixel / pixel value of the chroma component. A sample can be used as a term corresponding to a pixel or a pel.

[0098] The image / video encoder (300) can generate a residual signal (residual block, residual sample array) by subtracting a prediction signal (predicted block, prediction sample array) output from an inter-prediction unit (321) or an intra-prediction unit (322) from an input image signal (original block, original sample array), and the generated residual signal is transmitted to a conversion unit (332). In this case, as illustrated, a unit that subtracts a prediction signal (prediction block, prediction sample array) from an input image signal (original block, original sample array) within the image / video encoder (300) may be referred to as a subtraction unit (331). The prediction unit can perform prediction on a block to be processed (hereinafter, referred to as a current block) and generate a predicted block including prediction samples for the current block. The prediction unit can determine whether intra-prediction or inter-prediction is applied to the current block or CU unit. The prediction unit can generate various information regarding prediction, such as prediction mode information, and transmit it to the entropy encoding unit (340). The information regarding prediction can be encoded in the entropy encoding unit (340) and output in the form of a bitstream.

[0099] The intra prediction unit (322) can predict the current block by referring to samples within the current picture. At this time, the referenced samples may be located in the neighborhood of the current block or may be located away from it depending on the prediction mode. In intra prediction, the prediction modes may include multiple non-directional modes and multiple directional modes. The non-directional modes may include, for example, a DC mode and a planar mode. The directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes depending on the degree of detail in the prediction direction. However, this is only an example, and a greater or lesser number of directional prediction modes may be used depending on the settings. The intra prediction unit (322) may also determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.

[0100] The inter prediction unit (321) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between the neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on the inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring block can include a spatial neighboring block existing in the current picture and a temporal neighboring block existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. Temporal neighboring blocks may be referred to as collocated reference blocks, collocated CUs (colCUs), etc., and reference pictures including temporal neighboring blocks may be referred to as collocated pictures (colPic). For example, the inter prediction unit (321) may construct a motion information candidate list based on neighboring blocks, and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction may be performed based on various prediction modes, and for example, in the case of skip mode and merge mode, the inter prediction unit (321) may use the motion information of neighboring blocks as the motion information of the current block. In the case of skip mode, unlike the merge mode, a residual signal may not be transmitted.In the motion vector prediction (MVP) mode, the motion vector of the surrounding blocks is used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0101] The prediction unit (320) can generate a prediction signal based on various prediction methods. For example, the prediction unit can apply intra prediction or inter prediction for prediction of a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). In addition, the prediction unit can be based on an intra block copy (IBC) prediction mode or a palette mode for prediction of a block. The IBC prediction mode or palette mode can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in the present disclosure. The palette mode can be viewed as an example of intra coding or intra prediction. When palette mode is applied, sample values ​​within a picture can be signaled based on information about the palette table and palette index.

[0102] The prediction signal generated by the prediction unit (320) can be used to generate a reconstructed signal or a residual signal. The transformation unit (332) can apply a transformation technique to the residual signal to generate transform coefficients. For example, the transformation technique can include at least one of a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), a Karhunen-Loeve Transform (KLT), a Graph-Based Transform (GBT), or a Conditionally Non-linear Transform (CNT). Here, GBT refers to a transformation obtained from a graph when the relationship information between pixels is expressed as a graph. CNT refers to a transformation obtained based on generating a prediction signal using all previously reconstructed pixels. In addition, the transformation process can be applied to a pixel block having a square equal size, or can be applied to a block of a non-square variable size.

[0103] The quantization unit (333) quantizes the transform coefficients and transmits them to the entropy encoding unit (340), and the entropy encoding unit (340) can encode the quantized signal (information about the quantized transform coefficients) and output it as a bitstream. The information about the quantized transform coefficients may be called residual information. The quantization unit (333) can rearrange the quantized transform coefficients in the form of a block into the form of a one-dimensional vector based on the coefficient scan order, and can also generate information about the quantized transform coefficients based on the quantized transform coefficients in the form of a one-dimensional vector. The entropy encoding unit (340) can perform various encoding methods, such as, for example, exponential Golomb, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), etc. The entropy encoding unit (340) may encode, together or separately, information necessary for image / video restoration (e.g., values ​​of syntax elements, etc.) in addition to quantized transform coefficients. The encoded information (e.g., encoded image / video information) may be transmitted or stored in the form of a bitstream in units of NAL (network abstraction layer) units. The image / video information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the image / video information may further include general constraint information. In addition, the image / video information may further include a method of generating and using encoded information, a purpose, etc. In the present disclosure, information and / or syntax elements transmitted / signaled from an image / video encoder to an image / video decoder may be included in the image / video information.Image / video information can be encoded through the encoding process described above and included in a bitstream. The bitstream can be transmitted through a network or stored in a digital storage medium. Here, the network can include a broadcasting network and / or a communication network, and the digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmission unit (not shown) that transmits a signal output from an entropy encoding unit (340) and / or a storage unit (not shown) that stores the signal can be configured as an internal / external element of the image / video encoder (300), or the transmission unit can be included in the entropy encoding unit (340).

[0104] The quantized transform coefficients output from the quantization unit (333) can be used to generate a prediction signal. For example, by applying inverse quantization and inverse transformation to the quantized transform coefficients through the inverse quantization unit (334) and the inverse transform unit (335), a residual signal (residual block or residual samples) can be reconstructed. The addition unit (350) can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from the inter prediction unit (321) or the intra prediction unit (322). When there is no residual for the block to be processed, such as when skip mode is applied, the predicted block can be used as a reconstructed block. The addition unit (350) may be called a reconstructor or a reconstructed block generation unit. The generated restoration signal can be used for intra prediction of the next processing target block within the current picture, and can also be used for inter prediction of the next picture after filtering as described below.

[0105] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture encoding and / or restoration process.

[0106] The filtering unit (360) can improve subjective / objective picture quality by applying filtering to the restoration signal. For example, the filtering unit (360) can apply various filtering methods to the restoration picture to generate a modified restoration picture and store the modified restoration picture in the memory (370), specifically, in the DPB of the memory (370). The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filtering unit (360) can generate various information regarding filtering and transmit it to the entropy encoding unit (340). The information regarding filtering can be encoded by the entropy encoding unit (340) and output in the form of a bitstream.

[0107] The modified restored picture transmitted to the memory (370) can be used as a reference picture in the inter prediction unit (321). Through this, prediction mismatch in the encoder and decoder stages can be avoided, and encoding efficiency can also be improved.

[0108] The DPB of the memory (370) can store the modified restored picture to be used as a reference picture in the inter prediction unit (321). The memory (370) can store motion information of a block from which motion information in the current picture is derived (or encoded) and / or motion information of blocks in a picture that has already been restored. The stored motion information can be transferred to the inter prediction unit (321) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (370) can store restored samples of restored blocks in the current picture and can transfer the stored restored samples to the intra prediction unit (322).

[0109] Meanwhile, the VCM encoder (or feature / feature map encoder) may have a structure basically identical / similar to the image / video encoder (300) described with reference to FIG. 3 in that it performs a series of procedures such as prediction, transformation, and quantization to encode a feature / feature map. However, the VCM encoder differs from the image / video encoder (300) in that it targets a feature / feature map for encoding, and accordingly, the name of each unit (or component) (e.g., image segmentation unit (310), etc.) and its specific operation contents may be different from the image / video encoder (300). The specific operation contents of the VCM encoder will be described in detail later.

[0110] Decoder

[0111] FIG. 4 is a schematic diagram of an image / video decoder to which embodiments of the present disclosure can be applied.

[0112] Referring to FIG. 4, the image / video decoder (400) may include an entropy decoder (410), a residual processor (420), a predictor (430), an adder (440), a filter (450), and a memory (460). The predictor (430) may include an inter-prediction unit (431) and an intra-prediction unit (432). The residual processor (420) may include a dequantizer (421) and an inverse transformer (422). The entropy decoding unit (410), residual processing unit (420), prediction unit (430), addition unit (440), and filtering unit (450) described above may be configured by a single hardware component (e.g., a decoder chipset or processor) depending on the embodiment. In addition, the memory (460) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (460) as an internal / external component.

[0113] When a bitstream including video / image information is input, the video / video decoder (400) can restore the video / image in accordance with the process in which the video / image information is processed in the video / video encoder (300) of FIG. 3. For example, the video / video decoder (400) can derive units / blocks based on block division-related information obtained from the bitstream. The video / video decoder (400) can perform decoding using a processing unit applied in the video / video encoder. Accordingly, the processing unit of decoding may be, for example, a coding unit, and the coding unit may be divided from a coding tree unit or a maximum coding unit according to a quad tree structure, a binary tree structure, and / or a ternary tree structure. One or more transform units may be derived from the coding unit. Then, the restored video signal decoded and output by the video / video decoder (400) can be played back through a playback device.

[0114] The image / video decoder (400) can receive a signal output from the encoder of FIG. 3 in the form of a bitstream, and the received signal can be decoded through the entropy decoding unit (410). For example, the entropy decoding unit (410) can parse the bitstream to derive information (e.g., image / video information) necessary for image restoration (or picture restoration). The image / video information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the image / video information may further include general constraint information. In addition, the image / video information may include a method of generating and using decoded information, a purpose, etc. The image / video decoder (400) can decode a picture further based on information on the parameter set and / or the general constraint information. Signaling / received information and / or syntax elements can be decoded through a decoding procedure and obtained from the bitstream. For example, the entropy decoding unit (410) can decode information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output the values ​​of syntax elements necessary for image restoration and the quantized values ​​of transform coefficients for the residual. More specifically, the CABAC entropy decoding method receives a bin corresponding to each syntax element in the bitstream, determines a context model using information of the syntax element to be decoded and decoding information of the surrounding and decoding target blocks or information of symbols / bins decoded in the previous step, and predicts the occurrence probability of the bin according to the determined context model to perform arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element.At this time, the CABAC entropy decoding method can update the context model using the information of the decoded symbol / bin for the context model of the next symbol / bin after determining the context model. Information regarding prediction among the information decoded by the entropy decoding unit (410) is provided to the prediction unit (inter prediction unit (432) and intra prediction unit (431)), and residual values ​​on which entropy decoding is performed by the entropy decoding unit (410), i.e., quantized transform coefficients and related parameter information, can be input to the residual processing unit (420). The residual processing unit (420) can derive a residual signal (residual block, residual samples, residual sample array). In addition, information regarding filtering among the information decoded by the entropy decoding unit (410) can be provided to the filtering unit (450). Meanwhile, a receiving unit (not shown) that receives a signal output from an image / video encoder may be further configured as an internal / external element of the image / video decoder (400), or the receiving unit may be a component of an entropy decoding unit (410). Meanwhile, the image / video decoder according to the present disclosure may be called an image / video decoding device, and the image / video decoder may be divided into an information decoder (image / video information decoder) and a sample decoder (image / video sample decoder). In this case, the information decoder may include an entropy decoding unit (410), and the sample decoder may include at least one of an inverse quantization unit (321), an inverse transformation unit (322), an addition unit (440), a filtering unit (450), a memory (460), an inter prediction unit (432), and an intra prediction unit (431).

[0115] The inverse quantization unit (421) can inverse quantize the quantized transform coefficients and output the transform coefficients. The inverse quantization unit (421) can rearrange the quantized transform coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scanning order performed in the image / video encoder. The inverse quantization unit (321) can perform inverse quantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain transform coefficients.

[0116] In the inverse transform unit (422), the transform coefficients are inversely transformed to obtain a residual signal (residual block, residual sample array).

[0117] The prediction unit (430) can perform a prediction on the current block and generate a predicted block containing prediction samples for the current block. Based on the prediction information output from the entropy decoding unit (410), the prediction unit can determine whether intra-prediction or inter-prediction is applied to the current block, and can determine a specific intra / inter-prediction mode.

[0118] The prediction unit (420) can generate a prediction signal based on various prediction methods. For example, the prediction unit can apply intra prediction or inter prediction for prediction of a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). In addition, the prediction unit can be based on an intra block copy (IBC) prediction mode or a palette mode for prediction of a block. The IBC prediction mode or palette mode can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in this document. The palette mode can be viewed as an example of intra coding or intra prediction. When palette mode is applied, information about the palette table and palette index can be signaled by including it in the image / video information.

[0119] The intra prediction unit (431) can predict the current block by referring to samples within the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from it, depending on the prediction mode. In intra prediction, the prediction modes may include multiple non-directional modes and multiple directional modes. The intra prediction unit (431) can also determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.

[0120] The inter prediction unit (432) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between the neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on the inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring blocks can include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit (432) can construct a motion information candidate list based on the neighboring blocks, and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter prediction can be performed based on various prediction modes, and information about the prediction can include information indicating the mode of inter prediction for the current block.

[0121] The addition unit (440) can generate a restoration signal (restored picture, restoration block, restoration sample array) by adding the acquired residual signal to the prediction signal (predicted block, prediction sample array) output from the prediction unit (including the inter-prediction unit (432) and / or intra-prediction unit (431)). When there is no residual for the block to be processed, such as when skip mode is applied, the predicted block can be used as the restoration block.

[0122] The addition unit (440) may be referred to as a restoration unit or restoration block generation unit. The generated restoration signal may be used for intra prediction of the next processing target block within the current picture, may be output after filtering as described below, or may be used for inter prediction of the next picture.

[0123] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture decoding process.

[0124] The filtering unit (450) can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit (450) can apply various filtering methods to the restored picture to generate a modified restored picture, and transmit the modified restored picture to the memory (460), specifically, the DPB of the memory (460). The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0125] The (modified) reconstructed picture stored in the DPB of the memory (460) can be used as a reference picture in the inter prediction unit (432). The memory (460) can store motion information of a block from which motion information is derived (or decoded) within the current picture and / or motion information of blocks within a picture that has already been reconstructed. The stored motion information can be transmitted to the inter prediction unit (432) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (460) can store reconstructed samples of reconstructed blocks within the current picture and transmit them to the intra prediction unit (431).

[0126] Meanwhile, the VCM decoder (or feature / feature map decoder) may have a structure basically identical / similar to the image / video decoder (400) described above with reference to FIG. 4 in that it performs a series of procedures such as prediction, inverse transformation, and inverse quantization to decode a feature / feature map. However, the VCM decoder is different from the image / video decoder (400) in that it targets a feature / feature map for decoding, and accordingly, the name of each unit (or component) (e.g., DPB, etc.) and its specific operation contents may be different from the image / video decoder (400). The operation of the VCM decoder may correspond to the operation of the VCM encoder, and the specific operation contents will be described in detail later.

[0127] Feature / feature map encoding procedure

[0128] FIG. 5 is a flowchart schematically illustrating a feature / feature map encoding procedure to which embodiments of the present disclosure may be applied.

[0129] Referring to FIG. 5, the feature / feature map encoding procedure may include a prediction procedure (S510), a residual processing procedure (S520), and an information encoding procedure (S530).

[0130] The prediction procedure (S510) can be performed by the prediction unit (320) described above with reference to FIG. 3.

[0131] Specifically, the intra prediction unit (322) can predict the current block (i.e., the set of feature elements to be currently encoded) by referring to feature elements within the current feature / feature map. Intra prediction can be performed based on the spatial similarity of feature elements constituting the feature / feature map. For example, feature elements included in the same region of interest (RoI) within an image / video can be estimated to have similar data distribution characteristics. Accordingly, the intra prediction unit (322) can predict the current block by referring to the restored feature elements within the region of interest including the current block. At this time, the referenced feature elements may be positioned adjacent to the current block or spaced apart from the current block depending on the prediction mode. Intra prediction modes for feature / feature map encoding can include a plurality of non-directional prediction modes and a plurality of directional prediction modes. Non-directional prediction modes may include prediction modes corresponding to, for example, the DC mode and the planar mode of an image / video encoding procedure. Furthermore, directional modes may include prediction modes corresponding to, for example, the 33 directional modes or the 65 directional modes of an image / video encoding procedure. However, this is merely an example, and the types and number of intra-prediction modes may be set / changed in various ways depending on the embodiment.

[0132] The inter prediction unit (321) can predict the current block based on a reference block (i.e., a set of referenced feature elements) specified by motion information on a reference feature / feature map. Inter prediction can be performed based on the temporal similarity of feature elements constituting the feature / feature map. For example, temporally consecutive features can have similar data distribution characteristics. Therefore, the inter prediction unit (321) can predict the current block by referring to the restored feature elements of features that are temporally adjacent to the current feature. At this time, the motion information for specifying the referenced feature elements can include a motion vector and a reference feature / feature map index. The motion information can further include information regarding the inter prediction direction (e.g., L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, neighboring blocks may include spatial neighboring blocks existing in the current feature / feature map and temporal neighboring blocks existing in the reference feature / feature map. The reference feature / feature map including the reference block and the reference feature / feature map including the temporal neighboring blocks may be the same or different. The temporal neighboring blocks may be referred to as collocated reference blocks, and the reference feature / feature map including the temporal neighboring blocks may be referred to as collocated feature / feature maps. The inter prediction unit (321) may construct a motion information candidate list based on the neighboring blocks and generate information indicating which candidate is used to derive the motion vector and / or reference feature / feature map index of the current block.Inter prediction can be performed based on various prediction modes. For example, in the case of skip mode and merge mode, the inter prediction unit (321) can use the motion information of the surrounding blocks as the motion information of the current block. In the case of skip mode, unlike the merge mode, a residual signal may not be transmitted. In the case of motion vector prediction (MVP) mode, the motion vector of the surrounding blocks can be used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference. In addition to the intra prediction and inter prediction described above, the prediction unit (320) can generate a prediction signal based on various prediction methods.

[0133] The prediction signal generated by the prediction unit (320) can be used to generate a residual signal (residual block, residual feature elements) (S520). The residual processing procedure (S520) can be performed by the residual processing unit (330) described above with reference to FIG. 3. In addition, (quantized) transform coefficients can be generated through a transformation and / or quantization procedure for the residual signal, and the entropy encoding unit (340) can encode information about the (quantized) transform coefficients as residual information in the bitstream (S530). In addition to the residual information, the entropy encoding unit (340) can encode information necessary for feature / feature map restoration, such as prediction information (e.g., prediction mode information, motion information, etc.), in the bitstream.

[0134] Meanwhile, the feature / feature map encoding procedure may further include a procedure (S530) for encoding information for feature / feature map restoration (e.g., prediction information, residual information, partitioning information, etc.) and outputting it in the form of a bitstream, as well as a procedure for generating a restored feature / feature map for the current feature / feature map and a procedure (optional) for applying in-loop filtering to the restored feature / feature map.

[0135] The VCM encoder can derive (modified) residual feature(s) from the quantized transform coefficient(s) through inverse quantization and inverse transformation, and can generate a restored feature / feature map based on the predicted feature(s) and the (modified) residual feature(s) which are the outputs of step S510. The restored feature / feature map generated in this way can be identical to the restored feature / feature map generated by the VCM decoder. If an in-loop filtering procedure is performed on the restored feature / feature map, a modified restored feature / feature map can be generated through the in-loop filtering procedure on the restored feature / feature map. The modified restored feature / feature map can be stored in a decoded feature buffer (DFB) or memory, and can be used as a reference feature / feature map in a subsequent feature / feature map prediction procedure. In addition, (in-loop) filtering-related information (parameters) can be encoded and output in the form of a bitstream. Through the in-loop filtering procedure, noise that may occur during feature / feature map coding can be removed, and the performance of feature / feature map-based tasks can be improved. Furthermore, by performing the in-loop filtering procedure in both the encoder and decoder stages, the consistency of prediction results can be guaranteed, the reliability of feature / feature map coding can be improved, and the amount of data transmission for feature / feature map coding can be reduced.

[0136] Feature / feature map decoding procedure

[0137] FIG. 6 is a flowchart schematically illustrating a feature / feature map decoding procedure to which embodiments of the present disclosure may be applied.

[0138] Referring to FIG. 6, the feature / feature map decoding procedure may include an image / video information acquisition procedure (S610), a feature / feature map restoration procedure (S620 to S640), and an in-loop filtering procedure (S650) for the restored feature / feature map. The feature / feature map restoration procedure may be performed based on a prediction signal and a residual signal obtained through the inter / intra prediction (S620) and residual processing (S630), inverse quantization for quantized transform coefficients, and inverse transformation) processes described in the present disclosure. A modified restored feature / feature map may be generated through the in-loop filtering procedure for the restored feature / feature map, and the modified restored feature / feature map may be output as a decoded feature / feature map. The decoded feature / feature map is stored in the decoded feature buffer (DFB) or memory, and can be used as a reference feature / feature map in the inter prediction procedure during subsequent decoding of the feature / feature map. In some cases, the in-loop filtering procedure described above may be omitted. In this case, the reconstructed feature / feature map can be output as a decoded feature / feature map, and can be stored in the decoded feature buffer (DFB) or memory, and can be used as a reference feature / feature map in the inter prediction procedure during subsequent decoding of the feature / feature map.

[0139] VCM coding hierarchy and structure

[0140] Figure 7 is a diagram showing an example of a VCM hierarchical structure.

[0141] Referring to FIG. 7, the VCM hierarchy may be composed of a feature extraction layer (810), a neural network (feature) abstraction layer (720), and a feature coding layer (730).

[0142] The feature extraction layer (710) refers to a layer for extracting features from an input source, and may also include the results of the extraction. The feature coding layer (730) refers to a layer for compressing the extracted features, and may also include the results of the compression.

[0143] The neural network abstraction layer (720) can abstract information generated by the feature extraction layer (710) (e.g., information about extracted features / feature maps) and pass it to the feature coding layer (730). The neural network abstraction layer (720) can hide the internals of the feature extraction layer (710) through information abstraction and provide a consistent feature interface function. Accordingly, even if the compression target changes due to a change in the tool (e.g., CNN, DNN, etc.), the feature coding layer (730) can perform a consistent feature coding procedure. In the present disclosure, the neural network abstraction layer (NNAL) may also be referred to as a feature abstraction layer.

[0144] The interface between the feature extraction layer (710) and the neural network abstraction layer (720) and the interface between the feature coding layer (730) and the neural network abstraction layer (720) can be defined in advance, and the operation in the neural network abstraction layer (720) can be set to be changed later.

[0145] FIG. 8 is a diagram illustrating an example of a VCM bitstream composed of encoded abstract features and NNAL information.

[0146] A bitstream configured as illustrated in Fig. 8 may be referred to as an NNAL (Neural Network Abstraction Layer) unit. An NNAL unit may be an independent feature restoration unit. The input features for a single NNAL unit may be extracted from the same layer within a neural network. Accordingly, the input features for a single NNAL unit may be forced to have the same characteristics. For example, the same feature extraction method may be applied to the input features for a single NNAL unit.

[0147] An NNAL unit may include an NNAL unit header and an NNAL unit payload. The NNAL unit header may include all information necessary to utilize the encoded feature for a task. The NNAL unit payload may include abstracted feature information. The NNAL unit payload may include a group header and group data. The group header may include configuration information of feature group data, such as information about the temporal order, number, and common properties of feature channels constituting the feature group. A feature channel may mean an encoded feature unit. The group data may include a plurality of feature channels and coding instructions, and each feature channel may include type information, prediction information, side information, and residual information. In this case, the type information may indicate an encoding method, and the prediction information may indicate a prediction method. In addition, the side information may indicate additional information necessary for decoding (e.g., entropy coding, quantization-related information, etc.), and the residual information may include information about encoded feature elements (i.e., a set of feature value information).

[0148] Figure 9 is a drawing to explain various issues of the VCM system.

[0149] Referring to FIG. 9, the VCM system (900) may include a first device (910) and a second device (920). The first device (910) is a transmitting device and may include a source device (911) and an encoder (913). The source device (911) may obtain image information through processes such as image capture and synthesis. The encoder (913) may encode the image information obtained by the source device (911) to generate encoded image data (encoded image / feature information) and output it in the form of a bitstream. The second device (920) is a receiving device and may include a decoder (921) and a test network (923). The decoder (921) may decode data obtained from the first device (910) according to receiving requirements (e.g., bandwidth, frame rate) to obtain image / feature information. The task network (923) can perform a predetermined task (e.g., machine task, human vision task) using the acquired image / feature information.

[0150] A predetermined context information may be input to the source device (911). Here, the context information refers to information regarding the circumstances when acquiring image information, and may include information regarding the device environment (e.g., location, time) or task properties. Unnecessary noise may be added to the context information due to environmental factors (e.g., weather, light source, background). In addition, unnecessary noise may be added to the image information acquired based on the context information due to properties of the source device (911) (e.g., camera parameters, sensor noise). Therefore, a preprocessing (or optimization) process is required to remove the above-described noise.

[0151] Meanwhile, the information required for task performance may vary depending on the task's attributes or purpose. For example, the information required for machine tasks and human vision tasks may differ. Furthermore, the required information may vary depending on the type of machine task. Therefore, adaptive decoding or postprocessing processes that consider the task's attributes and purpose are required.

[0152] Accordingly, the present disclosure provides various embodiments of optimization techniques and their expression methods to address the aforementioned issues. The embodiments of the present disclosure may be implemented individually or in combination of two or more. Below, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0153] Example 1

[0154] Embodiment 1 of the present disclosure relates to a method for removing noise from an input source and a method for enhancing information to be suitable for a specific purpose task.

[0155] FIGS. 10 and 11 are drawings showing a VCM system according to one embodiment of the present disclosure.

[0156] First, referring to FIG. 10, the VCM system (1000) may include a first device (1010) and a second device (1020). The first device (1010) may refer to a component that provides information as a transmitting device (e.g., encoding and transmitting image information), and the second device (1020) may refer to a component that consumes information as a receiving device (e.g., decoding and utilizing image information).

[0157] The first device (1010) may include a source device (1011) and an encoder (1013). In addition, the first device (1010) may further include an optimizer (1012) for removing noise from image information (or input source) acquired by the source device (1011). In the present disclosure, the optimizer (1012) may be referred to by various terms such as a preprocessor or a noise remover.

[0158] The optimizer (1012) can perform a predetermined preprocessing operation (or optimization) on the image information to remove various noises added to the image information. In addition, the optimizer (1012) can improve information necessary for performing a task through the preprocessing operation. The preprocessing method, i.e., the optimization method, needs to be adaptively defined in consideration of the task purpose in order to improve task efficiency and accuracy. To this end, the second device (1020) can request (B) the first device (1010) to apply an appropriate optimization method, and the first device (1010) can transmit (A) information about the optimization method (e.g., whether to apply the optimization method, the optimization type and purpose, etc.) (hereinafter, optimization information) to the second device (1020). In one example, the optimization information can be transmitted to the second device (1020) via a bitstream. As described above with reference to FIG. 10, a bitstream can be composed of units for transmission (e.g., NNAL units), and attribute information of the transmission units (e.g., NNAL unit type) can be utilized to transmit the optimization information.

[0159] Meanwhile, optimization information may be generated by the optimizer (1012) and transmitted to the encoder (1013). For this purpose, an interface (INF) between the optimizer (1012) and the encoder (1013) may be defined. The interface (INF) may be composed of general information about the input source and optimization information. Here, the general information about the input source may include information necessary for the basic operation of the encoder (1013), such as input bit depth (InputBitDepth), luma to chroma sample ratio (InputFormat), frame rate (FrameRete), input source width (SourceWidth), and height (SourceHeight).

[0160] According to an embodiment, the first device (1010) described above may be expanded into multiple units, unlike that illustrated in FIG. 10. That is, the second device (1020) may communicate with multiple first devices. In this case, the multiple first devices may be interconnected to form a single logical entity. For example, the multiple first devices may be a collection of CCTVs installed in a wide area, and the second device (1020) may be a central control device that receives CCTV images from the multiple first devices.

[0161] As the first device (1010) is expanded to multiple units, the number of decoders and task networks included in the second device (1020) may also be expanded to multiple units. That is, the second device (1020) may also be a single logical entity in which multiple second devices are interconnected. In this case, as illustrated in FIG. 11, the second device (1120) may further include an optimization manager (1123) that controls the decoder (1121) and the task network (1122) and manages optimization information. The optimization manager (1123) may receive multiple output signals from the decoder (1121) and the task network (1122) (A), request optimization information corresponding to each signal from the first device (1110), and transmit a receiving-side requirement (B). The information transmitted between the first device (1110) and the second device (1120) may be all or part of the same as in case (B) of FIG. 10.

[0162] Meanwhile, optimization information is information required for preprocessing (or optimization) operations on input sources, and can be defined step-by-step (or hierarchically) as shown in FIGS. 12 and 13.

[0163] Referring first to FIG. 12, the first stage interface can define whether optimization is applied to an input source and the scope of application. The second stage interface can define optimization properties and types, or optimization objectives. The third stage interface can define additional information based on optimization properties and types. Depending on the embodiment, only some of the above stages may be defined, or all of the above stages may be defined. For example, only the first stage interface may be defined, the first and second stage interfaces may be defined, or all of the first to third stage interfaces may be defined.

[0164] Table 1 below shows an example where only the first-stage interface is defined.

[0165]

[0166] Referring to Table 1, optimization_info( ) can include the syntax elements optimization_enable_flag, level_2_information_enable_flag, and level_2_information.

[0167] The syntax element optimization_enable_flag can indicate whether optimization is applied to the input source. For example, optimization_enable_flag with a first value (e.g., 1) can indicate that optimization is applied to the input source. Conversely, optimization_enable_flag with a second value (e.g., 0) can indicate that optimization is not applied to the input source. If optimization_enable_flag does not exist, it is considered that optimization is not applied to the input source, and the value of optimization_enable_flag can be inferred as the second value (e.g., 0). Whether the encoded bitstream is for the original input source or the optimized input source can be determined through the optimization_enable_flag value. Meanwhile, unlike Table 1, whether optimization is applied can also be indicated using two syntax elements. For example, whether optimization is available may be indicated through an optimization enable flag (eg, optimization_enable_flag), and if optimization is available (eg, optimization_enable_flag = 1), whether optimization is to be applied to the input source may be indicated through an optimization flag (eg, optimization_flag).

[0168] The syntax elements level_2_infomation_enable_flag and level_2_information may indicate whether a second-level interface is defined and the corresponding information, respectively. In one example, the second-level interface information may include information identifying optimization properties and purposes.

[0169] Table 2 below shows examples where the first and second stage interfaces are defined.

[0170]

[0171] Referring to Table 2, optimization_info( ) can include the syntax elements optimization_enable_flag, optimization_type_flag, optimization_type, optimization_purpose_flag, and optimization_purpose.

[0172] The syntax element optimization_enable_flag is as described above with reference to Table 1 as the first stage interface information.

[0173] The syntax element optimization_type_flag can indicate whether optimization_type is defined. For example, an optimization_type_flag of the first value (e.g., 1) can indicate that optimization_type is defined. Conversely, an optimization_type_flag of the second value (e.g., 0) can indicate that optimization_type is not defined.

[0174] The syntax element optimization_type can be defined when optimization_type_flag is the first value (e.g., 1). optimization_type can indicate optimization properties applied to the input source. The optimization properties can be represented by the type of noise added to the input source, the method for removing / reducing it, the type of information to be enhanced, and the method for enhancing it.

[0175] The syntax element optimization_purpose_flag can indicate whether optimization_purpose is defined. For example, a first value (e.g., 1) of optimization_purpose_flag can indicate that optimization_purpose is defined. Conversely, a second value (e.g., 0) of optimization_purpose_flag can indicate that optimization_purpose is not defined.

[0176] The syntax element optimization_purpose can be defined when optimization_purpose_flag is the first value (e.g., 1). optimization_purpose can indicate the type of tasks that can be performed, i.e., the target application.

[0177] Meanwhile, receiver requirements may vary depending on the device's performance and operating environment. Consequently, even if the purpose is the same, applicable optimization properties may also vary, and optimization properties may not accurately represent the application that can be performed. Therefore, it is necessary to define both optimization properties (optimization_type) and optimization purpose (optimization_purpose) as second-stage interface information. This second-stage interface information allows users to select and request a bitstream appropriate for the device's performance and operating environment.

[0178] Specific examples expressing the optimization properties (optimization_type) are shown in Tables 3 and 4 below.

[0179] First, Table 3 shows the cases in which various optimization methods are applied.

[0180]

[0181] Referring to Table 3, if 'optimization_type == 0' is True, the optimization method determined by the application can be used as is. If 'optimization_type > 0 && ( optimization_type & 0x01 ) == 0' is True, temporal resampling may not be used as the optimization method. If '( optimization_type & 0x01 ) != 0' is True, temporal resampling may be used as the optimization method. If 'optimization_type > 0 && ( optimization_type & 0x02 ) == 0' is True, spatial resampling may not be used as the optimization method. If '( optimization_type & 0x02 ) != 0' is True, spatial resampling may be used as the optimization method. If 'optimization_type > 0 && ( optimization_type & 0x04 ) == 0' is True, Region of interest (ROI)-based optimization may not be used. If '( optimization_type & 0x04 ) != 0' is True, ROI-based optimization may be used. If 'optimization_type > 0 && ( optimization_type & 0x08 ) == 0' is True, spatial quality optimization may not be used. If '( optimization_type & 0x08 ) != 0' is True, quality optimization may be used. When quality optimization is used, pictures may be preprocessed to reduce unnecessary information or improve the quality of necessary information (e.g., noise reduction and speckle removal).

[0182] Table 4 shows the cases where only one optimization method is applied.

[0183]

[0184] Referring to Table 4, if the optimization_type value is '000', temporal resampling can be used as the optimization method. If the optimization_type value is '001', spatial resampling can be used as the optimization method. If the optimization_type value is '010', ROI-based optimization can be used. If the optimization_type value is '011', quality optimization can be used. When spatial quality optimization is used, the picture can be preprocessed to reduce unnecessary information or improve the quality of necessary information (e.g., noise reduction and speckle removal).

[0185] Meanwhile, the above-described optimization methods (e.g., temporal resampling, etc.) are merely examples, and the embodiments of the present disclosure are not limited thereto. That is, the number of bits in optimization_type and the expression methods in Tables 3 and 4 can be expanded to express various optimization methods.

[0186] Next, specific examples expressing the optimization purpose (optimization_purpose) are as shown in Tables 5 and 6 below.

[0187] First, Table 5 shows the cases in which various optimization methods are applied.

[0188]

[0189] Referring to Table 5, if 'optimization_purpose == 0' is True, the optimization objective determined by the application can be applied as is. If 'optimization_purpose > 0 && ( optimization_purpose & 0x01 ) == 0' is True, object detection may not be included in the optimization objective. If '( optimization_purpose & 0x01 ) != 0' is True, object detection may be included in the optimization objective. If 'optimization_purpose > 0 && ( optimization_purpose & 0x02 ) == 0' is True, object tracking may not be included in the optimization objective. If '( optimization_purpose & 0x02 ) != 0' is True, object tracking may be included in the optimization objective. If 'optimization_purpose > 0 && ( optimization_purpose & 0x04 ) == 0' is True, object segmentation may not be included in the optimization objective. If '( optimization_purpose & 0x04 ) != 0' is True, then object segmentation can be included in the optimization objective. If 'optimization_purpose > 0 && ( optimization_purpose & 0x08 ) == 0' is True, then face recognition can be excluded from the optimization objective. If '( optimization_purpose & 0x08 ) != 0' is True, then face recognition can be included in the optimization objective. If 'optimization_purpose > 0 && ( optimization_purpose & 0x10 ) == 0' is True, then human viewing can be excluded from the optimization objective. If '( optimization_purpose & 0x10 ) != 0' is True, then human viewing can be included in the optimization objective.If 'optimization_purpose > 0 && ( optimization_purpose & 0x20 ) == 0' is True, machine analysis may not be included in the optimization purpose. If '( optimization_purpose & 0x20 ) != 0' is True, machine analysis may be included in the optimization purpose.

[0190] Table 6 shows the cases where only one optimization method is applied.

[0191]

[0192] Referring to Table 6, if the optimization_purpose value is '000', object detection can be included in the optimization objective. If the optimization_purpose value is '001', object tracking can be included in the optimization objective. If the optimization_purpose value is '010', object segmentation can be included in the optimization objective. If the optimization_purpose value is '011', face recognition can be included in the optimization objective. If the optimization_purpose value is '100', human viewing can be included in the optimization objective. If the optimization_purpose value is '101', machine analysis can be included in the optimization objective.

[0193] Meanwhile, the above-described optimization purpose (e.g., object detection, etc.) is merely an example, and the embodiments of the present disclosure are not limited thereto. That is, the number of bits in optimization_purpose and the expression methods in Tables 5 and 6 can be expanded to express various optimization purposes.

[0194] Referring back to Table 2, which defines the first and second stage interfaces, optimization_info( ) can further include the syntax elements level_3_information_enable_flag and level_3_information.

[0195] The syntax elements level_3_infomation_enable_flag and level_3_information can indicate whether a third-level interface is defined and the corresponding information, respectively. In one example, the third-level interface information can include details of the optimization method applied to the input source.

[0196] The general information of the original input source (e.g., frame rate, source width / height, etc.) may be changed through optimization. Depending on the application, there may be situations where information regarding the change in the general information is required. For example, if the spatial resolution reduced due to optimization needs to be converted back to the original resolution, the encoder / decoder must know the resolution of the original input source or the resolution reduction ratio. Alternatively, if ROI-based optimization is applied, the encoder / decoder must know the ROI area and non-ROI area. Therefore, detailed information including information regarding the change in the general information may be additionally defined as a third-stage interface.

[0197] Tables 7 through 9 below show examples in which all of the first through third stage interfaces are defined.

[0198]

[0199]

[0200]

[0201] Table 7 is an example of expressing one optimization property (optimization_type) and optimization purpose (optimization_purpose) based on the aforementioned Tables 3 and 5, Table 8 is an example of expressing multiple optimization properties and optimization purposes based on the aforementioned Tables 4 and 6, and Table 9 is an example of expressing one optimization property and optimization purpose based on the aforementioned Tables 4 and 6. In the case of Table 8, in order to express multiple optimization properties and optimization purposes applied to the input source, a syntax element num_of_optimization_type indicating the number of optimization properties and a syntax element num_of_optimization_purpose indicating the optimization purpose may be additionally defined.

[0202] As described above with reference to FIG. 12 and the related tables, the extent to which optimization information is defined in each interface can be individually determined at each stage. For example, whether a second-level interface is defined (e.g., level_2_information_enable_flag) can be determined in a first-level interface, and whether a third-level interface is defined (e.g., level_3_information_enable_flag) can be determined in a second-level interface. Alternatively, the extent to which optimization information is defined in each interface can be determined all at once in the first-level interface, which is the highest level, as illustrated in FIG. 13. Table 10 shows an example of optimization_info( ) based on the interface structure of FIG. 13.

[0203]

[0204] Referring to Table 10, optimization_info( ) can include a syntax element optimization_info_level that indicates up to which level of interface optimization information is defined. optimization_info( ) can further include corresponding level interface information depending on the value of optimization_info_level. For example, if the value of optimization_info_level is 0, optimization_info( ) will only include level 1 interface information. In contrast, if the value of optimization_info_level is 1, optimization_info( ) will further include optimization_type and optimization_purpose as level 2 interface information. In contrast, if the value of optimization_info_level is 2, optimization_info( ) will further include level 3 interface information, such as ROI_info_flag, in addition to the level 2 interface information described above.

[0205] Meanwhile, in Tables 7 to 10 described above, the syntax element ROI_info_flag can indicate whether the syntax ROI_info( ) indicating ROI-related information is defined. For example, a ROI_info_flag of a first value (eg, 1) can indicate that ROI_info( ) is defined. Conversely, a ROI_info_flag of a second value (eg, 0) can indicate that ROI_info( ) is not defined. ROI_info_flag can be defined when the optimization property (optimization_type) includes ROI-based optimization. An example of the ROI_info( ) syntax structure is as shown in Table 11.

[0206]

[0207] Referring to Table 11, ROI_info( ) can include the syntax element num_of_ROI.

[0208] The syntax element num_of_ROI can indicate the number of ROIs within an input source (e.g., a picture). Region_info, which indicates the location and size of the ROI area, can be defined as many times as the number of ROIs indicated by num_of_ROI.

[0209] An example of expressing the position and size of the ROI area within the optimized input source in pixels is as shown in Fig. 14. In Fig. 14, the variable topX represents the horizontal position of the upper left point of the ROI area (the hatched area) in pixels, and the variable topY represents the vertical position of the upper left point of the ROI area in pixels. In addition, the variable width represents the width of the ROI area in pixels, and the variable height represents the height of the ROI area in pixels. Meanwhile, unlike as shown in Fig. 12, the position and size of the ROI area within the input source can also be expressed in a predetermined unit larger than pixels. For this purpose, an example of the ROI_info( ) syntax structure is as shown in Table 12.

[0210]

[0211] Referring to Table 12, ROI_info( ) may further include syntax elements unit_width and unit_height, which indicate the expression unit of the ROI area, unlike Table 11.

[0212] The syntax element unit_width can indicate the width of a predetermined unit that is divided into a specific size to represent the ROI area. In addition, the syntax element unit_height can indicate the height of the predetermined unit. If a suitable division unit is defined in advance considering the processing units of the encoder and decoder, unit_width and unit_height, which indicate the size of the division unit, may not be defined.

[0213] An example of expressing the position and size of the ROI area within the optimized input source in terms of the predetermined division units described above is illustrated in FIG. 15. In FIG. 15, the variable topUnitX represents the horizontal position of the upper left point of the ROI area (the hatched area) in terms of the predetermined division unit, and the variable topUnitY represents the vertical position of the upper left point of the ROI area in terms of the division unit. For example, the topUnitX of the ROI area may be 1, and topUnitY may be 2. In addition, the variable num_of_horizontal_unit represents the width of the ROI area in terms of the division unit, and the variable num_of_vertical_unit represents the height of the ROI area in terms of the division unit. For example, the num_of_horizontal_unit of the ROI area may be 2, and the num_of_vertical_unit may be 1.

[0214] Meanwhile, ROI_info( ) may further include attribute information of predefined regions within the input source. An example of the ROI_info( ) syntax structure further including the region attribute information is shown in Table 13.

[0215]

[0216] Referring to Table 13, ROI_info( ) can include an additional syntax element region_property, unlike Tables 11 and 12.

[0217] The syntax element region_property can represent properties of predefined regions within an input source. The properties of the predefined regions can be categorized as Non-ROI or ROI. That is, region_property can have a first value (eg, 0) representing Non-ROI (eg, background) or a second value (eg, 1) representing ROI (eg, foreground), as shown in Table 14 below.

[0218]

[0219] Another example of the ROI_info( ) syntax structure that further includes the above region attribute information is shown in Table 15.

[0220]

[0221] Referring to Table 15, ROI_info( ) can include an additional syntax element region_property, unlike Tables 11 and 12.

[0222] The syntax element region_property can represent properties of ROI regions within the input source. The properties of the ROI regions can be categorized based on their importance. That is, region_property can have any one of the following values: a first value (e.g., 0) indicating importance level 0, or a fourth value (e.g., 4) indicating importance level 3, as shown in Table 16 below.

[0223]

[0224] Referring back to Tables 7 to 10 mentioned above, optimization_info( ) may further include the syntax element temporal_optimization_info_flag.

[0225] The syntax element temporal_optimization_info_flag may indicate whether temporal_optimization_info( ) containing temporal resampling related information is defined. For example, temporal_optimization_info_flag with a first value (e.g., 1) may indicate that temporal_optimization_info( ) is defined. Conversely, temporal_optimization_info_flag with a second value (e.g., 0) may indicate that temporal_optimization_info( ) is not defined. temporal_optimization_info_flag may be defined only when the optimization property (optimization_type) includes temporal resampling. An example of temporal_optimization_info( ) is as shown in Table 17.

[0226]

[0227] Referring to Table 17, temporal_optimization_info( ) can include syntax elements original_temporal_info and optimization_method_idc.

[0228] The syntax element original_temporal_info may be information necessary to restore the temporal information of the optimized input source to that of the original input source or to compare the two. For example, original_temporal_info may include the frame rate and / or temporal resampling ratio of the original input source.

[0229] The syntax element optimization_method_idc can identify the temporal optimization method applied to the input source. If decrypted information needs to be restored to the temporal level of the original input source for a specific purpose, optimization_method_idc can be used.

[0230] Referring again to Tables 7 to 10, optimization_info( ) may further include the syntax element spatial_optimization_info_flag.

[0231] The syntax element spatial_optimization_info_flag can indicate whether spatial_optimization_info( ) containing spatial resampling (or spatial resampling) related information is defined. For example, a spatial_optimization_info_flag of a first value (eg, 1) can indicate that spatial_optimization_info( ) is defined. Conversely, a spatial_optimization_info_flag of a second value (eg, 0) can indicate that spatial_optimization_info( ) is not defined. spatial_optimization_info_flag can be defined only when the optimization property (optimization_type) includes spatial resampling. An example of spatial_optimization_info( ) is as shown in Table 18.

[0232]

[0233] Referring to Table 18, spatial_optimization_info( ) can include syntax elements original_spatial_info and optimization_method_idc.

[0234] The syntax element original_spatial_info may be information necessary to restore the spatial level of the optimized input source to that of the original input source or to compare the two. For example, original_spatial_info may include the resolution and / or spatial resampling ratio of the original input source.

[0235] The syntax element optimization_method_idc can identify the spatial optimization method applied to the input source. If decoded information needs to be restored to the spatial level of the original input source for a specific purpose, optimization_method_idc can be used.

[0236] Referring again to Tables 7 to 10, optimization_info( ) may further include the syntax element quality_optimization_info_flag.

[0237] The syntax element quality_optimization_info_flag can indicate whether quality_optimization_info( ) containing quality optimization related information is defined. For example, a quality_optimization_info_flag of the first value (e.g., 1) can indicate that quality_optimization_info( ) is defined. Conversely, a quality_optimization_info_flag of the second value (e.g., 0) can indicate that quality_optimization_info( ) is not defined. quality_optimization_info_flag can be defined only when the optimization property (optimization_type) includes quality optimization. An example of quality_optimization_info( ) is as shown in Table 19.

[0238]

[0239] Referring to Table 19, quality_optimization_info( ) can include syntax elements optimization_type and optimization_method_idc.

[0240] The syntax element optimization_type can be attribute information for the applied optimization method. The purpose of optimization may be to remove noise or to strengthen or weaken specific attribute information. Furthermore, the optimization method may alter the semantic information of the input source. Therefore, the information consumer can use the attribute information of the optimization method to determine whether the optimization method is suitable for the task's purpose and its impact.

[0241] The syntax element optimization_method_idc can identify the applied optimization method. Even optimization methods with the same purpose and similar properties can be implemented and applied differently. For example, the optimization method may be defined differently depending on the operating conditions and performance of the device to which it is applied. Therefore, the information consumer can use the optimization method identification information to specifically identify the tool used in the optimization method. An example of optimization_method_idc is shown in Table 20.

[0242]

[0243] Referring to Table 20, if the optimization_method_idc value is '00', a denoising filter can be used to optimize the input source. If the optimization_method_idc value is '01', an edge-preserving filter can be used to optimize the input source. If the optimization_method_idc value is '10', an image enhancement process can be used to optimize the input source. If the optimization_method_idc value is '11', a filter bank (bandpass filter array) can be used to optimize the input source.

[0244] In order to transmit the aforementioned optimization information to the receiving device (1020, 1120), it is necessary to include the optimization information transmitted to the encoder (1013, 1113) in the bitstream through the definition of the interface (INF) between the optimizer (1012, 1112) and the encoder (1013, 1113).

[0245] Figure 16 is a diagram illustrating the process of optimizing an input source and transmitting it to an encoder. In Figure 16, 0 to 8 represent the frame numbers of the input source, which may correspond to the input / output order of each frame. Additionally, point A represents the point in time when optimization method A is applied, and point B represents the point in time when a new optimization method B is applied.

[0246] Referring to FIG. 16, at time point A, optimization method A corresponding to optimization_info_A is applied to frame 0. In addition, the optimization method A is also applied to frames 1 to 3 following frame 0 in the input / output order. Thereafter, at time point B, the optimization method is changed, and optimization method B corresponding to optimization_info_B is applied to frame 4. In addition, the optimization method B is also applied to frames 5 to 8 following frame 4 in the input / output order.

[0247] Frames 0 to 8 optimized by the above optimization method A or B are sequentially transmitted from the optimizer to the encoder.

[0248] Figure 17 is a diagram showing the process of encoding the optimized input source of Figure 16.

[0249] Referring to FIG. 17, the encoding order (and decoding order) of the input source may be different from the order transmitted from the optimizer. For example, in FIG. 17, the optimized input source may be encoded in the order of 'frame 0 → 4 → 2 → 1 → 3 → 8 → 6 → 5 → 7'. As such, the encoding order of the input source may be different from the input / output order, and thus the change points of the optimization method may also be different during the optimization process and the encoding process. Specifically, in the example of FIG. 16, the discontinuity of the optimization method occurs only at time point B (i.e., frame time point 3 → 4), but in the example of FIG. 17, the discontinuity of the optimization method may occur at three time points (i.e., frame time points 0 → 4, 4 → 2, and 3 → 8).

[0250] When encoding frame 4, the only frame that can be referenced is frame 0. However, since the optimization methods applied to the two frames are different, the information properties may also be different. Therefore, referencing frame 0 to encode frame 4 may be inappropriate. Furthermore, when encoding frame 2, referencing frame 0 to which the same optimization method has been applied is acceptable, but referencing frame 4 to which a different optimization method has been applied may be inappropriate. Therefore, the encoder must be able to determine whether the optimization methods are the same between the encoding target frame and the reference candidate frame(s). Accordingly, information for identifying the optimization method applied to each frame of the input source may be defined. Examples of syntax structures including the identification information are shown in Tables 21 and 22.

[0251]

[0252]

[0253] First, referring to Table 21, input_optimization_info( ) can include a syntax element info_id. The syntax element info_id can be information for identifying the preprocessing information optimization_info( ) received from the optimizer (see Tables 1-2 and 7-10). In one example, the info_id value can be defined based on the order of receiving optimization_info( ). For example, the info_id value for the optimization_info( ) received first by the encoder can be 0, and the info_id value for the optimization_info( ) received second can be 1. In another example, the info_id value can also be defined based on whether the information of optimization_info( ) has changed. For example, if the optimization_info( ) received first by the encoder and the optimization_info( ) received second contain the same information, the info_id value can be the same as 0 for both. Conversely, if the second optimization_info( ) received by the encoder contains different information from the third optimization_info( ), the info_id value for the third optimization_info( ) received may be 1.

[0254] Next, referring to Table 22, picture_parameter_set_rbsp( ) can include the syntax element info_id. Table 22 is not a method of defining a new syntax structure as in Table 21, but a method of including the aforementioned info_id in the existing syntax structure. According to Table 22, the identification information of the optimization method can be defined on a picture basis. Of course, unlike Table 22, it is also possible to define the identification information in a specific unit other than a picture (e.g., a frame group).

[0255] Meanwhile, it can be assumed that the same optimization method is applied until new identification information is received. In this case, optimization information for consecutive frames in the input / output sequence can be obtained (or inferred) from the most recently received optimization_info() until a new optimization_info() is received.

[0256] Tables 23 and 24 show examples of how to express input_optimization_info( ) of Table 22 described above.

[0257]

[0258]

[0259] input_optimization_info( ) can be defined as a new NAL unit type as shown in Table 23. Alternatively, input_optimization_info( ) can also be defined via a SEI (Supplemental enhancement information) message as shown in Table 24. The payload type value (ie, 205) of input_optimization_info( ) in Table 24 is only an example and can be determined as an arbitrary value.

[0260] Meanwhile, if each frame of the input source has a different info_id, additional identification information may be defined to minimize inefficiency in the referencing process.

[0261] In one example, the additionally defined identification information may include an optimization_compensation_flag. The optimization_compensation_flag may indicate whether to compensate for differences in properties between the encoding target frame and the reference frame due to differences in optimization methods (e.g., different info_id values ​​or different methods applied to each ROI region). For example, an optimization_compensation_flag of a first value (e.g., 1) may indicate that a compensation operation is performed between the encoding target frame and the reference frame. Conversely, an optimization_compensation_flag of a second value (e.g., 0) may indicate that no compensation operation is performed between the encoding target frame and the reference frame. The optimization_compensation_flag may be defined on a per-frame basis through a picture header or a picture parameter set, or may be defined on a per-frame basis through a sequence parameter set, a picture group header, or the like.

[0262] In another example, the additionally defined identification information may include a constrained_prediction_flag. The constrained_prediction_flag may indicate whether a given reference constraint is applied. For example, a constrained_prediction_flag of a first value (e.g., 1) may indicate that the reference constraint is applied. Here, examples of the reference constraint may include a priority constraint (e.g., a reference candidate frame to which a different optimization method is applied is referenced later), a restricted reference (e.g., texture information of a reference candidate frame to which a different optimization method is applied is not referenced), etc. In contrast, a constrained_prediction_flag of a second value (e.g., 0) may indicate that the reference constraint is not applied. The constrained_prediction_flag may be defined on a frame-by-frame basis through a picture header or a picture parameter set, or may be defined on a frame-by-frame basis through a sequence parameter set, a picture group header, etc.

[0263] According to Embodiment 1 of the present disclosure, a preprocessing operation based on a predetermined optimization method can be performed on the input source of the VCM. Various information and interfaces for representing, identifying, and communicating the optimization method can be defined. In addition, an encoding method (e.g., reference constraint) that considers the optimization method can be provided. Through such a preprocessing operation (optimization process), noise contained in the input source can be effectively removed, and necessary information can be further enhanced to match the task properties and purpose.

[0264] Example 2

[0265] Multiple optimization methods can be applied to a single input source. However, if the order of application among these optimization methods is not defined, inefficiencies in the optimization process can arise. For example, if temporal resampling is performed after spatial resampling, unnecessary spatial resampling may occur even for information that is scheduled to be removed through temporal resampling. Furthermore, the influence of different optimization methods needs to be considered during the optimization process. For example, region-specific information defined in the ROI-based optimization process can be reused for region-specific adaptive processing in the quality optimization process. Furthermore, if post-processing is required (e.g., conversion to the original input frame rate, conversion to the original input resolution, etc.), the order of optimization application needs to be defined for more efficient post-processing.

[0266] Accordingly, according to Embodiment 2 of the present disclosure, when multiple optimization methods are applied to one input source, various methods for defining the application order among the multiple optimization methods can be provided.

[0267] FIG. 18 is a diagram showing the application order of an optimization method according to one embodiment of the present disclosure.

[0268] Referring to FIG. 18, each bit of the aforementioned optimization_type may be assigned a different optimization method. For example, temporal resampling may be assigned to the least significant bit (LSB) of optimization_type, spatial resampling to the second LSB, an ROI-based optimization method to the third LSB, and a quality optimization method to the fourth LSB. In this case, the plurality of optimization methods may be applied in bit-ascending order starting from the temporal resampling assigned to the LSB. Alternatively, the plurality of optimization methods may be applied in bit-descending order starting from the quality optimization method assigned to the fourth LSB. In this way, according to one embodiment, the application order among the plurality of optimization methods may be determined, and optimization_type may be defined based on the application order. In this case, there is no need to separately define information regarding the application order.

[0269] Meanwhile, if it is impossible to define optimization_type based on the above application order, or if the application order is likely to change, it is necessary to define information regarding the application order. Accordingly, in one embodiment, information regarding the application order may be additionally defined in the aforementioned preprocessing information optimization_info( ) (see Tables 1-2 and 7-10). Specific examples are provided in Tables 25 to 27.

[0270]

[0271]

[0272] First, referring to Table 25, optimization_info( ) can include syntax elements num_of_optimization and ordering_criterion_idc.

[0273] The syntax element num_of_optimization may indicate the number of optimization methods applied to the input source. In one embodiment, num_of_optimization may be derived from a predefined optimization_type. In this case, num_of_optimization may be set to, for example, the number of bit(s) with a value of 1 in optimization_type.

[0274] The syntax element ordering_criterion_idc can indicate reference information for the application order of multiple optimization methods. Table 26 shows an example of ordering_criterion_idc. Referring to Table 26 together with Table 25, if the ordering_criterion_idc value is '00', multiple optimization methods can be applied in the order of the LSB of optimization_type. If the ordering_criterion_idc value is '01', multiple optimization methods can be applied in the order of the MSB of optimization_type. If the ordering_criterion_idc value is '10', a separate application order (alternative_ordering) can be defined. If the ordering_criterion_idc value is '11', multiple optimization methods can be applied in any order.

[0275]

[0276] Referring to Table 27, optimization_info( ) can include the syntax elements num_of_optimization, ordering_criterion_idc, and alternative_ordering_flag. The syntax elements num_of_optimization and ordering_criterion_idc are as described above with reference to Table 25.

[0277] The syntax element alternative_ordering_flag may indicate whether the application order of multiple optimization methods is separately defined. For example, an alternative_ordering_flag of a first value (e.g., 1) may indicate that the application order is separately defined. Conversely, an alternative_ordering_flag of a second value (e.g., 0) may indicate that the application order is not separately applied. Unlike Table 28, alternative_ordering_flag may be defined only when multiple optimization methods are applied to the input source (i.e., num_of_optimization > 1). In addition, if the default application order is defined in advance based on MSB or LSB, alternative_ordering_flag may not be defined.

[0278] Table 28 shows examples of information utilization between optimization methods.

[0279]

[0280] Referring to Table 28, quality_optimization_info( ) regarding quality optimization methods can include syntax elements adaptive_quality_optimization_flag, optimization_types[i], optimization_methods[i], optimization_type, and optimization_method.

[0281] The syntax element adaptive_quality_optimization_flag can indicate whether the adaptive quality optimization method is applied to the ROI region. For example, adaptive_quality_optimization_flag with a first value (e.g., 1) can indicate that the adaptive quality optimization method is applied to the ROI region. Conversely, adaptive_quality_optimization_flag with a second value (e.g., 0) can indicate that the adaptive quality optimization method is not applied to the ROI region. adaptive_quality_optimization_flag can be defined only when optimization_type includes application of ROI-based optimization (i.e., ( optimization_type & 0x04 ) != 0). Furthermore, adaptive_quality_optimization_flag can be defined only when ROI-related details are defined (i.e., ROI_info_flag = True).

[0282] The syntax element optimization_methods[i] can indicate the optimization method applied to each ROI region. The value of optimization_methods[i] can also be defined using the region_property property included in ROI_info( ) in Table 13 described above. In this case, different optimization methods can be applied based on the properties of each region (e.g., ROI vs. non-ROI) or their importance.

[0283] The syntax element optimization_types[i] can indicate the properties of the optimization method applied to each ROI region. The value of optimization_types[i] can also be defined using the region_property included in ROI_info( ) in Table 13 described above. In this case, different optimization properties can be applied based on the properties of each region (e.g., ROI vs. non-ROI) or importance. For example, an optimization property aimed at removing unnecessary information (or noise) can be applied to a region with low importance, and an optimization property aimed at enhancing necessary information can be applied to a region with high importance.

[0284] The syntax element optimization_type can be attribute information for the applied optimization method. The purpose of optimization may be to remove noise or to strengthen or weaken specific attribute information. Furthermore, the optimization method may alter the semantic information of the input source. Therefore, the information consumer can use the attribute information of the optimization method to determine whether the optimization method is suitable for the task's purpose and its impact.

[0285] The syntax element optimization_method_idc may identify the applied optimization method. Optimization methods with the same purpose and similar properties may be implemented and applied differently. For example, the optimization method may be defined differently depending on the operating conditions and performance of the device to which it is applied. Therefore, the information consumer can use the optimization method identification information to specifically identify the tool used in the optimization method. Examples of optimization_method_idc are as described above with reference to Table 19.

[0286] According to the second embodiment of the present disclosure, various methods for defining the application order among multiple optimization methods can be provided. Accordingly, the efficiency of the optimization process, and further, the post-processing process on the receiver side, can be further improved.

[0287] Meanwhile, a post-processing process corresponding to the optimization (or pre-processing) process of the aforementioned embodiments 1 and 2 may be performed in the receiving device. For this purpose, the receiving device may further include a post-processor (1922) as illustrated in FIG. 19. The post-processor (1922) may identify an optimization method and properties applied to an input source based on optimization information obtained from a bitstream, and determine a specific post-processing method identical to or corresponding to the identified optimization method and properties. In addition, the post-processor (1922) may post-process the decoded image data based on the determined post-processing method.

[0288] Example 3

[0289] The optimization methods of the aforementioned Embodiments 1 and 2 may be defined differently depending on the purpose of the receiving device. This is because the required information may vary depending on the purpose of information consumption in the VCM system. Therefore, an interface must be defined for the receiving device to request an appropriate optimization method from the transmitting device. Furthermore, since the optimization method and its representation method (optimization information) may change according to the request of the receiving device, the interface must correspond to the interface defined between the optimizer and encoder of the transmitting device. Accordingly, Embodiment 3 of the present disclosure provides an interface between the receiving device and the transmitting device for optimization operations.

[0290] FIG. 20 is a diagram for explaining a setting process of an optimization method according to one embodiment of the present disclosure.

[0291] Referring to FIG. 20, a second device (2020) (e.g., client) must be able to obtain information regarding applicable optimization methods from a first device (2010) (e.g., server). To this end, the second device (2020) may request the first device (2010) to transmit information regarding applicable optimization methods via a describe function (a). In this case, the first device (2010) may transmit information regarding applicable optimization methods to the second device (2020) via a response A function (a').

[0292] Additionally, the second device (2020) must be able to request the first device (2010) to apply (including change / cancel application, etc.) an optimization method. To this end, the second device (2020) can request the first device (2010) to apply the optimization method through the setup function (b). In this case, the first device (2010) can apply the optimization method and transmit information about the applied optimization method to the second device (2020) through the response B function (b').

[0293] The above-described functions may be implemented through an interface (INF) between the first device (2010) and the second device (2020). Furthermore, the above-described functions may be implemented by extending / applying existing protocols (e.g., RTSP / RTCP / WebRTC, etc.) or defining new protocols.

[0294] An example of information transmitted through the above response A function is shown in Table 29.

[0295]

[0296] Referring to Table 29, available_optimization_info( ) for the response A function may include the syntax element available_optimization_type. available_optimization_type indicates available optimization properties. Examples of the available optimization properties are as shown in Table 30.

[0297]

[0298] Referring to Table 30, if 'available_optimization_type == 0' is True, the optimization method determined by the application can be used as is. If 'available_optimization_type > 0 && ( available_optimization_type & 0x01 ) == 0' is True, temporal resampling may not be available as an optimization method. If '( available_optimization_type & 0x01 ) != 0' is True, temporal resampling may be available as an optimization method. If 'available_optimization_type > 0 && ( available_optimization_type & 0x02 ) == 0' is True, spatial resampling may not be available as an optimization method. If '( available_optimization_type & 0x02 ) != 0' is True, spatial resampling may be available as an optimization method. If 'available_optimization_type > 0 && ( available_optimization_type & 0x04 ) == 0' is True, then Region of interest (ROI)-based optimization may not be available. If '( available_optimization_type & 0x04 ) != 0' is True, then ROI-based optimization may be available. If 'available_optimization_type > 0 && ( available_optimization_type & 0x08 ) == 0' is True, then quality optimization may not be available. If '( available_optimization_type & 0x08 ) != 0' is True, then quality optimization may be available.

[0299] Depending on the embodiment, the information transmitted through the above response A function may be defined through an existing protocol such as RTSP or RTCP, unlike Tables 29 and 30 described above. Specific examples thereof are as shown in Table 31.

[0300]

[0301] Meanwhile, an example of the information required when requesting deactivation of optimization settings is shown in Table 32.

[0302]

[0303] For deactivation requests for optimization settings, unlike activation requests, information regarding the deactivation target may not be necessary. Therefore, only a deactivation request message can be defined, as shown in Table 32. In this case, all optimization methods applied to the input source may be canceled.

[0304] Other examples of information required when requesting to disable optimization settings are shown in Table 33.

[0305]

[0306] Referring to Table 33, deactivation_request( ) can include the syntax elements deactivate_all_optimization_flag and deactivate_optimization_type.

[0307] The syntax element deactivate_all_optimization_flag can indicate whether optimization settings are to be selectively deactivated. For example, a deactivate_all_optimization_flag with a first value (e.g., 1) can indicate a request to deactivate all optimization settings. Conversely, a deactivate_all_optimization_flag with a second value (e.g., 0) can indicate a request to deactivate some (selectively) optimization settings.

[0308] The syntax element deactivate_optimization_type can indicate the optimization setting to be deactivated. deactivate_optimization_type can only be defined if a selective deactivation request exists, i.e., if the deactivate_all_optimization_flag has a second value (e.g., 0). An example of the selective deactivation request is shown in Table 34.

[0309]

[0310] Referring to Table 34, if 'deactivate_optimization_type == 0' is True, the optimization method determined by the application can be used as is. If 'deactivate_optimization_type > 0 && ( deactivate_optimization_type & 0x01 ) == 0' is True, there may not be any changes to the current state. If '( deactivate_optimization_type & 0x01 ) != 0' is True, the deactivation request may be for temporal resampling. If 'deactivate_optimization_type > 0 && ( deactivate_optimization_type & 0x02 ) == 0' is True, there may not be any changes to the current state. If '( deactivate_optimization_type & 0x02 ) != 0' is True, the deactivation request may be for spatial resampling. If 'deactivate_optimization_type > 0 && ( deactivate_optimization_type & 0x04 ) == 0' is True, there may be no changes to the current state. If '( deactivate_optimization_type & 0x04 ) != 0' is True, the deactivation request may be for a ROI-based optimization method. If 'deactivate_optimization_type > 0 && ( deactivate_optimization_type & 0x08 ) == 0' is True, there may be no changes to the current state. If '( deactivate_optimization_type & 0x08 ) != 0' is True, the deactivation request may be for a quality optimization.

[0311] Depending on the embodiment, the information required for the deactivation request may be defined through an existing protocol such as RTSP or RTCP, unlike Tables 32 to 34 described above. In this case, the optimization methods subject to the deactivation are listed as in Table 35.

[0312]

[0313] Meanwhile, it is also possible to request optional activation of optimization settings. Examples of the information required for such optional activation requests are shown in Table 36.

[0314]

[0315] According to Embodiment 3 of the present disclosure, an interface for optimization control can be defined between a VCM transmitter and a receiver. In this case, the control (particularly, a deactivation request) can be performed individually for each optimization method, or can be performed together for all optimization methods. Accordingly, appropriate optimization control can be performed according to the purpose of the receiver device.

[0316] Hereinafter, with reference to FIGS. 21 and 22, an image data processing method according to one embodiment of the present disclosure will be described in detail.

[0317] Fig. 21 is a flowchart illustrating an image data processing method according to one embodiment of the present disclosure. The image data processing method of Fig. 21 can be performed by a receiving device.

[0318] Referring to FIG. 21, a receiving device can decode image data acquired from a bitstream (S2110). Furthermore, the receiving device can acquire information regarding an optimization method for the image data (S2120) and, based on the acquired information regarding the optimization method, post-process the decoded image data (S2130). At this time, the information regarding the optimization method can include optimization attribute information indicating the type of at least one optimization method applied to the image data.

[0319] In one embodiment, the types of optimization methods may include temporal resampling, spatial resampling, region of interest (ROI)-based optimization, and quality optimization.

[0320] In one embodiment, the information regarding the optimization method may further include information indicating the number of types of optimization methods applied to the image data.

[0321] In one embodiment, the information regarding the optimization method may further include optimization purpose information indicating the purpose of at least one optimization method applied to the image data. Here, the purpose of the optimization method may include object detection, object tracking, object segmentation, face recognition, human viewing, and machine analysis.

[0322] In one embodiment, the information regarding the optimization method may further include information indicating the number of target optimization methods applied to the image data.

[0323] In one embodiment, when a region of interest (ROI)-based optimization method is applied to the image data, information about the optimization method may further include information indicating properties of each region included in the original image of the image data.

[0324] In one embodiment, the information regarding the optimization method further includes information regarding changes in general information of the image data, and the information regarding changes in general information can be used to restore the decrypted image data.

[0325] In one embodiment, when multiple optimization methods are applied to the image data, information about the optimization methods may further include information indicating the type of tool used in each of the multiple optimization methods.

[0326] In one embodiment, when multiple optimization methods are applied to the image data, information about the optimization methods may further include information about the order of application of the multiple optimization methods.

[0327] In one embodiment, the information regarding the optimization method may further include information indicating the types of optimization methods available for the image data. In this case, the available optimization methods may be individually controlled based on a selective deactivation request from the receiving device.

[0328] Fig. 22 is a flowchart illustrating an image data processing method according to one embodiment of the present disclosure. The image data processing method of Fig. 22 can be performed by a transmission device.

[0329] Referring to FIG. 22, a transmission device may determine an optimization method for image data (S2210) and preprocess the image data based on the determined optimization method (S2220). Furthermore, the transmission device may encode information regarding the preprocessed image data and the optimization method (S2230). At this time, the information regarding the optimization method may include optimization attribute information indicating the type of at least one optimization method applied to the image data.

[0330] Meanwhile, information regarding the optimization method encoded by the transmission device may be included in the bitstream. The bitstream may be stored in a non-transitory computer-readable recording medium and transmitted to a receiving device via a communication means (e.g., a wired / wireless network).

[0331] While the exemplary methods of this disclosure are presented as a series of operations for clarity of description, this is not intended to limit the order in which the steps are performed, and individual steps may be performed simultaneously or in different orders, if desired. To implement a method according to this disclosure, additional steps may be included in addition to the steps illustrated, some steps may be excluded and the remaining steps included, or some steps may be excluded and additional steps included.

[0332] In the present disclosure, a video encoding device or video decoding device performing a predetermined operation (step) may perform an operation (step) of checking the conditions or circumstances under which the operation (step) is performed. For example, if it is described that a predetermined operation is performed when a predetermined condition is satisfied, the video encoding device or video decoding device may perform an operation of checking whether the predetermined condition is satisfied and then perform the predetermined operation.

[0333] The various embodiments of the present disclosure are not intended to list all possible combinations but rather to illustrate representative aspects of the present disclosure, and the matters described in the various embodiments may be applied independently or in combination of two or more.

[0334] The embodiments described in this disclosure may be implemented and performed on a processor, microprocessor, controller, or chip. For example, the functional units depicted in each drawing may be implemented and performed on a computer, processor, microprocessor, controller, or chip. In this case, information (e.g., information on instructions) or algorithms for implementation may be stored on a digital storage medium.

[0335] In addition, the decoder (decoding device) and encoder (encoding device) to which the embodiment(s) of the present disclosure are applied may be included in a multimedia broadcasting transmitting / receiving device, a mobile communication terminal, a home cinema video device, a digital cinema video device, a surveillance camera, a video conversation device, a real-time communication device such as a video communication, a mobile streaming device, a storage medium, a camcorder, a video-on-demand (VoD) service providing device, an OTT video (Over the top video) device, an Internet streaming service providing device, a three-dimensional (3D) video device, a VR (virtual reality) device, an AR (argumente reality) device, a video phone video device, a transportation terminal (e.g., a vehicle (including an autonomous vehicle) terminal, a robot terminal, an airplane terminal, a ship terminal, etc.), and a medical video device, and may be used to process a video signal or a data signal. For example, OTT video (Over the top video) devices may include game consoles, Blu-ray players, Internet-connected TVs, home theater systems, smartphones, tablet PCs, and DVRs (Digital Video Recorders).

[0336] In addition, the processing method to which the embodiment(s) of the present disclosure are applied can be produced in the form of a computer-executable program and can be stored in a computer-readable recording medium. Multimedia data having a data structure according to the embodiment(s) of the present document can also be stored in a computer-readable recording medium. The computer-readable recording medium includes all types of storage devices and distributed storage devices in which computer-readable data is stored. The computer-readable recording medium can include, for example, a Blu-ray disc (BD), a universal serial bus (USB), a ROM, a PROM, an EPROM, an EEPROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device. In addition, the computer-readable recording medium includes a medium implemented in the form of a carrier wave (e.g., transmission via the Internet). In addition, a bitstream generated by an encoding method can be stored in a computer-readable recording medium or transmitted via a wired or wireless communication network.

[0337] Additionally, the embodiments of the present disclosure may be implemented as a computer program product by program code, and the program code may be executed on a computer by the embodiments of the present disclosure. The program code may be stored on a computer-readable carrier.

[0338] FIG. 23 is a diagram illustrating an example of a content streaming system to which embodiments of the present disclosure can be applied.

[0339] Referring to FIG. 23, a content streaming system to which an embodiment of the present disclosure is applied may largely include an encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.

[0340] The encoding server compresses content input from multimedia input devices, such as smartphones, cameras, and camcorders, into digital data, generates a bitstream, and transmits it to a streaming server. Alternatively, if multimedia input devices, such as smartphones, cameras, and camcorders, generate the bitstream directly, the encoding server may be omitted.

[0341] A bitstream can be generated by a video encoding method and / or a video encoding device to which an embodiment of the present disclosure is applied, and a streaming server can temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0342] A streaming server transmits multimedia data to a user device based on a user request via a web server, and the web server can act as an intermediary, informing the user of available services. When a user requests a desired service from the web server, the web server transmits the request to the streaming server, which then transmits the multimedia data to the user. The content streaming system may include a separate control server, in which case the control server may control commands and responses between each device within the content streaming system.

[0343] A streaming server can receive content from a media repository and / or encoding server. For example, when receiving content from an encoding server, the content can be received in real time. In this case, the streaming server can store the bitstream for a certain period of time to provide a smooth streaming service.

[0344] Examples of user devices include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, head mounted displays (HMDs)), digital TVs, desktop computers, and digital signage.

[0345] Each server within a content streaming system can be operated as a distributed server, in which case the data received from each server can be processed in a distributed manner.

[0346] FIG. 24 is a diagram showing another example of a content streaming system to which embodiments of the present disclosure can be applied.

[0347] Referring to FIG. 24, in an embodiment such as VCM, depending on the performance of the device, the user's request, the characteristics of the task to be performed, etc., the task may be performed on the user terminal or on an external device (e.g., a streaming server, an analysis server, etc.). In this way, in order to transmit the information required for performing the task to the external device, the user terminal may directly or through an encoding server generate a bitstream containing the information required for performing the task (e.g., information such as the task, neural network, and / or purpose).

[0348] The analysis server can decrypt encoded information received from the user terminal (or encoding server) and then perform a task requested by the user terminal. The analysis server can transmit the results obtained through the task performance back to the user terminal or to another linked service server (e.g., a web server). For example, the analysis server can perform a task to determine fire and transmit the results obtained to a fire-related server. The analysis server may include a separate control server, in which case the control server can control commands / responses between each device associated with the analysis server and the server. In addition, the analysis server can request desired information from the web server based on information about the tasks the user device wishes to perform and can perform. When the analysis server requests a desired service from the web server, the web server can transmit this request to the analysis server, and the analysis server can transmit data corresponding to the request to the user terminal. In this case, the control server of the content streaming system can control commands / responses between each device within the streaming system.

[0349] Embodiments according to the present disclosure can be used to process image data.

Claims

1. In a method for processing image data performed by a receiving device, A step of decoding image data obtained from a bitstream; A step of obtaining information on a method for optimizing the above image data; and A step of post-processing the decrypted image data based on information about the above optimization method, Information about the above optimization method includes optimization attribute information indicating the type of at least one optimization method applied to the image data. Method of processing image data.

2. In paragraph 1, The type of the above optimization method is determined for each region based on the region properties of the image data. Method of processing image data.

3. In paragraph 1, The above image data is decoded based on inter-frame prediction, The above inter-frame prediction is performed based on whether the same optimization method is applied to the current frame and the reference frame. Method of processing image data.

4. In paragraph 1, The information about the above optimization method further includes information indicating the number of types of optimization methods applied to the image data. Method of processing image data.

5. In paragraph 1, The information about the above optimization method further includes optimization purpose information indicating the purpose of at least one optimization method applied to the image data. Method of processing image data.

6. In paragraph 5, The information regarding the above optimization method further includes information indicating the number of optimization purposes applied to the image data. Method of processing image data.

7. In paragraph 1, Based on the application of the ROI (Region of interest)-based optimization method to the above image data, Information about the above optimization method further includes information indicating properties of each area included in the image data. Method of processing image data.

8. In paragraph 1, The information about the above optimization method further includes information about changes in general information of the image data, and the information about changes in general information is used to restore the decrypted image data. Method of processing image data.

9. In paragraph 1, Based on the application of multiple optimization methods to the above image data, The information about the above optimization method further includes information indicating the type of tool used in each of the plurality of optimization methods. Method of processing image data.

10. In paragraph 1, Based on the application of multiple optimization methods to the above image data, The information about the above optimization method further includes information about the order of application of the plurality of optimization methods. Method of processing image data.

11. In paragraph 1, The information about the above optimization method further includes information indicating the type of optimization method available for the image data. Method of processing image data.

12. In paragraph 11, The above available optimization methods are individually controlled based on the selective deactivation request of the receiving device. Method of processing image data.

13. In a method for processing image data performed by a transmission device, A step of determining a method for optimizing image data; A step of pre-processing the image data based on the above optimization method; and A step of encoding the preprocessed image data and information about the optimization method is included. Information about the above optimization method includes optimization attribute information indicating the type of at least one optimization method applied to the image data. Method of processing image data.

14. In paragraph 13, A computer-readable recording medium storing a bitstream generated by the above processing method.

15. In a method for transmitting a bitstream generated by an image data processing method, the image data processing method comprises: A step of determining a method for optimizing image data; A step of pre-processing the image data based on the above optimization method; and A step of encoding the preprocessed image data and information about the optimization method is included. Information about the above optimization method includes optimization attribute information indicating the type of at least one optimization method applied to the image data. Bitstream transmission method.

Images