Encoding / decoding method and apparatus based on video optimization according to video usage, and method for transmitting bitstream
The video encoding/decoding method optimizes image compression for either human recognition or machine tasks by signaling optimization information in the bitstream, addressing inefficiencies in existing technologies and improving performance for AI services.
Patent Information
- Application Number
- PCT/KR2024/007557
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-06-25
- Filing Date
- 2024-06-03
- Publication Date
- 2024-12-05
AI Technical Summary
Existing image compression technologies are not optimized for artificial intelligence services, as they focus on high-resolution, high-quality image processing for human vision, making them unsuitable for machine task performance, leading to inefficiencies in processing vast video data within limited resources.
A video encoding/decoding method and device that selects an image optimization method suitable for either human recognition or machine tasks, encoding data with optimization information into a bitstream, and transmitting it for efficient processing, enabling improved encoding/decoding efficiency and performance during human or machine perception.
The solution provides improved encoding/decoding efficiency and performance by optimizing image compression based on intended use, enhancing machine task performance and human recognition capabilities through targeted optimization methods signaled in the bitstream.
Smart Images

Figure KR2024007557_05122024_PF_FP_ABST
Abstract
Description
Encoding / decoding method, device, and bitstream transmission method based on image optimization according to image purpose
[0001] The present disclosure relates to a method and device for encoding / decoding data including images, features, etc., and more particularly, to a method and device for encoding / decoding based on image optimization according to image use, and a method for transmitting a bitstream generated by the encoding method / device of the present disclosure.
[0002] With the advancement of machine learning technology, demand for image processing-based AI services is increasing. To effectively process the massive amounts of image data required by AI services within limited resources, image compression technologies optimized for machine tasks are essential. However, existing image compression technologies have been developed with the goal of high-resolution, high-quality image processing for human vision, making them unsuitable for AI services. Consequently, active research and development is underway on new machine-oriented image compression technologies suitable for AI services.
[0003] The present disclosure aims to provide a video encoding / decoding method and device with improved encoding / decoding efficiency.
[0004] In addition, the present disclosure aims to provide a method and device for encoding / decoding an image based on image optimization according to the purpose of the image.
[0005] In addition, the present disclosure aims to provide a method and device for encoding / decoding an image by selecting an image optimization method suitable for human recognition or machine task (machine perception).
[0006] In addition, the present disclosure aims to provide a method and device for encoding / decoding a video based on a VCM (VIDEO CODING FOR MACHINE) bitstream based on video use.
[0007] In addition, the present disclosure aims to provide a method for defining properties of information required depending on the type of machine task.
[0008] In addition, the present disclosure aims to provide an image optimization method for improving performance in human processing or machine perception.
[0009] In addition, the present disclosure aims to provide a method for transmitting a bitstream generated by an image encoding method or device according to the present disclosure.
[0010] In addition, the present disclosure aims to provide a recording medium storing a bitstream generated by an image encoding method or device according to the present disclosure.
[0011] In addition, the present disclosure aims to provide a recording medium storing a bitstream received and decoded by an image decoding device according to the present disclosure and used for image restoration.
[0012] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present disclosure belongs from the description below.
[0013] A decoding method that can be performed by a data (e.g., image, feature, etc.) decoding device according to one aspect of the present disclosure includes a step of obtaining data optimization information related to an image data optimization method according to a purpose of the image data, and a step of restoring the image data based on the data optimization information, wherein the data optimization information may include data optimization application information indicating whether the image data is optimized for each purpose.
[0014] Meanwhile, the data optimization information may further include data optimization purpose information indicating the purpose of optimizing the image data.
[0015] Meanwhile, the data optimization purpose information may further include machine task optimization information indicating a method for optimizing image data for performing a machine task.
[0016] Meanwhile, the data optimization purpose information may further include human recognition optimization information indicating a method for optimizing image data for human recognition.
[0017] Meanwhile, the human recognition optimization information can be obtained based on the machine task optimization information.
[0018] Meanwhile, the machine task optimization information can be obtained based on the human recognition optimization information.
[0019] Meanwhile, the data optimization information may further include data optimization persistence information indicating the persistence of the optimization method of the image data.
[0020] Meanwhile, the data optimization persistence information may include persistence cancellation information indicating whether optimization persistence is canceled.
[0021] Meanwhile, based on the persistence cancellation information, at least one of persistence degree information indicating the degree of optimization persistence, target information indicating the optimization objective, or type information indicating the optimization property can be further acquired.
[0022] Meanwhile, the above data optimization information can be obtained based on optimization information indicating whether data is optimized.
[0023] Meanwhile, the above optimization information can be signaled in the bitstream.
[0024] Meanwhile, the above data optimization information can be signaled in a bitstream.
[0025] An encoding method that can be performed by a data (e.g., image, feature, etc.) encoding device according to one aspect of the present disclosure includes a step of determining data optimization information related to an image data optimization method according to a purpose of the image data, and a step of encoding the data optimization information into a bitstream, wherein the data optimization information may include data optimization application information indicating whether the image data is optimized for each purpose.
[0026] A recording medium according to another aspect of the present disclosure can store a bitstream generated by an image encoding method or an image encoding device of the present disclosure.
[0027] A bitstream transmission method according to another aspect of the present disclosure can transmit a bitstream generated by an encoding method or a data encoding device of the present disclosure to a data decoding device.
[0028] The features briefly summarized above regarding the present disclosure are merely exemplary aspects of the detailed description of the present disclosure that follows and do not limit the scope of the present disclosure.
[0029] According to the present disclosure, a video encoding / decoding method and device with improved encoding / decoding efficiency can be provided.
[0030] In addition, according to the present disclosure, a method and device for encoding / decoding an image that performs image optimization according to the image purpose can be provided.
[0031] Additionally, according to the present disclosure, image optimization according to image use can be applied.
[0032] Additionally, according to the present disclosure, a method and device for defining properties of information required for image use may be provided.
[0033] In addition, according to the present disclosure, a method and device for encoding / decoding an image with improved image coding performance in human recognition or machine recognition can be provided.
[0034] Additionally, according to the present disclosure, a VCM (VIDEO CODING FOR MACHINE) bitstream for video purposes can be generated.
[0035] In addition, according to the present disclosure, a method and device for encoding / decoding an image by selecting an image optimization method suitable for human recognition or a machine task can be provided.
[0036] Additionally, according to the present disclosure, information necessary to express an image optimization method suitable for human recognition or machine tasks can be signaled.
[0037] Additionally, according to the present disclosure, a method for transmitting a bitstream generated by an image encoding method or device according to the present disclosure can be provided.
[0038] In addition, according to the present disclosure, a recording medium storing a bitstream generated by an image encoding method or device according to the present disclosure can be provided.
[0039] In addition, according to the present disclosure, a recording medium storing a bitstream received and decoded by an image decoding device according to the present disclosure and used for image restoration can be provided.
[0040] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the description below.
[0041] FIG. 1 is a schematic diagram of a VCM system to which embodiments of the present disclosure can be applied.
[0042] FIG. 2 is a schematic diagram illustrating a VCM pipeline structure to which embodiments of the present disclosure can be applied.
[0043] FIG. 3 is a schematic diagram of an image / video encoder to which embodiments of the present disclosure can be applied.
[0044] FIG. 4 is a schematic diagram of an image / video decoder to which embodiments of the present disclosure can be applied.
[0045] FIG. 5 is a flowchart schematically illustrating a feature / feature map encoding procedure to which embodiments of the present disclosure may be applied.
[0046] FIG. 6 is a flowchart schematically illustrating a feature / feature map decoding procedure to which embodiments of the present disclosure may be applied.
[0047] FIG. 7 is a diagram illustrating an example of a feature extraction and restoration method to which embodiments of the present disclosure can be applied.
[0048] FIG. 8 is a diagram showing an example of an image segmentation method to which embodiments of the present disclosure can be applied.
[0049] FIGS. 9 and 10 are diagrams showing examples of a video encoding / decoding system including a VCM video encoder and decoder.
[0050] Figure 11 is a diagram showing an example of a VCM hierarchical structure.
[0051] Figure 12 is a diagram showing an example of a VCM bitstream composed of encoded abstract features and NNAL information.
[0052] FIG. 13 is a diagram for explaining an example in which a coding bitstream to which a coding optimization method according to one embodiment of the present disclosure is applied is used on a receiving target side.
[0053] FIG. 14 is a diagram for explaining a decryption method according to one embodiment of the present disclosure.
[0054] FIG. 15 is a drawing for explaining an encoding method according to one embodiment of the present disclosure.
[0055] FIG. 16 is a diagram illustrating an example of a content streaming system to which embodiments of the present disclosure can be applied.
[0056] FIG. 17 is a diagram showing another example of a content streaming system to which embodiments of the present disclosure can be applied.
[0057] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.
[0058] In describing embodiments of the present disclosure, detailed descriptions of known configurations or functions will be omitted if they are deemed to obscure the gist of the present disclosure. Furthermore, portions unrelated to the description of the present disclosure in the drawings have been omitted, and similar portions have been designated with similar reference numerals.
[0059] In the present disclosure, when a component is said to be "connected," "coupled," or "connected" to another component, this may include not only a direct connection, but also an indirect connection in which another component exists in between. Furthermore, when a component is said to "include" or "have" another component, unless otherwise specifically stated, this does not exclude the other component, but rather implies that the other component may be included.
[0060] In this disclosure, terms such as first, second, etc. are used solely to distinguish one component from another, and do not limit the order or importance of components unless specifically stated otherwise. Accordingly, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.
[0061] In this disclosure, distinct components are used to clearly illustrate their respective characteristics, and do not necessarily imply that the components are separated. That is, multiple components may be integrated into a single hardware or software unit, or a single component may be distributed into multiple hardware or software units. Therefore, even if not specifically mentioned, such integrated or distributed embodiments are also included within the scope of this disclosure.
[0062] In the present disclosure, the components described in various embodiments are not necessarily essential components, and some may be optional components. Therefore, embodiments comprising a subset of the components described in one embodiment are also within the scope of the present disclosure. Furthermore, embodiments including other components in addition to the components described in various embodiments are also within the scope of the present disclosure.
[0063] The present disclosure relates to encoding and decoding of images, and terms used in the present disclosure may have their usual meanings commonly used in the technical field to which the present disclosure belongs, unless newly defined in the present disclosure.
[0064] The present disclosure may be applied to a method disclosed in the Versatile Video Coding (VVC) standard and / or the Video Coding for Machines (VCM) standard. In addition, the present disclosure may be applied to a method disclosed in the essential video coding (EVC) standard, the AOMedia Video 1 (AV1) standard, the 2nd generation of audio video coding standard (AVS2), or the next generation video / image coding standard (e.g., H.267 or H.268, etc.).
[0065] The present disclosure presents various embodiments related to video / image coding, and unless otherwise stated, the embodiments may be performed in combination with each other. In the present disclosure, "video" may mean a set of a series of images over time. An "image" may be information generated by artificial intelligence (AI). Input information used in the process of AI performing a series of tasks, information generated during the information processing process, and output information may be used as images. A "picture" generally refers to a unit representing one image at a specific time point, and a slice / tile is a coding unit that constitutes a part of a picture in encoding. A picture may be composed of one or more slices / tiles. In addition, a slice / tile may include one or more coding tree units (CTUs). The CTUs may be divided into one or more units. A tile is a rectangular area existing within a specific tile row and a specific tile column within a picture, and may be composed of multiple CTUs. A tile column can be defined as a rectangular region of CTUs, having a height equal to the height of the picture, and a width specified by a syntax element signaled from a bitstream portion such as a picture parameter set. A tile row can be defined as a rectangular region of CTUs, having a width equal to the width of the picture, and a height specified by a syntax element signaled from a bitstream portion such as a picture parameter set. A tile scan is a predetermined sequential ordering method of CTUs that partition a picture. Here, CTUs can be sequentially ordered within a tile according to a CTU raster scan, and tiles within a picture can be sequentially ordered according to a raster scan order of tiles in the picture.A slice may contain an integer number of complete tiles, or a contiguous integer number of complete CTU rows within a tile of a picture. A slice may be exclusively contained in a single NAL unit. A picture may consist of one or more tile groups. A tile group may contain one or more tiles. A brick may represent a rectangular region of CTU rows within a tile within a picture. A tile may contain one or more bricks. A brick may represent a rectangular region of CTU rows within a tile. A tile may be divided into multiple bricks, and each brick may contain one or more CTU rows belonging to the tile. A tile that is not divided into multiple bricks may also be treated as a brick.
[0066] In the present disclosure, "pixel" or "pel" may refer to the smallest unit that constitutes a picture (or image). Additionally, "sample" may be used as a term corresponding to a pixel. A sample may generally represent a pixel or a pixel value, and may represent only a pixel / pixel value of a luma component or only a pixel / pixel value of a chroma component.
[0067] In one embodiment, particularly when applied to VCM, a pixel / pixel value may represent independent information of each component or a pixel / pixel value of a component generated through combination, synthesis, or analysis when there is a picture composed of a set of components with different characteristics and meanings. For example, in an RGB input, only the pixel / pixel value of R may be represented, only the pixel / pixel value of G may be represented, or only the pixel / pixel value of B may be represented. For example, only the pixel / pixel value of the Luma component synthesized using the R, G, and B components may be represented. For example, only the pixel / pixel value of the image information extracted through analysis of the R, G, and B components may be represented.
[0068] In the present disclosure, a "unit" may represent a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to the region. One unit may include one luma block and two chroma (e.g., Cb, Cr) blocks. In some cases, the term "unit" may be used interchangeably with terms such as "sample array," "block," or "area." In general, an MxN block may include a set (or array) of samples (or sample array) or transform coefficients consisting of M columns and N rows. In one embodiment, particularly when applied to VCM, a unit may represent a basic unit containing information for performing a specific task.
[0069] In the present disclosure, the "current block" may mean one of the following: a "current coding block," a "current coding unit," a "block to be encoded," a "block to be decoded," or a "block to be processed." When prediction is performed, the "current block" may mean a "current prediction block" or a "block to be predicted." When transformation (inverse transformation) / quantization (inverse quantization) is performed, the "current block" may mean a "current transformation block" or a "block to be transformed." When filtering is performed, the "current block" may mean a "block to be filtered."
[0070] Additionally, in the present disclosure, "current block" may mean "luma block of the current block" unless explicitly described as a chroma block. "Chroma block of the current block" may be expressed by explicitly including explicit description of chroma block, such as "chroma block" or "current chroma block."
[0071] In this disclosure, " / " and "," can be interpreted as "and / or". For example, "A / B" and "A, B" can be interpreted as "A and / or B". Additionally, "A / B / C" and "A, B, C" can mean "at least one of A, B, and / or C."
[0072] In this disclosure, "or" may be interpreted as "and / or." For example, "A or B" may mean 1) "A" only, 2) "B" only, or 3) "A and B." Alternatively, "or" in this disclosure may mean "additionally or alternatively."
[0073] The present disclosure relates to VCM (Video / image coding for machines).
[0074] VCM (Virtual Compression Modeling) is a compression technology that encodes / decodes a portion of a source image / video or information obtained from the source image / video for machine vision purposes. In VCM, the target of encoding / decoding can be referred to as a feature. A feature can refer to information extracted from the source image / video based on the task purpose, requirements, surrounding environment, etc. Features may have a different information format than the source image / video, and accordingly, the compression method and representation format of the feature may also differ from those of the video source.
[0075] VCMs can be applied to a variety of applications. For example, in surveillance systems that recognize and track objects or people, VCMs can be used to store or transmit object recognition information. Furthermore, in intelligent transportation or smart traffic systems, VCMs can be used to transmit vehicle location information collected from GPS, sensing information collected from LIDAR and radar, and various vehicle control information to other vehicles or infrastructure. Furthermore, in the smart city field, VCMs can be used to perform individual tasks for interconnected sensor nodes or devices.
[0076] The present disclosure provides various embodiments related to feature / feature map coding. Unless otherwise specified, the embodiments of the present disclosure may be implemented individually or in combination of two or more.
[0077] VCM System Overview
[0078] FIG. 1 is a schematic diagram of a VCM system to which embodiments of the present disclosure can be applied.
[0079] Referring to FIG. 1, the VCM system may include an encoding device (10) and a decoding device (20).
[0080] The encoding device (10) can compress / encode features / feature maps extracted from source images / videos to generate a bitstream, and transmit the generated bitstream to a decoding device (20) via a storage medium or a network. The encoding device (10) may also be referred to as a feature encoding device. In the VCM system, features / feature maps can be generated in each hidden layer of a neural network. The size and number of channels of the generated feature maps can vary depending on the type of neural network or the location of the hidden layer. In the present disclosure, a feature map may be referred to as a feature set, and a feature or feature map may be referred to as 'feature information'.
[0081] The encoding device (10) may include a feature acquisition unit (11), an encoding unit (12), and a transmission unit (13).
[0082] The feature acquisition unit (11) can acquire features / feature maps for source images / videos. In some embodiments, the feature acquisition unit (11) can acquire features / feature maps from an external device, for example, a feature extraction network. In this case, the feature acquisition unit (11) performs a feature reception interface function. Alternatively, the feature acquisition unit (11) can acquire features / feature maps by executing a neural network (e.g., CNN, DNN, etc.) using the source images / videos as input. In this case, the feature acquisition unit (11) performs a feature extraction network function.
[0083] According to an embodiment, the encoding device (10) may further include a source image generation unit (not shown) for obtaining a source image / video, or may include one instead of the feature acquisition unit (11). The source image generation unit may be implemented as an image sensor, a camera module, or the like, and may obtain the source image / video through a process of capturing, synthesizing, or generating an image / video. In this case, the generated source image / video may be transmitted to a feature extraction network and used as input data for extracting a feature / feature map.
[0084] The encoding unit (12) can encode the feature / feature map acquired by the feature acquisition unit (11). The encoding unit (12) can perform a series of procedures such as prediction, transformation, and quantization to increase encoding efficiency. The encoded data (encoded feature / feature map information) can be output in the form of a bitstream. A bitstream including the encoded feature / feature map information can be referred to as a VCM bitstream.
[0085] The transmission unit (13) can obtain feature / feature map information or data output in the form of a bitstream, and transmit it to a decoding device (20) or another external object through a digital storage medium or network in the form of a file or streaming. Here, the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit (13) may include elements for generating a media file having a predetermined file format or elements for transmitting data through a broadcast / communication network. The transmission unit (13) may be provided as a transmission device separate from the encoding unit (12), and in this case, the transmission device may include at least one processor for obtaining feature / feature map information or data output in the form of a bitstream, and a transmission unit for transmitting it in the form of a file or streaming.
[0086] The decoding device (20) can obtain feature / feature map information from the encoding device (10) and restore the feature / feature map based on the obtained information.
[0087] The decryption device (20) may include a receiving unit (21) and a decryption unit (22).
[0088] The receiving unit (21) can receive a bitstream from the encoding device (10), obtain feature / feature map information from the received bitstream, and transmit it to the decoding unit (22).
[0089] The decoding unit (22) can decode the feature / feature map based on the acquired feature / feature map information. The decoding unit (22) can perform a series of procedures, such as inverse quantization, inverse transformation, and prediction, corresponding to the operation of the encoding unit (14), to increase decoding efficiency.
[0090] According to an embodiment, the decryption device (20) may further include a task analysis / rendering unit (23).
[0091] The task analysis / rendering unit (23) can perform task analysis based on the decrypted feature / feature map. In addition, the task analysis / rendering unit (23) can render the decrypted feature / feature map into a form suitable for task execution. Various machine-oriented tasks can be performed based on the task analysis results and the rendered feature / feature map.
[0092] Above, the VCM system can encode / decode features extracted from source images / videos according to user and / or machine requests, task objectives, and surrounding environments, and perform various machine-directed tasks based on the decoded features. The VCM system can also be implemented by extending / redesigning a video / image coding system, and can perform various encoding / decoding methods defined in the VCM standard.
[0093] VCM pipeline
[0094] FIG. 2 is a schematic diagram illustrating a VCM pipeline structure to which embodiments of the present disclosure can be applied.
[0095] Referring to FIG. 2, the VCM pipeline (200) may include a first pipeline (210) for encoding / decoding images / videos and a second pipeline (220) for encoding / decoding features / feature maps. In the present disclosure, the first pipeline (210) may be referred to as a video codec pipeline, and the second pipeline (220) may be referred to as a feature codec pipeline.
[0096] The first pipeline (210) may include a first stage (211) for encoding input images / videos and a second stage (212) for decoding the encoded images / videos to generate restored images / videos. The restored images / videos may be used for human viewing, i.e., for human vision.
[0097] The second pipeline (220) may include a third stage (221) for extracting features / feature maps from input images / videos, a fourth stage (222) for encoding the extracted features / feature maps, and a fifth stage (223) for decoding the encoded features / feature maps to generate restored features / feature maps. The restored features / feature maps may be used for machine (vision) tasks. Here, the machine (vision) task may refer to a task in which images / videos are consumed by a machine. The machine (vision) task may be applied to service scenarios such as, for example, surveillance, intelligent transportation, smart city, intelligent industry, and intelligent content. In some embodiments, the restored features / feature maps may also be used for human vision.
[0098] In some embodiments, the encoded feature / feature map in the fourth stage (222) may be transferred to the first stage (221) and used to encode an image / video. In this case, an additional bitstream may be generated based on the encoded feature / feature map, and the generated additional bitstream may be transferred to the second stage (222) and used to decode the image / video.
[0099] In some embodiments, the decrypted feature / feature map in the fifth stage (223) may be transferred to the second stage (222) and used to decrypt an image / video.
[0100] Although FIG. 2 illustrates a case where the VCM pipeline (200) includes a first pipeline (210) and a second pipeline (220), this is merely exemplary and embodiments of the present disclosure are not limited thereto. For example, the VCM pipeline (200) may include only the second pipeline (220), or the second pipeline (220) may be extended into multiple feature codec pipelines.
[0101] Meanwhile, in the first pipeline (210), the first stage (211) may be performed by an image / video encoder, and the second stage (212) may be performed by an image / video decoder. In addition, in the second pipeline (220), the third stage (221) may be performed by a VCM encoder (or a feature / feature map encoder), and the fourth stage (222) may be performed by a VCM decoder (or a feature / feature map decoder). The encoder / decoder structure will be described in detail below.
[0102] Encoder
[0103] FIG. 3 is a schematic diagram of an image / video encoder to which embodiments of the present disclosure can be applied.
[0104] Referring to FIG. 3, the image / video encoder (300) may include an image partitioner (310), a prediction unit (predictor) 320, a residual processor (residual processor) 330, an entropy encoder (entropy encoder) 340, an adder (adder) 350, a filter (filter) 360, and a memory (memory) 370. The prediction unit (320) may include an inter prediction unit (321) and an intra prediction unit (322). The residual processor (330) may include a transformer (transformer) 332, a quantizer (quantizer) 333, a dequantizer (dequantizer) 334, and an inverse transformer (inverse transformer) 335. The residual processing unit (330) may further include a subtractor (331). The addition unit (350) may be referred to as a reconstructor or a recontructed block generator. The image segmentation unit (310), the prediction unit (320), the residual processing unit (330), the entropy encoding unit (340), the addition unit (350), and the filtering unit (360) described above may be configured by one or more hardware components (e.g., an encoder chipset or a processor) depending on the embodiment. In addition, the memory (370) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component described above may further include the memory (370) as an internal / external component.
[0105] The image segmentation unit (310) can segment an input image (or picture, frame) input to the image / video encoder (300) into one or more processing units. For example, the processing unit may be referred to as a coding unit (unit). The coding unit may be recursively segmented from a coding tree unit (CTU) or a largest coding unit (LCU, L unit) according to a Quad-tree binary-tree ternary-tree (QTBTTT) structure. For example, one coding unit may be segmented into a plurality of coding units of deeper depth based on a quad-tree structure, a binary tree structure, and / or a ternary structure. In this case, for example, the quad-tree structure may be applied first, and the binary tree structure and / or the ternary structure may be applied later. Alternatively, the binary tree structure may be applied first. The image / video coding procedure according to the present disclosure may be performed based on a final coding unit that is no longer divided. In this case, the largest coding unit may be used as the final coding unit based on coding efficiency according to image characteristics, etc., or, if necessary, the coding unit may be recursively divided into coding units of lower depths, and the coding unit with the optimal size may be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration, which will be described later. As another example, the processing unit may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may each be divided or partitioned from the final coding unit described above.The prediction unit may be a unit of sample prediction, and the transform unit may be a unit for deriving a transform coefficient and / or a unit for deriving a residual signal from a transform coefficient.
[0106] The term "unit" may be used interchangeably with terms such as "block" or "area" in some cases. In general, an MxN block can represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luma component or only the pixel / pixel value of the chroma component. A sample can be used as a term corresponding to a pixel or a pel.
[0107] The image / video encoder (300) can generate a residual signal (residual block, residual sample array) by subtracting a prediction signal (predicted block, prediction sample array) output from an inter-prediction unit (321) or an intra-prediction unit (322) from an input image signal (original block, original sample array), and the generated residual signal is transmitted to a conversion unit (332). In this case, as illustrated, a unit that subtracts a prediction signal (prediction block, prediction sample array) from an input image signal (original block, original sample array) within the image / video encoder (300) may be referred to as a subtraction unit (331). The prediction unit can perform prediction on a block to be processed (hereinafter, referred to as a current block) and generate a predicted block including prediction samples for the current block. The prediction unit can determine whether intra-prediction or inter-prediction is applied on a current block or unit basis. The prediction unit can generate various information regarding prediction, such as prediction mode information, and transmit it to the entropy encoding unit (340). The information regarding prediction can be encoded in the entropy encoding unit (340) and output in the form of a bitstream.
[0108] The intra prediction unit (322) can predict the current block by referring to samples within the current picture. At this time, the referenced samples may be located in the neighborhood of the current block or may be located away from it depending on the prediction mode. In intra prediction, the prediction modes may include multiple non-directional modes and multiple directional modes. The non-directional modes may include, for example, a DC mode and a planar mode. The directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes depending on the degree of detail in the prediction direction. However, this is only an example, and a greater or lesser number of directional prediction modes may be used depending on the settings. The intra prediction unit (322) may also determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.
[0109] The inter prediction unit (321) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between the neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on the inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring block can include a spatial neighboring block existing in the current picture and a temporal neighboring block existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. Temporal neighboring blocks may be referred to as collocated reference blocks, collocated units (colunits), etc., and a reference picture including temporal neighboring blocks may be referred to as a collocated picture (colPic). For example, the inter prediction unit (321) may construct a motion information candidate list based on neighboring blocks, and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction may be performed based on various prediction modes, and for example, in the case of skip mode and merge mode, the inter prediction unit (321) may use the motion information of neighboring blocks as the motion information of the current block. In the case of skip mode, unlike the merge mode, a residual signal may not be transmitted.In the motion vector prediction (MVP) mode, the motion vector of the surrounding blocks is used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.
[0110] The prediction unit (320) can generate a prediction signal based on various prediction methods. For example, the prediction unit can apply intra prediction or inter prediction for prediction of a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). In addition, the prediction unit can be based on an intra block copy (IBC) prediction mode or a palette mode for prediction of a block. The IBC prediction mode or palette mode can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in the present disclosure. The palette mode can be viewed as an example of intra coding or intra prediction. When palette mode is applied, sample values within a picture can be signaled based on information about the palette table and palette index.
[0111] The prediction signal generated by the prediction unit (320) can be used to generate a reconstructed signal or a residual signal. The transformation unit (332) can apply a transformation technique to the residual signal to generate transform coefficients. For example, the transformation technique can include at least one of a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), a Karhunen-Loeve Transform (KLT), a Graph-Based Transform (GBT), or a Conditionally Non-linear Transform (CNT). Here, GBT refers to a transformation obtained from a graph when the relationship information between pixels is expressed as a graph. CNT refers to a transformation obtained based on generating a prediction signal using all previously reconstructed pixels. In addition, the transformation process can be applied to a pixel block having a square equal size, or can be applied to a block of a non-square variable size.
[0112] The quantization unit (333) quantizes the transform coefficients and transmits them to the entropy encoding unit (340), and the entropy encoding unit (340) can encode the quantized signal (information about the quantized transform coefficients) and output it as a bitstream. The information about the quantized transform coefficients may be called residual information. The quantization unit (333) can rearrange the quantized transform coefficients in the form of a block into the form of a one-dimensional vector based on the coefficient scan order, and can also generate information about the quantized transform coefficients based on the quantized transform coefficients in the form of a one-dimensional vector. The entropy encoding unit (340) can perform various encoding methods, such as, for example, exponential Golomb, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), etc. The entropy encoding unit (340) may encode, together or separately, information necessary for image / video restoration (e.g., values of syntax elements, etc.) in addition to quantized transform coefficients. The encoded information (e.g., encoded image / video information) may be transmitted or stored in the form of a bitstream in units of NAL (network abstraction layer) units. The image / video information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the image / video information may further include general constraint information. In addition, the image / video information may further include a method of generating and using encoded information, a purpose, etc. In the present disclosure, information and / or syntax elements transmitted / signaled from an image / video encoder to an image / video decoder may be included in the image / video information.Image / video information can be encoded through the encoding process described above and included in a bitstream. The bitstream can be transmitted through a network or stored in a digital storage medium. Here, the network can include a broadcasting network and / or a communication network, and the digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmission unit (not shown) that transmits a signal output from an entropy encoding unit (340) and / or a storage unit (not shown) that stores the signal can be configured as an internal / external element of the image / video encoder (300), or the transmission unit can be included in the entropy encoding unit (340).
[0113] The quantized transform coefficients output from the quantization unit (333) can be used to generate a prediction signal. For example, by applying inverse quantization and inverse transformation to the quantized transform coefficients through the inverse quantization unit (334) and the inverse transform unit (335), a residual signal (residual block or residual samples) can be reconstructed. The addition unit (350) can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from the inter prediction unit (321) or the intra prediction unit (322). When there is no residual for the block to be processed, such as when skip mode is applied, the predicted block can be used as a reconstructed block. The addition unit (350) may be called a reconstructor or a reconstructed block generation unit. The generated restoration signal can be used for intra prediction of the next processing target block within the current picture, and can also be used for inter prediction of the next picture after filtering as described below.
[0114] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture encoding and / or restoration process.
[0115] The filtering unit (360) can improve subjective / objective picture quality by applying filtering to the restoration signal. For example, the filtering unit (360) can apply various filtering methods to the restoration picture to generate a modified restoration picture and store the modified restoration picture in the memory (370), specifically, in the DPB of the memory (370). The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filtering unit (360) can generate various information regarding filtering and transmit it to the entropy encoding unit (340). The information regarding filtering can be encoded by the entropy encoding unit (340) and output in the form of a bitstream.
[0116] The modified restored picture transmitted to the memory (370) can be used as a reference picture in the inter prediction unit (321). Through this, prediction mismatch in the encoder and decoder stages can be avoided, and encoding efficiency can also be improved.
[0117] The DPB of the memory (370) can store the modified restored picture to be used as a reference picture in the inter prediction unit (321). The memory (370) can store motion information of a block from which motion information in the current picture is derived (or encoded) and / or motion information of blocks in a picture that has already been restored. The stored motion information can be transferred to the inter prediction unit (321) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (370) can store restored samples of restored blocks in the current picture and can transfer the stored restored samples to the intra prediction unit (322).
[0118] Meanwhile, the VCM encoder (or feature / feature map encoder) may have a structure basically identical / similar to the image / video encoder (300) described with reference to FIG. 3 in that it performs a series of procedures such as prediction, transformation, and quantization to encode a feature / feature map. However, the VCM encoder differs from the image / video encoder (300) in that it targets a feature / feature map for encoding, and accordingly, the name of each unit (or component) (e.g., image segmentation unit (310), etc.) and its specific operation contents may be different from the image / video encoder (300). The specific operation contents of the VCM encoder will be described in detail later.
[0119] Decoder
[0120] FIG. 4 is a schematic diagram of an image / video decoder to which embodiments of the present disclosure can be applied.
[0121] Referring to FIG. 4, the image / video decoder (400) may include an entropy decoder (410), a residual processor (420), a predictor (430), an adder (440), a filter (450), and a memory (460). The predictor (430) may include an inter-prediction unit (431) and an intra-prediction unit (432). The residual processor (420) may include a dequantizer (421) and an inverse transformer (422). The entropy decoding unit (410), residual processing unit (420), prediction unit (430), addition unit (440), and filtering unit (450) described above may be configured by a single hardware component (e.g., a decoder chipset or processor) depending on the embodiment. In addition, the memory (460) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (460) as an internal / external component.
[0122] When a bitstream including video / image information is input, the video / video decoder (400) can restore the video / image in accordance with the process in which the video / image information is processed in the video / video encoder (300) of FIG. 3. For example, the video / video decoder (400) can derive units / blocks based on block division-related information obtained from the bitstream. The video / video decoder (400) can perform decoding using a processing unit applied in the video / video encoder. Accordingly, the processing unit of decoding may be, for example, a coding unit, and the coding unit may be divided from a coding tree unit or a maximum coding unit according to a quad tree structure, a binary tree structure, and / or a ternary tree structure. One or more transform units may be derived from the coding unit. Then, the restored video signal decoded and output by the video / video decoder (400) can be played back through a playback device.
[0123] The image / video decoder (400) can receive a signal output from the encoder of FIG. 3 in the form of a bitstream, and the received signal can be decoded through the entropy decoding unit (410). For example, the entropy decoding unit (410) can parse the bitstream to derive information (e.g., image / video information) necessary for image restoration (or picture restoration). The image / video information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the image / video information may further include general constraint information. In addition, the image / video information may include a method of generating and using decoded information, a purpose, etc. The image / video decoder (400) can decode a picture further based on information on the parameter set and / or the general constraint information. Signaling / received information and / or syntax elements can be decoded through a decoding procedure and obtained from the bitstream. For example, the entropy decoding unit (410) can decode information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output the values of syntax elements necessary for image restoration and the quantized values of transform coefficients for the residual. More specifically, the CABAC entropy decoding method receives a bin corresponding to each syntax element in the bitstream, determines a context model using information of the syntax element to be decoded and decoding information of the surrounding and decoding target blocks or information of symbols / bins decoded in the previous step, and predicts the occurrence probability of the bin according to the determined context model to perform arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element.At this time, the CABAC entropy decoding method can update the context model using the information of the decoded symbol / bin for the context model of the next symbol / bin after determining the context model. Information regarding prediction among the information decoded by the entropy decoding unit (410) is provided to the prediction unit (inter prediction unit (432) and intra prediction unit (431)), and residual values on which entropy decoding is performed by the entropy decoding unit (410), i.e., quantized transform coefficients and related parameter information, can be input to the residual processing unit (420). The residual processing unit (420) can derive a residual signal (residual block, residual samples, residual sample array). In addition, information regarding filtering among the information decoded by the entropy decoding unit (410) can be provided to the filtering unit (450). Meanwhile, a receiving unit (not shown) that receives a signal output from an image / video encoder may be further configured as an internal / external element of the image / video decoder (400), or the receiving unit may be a component of an entropy decoding unit (410). Meanwhile, the image / video decoder according to the present disclosure may be called an image / video decoding device, and the image / video decoder may be divided into an information decoder (image / video information decoder) and a sample decoder (image / video sample decoder). In this case, the information decoder may include an entropy decoding unit (410), and the sample decoder may include at least one of an inverse quantization unit (321), an inverse transformation unit (322), an addition unit (440), a filtering unit (450), a memory (460), an inter prediction unit (432), and an intra prediction unit (431).
[0124] The inverse quantization unit (421) can inverse quantize the quantized transform coefficients and output the transform coefficients. The inverse quantization unit (421) can rearrange the quantized transform coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scanning order performed in the image / video encoder. The inverse quantization unit (321) can perform inverse quantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain transform coefficients.
[0125] In the inverse transform unit (422), the transform coefficients are inversely transformed to obtain a residual signal (residual block, residual sample array).
[0126] The prediction unit (430) can perform a prediction on the current block and generate a predicted block containing prediction samples for the current block. Based on the prediction information output from the entropy decoding unit (410), the prediction unit can determine whether intra-prediction or inter-prediction is applied to the current block, and can determine a specific intra / inter-prediction mode.
[0127] The prediction unit (420) can generate a prediction signal based on various prediction methods. For example, the prediction unit can apply intra prediction or inter prediction for prediction of a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). In addition, the prediction unit can be based on an intra block copy (IBC) prediction mode or a palette mode for prediction of a block. The IBC prediction mode or palette mode can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in this document. The palette mode can be viewed as an example of intra coding or intra prediction. When palette mode is applied, information about the palette table and palette index can be signaled by including it in the image / video information.
[0128] The intra prediction unit (431) can predict the current block by referring to samples within the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from it, depending on the prediction mode. In intra prediction, the prediction modes may include multiple non-directional modes and multiple directional modes. The intra prediction unit (431) can also determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.
[0129] The inter prediction unit (432) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between the neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on the inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring blocks can include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit (432) can construct a motion information candidate list based on the neighboring blocks, and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter prediction can be performed based on various prediction modes, and information about the prediction can include information indicating the mode of inter prediction for the current block.
[0130] The addition unit (440) can generate a restoration signal (restored picture, restoration block, restoration sample array) by adding the acquired residual signal to the prediction signal (predicted block, prediction sample array) output from the prediction unit (including the inter-prediction unit (432) and / or intra-prediction unit (431)). When there is no residual for the block to be processed, such as when skip mode is applied, the predicted block can be used as the restoration block.
[0131] The addition unit (440) may be referred to as a restoration unit or restoration block generation unit. The generated restoration signal may be used for intra prediction of the next processing target block within the current picture, may be output after filtering as described below, or may be used for inter prediction of the next picture.
[0132] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture decoding process.
[0133] The filtering unit (450) can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit (450) can apply various filtering methods to the restored picture to generate a modified restored picture, and transmit the modified restored picture to the memory (460), specifically, the DPB of the memory (460). The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.
[0134] The (modified) reconstructed picture stored in the DPB of the memory (460) can be used as a reference picture in the inter prediction unit (432). The memory (460) can store motion information of a block from which motion information is derived (or decoded) within the current picture and / or motion information of blocks within a picture that has already been reconstructed. The stored motion information can be transmitted to the inter prediction unit (432) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (460) can store reconstructed samples of reconstructed blocks within the current picture and transmit them to the intra prediction unit (431).
[0135] Meanwhile, the VCM decoder (or feature / feature map decoder) may have a structure basically identical / similar to the image / video decoder (400) described above with reference to FIG. 4 in that it performs a series of procedures such as prediction, inverse transformation, and inverse quantization to decode a feature / feature map. However, the VCM decoder is different from the image / video decoder (400) in that it targets a feature / feature map for decoding, and accordingly, the name of each unit (or component) (e.g., DPB, etc.) and its specific operation contents may be different from the image / video decoder (400). The operation of the VCM decoder may correspond to the operation of the VCM encoder, and the specific operation contents will be described in detail later.
[0136] Feature / feature map encoding procedure
[0137] FIG. 5 is a flowchart schematically illustrating a feature / feature map encoding procedure to which embodiments of the present disclosure may be applied.
[0138] Referring to FIG. 5, the feature / feature map encoding procedure may include a prediction procedure (S510), a residual processing procedure (S520), and an information encoding procedure (S530).
[0139] The prediction procedure (S510) can be performed by the prediction unit (320) described above with reference to FIG. 3.
[0140] Specifically, the intra prediction unit (322) can predict the current block (i.e., the set of feature elements to be currently encoded) by referring to feature elements within the current feature / feature map. Intra prediction can be performed based on the spatial similarity of feature elements constituting the feature / feature map. For example, feature elements included in the same region of interest (RoI) within an image / video can be estimated to have similar data distribution characteristics. Accordingly, the intra prediction unit (322) can predict the current block by referring to the restored feature elements within the region of interest including the current block. At this time, the referenced feature elements may be positioned adjacent to the current block or spaced apart from the current block depending on the prediction mode. Intra prediction modes for feature / feature map encoding can include a plurality of non-directional prediction modes and a plurality of directional prediction modes. Non-directional prediction modes may include prediction modes corresponding to, for example, the DC mode and the planar mode of an image / video encoding procedure. Furthermore, directional modes may include prediction modes corresponding to, for example, the 33 directional modes or the 65 directional modes of an image / video encoding procedure. However, this is merely an example, and the types and number of intra-prediction modes may be set / changed in various ways depending on the embodiment.
[0141] The inter prediction unit (321) can predict the current block based on a reference block (i.e., a set of referenced feature elements) specified by motion information on a reference feature / feature map. Inter prediction can be performed based on the temporal similarity of feature elements constituting the feature / feature map. For example, temporally consecutive features can have similar data distribution characteristics. Therefore, the inter prediction unit (321) can predict the current block by referring to the restored feature elements of features that are temporally adjacent to the current feature. At this time, the motion information for specifying the referenced feature elements can include a motion vector and a reference feature / feature map index. The motion information can further include information regarding the inter prediction direction (e.g., L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, neighboring blocks may include spatial neighboring blocks existing in the current feature / feature map and temporal neighboring blocks existing in the reference feature / feature map. The reference feature / feature map including the reference block and the reference feature / feature map including the temporal neighboring blocks may be the same or different. The temporal neighboring blocks may be referred to as collocated reference blocks, and the reference feature / feature map including the temporal neighboring blocks may be referred to as collocated feature / feature maps. The inter prediction unit (321) may construct a motion information candidate list based on the neighboring blocks and generate information indicating which candidate is used to derive the motion vector and / or reference feature / feature map index of the current block.Inter prediction can be performed based on various prediction modes. For example, in the case of skip mode and merge mode, the inter prediction unit (321) can use the motion information of the surrounding blocks as the motion information of the current block. In the case of skip mode, unlike the merge mode, a residual signal may not be transmitted. In the case of motion vector prediction (MVP) mode, the motion vector of the surrounding blocks can be used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference. In addition to the intra prediction and inter prediction described above, the prediction unit (320) can generate a prediction signal based on various prediction methods.
[0142] The prediction signal generated by the prediction unit (320) can be used to generate a residual signal (residual block, residual feature elements) (S520). The residual processing procedure (S520) can be performed by the residual processing unit (330) described above with reference to FIG. 3. In addition, (quantized) transform coefficients can be generated through a transformation and / or quantization procedure for the residual signal, and the entropy encoding unit (340) can encode information about the (quantized) transform coefficients as residual information in the bitstream (S530). In addition to the residual information, the entropy encoding unit (340) can encode information necessary for feature / feature map restoration, such as prediction information (e.g., prediction mode information, motion information, etc.), in the bitstream.
[0143] Meanwhile, the feature / feature map encoding procedure may further include a procedure (S530) for encoding information for feature / feature map restoration (e.g., prediction information, residual information, partitioning information, etc.) and outputting it in the form of a bitstream, as well as a procedure for generating a restored feature / feature map for the current feature / feature map and a procedure (optional) for applying in-loop filtering to the restored feature / feature map.
[0144] The VCM encoder can derive (modified) residual feature(s) from the quantized transform coefficient(s) through inverse quantization and inverse transformation, and can generate a restored feature / feature map based on the predicted feature(s) and the (modified) residual feature(s) which are the outputs of step S510. The restored feature / feature map generated in this way can be identical to the restored feature / feature map generated by the VCM decoder. If an in-loop filtering procedure is performed on the restored feature / feature map, a modified restored feature / feature map can be generated through the in-loop filtering procedure on the restored feature / feature map. The modified restored feature / feature map can be stored in a decoded feature buffer (DFB) or memory, and can be used as a reference feature / feature map in a subsequent feature / feature map prediction procedure. In addition, (in-loop) filtering-related information (parameters) can be encoded and output in the form of a bitstream. Through the in-loop filtering procedure, noise that may occur during feature / feature map coding can be removed, and the performance of feature / feature map-based tasks can be improved. Furthermore, by performing the in-loop filtering procedure in both the encoder and decoder stages, the consistency of prediction results can be guaranteed, the reliability of feature / feature map coding can be improved, and the amount of data transmission for feature / feature map coding can be reduced.
[0145] Feature / feature map decoding procedure
[0146] FIG. 6 is a flowchart schematically illustrating a feature / feature map decoding procedure to which embodiments of the present disclosure may be applied.
[0147] Referring to FIG. 6, the feature / feature map decoding procedure may include an image / video information acquisition procedure (S610), a feature / feature map restoration procedure (S620 to S640), and an in-loop filtering procedure (S650) for the restored feature / feature map. The feature / feature map restoration procedure may be performed based on a prediction signal and a residual signal obtained through the inter / intra prediction (S620) and residual processing (S630), inverse quantization for quantized transform coefficients, and inverse transformation) processes described in the present disclosure. A modified restored feature / feature map may be generated through the in-loop filtering procedure for the restored feature / feature map, and the modified restored feature / feature map may be output as a decoded feature / feature map. The decoded feature / feature map is stored in the decoded feature buffer (DFB) or memory, and can be used as a reference feature / feature map in the inter prediction procedure during subsequent decoding of the feature / feature map. In some cases, the in-loop filtering procedure described above may be omitted. In this case, the reconstructed feature / feature map can be output as a decoded feature / feature map, and can be stored in the decoded feature buffer (DFB) or memory, and can be used as a reference feature / feature map in the inter prediction procedure during subsequent decoding of the feature / feature map.
[0148] Feature extraction methods and data distribution characteristics
[0149] Embodiments of the present disclosure propose a method for generating a prediction process and a related bitstream necessary for compressing an activation (feature) map generated in a hidden layer of a deep neural network.
[0150] Input data input into a deep neural network goes through the associative process of multiple hidden layers, and the computational results of each hidden layer are output as features / feature maps with various sizes and number of channels depending on the type of deep neural network being used and the location of the hidden layer within the deep neural network.
[0151] FIG. 7 is a diagram illustrating an example of a feature extraction and restoration method to which embodiments of the present disclosure can be applied.
[0152] Referring to FIG. 7, a feature extraction network (710) can extract an intermediate layer activation (feature) map of a deep neural network from a source image / video and output an extracted feature map. The feature extraction network (710) can be a set of consecutive hidden layers from the input of the deep neural network.
[0153] The encoding device (720) can compress the output feature map and output it in the form of a bitstream, and the decoding device (730) can restore the (compressed) feature map from the output bitstream. The encoding device (720) may correspond to the encoding unit (12) of FIG. 1, and the decoding device (720) may correspond to the decoding unit (22) of FIG. 1. The task network (740) can perform a task based on the restored feature map.
[0154] The number of channels in the feature map that is the compression target of VCM may vary depending on the network used for feature extraction and the extraction location, and may be greater than the number of channels in the input data.
[0155] FIG. 8 is a diagram illustrating an example of a method for segmenting an image to which embodiments of the present disclosure may be applied. As an example, the diagram illustrates CTUs, slices, and tiles within an image.
[0156] The video / image coding method according to this document can be performed based on the following partitioning structure. Procedures such as prediction, residual processing ((inverse) transformation, (inverse) quantization, etc.), syntax element coding, and filtering can be performed based on CTUs, CUs (and / or TUs, PUs) derived based on the partitioning structure of Fig. 8. The block partitioning procedure can be performed in an encoding device, and partitioning-related information can be encoded and transmitted to a decoding device in the form of a bitstream. The decoding device can derive a block partitioning structure of the current picture based on the partitioning-related information obtained from the bitstream, and perform a series of procedures for image decoding (e.g., prediction, residual processing, block / picture restoration, in-loop filtering, etc.) based on the block partitioning structure. The CU size can be the same as the TU size, and multiple TUs can exist within the CU area. Meanwhile, the CU size can generally represent the luma component (sample) CB size. TU size can generally represent the luma component (sample) TB size. The chroma component (sample) CB or TB size can be derived based on the luma component (sample) CB or TB size according to the component ratio according to the color format (chroma format, e.g., 4:4:4, 4:2:2, 4:2:0, etc.) of the picture / video, and conversion / inverse conversion can be performed in TU (TB) units.
[0157] In addition, in the coding of video / images according to this document, the image processing unit may have a hierarchical structure. One picture may be divided into one or more CUs, and one or more CUs may be grouped and distinguished into one or more tiles, bricks, slices, and / or tile groups. One slice may include one or more bricks. One brick may include one or more CTU rows within the tile. A slice may include an integer number of bricks of the picture. One tile group may include one or more tiles. One tile may include one or more CTUs. The CTUs may be divided into one or more CUs. A tile group may include an integer number of tiles according to a tile raster scan within the picture. A slice header may carry information / parameters applicable to the corresponding slice (blocks within the slice). A picture header may carry information / parameters applicable to the corresponding picture (or blocks within the picture). When the encoding / decoding device has a multi-core processor, the encoding / decoding procedure for the tile, slice, brick, and / or tile group can be processed in parallel. In this document, the terms slice and tile group may be used interchangeably. That is, the tile group header may be called a slice header. Here, the slice may have one of the following types: I slice, P slice, and B slice.
[0158] In an encoding device, the tile / tile group, brick, slice, maximum and minimum coding unit sizes may be determined based on the characteristics of the video image (e.g., resolution) or in consideration of coding efficiency or parallel processing, and information about this or information that can derive this may be included in the bitstream.
[0159] A decoding device can obtain information indicating whether the tile / tile group, brick, sliat, or CTU within a tile of the current picture is divided into multiple coding units. Efficiency can be improved by ensuring that this information is obtained (transmitted) only under certain conditions.
[0160] The above slice header (slice header syntax) may include information / parameters that can be commonly applied to the slices. The APS (APS syntax) or PPS (PPS syntax) may include information / parameters that can be commonly applied to one or more pictures. The SPS (SPS syntax) may include information / parameters that can be commonly applied to one or more sequences. The VPS (VPS syntax) may include information / parameters that can be commonly applied to multiple layers. The DPS (DPS syntax) may include information / parameters that can be commonly applied to the entire video. The DPS may include information / parameters related to the concatenation of CVS (coded video sequence).
[0161] In this document, the upper level syntax may include at least one of the APS syntax, PPS syntax, SPS syntax, VPS syntax, DPS syntax, picture header syntax, and slice header syntax.
[0162] Additionally, for example, information regarding the division and configuration of the above tiles / tile groups / bricks / slices, etc., can be configured in the encoding section through the above-mentioned upper-level syntax and transmitted to the decoding device in the form of a bitstream.
[0163] FIG. 9 is a diagram showing an example of a VCM video encoding / decoding system, and FIG. 10 is a diagram showing another example of a VCM video encoding / decoding system. As an example, FIG. 9 and FIG. 10 can be used to expand / redesign a video coding system (e.g., FIG. 1) so that only a portion of a video source can be used or necessary portions / information can be acquired from the video source according to the request and purpose of a user or machine, and the surrounding environment. In other words, FIG. 9 and FIG. 10 can be related to VCM (Video coding for machines).
[0164] VCM (Video coding for machines) can mean encoding / decoding the entire video, part of the video, and / or necessary information (features) from the video based on the request, purpose, or surrounding environment of the user and / or machine. In VCM, the target of encoding can be the video itself, or it can be information referred to as features extracted from the video based on the request, purpose, or surrounding environment of the user and / or machine, or it can mean a set of a series of information over time.
[0165] Referring to FIG. 9, the VCM system may include a video encoder and a video decoder for VCM. A source device (FIG. 1) may transmit encoded video information to a receiving device via a storage medium or a network. The users of the devices may be humans and / or machines, respectively.
[0166] Referring to FIG. 10, the VCM system may include a video encoder and a video decoder for VCM. A source device may transmit encoded video information (features) to a receiving device via a storage medium or a network. The users of the devices may be humans and / or machines, respectively.
[0167] For example, the process of extracting information, or features, from an image can be called feature extraction. Feature extraction can be performed on both video / image capture devices and / or video / image generation devices. Features can be information extracted / processed from an image based on the user and / or machine's request, purpose, or surrounding environment, and can refer to a collection of information over time.
[0168] A video encoder for VCM can perform a series of procedures, such as prediction, transformation, and quantization, to compress and code the entire image and / or a portion of the image and / or features efficiently. The encoded data can be output in the form of a bitstream.
[0169] A video decoder for VCM can decode video / images by performing a series of procedures such as inverse quantization, inverse transformation, and prediction corresponding to the operation of an encoding device, i.e., an encoder.
[0170] Decoded images and / or features can be rendered. They can also be used to perform tasks for users or machines. Examples of tasks include AI and computer vision tasks such as facial recognition, activity recognition, and lane detection.
[0171] The present disclosure presents various embodiments related to the acquisition and coding of all and / or portions of an image for VCM. Unless otherwise stated, the embodiments may be performed in combination with each other. The methods / embodiments of the present disclosure can be applied to methods disclosed in the VCM (Video Coding for Machines) standard.
[0172] VCM hierarchy
[0173] VCM can be based on a hierarchical structure consisting of a feature coding layer, a neural network (feature) abstraction layer, and a feature extraction layer. Fig. 11 is a diagram illustrating an example of a VCM hierarchical structure, and Fig. 12 is a diagram illustrating an example of a VCM bitstream composed of encoded abstract features and NNAL information.
[0174] As an example, referring to FIG. 11, the VCM hierarchy may be composed of a feature extraction layer (1110), a neural network (feature) abstraction layer (1120), and a feature coding layer (1130).
[0175] The feature extraction layer (1110) refers to a layer for extracting features from an input source, and may also include the results of the extraction. The feature coding layer (1130) refers to a layer for compressing the extracted features, and may also include the results of the compression.
[0176] The neural network abstraction layer (1120) can abstract information generated by the feature extraction layer (1110) (e.g., information about extracted features / feature maps) and pass it to the feature coding layer (1130). The neural network abstraction layer (1120) can hide the internals of the feature extraction layer (1110) through information abstraction and provide a consistent feature interface function. Accordingly, even if the compression target changes due to a change in the tool (e.g., CNN, DNN, etc.), the feature coding layer (1130) can perform a consistent feature coding procedure. In the present disclosure, the neural network abstraction layer (NNAL) may also be referred to as a feature abstraction layer.
[0177] The interface between the feature extraction layer (1110) and the neural network abstraction layer (1120) and the interface between the feature coding layer (1130) and the neural network abstraction layer (1120) can be defined in advance, and the operation in the neural network abstraction layer (1120) can be set to be changed later.
[0178] Referring to FIG. 12, a bitstream configured as illustrated may be referred to as an NNAL (Neural Network Abstraction Layer) unit. An NNAL unit may be an independent feature restoration unit. The input features for a single NNAL unit may be extracted from the same layer within a neural network. Accordingly, the input features for a single NNAL unit may be forced to have the same characteristics. For example, the same feature extraction method may be applied to the input features for a single NNAL unit.
[0179] An NNAL unit may include an NNAL unit header and an NNAL unit payload. The NNAL unit header may include all information necessary to utilize the encoded feature for a task. The NNAL unit payload may include abstracted feature information. The NNAL unit payload may include a group header and group data. The group header may include configuration information of feature group data, such as information about the temporal order, number, and common properties of feature channels constituting the feature group. A feature channel may refer to an encoded feature unit. The group data may include a plurality of feature channels and coding instructions, and each feature channel may include type information, prediction information, side information, and residual information. In this case, the type information may indicate an encoding method, and the prediction information may indicate a prediction method. In addition, the side information may indicate additional information necessary for decoding (e.g., entropy coding, quantization-related information, etc.), and the residual information may include information about encoded feature elements (i.e., a set of feature value information).
[0180] Meanwhile, with image processing technologies including VCM, the video, i.e., the target of the video reception (e.g., human or machine) or the purpose of reception (e.g., human recognition or machine processing) may change. In other words, as the target of the bitstream reception expands from users, i.e., humans, to machines, i.e., machines, various optimization methods may be applied to the video depending on each target of reception and the intention or purpose of the video. For example, an image optimization method includes all methods of adjusting the quality, performance, and efficiency that can be applied to the video, including pre / post processing that can be applied to the video, including the number of pixels in the video, bit rate, and color adjustment methods of the video, and for example, methods of distinguishing objects included in the video and increasing or decreasing the number of pixels for some objects may be included. In addition, image optimization may also be expressed in terms such as image adjustment, image adjustment, image change, and image correction. In other words, when describing an embodiment of the present disclosure below, the term image optimization may be replaced with terms such as image adjustment, image control, image change, and image modification.
[0181] Meanwhile, if the video reception target or the video reception purpose is changed, the video optimization method (e.g., video pre / post processing, etc.) for efficient video processing and improved processing efficiency may also need to be changed based on the purpose or the video reception target. However, the video optimization method that can be expressed in the bitstream and the properties and purposes of the coded picture (i.e., video, image, etc.) may not match the video reception purpose of the reception target, i.e., the video usage intention of the user or machine. This will be described in detail with reference to FIG. 13. FIG. 13 is a diagram for explaining an example in which a coded bitstream to which a coding optimization method according to an embodiment of the present disclosure is applied is used on the reception target side. First, as shown in FIG. 13, there may be an encoding component that encodes a video, i.e., an optimization video encoding component that determines whether to apply an optimization method, and a task component, i.e., an element that consumes the encoded video. For example, a task component may include a decoding component and may be a target for decoding encoded video. Meanwhile, when the task component receives a bitstream, it may be desirable to consider the characteristics of the task component that consumes the encoded video (e.g., the characteristics of the receiving target, etc.), or the video reception intention and usage purpose of the task component, for purposes of improving coding efficiency and performance. However, the encoding component and the element that consumes the encoded video (e.g., the decoding component) illustrated in FIG. 13 may be configured together or separately. That is, the encoding component and the task component may be configured independently, and each may be a different service provider.In this case, there may be a difference between the optimization method suitable for the task component and the optimization method applied by the encoding component during video coding, which may cause problems such as reduced coding performance or reduced efficiency. In addition, since the encoded video is used for multiple purposes, problems may arise due to differences in the optimization method suitable for some task components and the optimization method applied to the video. For example, in the example of FIG. 13, the encoded video may be used for various purposes such as machine tasks A, B, and / or human recognition (e.g., human viewing). In this case, the optimization method applied for encoding may be suitable for all of machine tasks A, B, and human recognition, but may not be. In this case, referring to FIG. 13, on the task component side, the decoded video may have some of the background removed, unlike the original. In this case, when the information of the input video is changed through optimization, the optimization method may not be suitable for some tasks that require the changed information, which may cause problems such as significantly degrading the quality of the restored image. Moreover, problems may arise when the intended use of the video by the recipient (e.g., user, machine, etc.) changes. For example, if the recipient changes or the intended use of the video by the recipient changes, pictures encoded using an image optimization method suitable for other existing task components may no longer be suitable for the new task component or new intended use of the video, which may degrade coding performance or reduce efficiency.
[0182] In order to solve the above-mentioned problem, the user, i.e., human, or the machine, i.e., the decoder side, must be able to identify whether the currently received bitstream is optimized, the optimization range, and the optimization method, so as to determine whether the received bitstream is suitable and request an optimization method that fits the purpose. The present disclosure relates to an optimization method and information related to optimization according to the purpose of an image during image processing that can solve the above-mentioned problem, and relates to a technology for signaling the information. More specifically, the present disclosure defines an optimization method that can be applied to an encoded picture, defines a method for expressing the optimization method applied to the encoded picture and a method for expressing the properties and purpose of the encoded picture, and describes signaling of information related to the optimization method. More specifically, the present disclosure may relate to the coding layer and structure described above, and in particular, to a coding optimization method applied according to whether a user, i.e., a human or a machine, recognizes, i.e., a receiving target of an image (or a purpose of the image, etc.), and a method for expressing suitable uses and properties of a coded bitstream. Hereinafter, a coding optimization method, a method for expressing suitable uses and properties of a coded bitstream, signaling of information about the coding optimization method, etc. will be described through various embodiments. According to the present disclosure, it may be determined whether an encoded video and a receiving target (e.g., a user (human) or a machine) use purpose of the video are consistent with the optimization method applied to the encoded picture, and the receiving target (e.g., a user or a machine, etc.) may select and request an encoded video that suits the purpose, thereby promoting improvements in system efficiency, quality, and performance of restored pictures.
[0183] Meanwhile, the names of syntax and NAL units used when describing the embodiments of the present disclosure in detail below are arbitrarily determined for clarity of explanation, and the present disclosure is not limited thereto.
[0184] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
[0185] First, a method for signaling information about an optimization method according to one embodiment of the present disclosure is described.
[0186] According to one embodiment of the present disclosure, information on an optimization method applied to an image may be signaled by extending and applying an existing NAL unit. As an example, information on an optimization method defined in the present embodiment (e.g., encoder_optimization_info, intended_purpose_info, etc.) may be added to and signaled in the existing NAL unit by extending and applying the existing NAL unit. As an example, information on the optimization method may be included in the existing NAL unit and signaled based on at least one of structures included in the existing NAL unit, such as decoding capability information (DCI), video usage information (VUI), sequence parameter set (SPS), picture parameter set (PPS), picture header, and / or slice header. Table 1 below shows a case where information on an optimization method is signaled in a picture parameter set, i.e., a PPS, according to one embodiment of the present disclosure.
[0187]
[0188] For example, information about the optimization method may include task information (e.g., task_info_flag), which is information about the task to be performed through the image, optimization information (e.g., encoder_optimization_info), which is information about the optimization method applied on the encoder side, and / or image purpose information (e.g., intended_purpose_info), which is information about the purpose of the image. For example, information about the optimization method (e.g., encoder_optimization_info and intended_purpose_info) may both be signaled, or only one of them may be defined and only one of them may be signaled. For example, information about the purpose of the image may be derived based on the optimization information, or optimization information may be derived based on the information about the purpose of the image. In other words, intended_purpose_info may be derived through encoder_optimization_info, or vice versa. For example, optimization information and / or image purpose information may be signaled based on task information (task_info_flag). For example, if the value of the task information is a specific value (e.g., 1), optimization information (e.g., encoder_optimization_info) or image purpose information (e.g., intended_purpose_info) may be defined and signaled, and if the value of the task information is a specific value (e.g., 0), optimization information (e.g., encoder_optimization_info) or image purpose information (e.g., intended_purpose_info) may not be defined and signaled.
[0189] Meanwhile, the transmission unit of the above encoder_optimization_info() and / or Intended_purpose_info is not limited to a picture, and can also be transmitted in a higher or lower level unit. In addition, in this case, it is possible to indicate whether the video optimization method is applied and continued for each unit at a higher or lower level than the current picture. For example, the unit may be a specific video section composed of multiple pictures, or a part (e.g., a slice, tile, coding unit, etc.) that constitutes one picture.
[0190] Meanwhile, as another example, a new NAL unit type may be defined, and information on an optimization method may be signaled within the NAL unit type. Tables 2 and 3 below illustrate examples of defining new NAL unit types, namely, Encoder Optimization Information (EOI) NAL unit and Intended Purpose Information (IPI) NAL unit types, respectively, according to one embodiment of the present disclosure.
[0191]
[0192]
[0193] As shown in Table 2 or Table 3 above, a new NAL unit may include information about an optimization method (e.g., encoder_optimizationf_info, intended_purpose_info). However, it is obvious that the index value (e.g., 26) for identifying the NAL unit type class (e.g., non-VCL) and the NAL unit type (e.g., nal_unit_type) described in Table 2 below is an example and may be changed, and this is also included in the present disclosure. Referring to Table 2, a new NAL unit type may be defined to express an optimization method and application scope, and the purpose and properties of an encoded picture.
[0194] Meanwhile, according to another example of the present disclosure, information on an optimization method (e.g., encoder_optimization_info, Intended_purpose_info, etc.) may be encoded by applying SEI (Supplemental enhancement information). Tables 4 and 5 below show examples of signaling of information on an SEI-based optimization method (e.g., encoder_optimization_info, Intended_purpose_info, etc.) according to one embodiment of the present disclosure.
[0195]
[0196]
[0197] As shown in Table 3 or Table 4 above, when the value of payloadType is a specific value (e.g., 205), it may indicate that an SEI payload indicating information about an optimization method (e.g., encoder_optimization_info, Intended_purpose_info) is encoded and signaled. Meanwhile, since Table 3 above is an example, it is obvious that the value for indicating that an SEI payload indicating information about an optimization method (e.g., encoder_optimization_info, Intended_purpose_info) is encoded and signaled, i.e., the value of payloadType, may be determined as any number, and this is also included in the present disclosure.
[0198] Below, optimization-related information that can be included in information about optimization methods (e.g., encoder_optimization_info, intended_purpose_info, etc.) is described. As an example, first, image optimization (application) information (or image optimization status information) indicating whether or not an image is optimized, image optimization (purpose) information indicating the intention of optimizing an image, i.e., the optimization purpose, and / or image optimization (method) information indicating the optimization method applied to an image are described. Tables 6 to 8 below show examples of the syntax of information about optimization methods.
[0199]
[0200]
[0201]
[0202] Table 6 above can show examples of how to express whether optimization is applied, the application method, and the purpose. Tables 6 and 7 can show examples of defining the optimization method and application scope according to the optimization purpose, respectively. Table 8 can show examples of defining the optimization method and application method by integrating the optimization purpose. Referring to Tables 6 to 8 above, optimization-related information can be signaled by being included in, for example, encoder_optimization_info. Since encoder_optimization_info is as described above, a duplicate description is omitted.
[0203] As an example, the optimization-related information may include optimization application information indicating whether optimization is applied. As an example, the optimization application information (e.g., the first syntax) may be expressed as encoder_optimization_flag, and if the value of the first syntax is a specific value (e.g., 0), it may indicate that optimization is not applied. If there is an optimization method that was previously applied (such as a block or picture), it may indicate that the optimization method is no longer applied, i.e., has ended. If there is no optimization method that was previously applied, it may indicate that optimization is not applied to the current block or picture. Meanwhile, if the value of the first syntax (e.g., encoder_optimization_flag) is a specific value (e.g., 1), it may indicate that an optimization method is applied, and the applied optimization method may be specified according to optimization-related information including whether machine-recognition-based optimization (e.g., optimization_for_machine, etc.), an optimization method (e.g., optimization_method, etc.), a human-recognition optimization method (e.g., human_optimization_method, etc.), a machine-recognition optimization method (e.g., machine_optimization_method, etc.), optimization persistence information (e.g., purpose_persistence_flag and / or optimization_persistence_flag, etc.), and / or optimization persistence cancellation information (e.g., optimization_persistence_cancel_flag, etc.), which may be defined and signaled after the first syntax. Each optimization-related information will be described in detail below.
[0204] For example, the optimization-related information may include optimization objective information indicating the purpose of image optimization. For example, the optimization objective information (e.g., the second syntax) may be obtained based on other optimization-related information (e.g., the first syntax, etc.). The optimization objective information may be expressed as, for example, optimization_for_machine or optimization_for_human. For example, if the optimization objective information is defined as optimization_for_machine and the value of the optimization objective information is a specific value (e.g., 1), this may indicate that the purpose of optimization is to perform a machine task. As another example, if the value of the optimization objective information is a specific value (e.g., 0), this may indicate that the purpose of optimization is not to perform a machine task, and may also indicate that the purpose of optimization is, for example, for human recognition. Meanwhile, as another example, if the second syntax is defined as optimization_for_human, the value of the second syntax can indicate that the purpose of optimization is for human recognition when it is a certain value (e.g., 1), and can indicate that the purpose of optimization is for the performance of a machine task when it is another value (e.g., 0).
[0205] For example, the optimization-related information may include information specifying the image optimization method and purpose. As an example, information specifying the image optimization method and purpose (e.g., the third syntax) may be obtained based on other optimization-related information (e.g., the first syntax and / or the second syntax, etc.). As an example, the third syntax may be expressed as machine_optimization_method and may indicate information for identifying the optimization method and purpose for performing a machine task. As an example, the optimization method for performing a machine task may be as shown in Table 9 below.
[0206]
[0207] The optimization method and purpose defined in Table 9 are only examples, and the conditions for distinguishing the optimization method and purpose are also only examples. In addition, it is self-evident that the third syntax can identify an optimization method and purpose for performing a machine task by including only some of the optimization methods defined in Table 9, or an optimization method that includes other optimization methods not defined in Table 9, and this is also included in the embodiments of the present disclosure. As an example, the third syntax can identify an image optimization method and purpose by including the optimization methods and purposes defined in the table. Meanwhile, in Table 9, the case where (machine_optimization_method & 0x40) == 0 may indicate that the applied optimization method may be unsuitable for human recognition, for example, this may include a case where some information is removed through optimization and is unsuitable for viewing. Meanwhile, the condition for distinguishing the optimization method and purpose in the table, for example, 0x40, is only an example, and its value may be defined differently depending on the type or number of optimization methods defined. As an example, in Table 9, if (machine_optimization_method & 0x40) != 0, it may indicate that the applied optimization method can be used for human recognition.
[0208] Meanwhile, as an example, the optimization-related information may include information specifying the image optimization method and purpose. As an example, information specifying the image optimization method and purpose (e.g., human_optimization_method, the fourth syntax) may be acquired based on other optimization-related information (e.g., the first syntax and / or the second syntax, etc.). The fourth syntax human_optimization_method may be information for identifying the optimization method and purpose for human viewing, i.e., human recognition. As an example, the optimization method for human recognition may be as shown in Table 10 below.
[0209]
[0210] As an example, the fourth syntax can identify the optimization method and purpose defined in Table 10. Meanwhile, the optimization method defined in Table 10 is an example, and the optimization method and purpose can be defined using the corresponding structure. The optimization method and purpose defined in Table 10 are also an example, and the conditions for distinguishing the optimization method and purpose are also only an example. In addition, it is obvious that the fourth syntax can identify the optimization method and purpose for human recognition by including only some of the optimization methods defined in Table 10, but not all of the optimization methods, or by including other optimization methods not defined in Table 10, and this is also included in the embodiments of the present disclosure. As an example, the fourth syntax can identify the image optimization method and purpose by including the optimization methods and purposes defined in the table. Meanwhile, as an example, the case where (human_optimization_method & 0x40) == 0 in Table 10 may correspond to a case where the applied optimization method may be unsuitable for the machine task, i.e., for example, when some information is changed due to optimization. Meanwhile, 0x40 in Table 10 is an example, and its value may be defined differently depending on the type or number of optimization methods defined. In Table 10, if (human_optimization_method & 0x40) != 0, it may indicate that the applied optimization method can also be used for performing machine tasks. Meanwhile, as an example, the optimization-related information may include information for identifying an image optimization method. As an example, information for identifying an image optimization method (e.g., the fifth syntax) may be obtained based on other optimization-related information (e.g., the first syntax, etc.). As an example, the fifth syntax optimization_method may indicate information for identifying an optimization method.As an example, the fifth syntax can define optimization methods that encompass both optimization objectives, such as machine task performance and human recognition. As an example, Table 11 below illustrates examples of optimization methods that the fifth syntax can identify.
[0211]
[0212] The optimization method and objective defined in Table 11 are only examples, and the conditions for distinguishing the optimization method and objective are also only examples. In addition, it is self-evident that the fifth syntax can identify an optimization method and objective for performing human recognition and / or machine tasks by including only some of the optimization methods defined in Table 11, or an optimization method that includes other optimization methods not defined in Table 11, and this is also included in the embodiments of the present disclosure. As an example, the fifth syntax can identify an image optimization method and objective by including the optimization methods and objectives defined in the table. As an example, in Table 11, the case where (optimization_method & 0x40) == 0 may indicate that the applied optimization method may be unsuitable for the machine task, for example, when some information is changed due to optimization. Meanwhile, 0x40 as a condition in Table 11 is only an example, and its value may be defined differently depending on the type or number of optimization methods defined. As an example, if (optimization_method & 0x100) != 0, this may indicate that the applied optimization method can be used for human recognition. As another example, if (optimization_method & 0x100) == 0, this may indicate that the applied optimization method may not be suitable for human recognition, for example, if some information is removed through optimization, making it unsuitable for viewing. In Table 11, 0x100 is just one example, and its value may be defined differently depending on the type or number of optimization methods defined.
[0213] Meanwhile, as an example, the optimization-related information may include information for the persistence of the image optimization method. As an example, information for the persistence of the image optimization method (e.g., the sixth syntax) may be acquired based on other optimization-related information (e.g., the first syntax, etc.). As an example, the sixth syntax optimization_persistence_flag may indicate information for the persistence of the image optimization method. As an example, if the value of the sixth syntax is a specific value (e.g., 0), it may indicate that the optimization method is applied only to the current picture. Alternatively, if the value of the sixth syntax is a specific value (e.g., 1), the optimization method may also be applied to other pictures following the current picture. In other words, the optimization method may persist.
[0214] Meanwhile, unlike the embodiment described above, information indicating whether the applied image optimization method is suitable for performing a machine task (e.g., not_optimal_for_machine_analysis_flag, optimization_for_machine_analaysis_flag, etc.) and information indicating whether the applied image optimization method is suitable for human recognition (e.g., human viewing, human viewing, etc.) (e.g., not_optimal_for_human_viewing_flag, optimization_for_human_viewing_flag, etc.) may be defined separately. As in the embodiment described above, the image optimization method and the usage restriction information according to the application of optimization may be defined as one syntax, but according to another embodiment of the present disclosure, since the properties of the two pieces of information may be different, they may be defined as independent syntaxes.
[0215] Below, we describe optimization-related information that may be included in information about optimization methods (e.g., encoder_optimization_info, intended_purpose_info, etc.). Any descriptions of optimization-related information that overlap with the above descriptions will be omitted. Tables 12 to 16 below illustrate examples of the syntax of information about optimization methods.
[0216]
[0217]
[0218]
[0219]
[0220]
[0221] Tables 12 to 16 above may represent examples of methods for expressing whether or not image optimization is applied, the application method, and the purpose. More specifically, according to the examples of Tables 12 and 13 above, the optimization method and application scope, machine task suitability, and human recognition suitability may be defined according to the image optimization purpose. According to the example of Table 14 above, the optimization method, application scope, machine task suitability, and human recognition suitability may be expressed by integrating the optimization purpose. According to the examples of Tables 15 and 16 above, machine task suitability and human recognition suitability may be defined respectively. Meanwhile, in another embodiment, since the information on machine task suitability and the information on human recognition suitability may be mutually exclusive, both pieces of information may not be defined.
[0222] Among the above Tables 12 to 16, the descriptions of the first syntax (e.g., encoder_optimization_flag), the second syntax (e.g., optimization_for_machine and / or optimization_for_human), the third syntax (e.g., machine_optimization_method), the fourth syntax (e.g., human_optimization_method), the fifth syntax (e.g., optimization_method), and / or the sixth syntax (e.g., optimization_persistence_flag) are the same as those described above, and therefore, duplicate descriptions are omitted.
[0223] Meanwhile, as an example, the optimization-related information may include information related to the purpose of image optimization, i.e., the intent. As an example, information related to the purpose of the image optimization method (e.g., the seventh syntax) may be obtained based on other optimization-related information (e.g., the second syntax, etc.). As an example, the seventh syntax may be expressed as not_optimal_for_machine_analysis_flag. As an example, the seventh syntax may be defined when the value of the second syntax (e.g., optimization_for_machine) is a specific value (e.g., 0), i.e., when the purpose of optimization is human recognition (e.g., human viewing, etc.). If the value of the seventh syntax is a specific value (e.g., 1), it indicates that the Picture encoded / decoded by applying the optimization method identified by human_optimization_method may not be suitable for performing the machine task. If the value is 0, it may indicate that the coded picture can also be used for machine tasks by applying optimization based on an optimization method identified by another syntax (e.g., the fourth syntax, human_optimization_method).
[0224] Meanwhile, as an example, the optimization-related information may include information related to the purpose of video optimization, i.e., the intent. As an example, information related to the purpose of the video optimization method (e.g., the eighth syntax) may be obtained based on other optimization-related information (e.g., the second syntax, etc.). As an example, the eighth syntax may be expressed as not_optimal_for_human_viewing_flag. As an example, the eighth syntax not_optimal_for_human_viewing_flag may be defined when the value of another syntax (e.g., the second syntax optimization_for_machine) is a specific value (e.g., 1), i.e., when the purpose of optimization is to perform a machine task. As an example, if the value of the eighth syntax not_optimal_for_human_viewing_flag is a specific value (e.g., 1), an optimization method identified by another syntax (e.g., the third syntax machine_optimization_method) may be applied to indicate that the coded picture may not be suitable for human recognition (e.g., human viewing), etc. As another example, if the value is a specific value (e.g., 0), an optimization method identified by another syntax (e.g., the third syntax machine_optimization_method) may be applied to indicate that the coded picture may be used for human recognition (e.g., human viewing), etc.
[0225] Meanwhile, as an embodiment of the present disclosure, when identifying an optimization method based on the third syntax, it may be based on Table 9, but the case where ( machine_optimization_method & 0x40 ) == 0 (Optimization methods may be unsuitable for human viewing) or ( machine_optimization_method & 0x40 ) != 0 (Can be used for human viewing purposes) may be excluded.
[0226] Meanwhile, as an embodiment of the present disclosure, when identifying an optimization method based on the fourth syntax, it may be based on Table 10, but the case where (human_optimization_method & 0x40) == 0 (May be unsuitable for performing machine tasks) or (human_optimization_method & 0x40) != 0 (Can be used for machine tasks) may be excluded.
[0227] Meanwhile, as an embodiment of the present disclosure, when identifying an optimization method based on the fifth syntax, it may be based on Table 11, but cases where (optimization_method & 0x40) == 0 (May be unsuitable for performing machine tasks), (optimization_method & 0x40) != 0 (Can be used for machine tasks), (optimization_method & 0x100) == 0 (Optimization methods may be unsuitable for human viewing), (optimization_method & 0x100) != 0 (Can be used for human viewing purposes) may be excluded.
[0228] Meanwhile, the image optimization process may result in changes to picture (video) information. For example, specific information about the image may be removed or reduced, while other information may be enhanced or added. These changes in information may impact the tasks that a user, i.e., a human or a machine, is attempting to perform. Therefore, it is necessary to specify the impact that such image optimization may have on task performance. Furthermore, considering that image optimization methods can be modified and added, it may be necessary to define information about the impact of optimization rather than information about the optimization method itself. The following examples describe in detail information about the impact of image optimization that may be included in information related to image optimization. Examples of information about the impact of image optimization are described with reference to Tables 17 and 18.
[0229]
[0230]
[0231] Table 17 above is a table showing an example of a method for expressing the purpose, influence, and scope of application of image optimization according to one embodiment of the present disclosure, and Table 18 is a table showing an example of a method for expressing the purpose and influence of image optimization.
[0232] Meanwhile, when describing one embodiment of the present disclosure with reference to Table 17 above, the description of the first syntax (e.g., encoder_optimization_flag) and the sixth syntax (optimization_persistence_flag) are the same as described above, so redundant description is omitted.
[0233] Meanwhile, information related to video optimization may include information about the purpose of video optimization. Information about the purpose of video optimization may be expressed as optimization_purpose, and may be signaled based on other syntaxes including information about whether video optimization has been performed (e.g., the first syntax). As an example, the ninth syntax optimization_purpose may be information about the purpose of video optimization and the impact of video optimization. As an example, the ninth syntax may indicate how information included in a bitstream encoded by applying video optimization has been changed for which purpose and by which optimization method. Meanwhile, referring to an example in Table 18, "optimization_purpose > 0 && (optimization_purpose & 0x04) != 0" may mean that the applied optimization method does not affect the identification of personal information. In addition, "optimization_purpose > 0 && (optimization_purpose & 0x04) = = 0" may mean that the applied optimization method affects the identification of personal information. In this case, personal information may be removed or altered during the video optimization process due to privacy concerns. In such cases, it may be processed in machine analysis related to personal information identification. As another example, "optimization_purpose > 0 && (optimization_purpose & 0x10) ! = 0" may indicate that the applied optimization method did not affect security and / or safety-related information.For example, if some areas in the image are identified as unnecessary areas and have been changed, for example, if information in background areas such as roads or facilities has been removed or changed, some tasks that require this information may require additional care, such as referencing other bitstreams, referring to other syntaxes, or processing other additional processes when processing the corresponding bitstream. Meanwhile, as another example, "optimization_purpose > 0 && (optimization_purpose & 0x40) != 0" may mean that the optimization method did not affect security-related information. Meanwhile, Table 18 is an example showing the purpose and impact of image optimization, and may identify the purpose and impact of image optimization by including only some of the cases defined in Table 18, or may be configured to include the purpose and impact of other optimizations not defined in Table 18, and this is also included in the present disclosure.
[0234] Meanwhile, in certain cases, from the perspective of the user (i.e., human or machine), it may be necessary to determine whether the encoded video is suitable for the task to be performed, i.e., suitable for the intended purpose, rather than the applied encoding and optimization method. The following examples provide detailed information regarding the purpose of image optimization that may be included in the information related to image optimization. Examples of information regarding the purpose of image optimization are described with reference to Tables 19 to 21.
[0235]
[0236]
[0237]
[0238] Table 19 above is an example of information expressing the intention and scope of application for optimization of a coded image according to one embodiment of the present disclosure, and Tables 20 and 21 above are examples of information expressing the intention and purpose for image optimization of a coded image.
[0239] First, referring to Table 19, the video optimization-related information may include information about the video optimization intent and scope of application, and information about whether coding settings for a specific purpose are applied. As an example, information indicating whether coding settings for a specific purpose were applied when coding a video may be expressed as purpose_oriented_encoding_flag and may be referred to as the tenth syntax. If the value of the tenth syntax purpose_oriented_encoding_flag is a specific value (e.g., 0), it may indicate that coding settings for a specific purpose (e.g., encoder configuration, encoder optimization) were not applied when the video was coded. In one example, further, if the value of the tenth syntax is a specific value (e.g., 0), it may also indicate that settings for a specific purpose that were previously applied to the video have been terminated. Meanwhile, if the value of the 10th syntax is a specific value (e.g., 1), it may indicate that a specific coding setting exists for a specific purpose, and that coding settings are applied according to other image optimization-related information defined subsequently. As an example, the coding setting may be expressed by other syntaxes (e.g., purpose_persistence_flag, intended_purpose, etc.) described below.
[0240] As an example, information about the intent and scope of application of video optimization may include information about the persistence of the video optimization purpose. As an example, information about the persistence of the video optimization purpose (e.g., syntax 11) may be obtained based on other optimization-related information (e.g., syntax 10, etc.). Information about the persistence of the video optimization purpose may be expressed, for example, by the purpose _persistence_flag syntax. As an example, the eleventh syntax purpose _persistence_flag may indicate information about the persistence of the video optimization purpose. As an example, if the value of the eleventh syntax is a specific value (e.g., 0), it may indicate that the persistence of the optimization purpose has ended. Alternatively, if the value of the eleventh syntax is a specific value (e.g., 1), it may indicate that the optimization purpose continues in other pictures following the current picture.
[0241] As an example, information about the intent and scope of video optimization may include information about the purpose of video optimization. As an example, information about the purpose of video optimization (e.g., syntax 12) may be obtained based on other optimization-related information (e.g., syntax 10, etc.). Information about the purpose of video optimization may be expressed, for example, by the syntax intended_purpose. As an example, the syntax intended_purpose of the 12th syntax indicates the purpose of the encoded video and may indicate whether it is suitable or unsuitable for a specific purpose. Meanwhile, Tables 20 and 21 are examples that integrate and express the properties of the video and the tasks that can be performed. In Table 20, "intended_purpose > 0 && (intended_purpose & 0x04) = = 0" may mean that the purpose of using the video does not include the identification of personal information. Also, in Table 20, "intended_purpose > 0 && (intended_purpose & 0x04) ! = 0" may mean that the purpose of the video includes the identification of personal information. As an example, "intended_purpose > 0 && (intended_purpose & 0x10) = = 0" in Table 20 may mean that the purpose of the video does not include security or safety-related things. Meanwhile, Tables 20 and 21 are only examples, and may not include all cases defined in each table, but may include only some cases to identify whether they are suitable or unsuitable for specific purposes and tasks, or may include other purposes and tasks not defined in Tables 20 and 21.
[0242] Meanwhile, information related to image optimization may include information about the properties of the image. In other words, image optimization may result in changes to the properties of the image (e.g., pixel values, segmentation method, etc.). Information about the changed properties of the image may be required to indicate the purpose, intent, and use of the image optimization. Tables 22 and 23 below are tables for indicating information about the properties of the image according to one embodiment of the present disclosure.
[0243]
[0244]
[0245] Table 22 above is a table showing information on image properties according to one embodiment of the present disclosure, and Table 23 is a table showing properties of a changed image according to one embodiment of the present disclosure.
[0246] First, the descriptions of the first syntax (e.g., encoder_optimization_flag) and the sixth syntax (e.g., optimization_persistence_flag) are the same as those described above, so duplicate descriptions are omitted.
[0247] Meanwhile, information related to image optimization may include information on image properties, and the information on image properties may be signaled based on other syntaxes (e.g., the first syntax, etc.). As an example, information on image properties may be referred to as picture_characteristics. The 13th syntax picture_characteristics may indicate the properties of an image changed by the application of image optimization, and may indicate changes in image properties as shown in Table 22. However, since Table 22 is only an example, changes in image properties may be identified by including only some of the cases defined in Table 22, and other properties not defined in Table 22 may be further identified, which will also be included in the present disclosure.
[0248] Meanwhile, information on the purpose, properties, and / or scope of image optimization that may be included in image optimization-related information according to another embodiment of the present disclosure is described based on Tables 24 to 26.
[0249]
[0250]
[0251]
[0252] Table 24 above is a table showing examples of information that may be included in image optimization-related information according to one embodiment of the present disclosure, and Tables 25 and 26 are tables showing examples of image optimization attribute expression methods. As an example, information related to image optimization may include information on the persistence of image optimization. As described above, information on the persistence of image optimization may include information on whether image optimization is continued (e.g., syntax 6, etc.) and may also include information on whether image optimization continuation is canceled. As an example, information on whether image continuation is minimum may be expressed as optimization_cancel_flag and may be expressed as syntax 14. If the value of syntax 14 optimization_cancel_flag is a specific value (e.g., 1), it may indicate that the persistence of image optimization that was previously continuously applied is canceled. On the other hand, if the value of syntax 14 is a specific value (e.g., 0), it may indicate that the persistence of other image optimizations is not canceled. In addition, in this case, more information related to image optimization can be signaled, for example, the sixth syntax optimization_persistence_flag, the fifteenth syntax optimization_for_machine_analysis_flag and the sixteenth syntax optimization_type described below can be defined subsequently. In addition, the image optimization method, purpose, and / or scope can be identified and applied based on the sixth syntax optimization_persistence_flag, the fifteenth syntax optimization_for_machine_analysis_flag and the sixteenth syntax optimization_type defined subsequently.
[0253] Meanwhile, the description of the sixth syntax (e.g., optimization_persistence_flag) among the syntaxes shown in Table 24 is the same as that described above, so a duplicate description is omitted. Meanwhile, referring to Table 24, the sixth syntax can be signaled based on another syntax (e.g., the fourteenth syntax optimization_cancel_flag).
[0254] As an example, information related to video optimization may include information about the purpose of video optimization. As described above, information about the purpose of video optimization may include information indicating the use and / or purpose of video optimization. As an example, the 15th syntax optimization_for_machine_analysis_flag may be included in the information about the purpose of video optimization. As an example, when the value of the 15th syntax optimization_for_machine_analysis_flag is a specific value (e.g., 1), it may indicate that the purpose of video optimization and the coded bitstream are for performing a machine task (e.g., machine analysis). That is, it may indicate that the bitstream is suitable for performing a machine task. As another example, when the value of the 15th syntax is a specific value (e.g., 0), it may indicate that the purpose of video optimization and the coded bitstream may be for a machine task, but may not be (it indicates it may / may not be for machine analysis).
[0255] As an example, information related to image optimization may include information about the purpose of image optimization. As described above, information about the purpose of image optimization may include information indicating the properties of the image according to the image optimization. As an example, the 16th syntax optimization_type may be included in information about the properties of the image. As an example, the 16th syntax may identify the optimization properties defined in Table 25 and / or Table 26. However, the optimization properties defined in Table 25 and / or Table 26 are only an example, and new optimization properties may be defined using the structure, and this is also included in the present disclosure. In addition, optimization properties and methods may be identified by including only some of the optimization properties defined in Table 25 and / or Table 26, or may be configured to include other optimization properties and methods not defined in Table 25 and / or Table 26, and this is also included in the present disclosure.
[0256] Meanwhile, in the embodiment of Table 24, the seventh and / or eighth syntaxes described above may be signaled instead of the fifteenth syntax. Furthermore, the seventeenth syntax optimization_for_human_viewing_flag, which is similar to the seventh and / or eighth syntaxes, may be signaled. Table 27 below is a table illustrating such an embodiment.
[0257]
[0258] As an example of Table 27, the 17th syntax may be included in the information about the purpose of image optimization, and may be information for identifying whether the purpose of the current image optimization is for machine task performance or human recognition, similar to the 7th syntax and / or the 8th syntax. As an example, the 17th syntax may be signaled based on other syntaxes (e.g., syntax 14, etc.), such that when the value of the 17th syntax is a specific value (e.g., 1), it may indicate that the purpose of image optimization and the coded bitstream are for human recognition (human viewing, etc.). As another example, when the value of the 17th syntax is a specific value (e.g., 0), the purpose of image optimization and the coded bitstream may or may not be for human recognition (If the value is '0', it may / may not be for human viewing).
[0259] Meanwhile, among the syntaxes shown in Table 27, the descriptions of the 14th syntax (e.g., optimization_cancel_flag), the 6th syntax (e.g., optimization_persistence_flag), and the 16th syntax (e.g., optimization_type) are the same as those described above, so redundant descriptions are omitted. Meanwhile, even with reference to Table 27, the 6th syntax and / or the 16th syntax may be signaled based on another syntax (e.g., the 14th syntax optimization_cancel_flag).
[0260] Additionally, in the embodiment of Table 27, the 16th syntax can identify the image optimization methods of Tables 25 and 26.
[0261] Meanwhile, as another example, Table 24 and Table 27 describe examples in which only one of the 15th syntax and the 17th syntax is signaled, but both the 15th syntax and the 17th syntax may be signaled, as shown in Table 28 below.
[0262]
[0263] However, since Table 28 corresponds to one embodiment of the present disclosure, the 15th syntax may be signaled first instead of the 17th syntax as shown in Table 28, and this may also be included in the present disclosure. Such an embodiment may be as shown in Table 29 below.
[0264]
[0265] The description of the 14th syntax (e.g., optimization_cancel_flag), the 6th syntax (e.g., optimization_persistence_flag), the 17th syntax (e.g., optimization_for_human_viewing_flag), the 15th syntax (e.g., optimization_for_machine_analysis_flag), and the 16th syntax (e.g., optimization_type), which may be included in the information related to video optimization disclosed in Table 28 and / or Table 29, is the same as described above, and therefore, redundant description is omitted. Meanwhile, even with reference to Table 28 and / or Table 29, the 6th syntax, the 17th syntax, the 15th syntax, and / or the 16th syntax may be signaled based on another syntax (e.g., the 14th syntax optimization_cancel_flag). Also, in the embodiments of Table 28 and / or Table 29, the 16th syntax can identify the image optimization methods of Tables 25 to 26. Meanwhile, unlike Table 28 and / or Table 29, the 15th syntax can be signaled based on the 17th syntax, or the 17th syntax can be signaled based on the 15th syntax. Tables 30 and 31 are tables for explaining these embodiments, respectively.
[0266]
[0267]
[0268] The description of the 14th syntax (e.g., optimization_cancel_flag), the 6th syntax (e.g., optimization_persistence_flag), the 17th syntax (e.g., optimization_for_human_viewing_flag), the 15th syntax (e.g., optimization_for_machine_analysis_flag), and the 16th syntax (e.g., optimization_type), which may be included in the information related to video optimization disclosed in Table 30 and / or Table 31, is the same as described above, and therefore, redundant description is omitted. Meanwhile, even with reference to Table 30 and / or Table 31, the 6th syntax, the 17th syntax, the 15th syntax, and / or the 16th syntax may be signaled based on another syntax (e.g., the 14th syntax optimization_cancel_flag). Additionally, in the embodiments of Table 30 and / or Table 31, the 16th syntax can identify the image optimization method of Tables 25 and 26.
[0269] Meanwhile, in addition to the above description of the 15th syntax and / or the 17th syntax, according to the embodiments of Table 30 and / or Table 31, the 15th syntax and the 17th syntax may be restricted from being a specific value (e.g., 0) at the same time. For example, according to the embodiment of Table 30, the 15th syntax may be signaled based on the 17th syntax. For example, when the value of the 17th syntax is a specific value (e.g., 0), the value of the 15th syntax may not be signaled, but may be derived to a specific value (e.g., 1) on the decoder side, for example. In addition, the value of the 15th syntax may also be determined on the encoder side according to the value of the 17th syntax. Meanwhile, as in Table 31, the 17th syntax may also be signaled based on the 15th syntax. For example, when the value of the 15th syntax is a specific value (e.g., 0), the value of the 17th syntax is not signaled and can be derived as a specific value (e.g., 1) on the decoder side, for example. In addition, the value of the 17th syntax can also be determined on the decoder side depending on the value of the 15th syntax.
[0270] Meanwhile, in one embodiment, at least one of the first to seventeenth syntaxes described above may be signaled in a Supplemental Enhancement Information (SEI) message or the like. For example, the SEI message may include information about the encoding preprocessing type applied to the picture associated with the SEI message, information about whether image optimization for human recognition is applied, and / or information about whether image optimization for machine task processing is applied. As an example, when the value of the sixth syntax optimization_persistence_flag, which may be included in the information about the persistence of image optimization, is a specific value (e.g., 1), it may indicate that image optimization may persist, and a specific image optimization method may be applied to subsequent pictures of the current picture of the current layer in output order until one or more of the following conditions are true:
[0271] - When a new CLVS (Coded Layer Video Sequence) of the current layer starts
[0272] - When the bitstream ends
[0273] - When the picture of the current layer associated with the encoder_optimization_info SEI message with optimization_cancel_flag set to 1 is output after the current picture in the output order.
[0274] FIG. 14 is a diagram illustrating an example of a decryption method that can be performed by a decryption device according to an embodiment of the present disclosure. As an example, a decryption device that can perform FIG. 14 includes a data decryption device, and may include the image decryption device, feature decryption device, etc. described above. In addition, the decryption method of FIG. 14 may be a data decryption method, and may include the image decryption method and feature decryption method described above. In addition, since the decryption method of FIG. 14 may be based on the embodiment described above, any description that overlaps with the above description will be omitted.
[0275] As an example, data optimization information related to an image data optimization method (or, an image data modification method, etc.) according to the purpose of the image data may be acquired (S1410). As an example, the image data or data may include an image, a feature, a feature map, etc. As an example, the data optimization information (or, data modification information, etc.) related to the image data optimization method may include the image optimization-related information (or, image modification-related information, etc.) described above, and may include information on the image optimization method (e.g., information on the image modification method, etc.) (e.g., the image optimization method, the image optimization purpose, the image optimization intention, the image optimization persistence, etc.). Since each piece of information is the same as described above, a duplicate description will be omitted.
[0276] Thereafter, the image data may be restored (S1420) based on the acquired data optimization information. In addition, the data optimization information may include data optimization application information indicating whether the image data is optimized for each purpose. In addition, the data optimization information may include data optimization purpose information indicating the purpose of optimizing the image data. In addition, as an example, the data optimization purpose information may include machine task optimization information indicating a method of optimizing image data for performing a machine task. In addition, as an example, the data optimization purpose information may include human recognition optimization information indicating a method of optimizing image data for human recognition. In addition, as an example, the human recognition optimization information may be acquired based on the machine task optimization information. In addition, as an example, the machine task optimization information may be acquired based on the human recognition optimization information. In addition, as an example, the data optimization information may include data optimization persistence information indicating the persistence of the optimization method for the image data. As described above, the data optimization persistence information may include persistence maintenance information and / or persistence cancellation information indicating whether to cancel optimization persistence. Additionally, as an example, based on the persistence cancellation information, at least one of the following may be obtained: persistence degree information indicating the degree of optimization persistence, target information indicating the optimization target, or type information indicating the optimization property. Additionally, as an example, data optimization information may be obtained based on optimization status information indicating whether data is optimized. As an example, information related to image optimization, i.e., data optimization information, may be signaled in the bitstream, for example, optimization status information may be signaled in the bitstream.As an example, information related to image optimization, for example, data optimization information, may include at least one of the first to eighteenth syntaxes described above, and may be signaled through an SEI message or the like as described above. As for other embodiments, the same is as described above, and therefore, redundant descriptions are omitted.
[0277] Meanwhile, since the decryption method of FIG. 14 corresponds to one embodiment of the present disclosure, the order of some steps may be changed or some steps may be added or deleted, and this will also be included in the embodiment of the present disclosure, and the names of information, etc. are for the purpose of clarity of explanation and are not intended to limit the present disclosure.
[0278] FIG. 15 is a diagram illustrating an example of an image encoding method that can be performed by an image encoding device according to an embodiment of the present disclosure. As an example, an encoding device that can perform FIG. 15 includes a data encoding device, and may include the image encoding device, feature encoding device, etc. described above. In addition, the encoding method of FIG. 15 may be a data encoding method, and may include the image encoding method and feature encoding method described above. In addition, the encoding method of FIG. 15 may be based on the embodiment described above, and any description that overlaps with the above description will be omitted.
[0279] As an example, data optimization information (e.g., data modification information, etc.) related to an image data optimization method (e.g., an image data modification method, etc.) according to the purpose of the image data may be determined (S1510). In addition, the data optimization information may be encoded into a bitstream (S1520). As an example, the data optimization information may include data optimization application information indicating whether the image data is optimized for each purpose. In addition, the image data or data may include an image, a feature, a feature map, etc. As an example, the data optimization information (or data modification information, etc.) related to the image data optimization method may include the image optimization-related information (or image modification-related information, etc.) described above, and may include information on the image optimization method (e.g., information on the image modification method, etc.) (e.g., the image optimization method, the image optimization purpose, the image optimization intention, the image optimization persistence, etc.). Since each piece of information is the same as described above, a duplicate description will be omitted.
[0280] In addition, the data optimization information may include data optimization objective information indicating the purpose of optimizing the image data. Furthermore, as an example, the data optimization objective information may include machine task optimization information indicating a method of optimizing image data for performing a machine task. Furthermore, as an example, the data optimization objective information may include human recognition optimization information indicating a method of optimizing image data for human recognition. Furthermore, as an example, the human recognition optimization information may be obtained based on the machine task optimization information. Furthermore, as an example, the machine task optimization information may be obtained based on the human recognition optimization information. Furthermore, as an example, the data optimization information may include data optimization persistence information indicating the persistence of the image data optimization method. As described above, the data optimization persistence information may include persistence maintenance information and / or persistence cancellation information indicating whether to cancel optimization persistence. Furthermore, as an example, at least one of persistence degree information indicating the degree of optimization persistence, target information indicating an optimization objective target, or type information indicating an optimization attribute may be signaled based on the persistence cancellation information. In addition, as an example, data optimization information may be signaled based on optimization status information indicating whether data is optimized. As an example, information related to image optimization, i.e., data optimization information, may be signaled in a bitstream, and for example, optimization status information may be signaled in a bitstream. As an example, information related to image optimization, e.g., data optimization information, may include at least one of the first to eighteenth syntaxes described above, and may be signaled via an SEI message or the like as described above. Since other embodiments are the same as described above, redundant descriptions are omitted.
[0281] According to one embodiment of the present disclosure, a bitstream generated by the encoding method may be stored / recorded on a medium or the like. As an example, the medium may be a non-transitory computer-readable medium, and may be a storage medium and / or a recording medium.
[0282] According to one embodiment of the present disclosure, a method for transmitting a bitstream generated by an encoding method may be disclosed.
[0283] Meanwhile, for the sake of clarity of explanation, the term “image optimization” was used only in the above description, but as mentioned above, all explanations based on image optimization can be replaced with terms such as image modification, image adjustment, image control, image change, and / or image processing.
[0284] All names of the syntax elements described above are arbitrarily assigned for clarity of explanation and do not limit the names of the syntax elements. In addition, the first to eighteenth syntaxes may be referred to as first information to eighteenth information, etc., respectively. In addition, the first to eighteenth syntaxes may be obtained from a bitstream, but may also be derived from other syntax elements, which may also be included in embodiments of the present disclosure.
[0285] Additionally, the bitstream generated by the image encoding method may be stored in a non-transitory computer-readable recording medium.
[0286] Additionally, as another example, a bitstream generated by a video encoding method may be transmitted to another device (e.g., a video decoding device, etc.). In this case, the method for transmitting the bitstream may include a process for transmitting the bitstream.
[0287] While the exemplary methods of this disclosure are presented as a series of operations for clarity of description, this is not intended to limit the order in which the steps are performed, and individual steps may be performed simultaneously or in different orders, if desired. To implement a method according to this disclosure, additional steps may be included in addition to the steps illustrated, some steps may be excluded and the remaining steps included, or some steps may be excluded and additional steps included.
[0288] In the present disclosure, a video encoding device or video decoding device performing a predetermined operation (step) may perform an operation (step) of checking the conditions or circumstances under which the operation (step) is performed. For example, if it is described that a predetermined operation is performed when a predetermined condition is satisfied, the video encoding device or video decoding device may perform an operation of checking whether the predetermined condition is satisfied and then perform the predetermined operation.
[0289] The various embodiments of the present disclosure are not intended to list all possible combinations but rather to illustrate representative aspects of the present disclosure, and the matters described in the various embodiments may be applied independently or in combination of two or more.
[0290] The embodiments described in this disclosure may be implemented and performed on a processor, microprocessor, controller, or chip. For example, the functional units depicted in each drawing may be implemented and performed on a computer, processor, microprocessor, controller, or chip. In this case, information (e.g., information on instructions) or algorithms for implementation may be stored on a digital storage medium.
[0291] In addition, the decoder (decoding device) and encoder (encoding device) to which the embodiment(s) of the present disclosure are applied may be included in a multimedia broadcasting transmitting and receiving device, a mobile communication terminal, a home cinema video device, a digital cinema video device, a surveillance camera, a video conversation device, a real-time communication device such as a video communication, a mobile streaming device, a storage medium, a camcorder, a video-on-demand (VoD) service providing device, an OTT video (Over the top video) device, an Internet streaming service providing device, a three-dimensional (3D) video device, a VR (virtual reality) device, an AR (argumente reality) device, a video phone video device, a transportation terminal (e.g., a vehicle (including an autonomous vehicle) terminal, a robot terminal, an airplane terminal, a ship terminal, etc.), and a medical video device, and may be used to process a video signal or a data signal. For example, OTT video (Over the top video) devices may include game consoles, Blu-ray players, Internet-connected TVs, home theater systems, smartphones, tablet PCs, and DVRs (Digital Video Recorders).
[0292] In addition, the processing method to which the embodiment(s) of the present disclosure are applied can be produced in the form of a computer-executable program and can be stored in a computer-readable recording medium. Multimedia data having a data structure according to the embodiment(s) of the present document can also be stored in a computer-readable recording medium. The computer-readable recording medium includes all types of storage devices and distributed storage devices in which computer-readable data is stored. The computer-readable recording medium can include, for example, a Blu-ray disc (BD), a universal serial bus (USB), a ROM, a PROM, an EPROM, an EEPROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device. In addition, the computer-readable recording medium includes a medium implemented in the form of a carrier wave (e.g., transmission via the Internet). In addition, a bitstream generated by an encoding method can be stored in a computer-readable recording medium or transmitted via a wired or wireless communication network.
[0293] Additionally, the embodiments of the present disclosure may be implemented as a computer program product by program code, and the program code may be executed on a computer by the embodiments of the present disclosure. The program code may be stored on a computer-readable carrier.
[0294] FIG. 16 is a diagram illustrating an example of a content streaming system to which embodiments of the present disclosure can be applied.
[0295] Referring to FIG. 16, a content streaming system to which an embodiment of the present disclosure is applied may largely include an encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.
[0296] The encoding server compresses content input from multimedia input devices, such as smartphones, cameras, and camcorders, into digital data, generates a bitstream, and transmits it to a streaming server. Alternatively, if multimedia input devices, such as smartphones, cameras, and camcorders, generate the bitstream directly, the encoding server may be omitted.
[0297] A bitstream can be generated by a video encoding method and / or a video encoding device to which an embodiment of the present disclosure is applied, and a streaming server can temporarily store the bitstream during the process of transmitting or receiving the bitstream.
[0298] A streaming server transmits multimedia data to a user device based on a user request via a web server, and the web server can act as an intermediary, informing the user of available services. When a user requests a desired service from the web server, the web server transmits the request to the streaming server, which then transmits the multimedia data to the user. The content streaming system may include a separate control server, in which case the control server may control commands and responses between each device within the content streaming system.
[0299] A streaming server can receive content from a media repository and / or encoding server. For example, when receiving content from an encoding server, the content can be received in real time. In this case, the streaming server can store the bitstream for a certain period of time to provide a smooth streaming service.
[0300] Examples of user devices include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, head mounted displays (HMDs)), digital TVs, desktop computers, and digital signage.
[0301] Each server within a content streaming system can be operated as a distributed server, in which case the data received from each server can be processed in a distributed manner.
[0302] FIG. 17 is a diagram showing another example of a content streaming system to which embodiments of the present disclosure can be applied.
[0303] Referring to FIG. 17, in an embodiment such as VCM, depending on the performance of the device, the user's request, the characteristics of the task to be performed, etc., the task may be performed on the user terminal or on an external device (e.g., a streaming server, an analysis server, etc.). In this way, in order to transmit the information required for performing the task to the external device, the user terminal may directly or through an encoding server generate a bitstream containing the information required for performing the task (e.g., information such as the task, neural network, and / or purpose).
[0304] The analysis server can decrypt encoded information received from the user terminal (or encoding server) and then perform a task requested by the user terminal. The analysis server can transmit the results obtained through the task performance back to the user terminal or to another linked service server (e.g., a web server). For example, the analysis server can perform a task to determine fire and transmit the results obtained to a fire-related server. The analysis server may include a separate control server, in which case the control server can control commands / responses between each device associated with the analysis server and the server. In addition, the analysis server can request desired information from the web server based on information about the tasks the user device wishes to perform and can perform. When the analysis server requests a desired service from the web server, the web server can transmit this request to the analysis server, and the analysis server can transmit data corresponding to the request to the user terminal. In this case, the control server of the content streaming system can control commands / responses between each device within the streaming system.
[0305] Embodiments according to the present disclosure can be used for image encoding / decoding.
Claims
1. In the decryption method, A step of obtaining data optimization information related to a method of optimizing image data according to the purpose of the image data; and A step of restoring the image data based on the above data optimization information; including, A decryption method, wherein the above data optimization information includes data optimization application information indicating whether the image data is optimized for each purpose.
2. In paragraph 1, A decryption method, wherein the above data optimization information further includes data optimization purpose information indicating the purpose of optimizing the image data.
3. In paragraph 2, A decoding method, wherein the above data optimization purpose information further includes machine task optimization information indicating a method for optimizing image data for performing a machine task.
4. In paragraph 3, A decoding method, wherein the above data optimization purpose information further includes human recognition optimization information indicating a method of optimizing image data for human recognition.
5. In paragraph 4, A decryption method wherein the above human recognition optimization information is obtained based on the above machine task optimization information.
6. In paragraph 4, A decryption method wherein the above machine task optimization information is obtained based on the above human recognition optimization information.
7. In paragraph 1, A decryption method, wherein the above data optimization information further includes data optimization persistence information indicating persistence of an optimization method of image data.
8. In paragraph 7, A decryption method wherein the above data optimization persistence information includes persistence cancellation information indicating whether optimization persistence is canceled.
9. In paragraph 8, A decryption method in which at least one of persistence degree information indicating the degree of optimization persistence, target information indicating the optimization objective, or type information indicating the optimization property is further acquired based on the above persistence cancellation information.
10. In paragraph 1, A decryption method in which the above data optimization information is obtained based on optimization information indicating whether data is optimized.
11. In paragraph 10, A decoding method in which the above optimization information is signaled in the bitstream.
12. In paragraph 1, A decoding method wherein the above data optimization information is signaled in a bitstream.
13. In the encoding method, A step of determining data optimization information related to a method of optimizing image data according to the purpose of the image data; and A step of encoding the above data optimization information into a bitstream; including: An encoding method wherein the above data optimization information includes data optimization application information indicating whether the image data is optimized for each purpose.
14. In a bitstream storage medium generated by an encoding method, the encoding method comprises: A step of determining data optimization information related to a method of optimizing image data according to the purpose of the image data; and A step of encoding the above data optimization information into a bitstream; including: The above data optimization information is a medium including data optimization application information indicating whether the image data is optimized for each purpose.
15. In a method for transmitting a bitstream generated by an encoding method, the encoding method comprises: A step of determining data optimization information related to a method of optimizing image data according to the purpose of the image data; and A step of encoding the above data optimization information into a bitstream; including: A bitstream transmission method, wherein the above data optimization information includes data optimization application information indicating whether the image data is optimized for each purpose.
Citation Information
Patent Citations
Video processing device and video stream processing method
JP2022028586A
Content-adaptive application of fixed transfer function to high dynamic range (HDR) and / or wide color gamut (WCG) video data
KR1020180016383A
Method of Preparing Shape-Variable Micropatterned Polymer Tactile Material Using Electric Field Application
KR1020220029377A
Battery module and method for evaluating swelling characteristics of lithium secondary battery using the same
KR1020240019519A
Probe card transporting apparatus and method thereof
KR1020240094137A
