Method for decoding image information, method for encoding image information, method for a bitstream, and computer-readable storage medium storing a bitstream

Abstract

A method according to the present disclosure includes acquiring image information including an encoder optimization information (EOI) supplemental enhancement information (SEI) message, and deriving information on an encoder optimization based on the EOI SEI message, wherein the EOI SEI message includes optimization type information indicating an optimization type including temporal resampling, and based on the optimization type information, deriving whether a current picture included in a same unit as the EOI SEI message is a source picture for temporal resampling.

Description

Technical Field

[0001] This disclosure relates to methods for decoding image information, methods for encoding image information, methods for using bitstreams, and computer-readable storage media for storing bitstreams. Background Technology

[0002] Recently, the demand for high-resolution and high-quality images, such as HD (high-definition) and UHD (ultra-high-definition) images, has been growing across various fields. As image data becomes higher in resolution and quality, the amount of information or bits transmitted increases relatively compared to traditional image data. This increase in the amount of information or bits transmitted leads to increased transmission and storage costs.

[0003] Therefore, there is a need for efficient image compression techniques for effectively transmitting, storing, and reproducing information from high-resolution and high-quality images. Summary of the Invention

[0004] Technical issues

[0005] This disclosure seeks to suppress, prevent, or minimize obfuscation and / or malfunctions in coding systems.

[0006] This disclosure seeks to improve the reliability of encoding systems, including encoding and decoding devices.

[0007] This disclosure seeks to improve the encoding speed and efficiency of encoding systems that include encoding and decoding devices.

[0008] This disclosure seeks to improve the data transmission efficiency of an encoding system, which includes an encoding device and a decoding device.

[0009] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains from the following description.

[0010] Technical solution

[0011] According to one aspect of this disclosure, a method for decoding image information includes: obtaining image information including an encoder optimization information (EOI) SEI (supplementary enhancement information) message; and deriving information about encoder optimization based on the encoder optimization information SEI message, wherein the encoder optimization information SEI message includes optimization type information indicating an optimization type including temporal resampling, and wherein, based on the optimization type information, it is derived whether the current image included in the same unit as the encoder optimization information SEI message is a source image used for temporal resampling.

[0012] According to one aspect of this disclosure, an apparatus for decoding image information includes a memory and a processor connected to the memory, wherein the processor is configured to: acquire image information including an encoder optimization information (EOI) SEI (supplementary enhancement information) message; and derive information about encoder optimization based on the encoder optimization information SEI message, wherein the encoder optimization information SEI message includes optimization type information indicating an optimization type including time resampling, and wherein, based on the optimization type information, it is derived whether the current image included in the same unit as the encoder optimization information SEI message is a source image for time resampling.

[0013] In a method or apparatus for decoding image information, the encoder optimization information (SEI) message may further include source image information that the current image included in the same unit as the encoder optimization information (SEI) message is a source image used for time resampling.

[0014] In a method or apparatus for decoding image information, the encoder optimization information (SEI) message may further include temporal resampling type information about temporal upsampling and image quantity information about the number of images added or excluded through temporal resampling.

[0015] In a method or apparatus for decoding image information, source image information can be obtained based on time resampling type information and image quantity information.

[0016] In a method or apparatus for decoding image information, source image information can be obtained based on time resampling type information indicating time upsampling and image quantity information with a value greater than 0.

[0017] In a method or apparatus for decoding image information, temporal resampling type information and image quantity information can be obtained based on optimization type information indicating temporal resampling.

[0018] According to one aspect of this disclosure, a method for encoding image information includes: performing encoder optimization; and encoding image information including an encoder optimization information (EOI) SEI (supplementary enhancement information) message generated based on information about encoder optimization, wherein the encoder optimization information SEI message includes optimization type information indicating an optimization type including temporal resampling, and wherein, based on the optimization type information, it is determined whether a current image included in the same unit as the encoder optimization information SEI message is a source image for temporal resampling.

[0019] According to one aspect of this disclosure, an apparatus for encoding image information includes a memory and a processor connected to the memory, wherein the processor is configured to: perform encoder optimization; and encode image information including an encoder optimization information (EOI) SEI (supplementary enhancement information) message generated based on information about the encoder optimization, wherein the encoder optimization information SEI message includes optimization type information indicating an optimization type including temporal resampling, and wherein, based on the optimization type information, it is determined whether a current image included in the same unit as the encoder optimization information SEI message is a source image for temporal resampling.

[0020] In a method or apparatus for encoding image information, the encoder optimization information (SEI) message may further include source image information that the current image included in the same unit as the encoder optimization information (SEI) message is a source image used for temporal resampling.

[0021] In a method or apparatus for encoding image information, the encoder optimization information (SEI) message may further include temporal resampling type information about temporal upsampling and image quantity information about the number of images added or excluded through temporal resampling.

[0022] In a method or apparatus for encoding image information, source image information can be obtained based on time resampling type information and image quantity information.

[0023] In a method or apparatus for encoding image information, source image information can be obtained based on time resampling type information indicating time upsampling and image quantity information with a value greater than 0.

[0024] In a method or apparatus for encoding image information, temporal resampling type information and image quantity information can be obtained based on optimization type information indicating temporal resampling.

[0025] According to one aspect of this disclosure, a method for a bitstream includes: performing encoder optimization; generating a bitstream of image information including an encoder optimization information (EOI) SEI (supplementary enhancement information) message generated based on information about the encoder optimization; and transmitting data for the bitstream, wherein the encoder optimization information SEI message includes optimization type information indicating an optimization type including time resampling, and wherein, based on the optimization type information, it is determined whether a current image included in the same unit as the encoder optimization information SEI message is a source image for time resampling.

[0026] According to one aspect of this disclosure, an apparatus for a bitstream includes: at least one processor configured to perform encoder optimization and generate a bitstream of image information including an encoder optimization information (EOI) SEI (supplementary enhancement information) message generated based on information about the encoder optimization; and a transmitter configured to transmit data for the bitstream, wherein the encoder optimization information SEI message includes optimization type information indicating an optimization type including time resampling, and wherein, based on the optimization type information, it is determined whether a current image included in the same unit as the encoder optimization information SEI message is a source image for time resampling.

[0027] According to one aspect of this disclosure, in a computer-readable storage medium storing a bitstream, the storage medium stores a bitstream of image information including an encoder optimization information (EOI) SEI (supplementary enhancement information) message generated based on information about encoder optimization, wherein the encoder optimization information SEI message includes optimization type information indicating an optimization type including time resampling, and wherein, based on the optimization type information, it is determined whether the current image included in the same unit as the encoder optimization information SEI message is a source image for time resampling.

[0028] The features briefly summarized above are merely exemplary aspects of the detailed description of this disclosure that follows, and do not limit the scope of this disclosure.

[0029] Beneficial effects

[0030] According to this disclosure, source image information can be used to provide information about whether the current image is the original source image.

[0031] According to this disclosure, the reliability of image information delivered by the encoding system can be improved by enabling the encoding system to provide image information that is more similar to the original image.

[0032] According to this disclosure, the encoding speed and efficiency of the encoding system can be improved by enabling the encoding system to identify the original source image more quickly.

[0033] According to this disclosure, the data transmission efficiency of the encoding system can be improved by enabling the encoding system to identify the original source image using less data.

[0034] The effects that can be obtained in this disclosure are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains from the following description. Attached Figure Description

[0035] Figure 1 This is a diagram schematically illustrating a VCM system to which embodiments of the present disclosure can be applied.

[0036] Figure 2 This is a schematic diagram illustrating a VCM pipeline structure to which embodiments of the present disclosure can be applied.

[0037] Figure 3 This is a diagram schematically illustrating an image / video encoder to which embodiments of the present disclosure may be applied.

[0038] Figure 4 This is a diagram schematically illustrating an image / video decoder to which embodiments of the present disclosure may be applied.

[0039] Figure 5 This is a flowchart schematically illustrating a feature / feature map encoding process to which embodiments of the present disclosure may be applied.

[0040] Figure 6 This is a flowchart illustrating, schematically, the feature / feature map decoding process to which embodiments of the present disclosure may be applied.

[0041] Figure 7 This is a diagram illustrating a method for decoding image information according to an embodiment of the present disclosure.

[0042] Figure 8 This is a diagram illustrating a method for encoding image information according to an embodiment of the present disclosure.

[0043] Figure 9 This is a diagram illustrating an example of a content streaming system to which embodiments of the present disclosure can be applied.

[0044] Figure 10 This is a diagram illustrating another example of a content streaming system to which embodiments of the present disclosure can be applied. Detailed Implementation

[0045] In the following description, embodiments of the present disclosure will be detailed with reference to the accompanying drawings to facilitate implementation by those skilled in the art. However, the present disclosure can be implemented in various different forms and is not limited to the embodiments described herein.

[0046] In describing embodiments of this disclosure, detailed explanations of well-known configurations or functions have been omitted where they are deemed to obscure the essential points of this disclosure. Additionally, portions irrelevant to the description of this disclosure have been omitted from the accompanying drawings, and similar reference numerals have been assigned to similar portions.

[0047] In this disclosure, when certain components are described as “connected,” “coupled,” or “linked” to another component, this may include not only direct connections but also indirect connections in which another component may be present in between. Additionally, when a particular component is described as “including” or “having” another component, this means that, unless otherwise expressly stated, it does not exclude other components but may further include additional components.

[0048] In this disclosure, unless otherwise expressly stated, the terms first, second, etc., are used only to distinguish one component from another and do not limit the order or importance of the components. Therefore, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.

[0049] In this disclosure, distinguishable components are described to clearly explain their respective characteristics, and this does not necessarily mean that the components are separate. In other words, multiple components may be integrated into a single hardware or software unit, or a single component may be distributed across multiple hardware or software units. Therefore, such integrated or distributed embodiments are also included within the scope of this disclosure unless they are explicitly described.

[0050] In this disclosure, the components described in the various embodiments are not necessarily essential components, and some components may be optional components. Therefore, embodiments comprising a subset of the components described in one embodiment are also included within the scope of this disclosure. Furthermore, embodiments that include additional components in addition to those described in the various embodiments are also included within the scope of this disclosure.

[0051] This disclosure relates to the encoding and decoding of images, and the terms used herein may have their common meanings as commonly used in the art to which this disclosure pertains, unless these terms are redefined in this disclosure.

[0052] This disclosure can be applied to methods disclosed in Versatile Video Coding (VVC) standards and / or Video Coding for Machines (VCM) standards. Additionally, this disclosure can be applied to methods disclosed in Basic Video Coding (EVC) standards, AOMedia Video 1 (AV1) standards, second-generation Audio Video Coding (AVS2) standards, or next-generation video / image coding standards (e.g., H.267 or H.268).

[0053] This disclosure presents various embodiments related to video / image encoding and decoding, and unless otherwise stated, embodiments may be performed in combination with each other. In this disclosure, "video" can refer to a collection of images in chronological order. "Image" can be information generated by artificial intelligence (AI). Input information used by AI in performing a series of tasks, information generated during information processing, and output information can be used as images. In this disclosure, "picture" generally refers to a unit indicating a single image at a specific point in time, and a slice / tile is a coding unit that constitutes a part of a picture. A picture can consist of at least one slice / tile. Additionally, a slice / tile can include at least one codec tree unit (CTU). A CTU can be partitioned into at least one CU. A tile is a rectangular region existing within a specific tile row and a specific tile column within a picture, and can consist of multiple CTUs. A tile column can be defined as a rectangular region of a CTU and can have the same height as the picture and a width specified by syntax elements signaled from a bitstream portion (such as a picture parameter set). A tile row can be defined as a rectangular region of a CTU and can have the same width as the image and a height specified by a syntax element signaled from a bitstream portion (such as an image parameter set). Tile scanning is a method of predefined ordering of CTUs within a tile partition. Here, CTUs can be ordered according to the raster scan order of CTUs within a tile, and tiles within an image can be ordered sequentially according to the raster scan order of tiles in the image. A slice can include an integer number of complete tiles or an integer number of sequential complete CTU rows within a tile of an image. Slices can be specifically included in a single NAL unit. An image can consist of at least one tile group. A tile group can include at least one tile. A brick can indicate a rectangular region of a CTU row within a tile of an image. A tile can include at least one brick. A brick can indicate a rectangular region of a CTU row within a tile. A tile can be partitioned into multiple bricks, and each brick can include at least one CTU row belonging to the tile. A tile that is not partitioned into multiple bricks can also be considered a brick.

[0054] In this disclosure, a "pixel" or "cell" can refer to the smallest unit that constitutes a picture (or image). Additionally, the term "sample" can be used as the corresponding term for a pixel. A sample can generally indicate a pixel or a pixel value, and can indicate only the pixel / pixel value of the luminance component or only the pixel / pixel value of the chrominance component.

[0055] In embodiments, particularly when applied to VCM, when an image exists consisting of a set of components with different characteristics and meanings, the pixel / pixel value can indicate the pixel / pixel value of the component generated by combining, synthesizing, and analyzing the independent information of each component. For example, in RGB input, it can indicate only the pixel / pixel value of R, only the pixel / pixel value of G, or only the pixel / pixel value of B. For example, it can indicate only the pixel / pixel value of the luminance component synthesized using the R, G, and B components. For example, it can indicate only the pixel / pixel value of the information or image extracted by analyzing the R, G, and B components.

[0056] In this disclosure, "unit" can refer to a basic unit of image processing. A unit may include at least one of a specific region of an image or information associated with that region. A unit may include a luminance block and two chrominance (e.g., Cb, Cr) blocks. Depending on the context, the term "unit" may be used interchangeably with "sample array," "block," "region," etc. Typically, an M×N block may include a set (or array) of samples (or sample arrays) or a set (or array) of transform coefficients consisting of M columns and N rows. In embodiments, particularly when applied to VCM, a unit may refer to a basic unit that includes information for performing a specific task.

[0057] In this disclosure, the term "current block" may refer to one of "current codec block," "current codec unit," "encoding target block," "decoding target block," or "processing target block." When performing prediction, "current block" may refer to "current prediction block" or "prediction target block." When performing transform (inverse transform) / quantization (dequantization), "current block" may refer to "current transform block" or "transform target block." When performing filtering, "current block" may refer to "filter target block."

[0058] Additionally, in this disclosure, unless explicitly stated as a chroma block, "current block" may refer to "the luminance block of the current block". "The chroma block of the current block" can be expressed by explicitly including an explicit description of a chroma block such as "chroma block" or "current chroma block".

[0059] In this disclosure, " / " and "," can mean "and / or". For example, "A / B" and "A, B" can mean "A and / or B". In addition, "A / B / C" and "A, B, C" can mean "at least one of A, B and / or C".

[0060] In this disclosure, "or" can mean "and / or". For example, "A or B" can mean 1) only "A", 2) only "B", or 3) "A and B". Alternatively, in this disclosure, "or" can also mean "additionally or alternatively".

[0061] This disclosure relates to video / image encoding and decoding (VCM) for machines.

[0062] VCM refers to compression techniques that encode / decode a portion of a source image / video or information for use in machine vision. In VCM, the encoding / decoding target can be called a feature. Features can refer to information extracted from the source image / video based on task objectives, requirements, and the surrounding environment. Features can have different information formats than the source image / video, and therefore, feature compression methods and representation formats can also differ from the video source.

[0063] VCMs can be applied to a wide range of applications. For example, in surveillance systems that identify and track objects or people, VCMs can be used to store or transmit object identification information. Additionally, in intelligent transportation or smart mobility systems, VCMs can be used to transmit vehicle location information collected from GPS, sensor information collected from LiDAR, radar, etc., and various vehicle control information to other vehicles or infrastructure. Furthermore, in the field of smart cities, VCMs can be used to perform individual tasks for interconnected sensor nodes or devices.

[0064] This disclosure provides various embodiments regarding feature / feature map encoding and decoding. Unless otherwise specifically stated, embodiments of this disclosure can be implemented individually or in combination of at least two.

[0065] Figure 1 This is a schematic diagram illustrating a VCM system to which embodiments of the present disclosure can be applied.

[0066] refer to Figure 1 The VCM system may include an encoding device 10 and a decoding device 20.

[0067] Encoding device 10 can compress / encode features / feature maps extracted from source images / videos to generate a bitstream, and send the generated bitstream to decoding device 20 via a storage medium or network. Encoding device 10 can also be referred to as a feature encoding device. In a VCM system, features / feature maps can be generated in each hidden layer of a neural network. The size and number of channels of the generated feature map can vary depending on the type of neural network or the location of the hidden layers. In this disclosure, the feature map can be referred to as a feature set.

[0068] The encoding device 10 may include a feature acquisition unit 11, an encoder 12, and a transmitter 13.

[0069] Feature acquirer 11 can obtain features / feature maps from the source image / video. According to one embodiment, feature acquirer 11 can obtain features / feature maps from an external device (e.g., a feature extraction network). In this case, feature acquirer 11 performs a feature receiving interface function. Alternatively, feature acquirer 11 can obtain features / feature maps by executing a neural network (e.g., CNN, DNN, etc.) using the source image / video as input. In this case, feature acquirer 11 performs a feature extraction network function.

[0070] According to an embodiment, the encoding device 10 may further include a source image generator (not shown) for obtaining source images / videos. The source image generator can be implemented using an image sensor, camera module, etc., and the source images / videos can be obtained through a process of capturing, synthesizing, or generating images / videos. In this case, the generated source images / videos can be sent to a feature extraction network and used as input data for extracting features / feature maps.

[0071] Encoder 12 can encode the features / feature maps obtained by feature acquirer 11. Encoder 12 can perform a series of processes, such as prediction, transformation, and quantization, to increase encoding efficiency. The encoded data (encoded feature / feature map information) can be output in the form of a bitstream. The bitstream including the encoded feature / feature map information can be referred to as the VCM bitstream.

[0072] The transmitter 13 can acquire feature / feature map information or data output in bitstream form, and can transmit the acquired information or data to the decoding device 20 in the form of a file or streaming via a digital storage medium or network. Here, the digital storage medium can include various storage media, such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitter 13 can include elements for generating media files with a predetermined file format, or elements for transmitting data via a broadcast / communication network.

[0073] The decoding device 20 can obtain feature / feature map information from the encoding device 10 and reconstruct the feature / feature map based on the obtained information.

[0074] The decoding device 20 may include a receiver 21 and a decoder 22.

[0075] Receiver 21 can receive bit streams from encoding device 10 and obtain feature / feature map information from the received bit streams to send to decoder 22.

[0076] Decoder 22 can decode features / feature maps based on the obtained feature / feature map information. Decoder 22 can perform a series of processes corresponding to the operations of encoder 14, such as dequantization, inverse transform, prediction, etc., to increase decoding efficiency.

[0077] According to an embodiment, the decoding device 20 may further include a task analysis / rendering unit 23.

[0078] The task analysis / rendering unit 23 can perform task analysis based on decoded features / feature maps. Furthermore, the task analysis / rendering unit 23 can render decoded features / feature maps into a form suitable for task execution. Based on the task analysis results and the rendered features / feature maps, various (machine-oriented) tasks can be executed.

[0079] Therefore, a VCM system can encode / decode features extracted from source images / videos based on user and / or machine requests, task objectives, and the surrounding environment, and perform various (machine-oriented) tasks based on the decoded features. A VCM system can also be implemented by extending / redesigning the video / image encoding / decoding system and can execute various encoding / decoding methods defined in the VCM standard.

[0080] Figure 2 This is a schematic diagram illustrating a VCM pipeline structure to which embodiments of the present disclosure can be applied.

[0081] refer to Figure 2 The VCM pipeline 200 may include a first pipeline 210 for encoding / decoding images / videos and a second pipeline 220 for encoding / decoding features / feature maps. In this disclosure, the first pipeline 210 may be referred to as a video codec pipeline, and the second pipeline 220 may be referred to as a feature codec pipeline.

[0082] The first pipeline 210 may include a first stage 211 for encoding the input image / video and a second stage 212 for decoding the encoded image / video to generate a reconstructed image / video. The reconstructed image / video can be used for human viewing, i.e., human vision.

[0083] The second pipeline 220 may include a third stage 221 for extracting features / feature maps from the input image / video, a fourth stage 222 for encoding the extracted features / feature maps, and a fifth stage 223 for decoding the encoded features / feature maps to generate reconstructed features / feature maps. The reconstructed features / feature maps can be used for machine (vision) tasks. Here, a machine (vision) task can refer to a task where a machine consumes images / videos. Machine vision tasks can be applied to service scenarios such as, for example, surveillance, intelligent transportation, smart cities, smart industries, and smart content. According to embodiments, the reconstructed features / feature maps can also be used for human vision.

[0084] According to an embodiment, the features / feature maps encoded in the fourth stage 222 can be sent to the first stage 221 and used to encode the image / video. In this case, an additional bitstream can be generated based on the encoded features / feature maps, and the generated additional bitstream can be sent to the second stage 222 and used to decode the image / video.

[0085] According to an embodiment, the features / feature maps decoded in the fifth stage 223 can be sent to the second stage 222 and used to decode the image / video.

[0086] although Figure 2 The illustration shows a VCM pipeline 200 including a first pipeline 210 and a second pipeline 220, but this is merely exemplary, and embodiments of this disclosure are not limited thereto. For example, the VCM pipeline 200 may include only the second pipeline 220, or the second pipeline 220 may be extended to multiple feature codec pipelines.

[0087] Meanwhile, in the first pipeline 210, the first stage 211 can be executed by an image / video encoder, and the second stage 212 can be executed by an image / video decoder. Additionally, in the second pipeline 220, the third stage 221 can be executed by a VCM encoder (or a feature / feature map encoder), and the fourth stage 222 can be executed by a VCM decoder (or a feature / feature map decoder). The encoder / decoder structure is described in detail below.

[0088] Figure 3 This is a schematic diagram illustrating an image / video encoder to which embodiments of the present disclosure may be applied.

[0089] refer to Figure 3 The image / video encoder 300 may include an image partitioner 310, a predictor 320, a residual processor 330, an entropy encoder 340, an adder 350, a filter 360, and a memory 370. The predictor 320 may include an inter-frame predictor 321 and an intra-frame predictor 322. The residual processor 330 may include a transformer 332, a quantizer 333, a dequantizer 334, and an inverse transformer 335. The residual processor 330 may also include a subtractor 331. The adder 350 may be referred to as a reconstructor or a reconstruction block generator. According to embodiments, the image partitioner 310, predictor 320, residual processor 330, entropy encoder 340, adder 350, and filter 360 may be configured by at least one hardware component (e.g., an encoder chipset or processor). Additionally, the memory 370 may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The aforementioned hardware component may further include the memory 370 as an internal / external component.

[0090] Image partitioner 310 can partition an input image (or picture, frame) input to image / video encoder 300 into at least one processing unit. As an example, a processing unit may be referred to as a codec unit (CU). Codec units can be recursively partitioned from codec tree units (CTUs) or maximum codec units (LCUs) according to a quadtree-binary-trinary tree (QTBTTT) structure. For example, a codec unit can be partitioned into multiple codec units with greater depth based on a quadtree structure, a binary tree structure, and / or a ternary structure. In this case, for example, a quadtree structure can be applied first, followed by a binary tree structure and / or a ternary structure. Alternatively, a binary tree structure can be applied first. The image / video encoding / decoding process according to this disclosure can be performed based on the final codec unit that is no longer partitioned. In this case, the maximum codec unit can be used as the final codec unit based on factors such as encoding / decoding efficiency according to image features, or, if necessary, the codec unit can be recursively partitioned into deeper codec units, and the optimally sized codec unit can be used as the final codec unit. Here, the encoding / decoding process may include processes such as prediction, transformation, and reconstruction as described later. As another example, the processing unit may further include a prediction unit (PU) or a transformation unit (TU). In this case, the prediction unit and the transformation unit may be partitioned or divided from the final encoding / decoding unit described above, respectively. The prediction unit may be a unit for predicting samples, and the transformation unit may be a unit for deriving transform coefficients and / or a unit for deriving residual signals from transform coefficients.

[0091] In some cases, a unit can be used interchangeably with terms such as block, region, etc. Typically, an MxN block can refer to a set of transform coefficients or a set of samples consisting of M columns and N rows. Samples can typically indicate pixels or pixel values, or they can indicate only pixel / pixel values ​​of the luminance component, or only pixel / pixel values ​​of the chrominance component. Samples can be used as items corresponding to pixels or cells.

[0092] The image / video encoder 300 generates a residual signal (residual block, residual sample array) by subtracting the prediction signal (prediction block, prediction sample array) output from the inter-frame predictor 321 or intra-frame predictor 322 from the input image signal (original block, original sample array), and the generated residual signal is sent to the converter 332. In this case, as shown, the unit that subtracts the prediction signal (prediction block, prediction sample array) from the input image signal (original block, original sample array) within the image / video encoder 300 can be called the subtractor 331. The predictor can perform prediction on the block to be processed (hereinafter referred to as the current block) and generate a prediction block that includes the prediction samples of the current block. The predictor can determine whether intra-frame prediction or inter-frame prediction is applied to the current block or the unit of the CU. The predictor can generate various prediction-related information, such as prediction mode information, and send it to the entropy encoder 340. The prediction-related information can be encoded by the entropy encoder 340 and output in the form of a bitstream.

[0093] Intra-predictor 322 can predict the current block by referencing samples within the current image. In this case, the reference samples can be located in a neighboring region of the current block, or they can be located further away depending on the prediction mode. In intra-prediction, the prediction modes can include multiple non-directional modes and multiple directional modes. Non-directional modes can include, for example, DC modes and planar modes. Directional modes can include, for example, 33 or 65 directional prediction modes depending on the granularity of the prediction orientation. However, this is just an example, and more or fewer directional prediction modes can be used depending on the configuration. Intra-predictor 322 can also determine the prediction mode applied to the current block by using prediction modes applied to neighboring blocks.

[0094] Inter-frame predictor 321 can derive the predicted block of the current block based on a reference block (reference sample array) specified by motion vectors on a reference image. In this case, to reduce the amount of motion information transmitted in the inter-frame prediction mode, motion information can be predicted at the block, sub-block, or sample level based on the correlation of motion information between neighboring blocks and the current block. Motion information may include motion vectors and reference image indices. Motion information may further include information about the inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In inter-frame prediction, neighboring blocks may include spatially neighboring blocks existing within the current image and temporally neighboring blocks existing in the reference image. The reference image including the reference block and the reference image including the temporally neighboring block may be the same or different. The temporally neighboring block may be referred to as a co-located reference block or co-located CU (colCU), and the reference image including the temporally neighboring block may be referred to as a co-located image (colPic). For example, inter-frame predictor 321 can construct a motion information candidate list based on neighboring blocks and generate information indicating which candidate is used to derive the motion vector and / or reference image index of the current block. Inter-frame prediction can be performed based on various prediction modes, and for example, in skip mode and merge mode, the inter-frame predictor 321 can use motion information of neighboring blocks as motion information of the current block. In skip mode, unlike merge mode, residual signals may not be sent. In motion vector prediction (MVP) mode, motion vectors of neighboring blocks can be used as motion vector predictors, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0095] Predictor 320 can generate prediction signals based on various prediction methods. For example, the predictor can apply intra-frame prediction or inter-frame prediction for the prediction of a block, and can also apply both intra-frame prediction and inter-frame prediction simultaneously. This can be referred to as combined inter-frame and intra-frame prediction (CIIP). Alternatively, the predictor can be based on an intra-block copy (IBC) prediction mode or a palette mode for the prediction of a block. The IBC prediction mode or palette mode can be used for content image / video codecs, such as Screen Content Codec (SCC). IBC essentially performs prediction within the current frame, but because it derives a reference block within the current frame, it can be similar to inter-frame prediction operation. In other words, IBC can use at least one of the inter-frame prediction methods described in this disclosure. The palette mode can be considered as an example of intra-frame codec or intra-frame prediction. When a palette mode is applied, sample values ​​within the frame can be signaled based on information associated with the palette table and palette index.

[0096] The predicted signal generated by predictor 320 can be used to generate a reconstructed signal or a residual signal. Transformer 332 can generate transform coefficients by applying a transform method to the residual signal. For example, the transform method can include at least one of Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), Karhunen-Loeve Transform (KLT), Graphical Based Transform (GBT), or Conditional Nonlinear Transform (CNT). Here, GBT refers to the transform obtained from a graph when the relationship information between pixels is indicated as a graph. CNT refers to the transform obtained based on the predicted signal generated using all previously reconstructed pixels. Furthermore, the transform process can be applied to pixel blocks of the same square size or non-square, variable-size blocks.

[0097] The quantizer 333 quantizes the transform coefficients and sends them to the entropy encoder 340, which encodes the quantized signal (information about the quantized transform coefficients) and outputs it as a bitstream. This information about the quantized transform coefficients can be referred to as residual information. The quantizer 333 can reorder the block-like quantized transform coefficients in a one-dimensional vector form based on the coefficient scan order, and can generate information about the quantized transform coefficients based on this one-dimensional vector form. The entropy encoder 340 can perform various encoding methods, such as Exponential Columbus, Context Adaptive Variable Length Codec (CAVLC), Context Adaptive Binary Arithmetic Codec (CABAC), etc. The entropy encoder 340 can encode the quantized transform coefficients together with or separately from information necessary for video / image reconstruction (e.g., values ​​of syntax elements). The encoded information (e.g., encoded video / image information) can be sent or stored as a bitstream in a Network Abstraction Layer (NAL) unit. The image / video information may further include information about various parameter sets, such as Adaptation Parameter Set (APS), Picture Parameter Set (PPS), Sequence Parameter Set (SPS), or Video Parameter Set (VPS). Additionally, the video / image information may further include general constraint information. Furthermore, the image / video information may further include methods for generating and using encoded information, their purpose, etc. In this disclosure, information and / or syntax elements sent / signed from the image / video encoder to the image / video decoder may be included in the image / video information. The image / video information may be encoded using the encoding process described above and included in the bitstream. The bitstream may be transmitted over a network or stored in a digital storage medium. Here, the network may include broadcast networks and / or communication networks, etc., and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmitter (not shown) for transmission and / or a storage unit (not shown) for storing the signal output from the entropy encoder 340 may be configured as an internal / external element of the image / video encoder 300, or the transmitter may be included in the entropy encoder 340.

[0098] The quantization transform coefficients output from quantizer 333 can be used to generate a prediction signal. For example, the residual signal (residual block or residual sample) can be reconstructed by applying dequantization and inverse transform to the quantization transform coefficients through dequantizer 334 and inverse transform 335. Adder 350 can generate a reconstructed signal (reconstructed image, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from inter-frame predictor 321 or intra-frame predictor 322. When there is no residual for the processing target block, such as when a skip mode is applied, the prediction block can be used as a reconstructed block. Adder 350 can be referred to as a reconstructor or reconstructed block generator. The generated reconstructed signal can be used for intra-frame prediction of the next processing target block within the current image, and, as described later, can also be used for inter-frame prediction of the next image by filtering.

[0099] Simultaneously, luminance mapping with chroma scaling can be applied during image encoding and / or reconstruction.

[0100] Filter 360 can apply filtering to the reconstructed signal to enhance subjective / objective image quality. For example, filter 360 can apply various filtering methods to the reconstructed image to generate a modified reconstructed image, and the modified reconstructed image can be stored in memory 370, specifically in the DPB of memory 370. Various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filtering, bilateral filtering, etc. Filter 360 can generate various filtering-related information and send it to entropy encoder 340. The filtering-related information can be encoded by entropy encoder 340 and output as a bitstream.

[0101] The modified reconstructed image sent to memory 370 can be used as a reference image in inter-frame predictor 321. This avoids prediction mismatch between the encoder and decoder sides and improves coding efficiency.

[0102] The DPB of memory 370 can store modified reconstructed images for use as reference images in inter-frame predictor 321. Memory 370 can store motion information of blocks in which motion information within the current image and / or motion information of blocks within reconstructed images is derived (or encoded). The stored motion information can be sent to inter-frame predictor 321 as motion information for spatially or temporally neighboring blocks. Memory 370 can store reconstructed samples of reconstructed blocks in the current image and send the stored reconstructed samples to intra-frame predictor 322.

[0103] At the same time, the VCM encoder (or feature / feature map encoder) can have essentially the same reference Figure 3The image / video encoder 300 described has the same / similar structure as the image / video encoder 300 because it performs a series of processes such as prediction, transformation, quantization, etc., to encode features / feature maps. However, the VCM encoder differs from the image / video encoder 300 because it targets the features / feature maps to be encoded, and therefore, its name in each unit (or component) (e.g., image partitioner 310, etc.) and its specific operational details derived from the image / video encoder 300 may differ. The specific operational details of the VCM encoder will be described in detail later.

[0104] Figure 4 This diagram schematically illustrates an image / video decoder to which embodiments of the present disclosure may be applied.

[0105] refer to Figure 4 The image / video decoder 400 may include an entropy decoder 410, a residual processor 420, a predictor 430, an adder 440, a filter 450, and a memory 460. The predictor 430 may include an inter-frame predictor 431 and an intra-frame predictor 432. The residual processor 420 may include a dequantizer 421 and an inverse transformer 422. According to an embodiment, the entropy decoder 410, residual processor 420, predictor 430, adder 440, and filter 450 may be configured by a single hardware component (e.g., a decoder chipset or processor). Additionally, the memory 460 may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may also include the memory 460 as an internal / external component.

[0106] When the input includes a bitstream containing video / image information, the image / video decoder 400 can respond to... Figure 3 The image / video encoder 300 reconstructs the image / video through a process of processing image / video information. For example, the image / video decoder 400 can derive units / blocks based on block partitioning information obtained from the bitstream. The image / video decoder 400 can perform decoding using processing units applied in the image / video encoder. Therefore, the decoding processing unit can be, for example, an encoding / decoding unit, and the encoding / decoding unit can be partitioned according to a quadtree structure, binary tree structure, and / or ternary tree structure from the encoding / decoding tree unit or the maximum encoding / decoding unit. At least one transform unit can be derived from the encoding / decoding unit. Furthermore, the reconstructed image signal decoded and output by the image / video decoder 400 can be played by a playback device.

[0107] The image / video decoder 400 can be used in bitstream form from... Figure 3The encoder in the process receives the signal output and can decode the received signal using the entropy decoder 410. For example, the entropy decoder 410 can parse the bitstream to derive the information necessary for image reconstruction (or picture reconstruction) (e.g., video / image information). The video / image information may further include information about various parameter sets, such as adaptation parameter sets (APS), picture parameter sets (PPS), sequence parameter sets (SPS), video parameter sets (VPS), etc. Additionally, the video / image information may further include general constraint information. Furthermore, the image / video information may include the method of generating the decoding information, its usage, purpose, etc. The image / video decoder 400 can further decode the picture based on information about the parameter sets and / or general constraint information. The signal notification / received information and / or syntax elements can be decoded and obtained from the bitstream through the decoding process. For example, the entropy decoder 410 can decode the information in the bitstream based on encoding and decoding methods such as exponential Golomb coding, CAVLC, or CABAC, and can output the values ​​of the syntax elements necessary for image reconstruction and the quantized values ​​of the transform coefficients associated with the residuals. More specifically, the CABAC entropy decoding method can receive bins corresponding to each syntax element in the bitstream, determine a context model by using information about the target syntax element, the decoding information of neighboring and target blocks, or information about symbols / bins decoded in previous steps, predict the probability of bin occurrence based on the determined context model, and perform arithmetic decoding of the bins to generate symbols corresponding to the values ​​of each syntax element. In this case, the CABAC entropy decoding method can update the context model after determining it by using information about decoded symbols / bins of the context model for the next symbol / bin. In the information decoded by the entropy decoder 410, prediction-related information can be provided to the predictors (inter-frame predictor 432 and intra-frame predictor 431), and the residual values ​​(i.e., quantization transform coefficients and related parameter information) entropied by the entropy decoder 410 can be input to the residual processor 420. The residual processor 420 can derive residual signals (residual blocks, residual samples, residual sample arrays). Additionally, in the information decoded by the entropy decoder 410, filtering-related information can be provided to the filter 450. Meanwhile, the receiver (not shown) that receives the signal output from the image / video encoder can be additionally configured as an internal / external component of the image / video decoder 400, or the receiver can be a component of the entropy decoder 410. Furthermore, the image / video decoder according to this disclosure can also be referred to as an image / video decoding device, and the image / video decoder can be divided into an information decoder (image / video information decoder) and / or a sample decoder (image / video sample decoder).In this case, the information decoder may include an entropy decoder 410, and the sample decoder may include at least one of a dequantizer 321, an inverse transformer 322, an adder 440, a filter 450, a memory 460, an inter-frame predictor 432, and an intra-frame predictor 431.

[0108] Dequantizer 421 can dequantize the quantized transform coefficients and the output transform coefficients. Dequantizer 421 can reorder the quantized transform coefficients in the form of two-dimensional blocks. In this case, the reordering can be performed based on the coefficient scan order performed in the image / video encoder. Dequantizer 321 can perform dequantization on the quantized transform coefficients using quantization parameters (i.e., quantization step size information) and obtain the transform coefficients.

[0109] The inverse transformer 422 can perform an inverse transformation on the transform coefficients to obtain the residual signal (residual block, residual sample array).

[0110] Predictor 430 can perform prediction for the current block and generate a prediction block that includes prediction samples of the current block. The predictor can determine whether intra-frame prediction or inter-frame prediction is applied to the current block based on prediction-related information output from entropy decoder 410, and can determine a specific intra-frame / inter-frame prediction mode (prediction method).

[0111] Predictor 420 can generate prediction signals based on various prediction methods. For example, the predictor can apply not only intra-frame prediction or inter-frame prediction, but also both intra-frame and inter-frame prediction simultaneously for the prediction of a block. This can be referred to as combined intra-frame and inter-frame prediction (CIIP). Alternatively, the predictor can be based on an intra-block copy (IBC) prediction mode or a palette mode for block prediction. The IBC prediction mode or palette mode can be used for content image / video encoding / decoding in games, such as Screen Content Codec (SCC). IBC essentially performs prediction within the current frame, but it can be performed similarly to inter-frame prediction because it derives a reference block within the current frame. In other words, IBC can use at least one of the inter-frame prediction techniques described in this document. The palette mode can be considered an example of intra-frame encoding / decoding or intra-frame prediction. When a palette mode is applied, information related to the palette table and palette index can be included in the image / video information and signaled.

[0112] The intra-predictor 431 can predict the current block by referencing samples within the current image. The reference samples can be located in the neighborhood of the current block or positioned far from the current block based on the prediction pattern. In intra-prediction, the prediction pattern can include multiple non-directional patterns and multiple directional patterns. The intra-predictor 431 can determine the prediction pattern applied to the current block by using prediction patterns applied to neighboring blocks.

[0113] Inter-frame predictor 432 can derive the predicted block of the current block based on a reference block (reference sample array) specified by motion vectors on a reference image. In this case, to reduce the amount of motion information transmitted in the inter-frame prediction mode, motion information can be predicted at the block, sub-block, or sample level based on the correlation of motion information between neighboring blocks and the current block. Motion information may include motion vectors and reference image indices. Motion information may further include information about the inter-frame prediction orientation (i.e., L0 prediction, L1 prediction, Bi prediction, etc.). In inter-frame prediction, neighboring blocks may include spatially neighboring blocks within the current image and temporally neighboring blocks in the reference image. For example, inter-frame predictor 432 can construct a motion information candidate list based on neighboring blocks and derive the motion vector and / or reference image index of the current block based on received candidate selection information. Inter-frame prediction can be performed based on various prediction modes, and prediction-related information may include information indicating the inter-frame prediction mode of the current block.

[0114] Adder 440 can generate a reconstruction signal (reconstructed image, reconstruction block, reconstruction sample array) by adding the obtained residual signal to the prediction signal (prediction block, prediction sample array) output from the predictors (including inter-frame predictor 432 and / or intra-frame predictor 431). When there is no residual for the processing target block, such as when a skip mode is applied, the prediction block can be used as the reconstruction block.

[0115] Adder 440 can be referred to as a reconstructor or reconstruction block generator. The generated reconstructed signal can be used for intra-frame prediction of the next processing target block in the current image, or, as described later, can be filtered out, or used for inter-frame prediction of the next image.

[0116] Simultaneously, luminance mapping with chroma scaling can be applied during image decoding.

[0117] Filter 450 can apply filtering to the reconstructed signal to enhance subjective / objective image quality. For example, filter 450 can apply various filtering methods to the reconstructed image to generate a modified reconstructed image, and can send the modified reconstructed image to memory 460, specifically to the DPB in memory 460. Various filtering methods may include, for example, unblocking filtering, adaptive sample shifting, adaptive loop filtering, bilateral filtering, etc.

[0118] The (modified) reconstructed image stored in the DPB of memory 460 can be used as a reference image in inter-frame predictor 432. Memory 460 can store motion information of blocks in which motion information within the current image is derived (or decoded) and / or the motion information of blocks in the reconstructed image is sent to inter-frame predictor 432 as motion information for spatially or temporally neighboring blocks. Memory 460 can store reconstructed samples of reconstructed blocks in the current image and send them to intra-frame predictor 431.

[0119] Meanwhile, the VCM decoder (or feature / feature map decoder) can have the same features as described above (see reference). Figure 4 The image / video decoder 400 described below has the same / similar structure as the image / video decoder 400 because it performs a series of processes, such as prediction, inverse transform, dequantization, etc., to decode features / feature maps. However, the VCM decoder differs from the image / video decoder 400 because it targets the features / feature maps used for decoding, and therefore, its name in each unit (or component) (e.g., DPB, etc.) and its specific operational details from those of the image / video decoder 400 may differ. The operations of the VCM decoder can correspond to the operations of the VCM encoder, and their specific operational details will be described in detail later.

[0120] Figure 5 This is a flowchart illustrating a feature / feature map encoding process to which embodiments of the present disclosure can be applied.

[0121] refer to Figure 5 The feature / feature map encoding process may include a prediction process S510, a residual processing process S520, and an information encoding process S530.

[0122] The prediction process S510 can be referenced above. Figure 3 The predictor 320 is described and executed.

[0123] Specifically, the intra predictor 322 can predict the current block (i.e., the currently encoded set of feature elements) by referencing feature elements in the current feature / feature map. Intra-prediction can be performed based on the spatial similarity of feature elements in the configured feature / feature map. For example, it can be estimated that feature elements included in the same region of interest (RoI) within an image / video have similar data distribution characteristics. Therefore, the intra predictor 322 can predict the current block by referencing pre-reconstructed feature elements in the RoI that includes the current block. In this case, the referenced feature elements can be located near the current block or can be located separately from the current block depending on the prediction mode. The intra-prediction modes used for feature / feature map encoding can include multiple non-directional prediction modes and multiple directional prediction modes. Non-directional prediction modes can include, for example, prediction modes corresponding to DC mode and planar mode in the image / video encoding process. In addition, directional modes can include, for example, prediction modes corresponding to 33 directional modes or 65 directional modes in the image / video encoding process. However, this is only an example, and according to embodiments, the type and number of intra-prediction modes can be configured / changed in various ways.

[0124] Inter-frame predictor 321 can predict the current block based on a reference block (i.e., a set of reference feature elements) specified by motion information on a reference feature / feature map. Inter-frame prediction can be performed based on the temporal similarity of feature elements in the configuration feature / feature map. For example, temporally consecutive features may have similar data distribution characteristics. Therefore, inter-frame predictor 321 can predict the current block by referencing pre-reconstructed feature elements of the current feature and temporally adjacent features. In this case, the motion information used to specify the reference feature elements may include motion vectors and reference feature / feature map indices. The motion information may further include information related to inter-frame prediction orientation (e.g., L0 prediction, L1 prediction, Bi prediction, etc.). For inter-frame prediction, neighboring blocks may include spatially neighboring blocks existing in the current feature / feature map and temporally neighboring blocks existing in the reference feature / feature map. The reference feature / feature map including the reference block and the reference feature / feature map including the temporally neighboring block may be the same or different. The temporally neighboring block may be referred to as a co-located reference block, etc., and the reference feature / feature map including the temporally neighboring block may be referred to as a co-located feature / feature map. Inter-frame predictor 321 can configure a candidate list of motion information based on neighboring blocks and generate information indicating which candidate is used to derive the motion vector and / or reference feature / feature map index of the current block. Inter-frame prediction can be performed based on various prediction modes, and for example, for skip mode and merge mode, inter-frame predictor 321 can use the motion information of neighboring blocks as the motion information of the current block. For skip mode, unlike merge mode, residual signals may not be sent. For motion vector prediction (MVP) mode, motion vectors of neighboring blocks can be used as motion vector predictors, and the motion vector of the current block can be indicated by signaling the motion vector difference. In addition to the intra-frame prediction and inter-frame prediction described above, predictor 320 can also generate prediction signals based on various prediction methods.

[0125] The prediction signal generated by predictor 320 can be used to generate the residual signal (residual block, residual eigenvalue) S520. This can be seen from the above reference... Figure 3 The described residual processor 330 executes the residual processing procedure S520. Furthermore, the (quantization) transform coefficients can be generated through a transform and / or quantization process for the residual signal, and the entropy encoder 340 can encode the information associated with the (quantization) transform coefficients into residual information in the bitstream S530. In addition to the residual information in the bitstream, the entropy encoder 340 can also encode information necessary for feature / feature map reconstruction, such as prediction information (e.g., prediction pattern information, motion information, etc.).

[0126] Meanwhile, the feature / feature map encoding process may further include a process for generating a reconstructed feature / feature map for the current feature / feature map, and a process for applying in-loop filtering to the reconstructed feature / feature map (optional), as well as a process S530 for encoding information for feature / feature map reconstruction (e.g., prediction information, residual information, partitioning information, etc.) and outputting it in the form of a bit stream.

[0127] The VCM encoder can derive (modified) residual features from the quantization transform coefficients through dequantization and inverse transform, and can generate reconstructed features / feature maps based on the predicted features as the output of S510 and the (modified) residual features. The reconstructed features / feature maps generated in this way can be identical to those generated by the VCM decoder. When an in-loop filtering process is performed on the reconstructed features / feature maps, a modified reconstructed features / feature maps can be generated through this in-loop filtering process. The modified reconstructed features / feature maps can be stored in the decoded feature buffer (DFB) or memory, and then used as reference features / feature maps during feature / feature map prediction. Furthermore, (in-loop) filtering-related information (parameters) can be encoded and output as a bitstream. The in-loop filtering process removes noise that may occur during feature / feature map encoding and decoding, and improves the performance of feature / feature map-based tasks. Additionally, the in-loop filtering process can be performed on both the encoder and decoder sides to ensure the recognition of prediction results, improve the reliability of feature / feature map encoding and decoding, and reduce the amount of data transmitted for feature / feature map encoding and decoding.

[0128] Figure 6 This is a flowchart illustrating a feature / feature map decoding process to which embodiments of the present disclosure can be applied.

[0129] refer to Figure 6The feature / feature map decoding process may include an image / video information acquisition process S610, feature / feature map reconstruction processes S620 to S640, and an intra-loop filtering process S650 for reconstructing the feature / feature map. The feature / feature map reconstruction process can be performed based on the prediction signal and residual signal obtained through inter-frame / intra-frame prediction S620, residual processing S630, and dequantization and inverse transform processes for the quantization transform coefficients described in this disclosure. A modified reconstructed feature / feature map can be generated through the intra-loop filtering process for reconstructing the feature / feature map, and the modified reconstructed feature / feature map can be output as a decoded feature / feature map. The decoded feature / feature map can be stored in a decoded feature buffer (DFB) or memory, and then used as a reference feature / feature map in the inter-frame prediction process when decoding the feature / feature map. In some cases, the above-described intra-loop filtering process may be omitted. In this case, the reconstructed features / feature maps can be output as decoded features / feature maps and stored in the decoded feature buffer (DFB) or memory, and then used as reference features / feature maps in the inter-frame prediction process when decoding the features / feature maps.

[0130] The following describes the SEI messages that are relevant to this disclosure.

[0131] Table 1 shows an example of the syntax for the encoder optimization information (EOI) SEI (supplementary enhancement information) message according to an embodiment.

[0132] [Table 1]

[0133] An example is described of the semantics of the encoder optimization information SEI message according to an embodiment.

[0134] The Encoder Optimization Information (SEI) message is used to indicate whether the video has been optimized for human viewing or machine analysis, and what types of optimizations have been applied in preprocessing or encoding.

[0135] An eoi_cancel_flag value of 1 indicates that the persistence of any encoder optimization information SEI messages included in any previous PUs in the output order is canceled. An eoi_cancel_flag value of 0 indicates that information about optimizations already applied in preprocessing or encoding follows.

[0136] `eoi_persistence_flag` specifies the persistence of the optimization information provided in this SEI message. `eoi_persistence_flag` equal to 0 specifies that the optimization information applies only to the current image. `eoi_persistence_flag` equal to 1 specifies that the optimization information applies to the current image of the current layer in output order, as well as all subsequent images, until one or more of the following conditions are true: - A new CLVS begins in the current layer.

[0137] - End of bitstream.

[0138] - Outputs the image in the current layer associated with the encoder optimization information (SEI) message, following the current images in the output order.

[0139] `eoi_for_human_viewing_idc` equal to 3 indicates that the optimizations applied are intended for human viewing. `eoi_for_human_viewing_idc` equal to 2 indicates that the video is suitable for human viewing but not specifically optimized for it. `eoi_for_human_viewing_idc` equal to 1 indicates that the video is not suitable for human viewing. `eoi_for_human_viewing_idc` equal to 0 indicates that whether the video is suitable for human viewing is unknown.

[0140] `eoi_for_machine_analysis_idc` equal to 3 indicates that the purpose of the applied optimization includes machine analysis. `eoi_for_machine_analysis_idc` equal to 2 indicates that the video is suitable but not specifically optimized for machine analysis. `eoi_for_machine_analysis_idc` equal to 1 indicates that the video is not suitable for machine analysis. `eoi_for_machine_analysis_idc` equal to 0 indicates that whether the video is suitable for machine analysis is unknown.

[0141] The requirement for bitstream consistency is that the values ​​of eoi_for_human_viewing_idc and eoi_for_machine_analysis_idc should not both be equal to 1.

[0142] `eoi_type` indicates the type of optimization method specified in Table 2, where `(eoi_type &bitMask)` not equal to 0 indicates that an optimization type with the bitmask value in Table 2 has been applied. When `eoi_type` is greater than 0 and `(eoi_type &bitMask)` is equal to 0, an optimization type with the bitmask value has not been applied. When `eoi_type` is equal to 0, an optimization as determined by the application has been used.

[0143] [Table 2]

[0144] The variables `EoiObjectBasedFlag`, `EoiTemporalResamplingFlag`, `EoiSpatialResamplingFlag`, `EoiTemporalQualityFlag`, `EoiSpatialQualityFlag`, and `EoiPrivacyProtectionFlag`, which specify whether `eoi_type` indicates the type of optimization, including object-based optimization, temporal resampling optimization, spatial resampling optimization, temporal quality optimization, spatial quality optimization, and privacy protection optimization, can be exported as follows: [Formula 1]

[0145] Note that, for example, when some highest time sublayers are encoded using a coarse quantization that leverages the annoyance of human viewer perception quality fluctuations without impairing machine task performance, eoi_for_human_viewing_flag and eoi_for_machine_analysis_flag can be set to 0 and 1 respectively, and eoi_type can be set to a value that makes EoiTemporalQualityFlag equal to 1.

[0146] When eoi_persistence_flag equals 0, the requirements for bitstream consistency are that EoiTemporalResampingFlag should be equal to 0 and EoiTemporalQualityFlag should be equal to 0.

[0147] When present, `eoi_object_based_idc` indicates the type of object-based optimization as specified in Table 3, where `(eoi_object_based_idc & bitMask)` not equal to 0 indicates that the object-based optimization type associated with the bitmask value in Table 3 has been applied. When `eoi_object_based_idc` is greater than 0 and `(eoi_object_based_idc & bitMask)` is equal to 0, the object-based optimization type associated with the bitmask value has not been applied. When `eoi_object_based_idc` is equal to 0, the object-based optimization of the defined type has been applied. In a bitstream conforming to this disclosure, the value of `eoi_object_based_idc` should be in the range of 0 to 7 (inclusive). Values ​​of 8 to 65,535 (inclusive) for `eoi_object_based_idc` are reserved and should not exist in a bitstream conforming to this disclosure. When the value of eoi_object_based_idc is in the range of 8 to 65,535 (inclusive), decoders conforming to this disclosure should ignore eoi_object_based_idc.

[0148] [Table 3]

[0149] When `eoi_temporal_resaming_type_flag` is equal to 0, the time-based resampling optimization is a subsampling operation. When `eoi_temporal_resaming_type_flag` is equal to 1, the time-based resampling optimization is an upsampling operation.

[0150] A value greater than 0 for `eoi_num_int_pics` indicates that the count of pictures excluded by the encoding system between each pair of encoded pictures in output order (when `eoi_temporal_resaming_type_flag` equals 0) or added between each pair of source pictures for encoding (when `eoi_temporal_resaming_type_flag` equals 1) is constant within the persistence of this SEI message. When `eoi_temporal_resaming_type_flag` equals 0 and `eoi_num_int_pics` is greater than 0, `eoi_num_int_pics` specifies the count of pictures excluded by the encoding system between each pair of encoded pictures in output order. When `eoi_temporal_resaming_type_flag` equals 1 and `eoi_num_int_pics` is greater than 0, `eoi_num_int_pics` specifies the count of pictures added by the encoding system between each pair of source pictures for encoding.

[0151] An eoi_num_int_pics equal to 0 indicates that the count of pictures excluded between each pair of encoded pictures in output order (when eoi_temporal_resampling_type_flag equals 0) or added between each pair of source pictures for encoding (when eoi_temporal_resampling_type_flag equals 1) within the persistence of this SEI message is unknown or variable.

[0152] The value of eoi_num_int_pics should be in the range of 0 to 63 (inclusive).

[0153] An eoi_orig_pic_dimensions_flag value of 1 indicates that the eoi_orig_pic_width and eoi_orig_pic_height syntax elements exist. An eoi_orig_pic_dimensions_flag value of 0 indicates that eoi_orig_pic_width and eoi_orig_pic_height do not exist.

[0154] When present, eoi_orig_pic_width and eoi_orig_pic_height indicate the width and height of the original source image in units of luminance samples, respectively.

[0155] When `eoi_spatial_resampling_type_flag` is equal to 0, spatial resampling optimization is a subsampling operation. When `eoi_spatial_resampling_type_flag` is equal to 1, spatial resampling optimization is an upsampling operation.

[0156] When present, eoi_privacy_protection_method_idc indicates the method / algorithm used for applying privacy protection optimizations, as specified in Table 4.

[0157] [Table 4]

[0158] When present, `eoi_privacy_info_type` indicates the type of protected information as specified in Table 5, where `eoi_privacy_info_type` greater than 0 and `(eoi_privacy_info_type& bitMask)` not equal to 0 indicates that the information type with the bitmask value in Table 5 is protected. When `eoi_privacy_info_type` equals 0, the application-defined information type is protected. In a bitstream conforming to this disclosure, the value of `eoi_privacy_info_type` should be in the range of 0 to 7 (inclusive). Values ​​of 8 to 255 (inclusive) for `eoi_privacy_info_type` are reserved for future use and should not exist in a bitstream conforming to this disclosure. When the value of `eoi_privacy_info_type` is in the range of 8 to 255 (inclusive), a decoder conforming to this disclosure should ignore `eoi_privacy_info_type`.

[0159] [Table 5]

[0160] The traditional design of Encoder Optimization Information (EOI) SEI messages includes signaling information indicating the application's temporal upsampling. However, EOI SEI messages do not include signaling information indicating the position of the source image within the bitstream. This makes it difficult (if not impossible) for the decoder / application to distinguish between the source image and the image generated from the temporal upsampling process. Such distinction may be useful / necessary when the decoder / application needs to discard / remove images from the bitstream, and naturally, they would choose to discard the generated image rather than the source image.

[0161] Each invention can be applied individually or in combination.

[0162] - Specify a constraint such that when the EOI SEI message includes temporal upsampling optimization as one of its optimization types, the first image in the decoding order in CLVS should be the original source image.

[0163] - Specify a constraint such that when the EOI SEI message includes temporal upsampling optimization as one of its optimization types, the first image in CLVS in output order should be the original source image.

[0164] - Specifies a syntax element that, if equal to 1, indicates that the first image in the decoding order in CLVS should be the original source image; otherwise (i.e., if it is equal to 0), it means there is no such indication (i.e., whether the first image in the decoding order in CLVS is the source image or whether the added image is unknown).

[0165] - Specifies a syntax element that, if equal to 1, indicates that the image present in the same access unit as the SEI message is the original source image; otherwise (i.e., if it is equal to 0), it means that there is no such indication (i.e., whether the first image in the CLVS in decoding order is the source image or whether the added image is unknown).

[0166] - Add a syntax element that indicates the maximum TemporalId value of the time sublayer to which the original source image belongs.

[0167] - Add a syntax element indicating whether the original source image belongs to each time sublayer.

[0168] - Specify a constraint that when an EOI SEI message includes temporal upsampling optimization as one of its optimization types, it should exist in the access unit containing the source image used for temporal upsampling optimization.

[0169] An example of the semantics of the encoder optimization information (SEI) message according to the embodiment is described. Semantics different from those previously described for the encoder optimization information (SEI) message semantics are described, and descriptions of semantics identical to those previously described for the encoder optimization information (SEI) message semantics are replaced with the descriptions of the previously described encoder optimization information (SEI) message semantics. Specifically, in the previously described encoder optimization information (SEI) message semantics, the semantics concerning the temporally upsampled original source image are described, and descriptions of other semantics are replaced with the descriptions of the previously described encoder optimization information (SEI) message semantics.

[0170] A value greater than 0 for `eoi_num_int_pics` indicates that the count of pictures excluded by the encoding system between each pair of encoded pictures in output order (when `eoi_temporal_resaming_type_flag` equals 0) or added between each pair of source pictures for encoding (when `eoi_temporal_resaming_type_flag` equals 1) is constant within the persistence of this SEI message. When `eoi_temporal_resaming_type_flag` equals 0 and `eoi_num_int_pics` is greater than 0, `eoi_num_int_pics` specifies the count of pictures excluded by the encoding system between each pair of encoded pictures in output order. When `eoi_temporal_resaming_type_flag` equals 1 and `eoi_num_int_pics` is greater than 0, `eoi_num_int_pics` specifies the count of pictures added by the encoding system between each pair of source pictures for encoding.

[0171] An eoi_num_int_pics equal to 0 indicates that the count of pictures excluded between each pair of encoded pictures in output order (when eoi_temporal_resampling_type_flag equals 0) or added between each pair of source pictures for encoding (when eoi_temporal_resampling_type_flag equals 1) within the persistence of this SEI message is unknown or variable.

[0172] The value of eoi_num_int_pics should be in the range of 0 to 63 (inclusive).

[0173] When EoiTemporalResamplingFlag equals 1, eoi_temporal_resampling_type_flag equals 1, and eoi_num_int_pics is greater than 0, the requirement for bitstream consistency is that the first image in the decoding order in CLVS should be the source image.

[0174] An example of the semantics of the encoder optimization information (SEI) message according to the embodiment is described. Semantics different from those previously described for the encoder optimization information (SEI) message semantics are described, and descriptions of semantics identical to those previously described for the encoder optimization information (SEI) message semantics are replaced with the descriptions of the previously described encoder optimization information (SEI) message semantics. Specifically, in the previously described encoder optimization information (SEI) message semantics, the semantics concerning the temporally upsampled original source image are described, and descriptions of other semantics are replaced with the descriptions of the previously described encoder optimization information (SEI) message semantics.

[0175] A value greater than 0 for `eoi_num_int_pics` indicates that the count of pictures excluded by the encoding system between each pair of encoded pictures in output order (when `eoi_temporal_resampling_type_flag` equals 0) or added between each pair of source pictures for encoding (when `eoi_temporal_resampling_type_flag` equals 1) is constant throughout the persistence of this SEI message. When `eoi_temporal_resampling_type_flag` equals 0 and `eoi_num_int_pics` is greater than 0, `eoi_num_int_pics` specifies the count of pictures excluded by the encoding system between each pair of encoded pictures in output order. When `eoi_temporal_resampling_type_flag` equals 1 and `eoi_num_int_pics` is greater than 0, `eoi_num_int_pics` specifies the count of pictures added by the encoding system between each pair of source pictures for encoding.

[0176] An eoi_num_int_pics equal to 0 indicates that the count of pictures excluded between each pair of encoded pictures in output order (when eoi_temporal_resampling_type_flag equals 0) or added between each pair of source pictures for encoding (when eoi_temporal_resampling_type_flag equals 1) within the persistence of this SEI message is unknown or variable.

[0177] The value of eoi_num_int_pics should be in the range of 0 to 63 (inclusive).

[0178] When EoiTemporalResamplingFlag equals 1, eoi_temporal_resampling_type_flag equals 1, and eoi_num_int_pics is greater than 0, the requirement for bitstream consistency is that the first image in the decoding order in CLVS should be the source image.

[0179] An example of the semantics of the encoder optimization information (SEI) message according to the embodiment is described. Semantics different from those previously described for the encoder optimization information (SEI) message semantics are described, and descriptions of semantics identical to those previously described for the encoder optimization information (SEI) message semantics are replaced with the descriptions of the previously described encoder optimization information (SEI) message semantics. Specifically, in the previously described encoder optimization information (SEI) message semantics, the semantics concerning the temporally upsampled original source image are described, and descriptions of other semantics are replaced with the descriptions of the previously described encoder optimization information (SEI) message semantics.

[0180] [Table 6]

[0181] `eoi_clvs_start_pic_is_src_pic_flag` equal to 1 indicates that the first image in CLVS in decoding order is the source image. `eoi_clvs_start_pic_is_src_pic_flag` equal to 0 does not provide this indication.

[0182] An example of the semantics of the encoder optimization information (SEI) message according to the embodiment is described. Semantics different from those previously described for the encoder optimization information (SEI) message semantics are described, and descriptions of semantics identical to those previously described for the encoder optimization information (SEI) message semantics are replaced with the descriptions of the previously described encoder optimization information (SEI) message semantics. Specifically, in the previously described encoder optimization information (SEI) message semantics, the semantics concerning the temporally upsampled original source image are described, and descriptions of other semantics are replaced with the descriptions of the previously described encoder optimization information (SEI) message semantics.

[0183] [Table 7]

[0184] An eoi_src_pic_flag value of 1 indicates that the image in the same access cell containing the EOI SEI message is the source image. An eoi_src_pic_flag value of 0 does not provide such an indication.

[0185] An example of the semantics of the encoder optimization information (SEI) message according to the embodiment is described. Semantics different from those previously described for the encoder optimization information (SEI) message semantics are described, and descriptions of semantics identical to those previously described for the encoder optimization information (SEI) message semantics are replaced with the descriptions of the previously described encoder optimization information (SEI) message semantics. Specifically, in the previously described encoder optimization information (SEI) message semantics, the semantics concerning the temporally upsampled original source image are described, and descriptions of other semantics are replaced with the descriptions of the previously described encoder optimization information (SEI) message semantics.

[0186] [Table 8]

[0187] `eoi_max_tid_src_pic_present_flag` equal to 1 indicates that the syntax element `eoi_max_tid_src_pic` exists in the SEI message. `eoi_max_tid_src_pic_present_flag` equal to 0 indicates that the syntax element `eoi_max_tid_src_pic` does not exist in the SEI message. When `eoi_max_tid_src_pic_present_flag` equals 1, the bitstream consistency requirement is that the source image and the upsampled image should not belong to the same temporal sublayer.

[0188] When present, eoi_max_tid_src_pic specifies the maximum TemporalId value of the time sublayer to which the original source image belongs.

[0189] An example of the semantics of the encoder optimization information (SEI) message according to the embodiment is described. Semantics different from those previously described for the encoder optimization information (SEI) message semantics are described, and descriptions of semantics identical to those previously described for the encoder optimization information (SEI) message semantics are replaced with the descriptions of the previously described encoder optimization information (SEI) message semantics. Specifically, in the previously described encoder optimization information (SEI) message semantics, the semantics concerning the temporally upsampled original source image are described, and descriptions of other semantics are replaced with the descriptions of the previously described encoder optimization information (SEI) message semantics.

[0190] [Table 9]

[0191] `eoi_sublayer_upsampling_info_present_flag` equal to 1 indicates that the syntax elements `eoi_max_sublayer_minus1` and / or `eoi_sublayer_upsampled_flag[i]` exist in the SEI message. `eoi_max_tid_src_pic_present_flag` equal to 0 indicates that the syntax elements `eoi_max_sublayer_minus1` and `eoi_sublayer_upsampled_flag[i]` do not exist in the SEI message. When `eoi_sublayer_upsampling_info_present_flag` equals 1, the bitstream consistency requirement is that the source image and the upsampled image should not both belong to the same temporal sublayer.

[0192] Increasing `eoi_max_sublayers_minus1` by 1 specifies the maximum number of time sublayers for which `eoi_sublayer_upsampled_flag[i]` is sent using the signal. When `eoi_max_sublayers_minus1` does not exist, it is inferred to be equal to `TemporalId`. The value of `eoi_max_sublayers_minus1` should be equal to or greater than the `TemporalId` of the `Eoi` SEI message.

[0193] The variable eoiMinTemporalSublayer is set as follows: - If eoi_persistence_flag equals 1, then eoiMinTemporalSublayer equals 0.

[0194] Otherwise, eoiMinTemporalSublayer equals eoi_max_sublayers_minus1.

[0195] When present, `eoi_sublayer_upsampled_picture_flag[i]` equal to 1 indicates that the output image of the decoded image belonging to the i-th temporal sublayer was upsampled and does not correspond to the unmodified original source image. `eoi_sublayer_upsampled_picture_flag[i]` equal to 0 does not provide such an indication. When absent, the value of `eoi_sublayer_upsampled_picture_flag[i]` is inferred to be 0.

[0196] Note that when the TemporalId of the Eoi SEI message is greater than 0, and the Eoi SEI message is continuously used for one or more pictures with a lower TemporalId, the encoder can repeat the information of the EOI SEI message by including it in one or more EOI SEI messages with a lower TemporalId, so as to avoid the loss of information when the pictures in the temporal sublayer are lost or removed.

[0197] An example of the semantics of the encoder optimization information (SEI) message according to the embodiment is described. Semantics different from those previously described for the encoder optimization information (SEI) message semantics are described, and descriptions of semantics identical to those previously described for the encoder optimization information (SEI) message semantics are replaced with the descriptions of the previously described encoder optimization information (SEI) message semantics. Specifically, in the previously described encoder optimization information (SEI) message semantics, the semantics concerning the temporally upsampled original source image are described, and descriptions of other semantics are replaced with the descriptions of the previously described encoder optimization information (SEI) message semantics.

[0198] A value greater than 0 for `eoi_num_int_pics` indicates that the count of pictures excluded by the encoding system between each pair of encoded pictures in output order (when `eoi_temporal_resampling_type_flag` equals 0) or added between each pair of source pictures for encoding (when `eoi_temporal_resampling_type_flag` equals 1) is constant throughout the persistence of this SEI message. When `eoi_temporal_resampling_type_flag` equals 0 and `eoi_num_int_pics` is greater than 0, `eoi_num_int_pics` specifies the count of pictures excluded by the encoding system between each pair of encoded pictures in output order. When `eoi_temporal_resampling_type_flag` equals 1 and `eoi_num_int_pics` is greater than 0, `eoi_num_int_pics` specifies the count of pictures added by the encoding system between each pair of source pictures for encoding.

[0199] An eoi_num_int_pics equal to 0 indicates that the count of pictures excluded between each pair of encoded pictures in output order (when eoi_temporal_resampling_type_flag equals 0) or added between each pair of source pictures for encoding (when eoi_temporal_resampling_type_flag equals 1) within the persistence of this SEI message is unknown or variable.

[0200] The value of eoi_num_int_pics should be in the range of 0 to 63 (inclusive).

[0201] When EoiTemporalResamplingFlag equals 1 and eoi_temporal_resampling_type_flag equals 1, the constraint is that the access unit containing the EOI SEI message should contain the image as the source image.

[0202] Note that `eoi_temporal_int_pics` can be used to identify images generated from temporal sampling optimization. The next (eoi_temporal_int_pics)th image from the current image (i.e., the image associated with the EOI SEI message) is an image generated from temporal sampling optimization.

[0203] The terms or names described below (e.g., names of grammatical elements or variable names, etc.) are merely examples, and the technical features of this disclosure are not limited to the terms described below. For example, the image information described below may include various information according to the embodiments described in this disclosure, and may include information described in at least one of the tables described above.

[0204] The operations described below do not correspond to the basic components of the embodiments, and at least some of the operations described below may be omitted. Furthermore, the operations described below do not correspond to a sufficient component of the embodiments, and the operations described above may be added. Moreover, unless the operations described below contradict those described above, the operations described below form an embodiment as a whole with the operations described above, and do not form a separate embodiment different from the operations described above.

[0205] Figure 7 This is a diagram illustrating a method for decoding image information according to an embodiment of the present disclosure.

[0206] The terms or names (e.g., names of grammatical elements or variable names, etc.) are merely examples, and the technical features of this disclosure are not limited to these examples. Figure 7 Terms described in the text, etc. For example, Figure 7 The image information described herein may include various information according to the embodiments described herein, and may include information described in at least one of the tables described above.

[0207] The decoding method (S700) may include the operations described below. The operations described below do not correspond to the essential components of the decoding method according to the embodiment, and at least some of the operations described below may be omitted. Furthermore, the operations described below do not correspond to a sufficient component of the decoding method according to the embodiment, and the operations described above may be added. Moreover, unless the operations described below contradict those described above, the operations described below form an embodiment integral with the operations described above, and do not form a separate embodiment different from the operations described above.

[0208] The decoding method (S700) can be executed by a decoding device including a memory and a processor electrically connected to the memory, and for example, it can be executed by a processor.

[0209] The decoding device can obtain image information (S710).

[0210] For example, the processor of the decoding device can obtain image information. This image information may include SEI (Supplemental Enhancement Information) messages. SEI messages can carry specific types of information that assist in processes related to the decoding, display, or other purposes of the image information. Here, the SEI message may not be necessary for the decoding process to determine sample values ​​of the decoded image.

[0211] For example, an SEI message may include an Encoder Optimization Information (EOI) SEI (Supplemental Enhancement Information) message.

[0212] The Encoder Optimization Information (SEI) message can include information about SEI message processing and information about encoder optimization. For example, the SEI message can provide encoder optimization information indicating whether the image is optimized for human viewing or machine analysis, and what type of optimization is applied during preprocessing or encoding. Encoder optimization could include optimization for human viewing, optimization for machine analysis, object-based optimization, temporal resampling optimization, spatial resampling optimization, temporal quality optimization, spatial quality optimization, or privacy-preserving optimization, etc.

[0213] Encoder optimization information (SEI) messages can have various names. For example, they can be called EOI SEI messages, EOI SEI related messages, EOI SEI information, EOI SEI related information, encoder optimization information messages, EOI messages, EOI related messages, EOI information, EOI related information, etc.

[0214] Encoder optimization information (SEI) messages can take various forms. For example, an SEI message can be a syntax element or a syntax structure that includes one or more syntax elements. Alternatively, an SEI message can be an RBSP (Raw Byte Sequence Payload) that includes one or more syntax elements or one or more syntax structures. For example, an SEI message can be expressed as encoder_optimization_info(payloadSize), but is not limited to these.

[0215] The encoder optimization information (SEI) message may include the cancellation information of the encoder optimization information SEI message, the persistence information of the encoder optimization information SEI message, the human viewing optimization information, the machine analysis optimization information, the optimization type information, the object-based optimization type information, the temporal resampling type information, the number of images information, the source image information, the information about the spatial resampling type, the privacy protection method type information, and / or the privacy information type information.

[0216] Human viewing optimization information can indicate whether the optimization purpose includes human viewing. In other words, human viewing optimization information can indicate whether the optimized image is optimized for human viewing, suitable for human viewing, unsuitable for human viewing, or unknown for human viewing.

[0217] Human viewing optimization information can take various forms and be expressed by various names. For example, human viewing optimization information can be a syntactic element or a syntactic structure that includes one or more syntactic elements. For example, human viewing optimization information as a syntactic element can include a flag consisting of one bit or an indicator consisting of two or more bits or variable-length bits. Human viewing optimization information as a syntactic element can be expressed as eoi_for_human_viewing_idc or eoi_for_human_viewing_flag, etc., but is not limited to these.

[0218] Machine analysis optimization information can indicate whether the optimization objective includes machine analysis. In other words, machine analysis optimization information can indicate whether the optimized image is optimized for machine analysis, suitable for human viewing, unsuitable for human viewing, or unknown for human viewing.

[0219] Machine analysis optimization information can take various forms and be expressed by various names. For example, machine analysis optimization information can be a syntax element or a syntax structure that includes one or more syntax elements. For example, machine analysis optimization information as a syntax element can include a flag consisting of one bit or an indicator consisting of two or more bits or variable-length bits. Machine analysis optimization information as a syntax element can be expressed as eoi_for_machine_analysis_idc or eoi_for_machine_analysis_flag, etc., but is not limited to these.

[0220] The optimization type information indicates the type of optimization applied. Encoder optimizations may include object-based optimization, temporal resampling optimization, spatial resampling optimization, temporal quality optimization, spatial quality optimization, and / or privacy protection optimization. One or more of these optimizations may be applied, and the optimization type information indicates the type of optimization applied for all purposes.

[0221] Optimized type information can take various forms and be expressed by various names. For example, optimized type information can be a syntax element or a syntax structure that includes one or more syntax elements. For instance, optimized type information as a syntax element can include a flag consisting of one bit or an indicator consisting of two or more bits (e.g., 16 bits) or variable-length bits. Optimized type information as a syntax element can be expressed as eoi_type, eoi_type_flag, or eoi_type_idc, but is not limited to these.

[0222] Based on the optimization type information, multiple optimization type variables are used to indicate whether to apply each of the following: object-based optimization, temporal resampling optimization, spatial resampling optimization, temporal quality optimization, spatial quality optimization, and privacy protection optimization. For example, multiple optimization type variables may include, but are not limited to, EoiObjectBasedFlag, EoiTemporalResamplingFlag, EoiSpatialResamplingFlag, EoiTemporalQualityFlag, EoiSpatialQualityFlag, and EoiPrivacyProtectionFlag.

[0223] Object-based optimization type information is obtained when the optimization type indicated by the optimization type information includes object-based optimization, and may indicate the type of object-based optimization including fuzzing, quantization and / or overlay.

[0224] Object-based optimization type information can take various forms and be expressed by various names. For example, object-based optimization type information can be a syntax element or a syntax structure that includes one or more syntax elements. For example, object-based optimization type information as a syntax element can include multiple flags, each consisting of one bit, or it can include an indicator consisting of variable-length bits. Optimization type information as a syntax element can be expressed as eoi_object_based_flag or eoi_object_based_idc, etc., but is not limited to these.

[0225] The time resampling type information is obtained when the optimization type indicated by the optimization type information includes time resampling, and can indicate the type of time resampling. Time resampling can include time upsampling and time subsampling. The time resampling type information can indicate whether the applied time resampling is time upsampling or time subsampling. For example, a time resampling type information of 0 indicates that the applied time resampling is time subsampling. A time resampling type information of 1 indicates that the applied time resampling is time upsampling. However, it is not limited to this, and the content indicated by a time resampling type information of 1 can be toggled with the content indicated by a time resampling type information of 0.

[0226] Temporal resampling type information can take various forms and be represented by various names. For example, temporal resampling type information can be a syntax element or a syntax structure that includes one or more syntax elements. For instance, temporal resampling type information as a syntax element can include a flag consisting of one bit or an indicator consisting of two or more bits or variable-length bits. Temporal resampling type information as a syntax element can be expressed as eoi_temporal_resampling_type_flag or eoi_temporal_resampling_type_idc, etc., but is not limited to these.

[0227] The image count information is obtained when the optimization type, indicated by the optimization type information, includes temporal resampling. It indicates whether any images were added between a pair of temporally adjacent original source images or excluded between a pair of temporally adjacent decoded images via temporal resampling. It can also indicate the number of images added between a pair of temporally adjacent original source images via temporal upsampling or the number of images excluded between a pair of temporally adjacent decoded images via temporal subsampling. For example, an image count of 0 indicates that no images were added or excluded by temporal resampling. Conversely, an image count greater than 0 indicates the number of images added via temporal upsampling or excluded via temporal subsampling.

[0228] Image quantity information can take various forms and be expressed by various names. For example, image quantity information can be a syntax element or a syntax structure that includes one or more syntax elements. For instance, image quantity information as a syntax element can include a flag consisting of one bit or an indicator consisting of two or more bits or variable-length bits. Image quantity information as a syntax element can be expressed as eoi_num_int_pics, etc., but is not limited to these.

[0229] The source image information is obtained when the optimization type, indicated by the optimization type information, includes time resampling, and can indicate whether the current image is the original source image.

[0230] When adding images via temporal upsampling, the source image information can provide information about whether the current image is the original source image or an image added via temporal upsampling.

[0231] When adding images via temporal upsampling, information about whether the current image is the original source image or an image added via temporal upsampling can be provided in various ways.

[0232] For example, source image information can indicate whether an image included in the same image unit or access unit as the encoder optimization information (SEI) message is the original source image. For instance, source image information equal to 1 can indicate that an image included in the same image unit or access unit as the encoder optimization information (SEI) message is the original source image. Conversely, source image information equal to 0 can indicate that no information about the original source image is provided.

[0233] Based on source image information, the original source images can be identified. For example, when adding an image between a pair of temporally adjacent original source images through temporal upsampling, the image included in the same image unit or access unit as the encoder optimization information (SEI) message is the original source image, the image following the original source image is the added image, and the image following the added image can also be the original source image. Similarly, when adding two images between a pair of temporally adjacent original source images through temporal upsampling, the image included in the same image unit or access unit as the encoder optimization information (SEI) message is the original source image, the two consecutive images following the original source image are the added images, and the image following the added image can also be the original source image.

[0234] As another example, source image information can indicate whether the first image in the decoding order within CLVS is the source image. Alternatively, the first image in the decoding order within CLVS can be restricted to being the source image.

[0235] As another example, source image information can indicate whether the first image in the output order within CLVS is the source image. Alternatively, the first image in the output order within CLVS can be restricted to being the source image.

[0236] As another example, source image information can indicate the maximum TemporalId value of the time sublayer to which the original source image belongs.

[0237] As another example, source image information can indicate a temporal sublayer of the image, including images added by upsampling from the encoding device.

[0238] As another example, source image information can indicate the number of images added up to the upsampling process via the encoding device.

[0239] Because source image information provides information about whether the current image is the original source image when adding images via temporal upsampling, obtaining source image information is only valid when images are added via temporal upsampling. For example, source image information can be obtained when the optimization type indicated by the optimization type information includes temporal resampling and the image quantity information indicates that there are images added or excluded via temporal resampling.

[0240] Source image information can take various forms and be expressed by various names. For example, source image information can be a syntax element or a syntax structure that includes one or more syntax elements. For example, source image information as a syntax element can include a flag consisting of one bit or can include an indicator consisting of two or more bits or variable-length bits. Source image information as a syntax element can be expressed as eoi_clvs_start_pic_is_src_pic_flag, eoi_src_pic_flag, eoi_max_tid_src_pic, eoi_sublayer_upsampled_picture_flag[i], or eoi_temporal_int_pics, but is not limited to these.

[0241] Information regarding the spatial resampling type is obtained when the optimization type, indicated by the optimization type information, includes spatial resampling, and can indicate the type of spatial resampling. Spatially, this spatial resampling includes resampling in the horizontal direction (width direction) and resampling in the vertical direction (height direction), and in terms of sampling, it can include spatial upsampling and spatial subsampling. In other words, spatial resampling can include upsampling / subsampling in the horizontal direction (width direction) and upsampling / subsampling in the vertical direction (height direction).

[0242] Information about spatial resampling types can be provided in various ways. For example, by providing information about the width and height of the original source image, information about upsampling / subsampling in the horizontal direction (width direction) and upsampling / subsampling in the vertical direction (height direction) can be provided. Alternatively, resampling information indicating upsampling / subsampling in the horizontal direction (width direction) and resampling information indicating upsampling / subsampling in the vertical direction (height direction) can be provided directly.

[0243] For example, information about the spatial resampling type may include information that provides the spatial resampling type (e.g., eoi_orig_pic_dimensions_flag), information about the width and height of the original source image (e.g., eoi_orig_pic_width and eoi_orig_pic_height), and / or spatial resampling type information (e.g., eoi_spatial_resampling_type_flag).

[0244] Information about spatial resampling types can take various forms and be expressed by various names. For example, information about spatial resampling types can be syntax elements or syntax structures that include one or more syntax elements. For example, information about spatial resampling types as syntax elements can include a flag consisting of one bit or an indicator consisting of two or more bits or variable-length bits. Information about spatial resampling types as syntax elements can include, but is not limited to, syntax elements expressed as eoi_orig_pic_dimensions_flag, eoi_orig_pic_width, eoi_orig_pic_height, and eoi_spatial_resampling_type_flag, etc.

[0245] Privacy protection method type information indicates the method / algorithm used to apply privacy protection optimizations. The method / algorithm used for privacy protection optimizations may include authorizations for applications, obfuscation, replacement, masking, etc.

[0246] Privacy protection method type information can take various forms and be expressed by various names. For example, privacy protection method type information can be a syntax element or a syntax structure that includes one or more syntax elements. For example, privacy protection method type information as a syntax element can include a flag consisting of one bit or an indicator consisting of two or more bits or variable-length bits. Privacy protection method type information as a syntax element can be expressed as eoi_privacy_protection_method_flag or eoi_privacy_protection_method_idc, etc., but is not limited to these.

[0247] Privacy information type information indicates the type of privacy information protected through coding optimization. Types of privacy information protected through coding optimization can include a person's face, a vehicle's license plate, location information, etc.

[0248] Privacy information type information can take various forms and be expressed by various names. For example, privacy information type information can be a syntax element or a syntax structure that includes one or more syntax elements. For example, privacy information type information as a syntax element can include a flag consisting of one bit or an indicator consisting of two or more bits or variable-length bits. Privacy information type information as a syntax element can be expressed as eoi_privacy_info_type, eoi_privacy_info_type_flag, or eoi_privacy_info_type_idc, etc., but is not limited to these.

[0249] The decoding device can export encoder optimization information (S720).

[0250] For example, the processor of the decoding device can process the encoder optimization information (SEI) message. Based on processing the SEI message, the decoding device can derive information about encoder optimization.

[0251] The Encoder Optimization Information (SEI) message can include information about temporal resampling. Additionally, when temporal resampling indicates temporal upsampling, the SEI message can further include source image information used to identify the original source image from the image added via temporal upsampling.

[0252] Therefore, identifying the original source image can provide various technical effects. The original source image can include image information that is more similar to the original image compared to the image added through upsampling by the encoding device. Therefore, when subsampling is required at the decoding device, the decoding device can remove the image added through upsampling by the encoding device and retain the original source image. Thus, the decoding device can provide the user with image information that is more similar to the original image.

[0253] Therefore, using source image information to provide information about whether the current image is the original source image enables the encoding system to provide image information that is more similar to the original image, thereby improving the reliability of image information delivered by the encoding system.

[0254] Furthermore, using source image information to provide information about whether the current image is the original source image enables the encoding system to identify the original source image more quickly, thereby improving the encoding speed and efficiency of the encoding system.

[0255] Furthermore, using source image information to provide information about whether the current image is the original source image allows the encoding system to identify the original source image with less data, thereby improving the data transmission efficiency of the encoding system.

[0256] Figure 8This is a diagram illustrating a method for encoding image information according to an embodiment of the present disclosure.

[0257] exist Figure 8 The terms or names described herein (e.g., names of grammatical elements or variable names, etc.) are merely examples, and the technical features of this disclosure are not limited to them. Figure 8 Terms described in the text, etc. For example, Figure 8 The image information described herein may include various information according to the embodiments described herein, and may include information described in at least one of the tables described above.

[0258] The encoding method (S800) may include the operations described below. The operations described below do not correspond to the essential components of the decoding method according to the embodiment, and at least some of the operations described below may be omitted. Furthermore, the operations described below do not correspond to a sufficient component of the encoding method according to the embodiment, and the operations described above may be added. Moreover, unless the operations described below contradict those described above, the operations described below form an embodiment integral with the operations described above, and do not form a separate embodiment different from the operations described above.

[0259] The encoding method (S800) can be executed by an encoding device including a memory and a processor electrically connected to the memory, and for example, it can be executed by a processor.

[0260] The encoding device can perform encoder optimization (S810).

[0261] For example, the processor of the encoding device can perform encoder optimization. Encoder optimization may include optimization for human viewing, optimization for machine analysis, object-based optimization, temporal resampling optimization, spatial resampling optimization, temporal quality optimization, spatial quality optimization, or privacy protection optimization, etc.

[0262] The encoding device can encode image information (S820).

[0263] For example, the processor of the encoding device can generate information about encoder optimization based on the encoder optimization performed, generate encoder optimization information (EOI) SEI (supplementary enhancement information) messages based on the generated information about encoder optimization, and encode image information including the encoder optimization information SEI messages.

[0264] The Encoder Optimization Information (SEI) message can include information about SEI message processing and information about encoder optimization. For example, the SEI message can provide encoder optimization information indicating whether the image is optimized for human viewing or machine analysis, and what type of optimization was applied during preprocessing or encoding.

[0265] Encoder optimization information (SEI) messages can have various names. For example, they can be called EOI SEI messages, EOI SEI related messages, EOI SEI information, EOI SEI related information, encoder optimization information messages, EOI messages, EOI related messages, EOI information, EOI related information, etc.

[0266] Encoder optimization information (SEI) messages can take various forms. For example, an SEI message can be a syntax element or a syntax structure that includes one or more syntax elements. Alternatively, an SEI message can be an RBSP (Raw Byte Sequence Payload) that includes one or more syntax elements or one or more syntax structures. For example, an SEI message can be expressed as encoder_optimization_info(payloadSize), but is not limited to these.

[0267] The encoder optimization information (SEI) message may include the cancellation information of the encoder optimization information SEI message, the persistence information of the encoder optimization information SEI message, the human viewing optimization information, the machine analysis optimization information, the optimization type information, the object-based optimization type information, the temporal resampling type information, the number of images information, the source image information, the information about the spatial resampling type, the privacy protection method type information, and / or the privacy information type information.

[0268] Human-viewed optimization information, machine-analyzed optimization information, optimization type information, object-based optimization type information, temporal resampling type information, image quantity information, source image information, information about spatial resampling type, privacy protection method type information, and / or privacy information type information can be related to... Figure 7 The information described above in Operation 710 includes human viewing optimization information, machine analysis optimization information, optimization type information, object-based optimization type information, temporal resampling type information, image quantity information, source image information, information about spatial resampling type, privacy protection method type information, and / or privacy information type information.

[0269] The descriptions of human-viewed optimization information, machine-analyzed optimization information, optimization type information, object-based optimization type information, temporal resampling type information, image quantity information, source image information, information about spatial resampling type, privacy protection method type information, and / or privacy information type information have been replaced with... Figure 7 The descriptions above in Operation 710 include human viewing optimization information, machine analysis optimization information, optimization type information, object-based optimization type information, temporal resampling type information, image quantity information, source image information, information about spatial resampling type, privacy protection method type information, and / or privacy information type information.

[0270] Therefore, the encoding device can encode image information including encoder optimization information (SEI) messages.

[0271] Here, the encoder optimization information (SEI) message may include information about temporal resampling. Additionally, when temporal resampling indicates temporal upsampling, the SEI message may further include source image information used to identify the original source image from the image added via temporal upsampling.

[0272] Therefore, identifying the original source image can provide various technical effects. The original source image can include image information that is more similar to the original image compared to the image added through upsampling by the encoding device. Therefore, when subsampling is required at the decoding device, the decoding device can remove the image added through upsampling by the encoding device and retain the original source image. Thus, the decoding device can provide the user with image information that is more similar to the original image.

[0273] Therefore, using source image information to provide information about whether the current image is the original source image enables the encoding system to provide image information that is more similar to the original image, thereby improving the reliability of delivering image information through the encoding system.

[0274] Furthermore, using source image information to provide information about whether the current image is the original source image enables the encoding system to identify the original source image more quickly, thereby improving the encoding speed and efficiency of the encoding system.

[0275] Furthermore, using source image information to provide information about whether the current image is the original source image allows the encoding system to identify the original source image with less data, thereby improving the data transmission efficiency of the encoding system.

[0276] A bitstream is generated based on image information encoded according to the encoding method (S800) described above, and the bitstream can be stored in a computer-readable storage medium.

[0277] In addition, a bit stream is generated based on the image information encoded according to the encoding method (S800) described above, and the bit stream can be transmitted by a transmitter and / or a transmission medium.

[0278] In this disclosure, by way of example, the names of the syntax elements described above are arbitrarily assigned for clarity of description and are not limited to any particular syntax element. Additionally, each syntax element may also be referred to as information. Furthermore, syntax elements can be obtained from a bitstream, but can also be derived from another syntax element, and this may also be included in embodiments of this disclosure.

[0279] In addition, bitstreams generated by image encoding methods can also be stored in non-transitory computer-readable recording media.

[0280] Alternatively, as another example, a bitstream generated by an image encoding method can be sent to another device (e.g., an image decoding device). In this case, the method of transmitting the bitstream can include the process of sending the bitstream.

[0281] Although the exemplary methods of this disclosure are presented as a series of operations for clarity of description, this is not intended to limit the order in which the steps are performed, and each step may be performed simultaneously or in a different order if necessary. To implement the methods according to this disclosure, the illustrated steps may additionally include other steps, may include remaining steps excluding some steps, or may include additional steps excluding some steps.

[0282] In this disclosure, an image encoding device or an image decoding device that performs a predetermined operation (step) can also perform an operation (step) that confirms the execution conditions or circumstances of the corresponding operation (step). For example, when describing the execution of a predetermined operation when predetermined conditions are met, the image encoding device or the image decoding device can perform the predetermined operation after performing an operation to confirm whether the predetermined conditions are met.

[0283] The various embodiments disclosed herein are not a list of all possible combinations, but are intended to describe representative aspects of the disclosure, and the matters described in the various embodiments may be applied independently or in combination of two or more.

[0284] The embodiments described in this disclosure can be implemented and executed on a processor, microprocessor, controller, or chip. For example, the functional units illustrated in each figure can be implemented and executed on a computer, processor, microprocessor, controller, or chip. In this case, information for implementation (e.g., information about instructions) or algorithms can be stored in a digital storage medium.

[0285] Furthermore, the decoders (decoding devices) and encoders to which the embodiments of this disclosure are applied can be included in multimedia broadcasting transmitting and receiving devices, mobile communication terminals, home theater video devices, digital cinema video devices, surveillance cameras, video conferencing devices, real-time communication devices such as video communication, mobile streaming media devices, storage media, cameras, video-on-demand (VoD) service providers, OTT video (over-the-top video) devices, internet streaming media service providers, three-dimensional (3D) video devices, VR (virtual reality) devices, AR (augmented reality) devices, video telephony devices, vehicle terminals (e.g., vehicle (including autonomous vehicles) terminals, robot terminals, aircraft terminals, ship terminals, etc.), and medical video devices, and can be used to process video signals or data signals. For example, OTT video (over-the-top video) devices can include game consoles, Blu-ray players, internet-connected televisions, home theater systems, smartphones, tablet PCs, DVRs (digital video recorders), etc.

[0286] Furthermore, the processing methods to which the embodiments of this disclosure can be applied can be generated in the form of a program executed by a computer and can be stored in a computer-readable recording medium. Multimedia data with data structures according to embodiments of this document can also be stored in a computer-readable recording medium. Computer-readable recording media include all kinds of storage devices and distributed storage devices in which computer-readable data is stored. For example, computer-readable recording media can include, for example, Blu-ray discs (BD), Universal Serial Bus (USB), ROM, PROM, EPROM, EEPROM, RAM, CD-ROM, magnetic tape, floppy disks, and optical data storage devices. Additionally, computer-readable recording media include media implemented in the form of carrier waves (e.g., transmission over the Internet). Furthermore, bitstreams generated by encoding methods can be stored in computer-readable recording media or transmitted via wired / wireless communication networks.

[0287] Furthermore, the embodiments of this disclosure can be implemented as a computer program product by program code, and the program code can be executed on a computer by means of the embodiments of this disclosure. The program code can be stored on a computer-readable medium.

[0288] Figure 9 This is a diagram illustrating an example of a content streaming system to which embodiments of the present disclosure can be applied.

[0289] refer to Figure 9 The content streaming system using embodiments of this disclosure can broadly include encoding servers, streaming servers, web servers, media storage, user equipment, and multimedia input devices.

[0290] An encoding server compresses content input from multimedia input devices such as smartphones, cameras, or camcorders into digital data, generates a bitstream, and sends it to a streaming server. As another example, when multimedia input devices such as smartphones, cameras, or camcorders directly generate bitstreams, the encoding server can be omitted.

[0291] The bitstream can be generated by the image encoding method and / or image encoding apparatus of the embodiments of this disclosure, and the streaming server can temporarily store the bitstream during the process of sending or receiving the bitstream.

[0292] A streaming server can send multimedia data to a user's device based on a user request via a web server, and the web server can act as a medium to notify the user of available services. When a user requests a service from the web server, the web server can send the request to the streaming server, and the streaming server can transmit the multimedia data to the user. In this scenario, the content streaming system may include a separate control server, which in this case can control the commands / responses between devices within the content streaming system.

[0293] A streaming server can receive content from media storage and / or encoding servers. For example, content can be received in real time when it is received from an encoding server. In this case, to provide a seamless streaming service, the streaming server can store a bitstream for a specific period of time.

[0294] Examples of user devices may include mobile phones, smartphones, laptops, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, slate PCs, tablet PCs, ultrabooks, wearable devices (i.e., smartwatches, smart glass, head-mounted displays (HMDs)), digital televisions, desktop computers, digital signage, etc.

[0295] In a content streaming system, each server can operate as a distributed server, in which case the data received by each server can be processed in a distributed manner.

[0296] Figure 10 This is a diagram illustrating another example of a content streaming system to which embodiments of the present disclosure can be applied.

[0297] refer to Figure 10In embodiments such as VCM, the task can be executed by a user terminal, or by an external device (e.g., a streaming server, analytics server, etc.) based on the device's performance, the user's request, the characteristics of the task to be performed, etc. In this way, in order to send the information necessary for executing the task to the external device, the user terminal can generate a bitstream directly or through an encoding server, which includes the information necessary for executing the task (e.g., information such as the task, neural network, and / or information used).

[0298] An analysis server can execute tasks requested by the user after decoding encoded information sent from a user terminal (or an encoding server). The analysis server can then send the results obtained from executing the tasks back to the user terminal or another linked service server (e.g., a web server). For example, the analysis server can send the results obtained from executing a task to determine a fire to a fire-related server. The analysis server may include a separate control server, in which case the control server can play a role in controlling the commands / responses between the analysis server and each device associated with it. Additionally, the analysis server can request desired information from a web server based on information about the tasks the user device wants to perform and the tasks the user device can perform. When the analysis server requests the desired service from the web server, the web server can forward it to the analysis server, and the analysis server can then send its data to the user terminal. In this scenario, the control server of the content streaming system can play a role in controlling the commands / responses between each device within the streaming system.

[0299] Industrial applicability

[0300] Embodiments of this disclosure can be used to encode / decode features / feature maps.

Claims

1. A method for decoding image information, the method comprising: Obtain the image information including encoder optimization information (EOI) and SEI (supplementary enhancement information) messages; as well as Information about encoder optimization is derived based on the encoder optimization information (SEI) message. The encoder optimization information (SEI) message includes optimization type information, which indicates the optimization type of time resampling. Specifically, based on the optimization type information, it is determined whether the current image included in the same unit as the encoder optimization information SEI message is the source image used for the time resampling.

2. The method according to claim 1, wherein, The encoder optimization information (SEI) message further includes source image information, which refers to the current image included in the same unit as the encoder optimization information (SEI) message as the source image used for the temporal resampling.

3. The method according to claim 2, wherein, The encoder optimization information (SEI) message further includes information about the temporal resampling type of the temporal upsampling and information about the number of images added or excluded through the temporal resampling.

4. The method according to claim 3, wherein, The source image information is obtained based on the time resampling type information and the number of images.

5. The method according to claim 4, wherein, The source image information is obtained based on the time resampling type information indicating the time upsampling and the number of images with a value greater than 0.

6. The method according to claim 3, wherein, The time resampling type information and the number of images are obtained based on the optimization type information indicating the time resampling.

7. A method for encoding image information, the method comprising: Perform encoder optimization; as well as The image information is encoded, and the image information includes encoder optimization information (EOI) SEI (supplementary enhancement information) messages generated based on information about encoder optimization. The encoder optimization information (SEI) message includes optimization type information, which indicates the optimization type of time resampling. Specifically, based on the optimization type information, it is determined whether the current image included in the same unit as the encoder optimization information SEI message is the source image used for the temporal resampling.

8. The method according to claim 7, wherein, The encoder optimization information (SEI) message further includes source image information, which refers to the current image included in the same unit as the encoder optimization information (SEI) message as the source image used for the temporal resampling.

9. The method according to claim 8, wherein, The encoder optimization information (SEI) message further includes information about the temporal resampling type of the temporal upsampling and information about the number of images added or excluded through the temporal resampling.

10. The method according to claim 9, wherein, The source image information is obtained based on the time resampling type information and the number of images.

11. The method according to claim 10, wherein, The source image information is obtained based on the time resampling type information indicating the time upsampling and the number of images with a value greater than 0.

12. The method according to claim 9, wherein, The time resampling type information and the number of images are obtained based on the optimization type information indicating the time resampling.

13. A method for a bitstream, the method comprising: Perform encoder optimization; Generate a bitstream for image information, which includes encoder optimization information (EOI) SEI (supplementary enhancement information) messages generated based on information about encoder optimization; as well as Send data for the bitstream. The encoder optimization information (SEI) message includes optimization type information, which indicates the optimization type of time resampling. Specifically, based on the optimization type information, it is determined whether the current image included in the same unit as the encoder optimization information SEI message is the source image used for the temporal resampling.

14. A computer-readable storage medium for storing a bit stream, wherein, The storage medium stores a bitstream for image information, which includes encoder optimization information (EOI) SEI (supplementary enhancement information) messages generated based on information about encoder optimization. The encoder optimization information (SEI) message includes optimization type information, which indicates the optimization type of time resampling. Specifically, based on the optimization type information, it is determined whether the current image included in the same unit as the encoder optimization information SEI message is the source image used for the temporal resampling.

Images