Method, apparatus, and recording medium for adaptive image reconstruction for machines

Abstract

An embodiment of the present disclosure proposes a decoding device for video coding for machines (VCM). This VCM decoding device may comprise: an internal decoding performer for generating a reconstructed image decoded from a bitstream, and parsing information on the reconstructed image; a region-of-interest-based reconstructor for reconfiguring the reconstructed image on the basis of information related to a region-of-interest included in the information on the reconstructed image; a spatial reconstruction performer for reconstructing a resolution of at least a portion of the reconstructed image on the basis of information related to spatial reconstruction included in the information on the reconstructed image; and a temporal reconstruction performer for performing temporal reconstruction by generating at least one frame for the reconstructed image on the basis of information related to temporal reconstruction included in the information on the reconstructed image, wherein the execution order of at least two of the region-of-interest-based reconstructor, the spatial reconstruction performer, and the temporal reconstruction performer may be adaptively determined on the basis of the information on the reconstructed image.

Description

Adaptive image restoration method, device and recording medium for machines

[0001] The present disclosure relates to an adaptive image restoration method, device, and recording medium in an image encoding / decoding method for a machine.

[0002] With the continuous development of the information and communication industry, broadcasting services with HD (High Definition) resolution have spread worldwide.

[0003] Through this proliferation, many users have become accustomed to high-resolution and high-quality images and / or videos, and the demand for higher-resolution and high-quality images / videos, such as 4K or 8K or higher UHD (Ultra High Definition) images / videos, has increased in various fields.

[0004] The technology for coding this UHD video data was completed in 2013 through HEVC (High Efficiency Video Coding), a standard technology.

[0005] HEVC is a next-generation video compression technology with a higher compression ratio and lower complexity than the previous H.264 / AVC technology, and is a key technology for effectively compressing the massive data of HD and UHD video.

[0006] HEVC performs block-by-block encoding, like previous compression standards.

[0007] However, unlike H.264 / AVC, there is only one profile. The core encoding technologies included in HEVC's sole profile are divided into eight areas: hierarchical encoding structure technology, transform technology, quantization technology, intra-frame prediction encoding technology, inter-frame motion prediction technology, entropy encoding technology, loop filter technology, and other technologies.

[0008] Since the establishment of the HEVC video codec in 2013, the Versatile Video Coding (VVC) standard, a next-generation video codec that aims to improve performance by more than twice that of HEVC, has been developed to address the expansion of realistic video and virtual reality services utilizing 4K and 8K video images. VVC is called H.266.

[0009] H.266 (VVC) was developed with the goal of being more than twice as efficient as the previous generation codec, H.265 (HEVC). VVC was initially developed with resolutions over 4K in mind, but it was also developed for ultra-high-resolution video processing at a whopping 16K level to support 360-degree videos due to the expansion of the VR market. In addition, as the HDR market is expanding due to the development of display technology, it supports 16-bit color depth as well as 10-bit color depth to respond to this, and supports brightness expressions of 1000 nits, 4000 nits, and 10000 nits. In addition, since it is being developed with the VR market and 360-degree video market in mind, it supports partial frame rates in the range of 0 to 120 FPS.

[0010] Advances in Artificial Intelligence

[0011] Artificial intelligence (AI) is also steadily developing. AI refers to the artificial imitation of human intelligence, including the ability to recognize, classify, infer, predict, and control / decision-making.

[0012] With the advancement of artificial intelligence technology and the increase in Internet of Things (IoT) devices, machine-to-machine traffic is expected to explode, and machine-dependent image analysis is expected to become widely used.

[0013] However, as the amount of images to be analyzed by machines is expected to increase exponentially, issues with server load and power consumption are expected to arise.

[0014] Accordingly, the present disclosure aims to provide an adaptive image restoration method, device and recording medium for a machine, so as to enable effective image analysis by the machine.

[0015] In order to achieve the above-mentioned purpose, according to one disclosure of the present specification, an adaptive image restoration method, device and recording medium for a machine can be proposed.

[0016] A VCM decoding device according to one disclosure of the present specification comprises: an internal decoding performer that generates a decoded restored image from a bitstream and parses information about the restored image; a region-of-interest-based restorer that reconstructs the restored image based on information related to a region of interest included in the information about the restored image; a spatial restoration performer that restores a resolution for at least a portion of the restored image based on information related to spatial restoration included in the information about the restored image; and a temporal restoration performer that generates at least one frame of the restored image and performs temporal restoration based on information related to temporal restoration included in the information about the restored image, wherein an execution order of at least two of the region-of-interest-based restorer, the spatial restoration performer, and the temporal restoration performer can be adaptively determined based on the information about the restored image.

[0017] A VCM encoding device according to one disclosure of the present specification includes a temporal resampling performer that changes a frame rate for an input image; a spatial resampling performer that changes a spatial resolution for each frame or a series of frames of the input image; a region-of-interest-based processor that extracts one or more regions of interest included in each frame of the input image and generates a processed image based on the one or more regions of interest; and an internal encoding performer that encodes the region-of-interest-based processed image, a non-region-of-interest image, and information for restoration to generate a bitstream, wherein the information for restoration includes index information for a plurality of decoding order definition tables, and the plurality of decoding order definition tables can be shared in advance with a VCM decoding device.

[0018] A non-transitory computer-readable recording medium having recorded thereon instructions according to one disclosure of the present specification, wherein the instructions, when executed by one or more processors, cause the one or more processors to: an internal decoding step of generating a decoded reconstructed image from a bitstream and parsing information about the reconstructed image; a region-of-interest-based reconstruction step of reconstructing the reconstructed image based on information related to a region of interest included in the information about the reconstructed image; a step of performing spatial reconstruction by changing a resolution of at least a portion of the reconstructed image based on information related to spatial reconstruction included in the information about the reconstructed image; and a step of performing temporal reconstruction by generating at least one frame of the reconstructed image based on information related to temporal reconstruction included in the information about the reconstructed image, wherein after performing the internal decoding step, an execution order of at least two of the region-of-interest-based reconstruction step, the spatial reconstruction step, and the temporal reconstruction step can be adaptively determined based on information about the reconstructed image.

[0019] According to the present disclosure, image analysis by a machine can be effectively performed.

[0020] Figure 1 schematically illustrates an example of a video / image coding system.

[0021] Figure 2 is a drawing schematically illustrating the configuration of a video / image encoding device.

[0022] Figure 3 is a drawing schematically illustrating the configuration of a video / image decoding device.

[0023] Figures 4a to 4d are exemplary diagrams showing a VCM encoder and a VCM decoder.

[0024] FIG. 5 illustrates a block diagram of an encoding device according to an embodiment of the present disclosure.

[0025] FIG. 6 illustrates detailed operations of an area-of-interest-based processor according to one embodiment of the present disclosure.

[0026] FIG. 7 illustrates detailed operations of an internal encoding performer according to an embodiment of the present disclosure.

[0027] FIG. 8 illustrates detailed operations of an area of ​​interest information encoding performing unit according to one embodiment of the present disclosure.

[0028] FIG. 9 shows a block diagram of a decoding device according to an embodiment of the present disclosure.

[0029] FIG. 10 is an example of a decoding order definition table of a decoding device according to an embodiment of the present disclosure.

[0030] FIG. 11 illustrates detailed operations of an internal decryption performer according to an embodiment of the present disclosure.

[0031] FIG. 12 illustrates detailed operations of an area of ​​interest information decryption performer according to one embodiment of the present disclosure.

[0032] Figures 13a and 13b illustrate detailed operations of a spatial restoration performer according to an embodiment of the present disclosure.

[0033] Figures 14a and 14b illustrate detailed operations of a temporal restoration performer according to an embodiment of the present disclosure.

[0034] Specific structural or step-by-step descriptions of embodiments according to the concept of the present disclosure disclosed in this specification or application are merely illustrative for the purpose of explaining embodiments according to the concept of the present disclosure, and embodiments according to the concept of the present disclosure may be implemented in various forms, and embodiments according to the concept of the present disclosure may be implemented in various forms and should not be construed as being limited to the embodiments described in this specification or application.

[0035] Embodiments according to the concept of the present disclosure may have various modifications and take various forms. Therefore, specific embodiments are illustrated in the drawings and described in detail in this specification or application. However, this is not intended to limit embodiments according to the concept of the present disclosure to specific disclosed forms, and it should be understood that all modifications, equivalents, and alternatives included within the spirit and technical scope of the present disclosure are included.

[0036] While terms such as "first" and / or "second" may be used to describe various components, these components should not be limited by these terms. These terms are only intended to distinguish one component from another; for example, without departing from the scope of the present disclosure, a first component may be referred to as a "second component," and similarly, a second component may also be referred to as a "first component."

[0037] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. Conversely, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Other expressions that describe the relationship between components, such as "between" and "directly between" or "adjacent to" and "directly adjacent to", should be interpreted similarly.

[0038] The terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit the present disclosure. The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, it should be understood that the terms "comprises" or "has" indicate the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0039] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0040] Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0041] In describing the embodiments, description of technical contents that are well known in the technical field to which the present disclosure belongs and are not directly related to the present disclosure will be omitted.

[0042] This is to convey the gist of the present disclosure more clearly without obscuring it by omitting unnecessary explanations.

[0043] This document relates to video / image coding. For example, the method / embodiment disclosed in this document may be related to the Versatile Video Coding (VVC) standard (ITU-T Rec. H.266), the next-generation video / image coding standard after VVC, or other video coding-related standards (e.g., the High Efficiency Video Coding (HEVC) standard (ITU-T Rec. H.265), the essential video coding (EVC) standard, the AVS2 standard, etc.).

[0044] This document presents various embodiments of video / image coding, and unless otherwise stated, the embodiments may be performed in combination with each other.

[0045] In this document, "video" can refer to a series of images over time. "Picture" generally refers to a unit representing a single image from a specific time period, and "slice" / "tile" are units that constitute part of a picture in coding.

[0046] A slice / tile can contain one or more coding tree units (CTUs). A picture can consist of one or more slices / tiles. A picture can consist of one or more tile groups. A tile group can contain one or more tiles.

[0047] A pixel or pel can mean the smallest unit that constitutes a picture (or image). Additionally, the term "sample" can be used as a counterpart to a pixel. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luma component, or only the pixel / pixel value of the chroma component. Alternatively, a sample can mean a pixel value in the spatial domain, or when such a pixel value is converted to the frequency domain, it can mean a transform coefficient in the frequency domain.

[0048] A unit may represent a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to that region.

[0049] A unit can contain one luma block and two chroma (e.g., cb, cr) blocks. The term "unit" may sometimes be used interchangeably with the terms "block" or "area." In general, an MxN block can contain a set (or array) of samples (or array of samples) or transform coefficients, each consisting of M columns and N rows.

[0050] Figure 1 schematically illustrates an example of a video / image coding system.

[0051] Referring to FIG. 1, a video / image coding system may include a source device and a receiving device. The source device may transmit encoded video / image information or data to the receiving device via a digital storage medium or a network in the form of a file or streaming.

[0052] The source device may include a video source, an encoding device, and a transmission unit. The receiving device may include a reception unit, a decoding device, and a renderer.

[0053] The encoding device may be referred to as a video / image encoding device, and the decoding device may be referred to as a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display unit, and the display unit may be configured as a separate device or an external component.

[0054] A video source may obtain video / images through a process of capturing, synthesizing, or generating video / images. The video source may include a video / image capture device and / or a video / image generation device. A video / image capture device may include, for example, one or more cameras, a video / image archive containing previously captured video / images, etc. A video / image generation device may include, for example, a computer, a tablet, a smartphone, etc., and may (electronically) generate video / images. For example, a virtual video / image may be generated through a computer, etc., in which case the video / image capture process may be replaced by a process of generating related data.

[0055] An encoding device can encode input video / images. The encoding device can perform a series of procedures, such as prediction, transformation, and quantization, to improve compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0056] The transmission unit can transmit encoded video / image information or data output in bitstream form to the receiving unit of the receiving device via a digital storage medium or network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit can include an element for generating a media file using a predetermined file format and an element for transmission via a broadcasting / communication network.

[0057] The receiving unit can receive / extract the bitstream and transmit it to a decoding device.

[0058] The decoding device can decode the video / image by performing a series of procedures such as inverse quantization, inverse transformation, and prediction corresponding to the operation of the encoding device.

[0059] The renderer can render decoded video / images. The rendered video / images can be displayed through the display unit.

[0060] Figure 2 is a drawing schematically illustrating the configuration of a video / image encoding device.

[0061] The term “video encoding device” hereinafter may include a video encoding device.

[0062] Referring to FIG. 2, the encoding device (10a) may be configured to include an image partitioner (10a-10), a prediction unit (predictor) (10a-20), a residual processor (residual processor) (10a-30), an entropy encoder (entropy encoder) (10a-40), an adder (adder) (10a-50), a filter (filter) (10a-60), and a memory (10a-70). The prediction unit (10a-20) may include an inter prediction unit (10a-21) and an intra prediction unit (10a-22). The residual processing unit (10a-30) may include a transformer (10a-32), a quantizer (10a-33), a dequantizer (10a-34), and an inverse transformer (10a-35). The residual processing unit (10a-30) may further include a subtractor (10a-31). The addition unit (10a-50) may be called a reconstructor or a reconstructed block generator. The above-described image segmentation unit (10a-10), prediction unit (10a-20), residual processing unit (10a-30), entropy encoding unit (10a-40), addition unit (10a-50), and filtering unit (10a-60) may be configured by one or more hardware components (e.g., encoder chipset or processor) according to an embodiment. In addition, the memory (10a-70) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (10a-70) as an internal / external component.

[0063] The image segmentation unit (10a-10) can segment an input image (or picture, frame) input to the encoding device (10a) into one or more processing units.

[0064] For example, the processing unit may be called a coding unit (CU). In this case, the coding unit may be recursively split from a coding tree unit (CTU) or a largest coding unit (LCU) according to a Quad-tree binary-tree ternary-tree (QTBTTT) structure. For example, one coding unit may be split into multiple coding units of deeper depth based on a quad-tree structure, a binary-tree structure, and / or a ternary structure. In this case, for example, the quad-tree structure may be applied first, and the binary-tree structure and / or the ternary structure may be applied later. Alternatively, the binary-tree structure may be applied first. The coding procedure according to the present document may be performed based on the final coding unit that is no longer split. In this case, based on coding efficiency according to image characteristics, etc., the maximum coding unit can be used as the final coding unit, or, if necessary, the coding unit can be recursively divided into coding units of lower depths, and the coding unit of the optimal size can be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration described below. As another example, the processing unit may further include a prediction unit (PU) or a transformation unit (TU). In this case, the prediction unit and the transformation unit may each be divided or partitioned from the final coding unit described above. The prediction unit may be a unit of sample prediction, and the transformation unit may be a unit for deriving a transform coefficient and / or a unit for deriving a residual signal from a transform coefficient.

[0065] The term "unit" may be used interchangeably with terms such as "block" or "area" depending on the case. In general, an MxN block can represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luminance component, or only the pixel / pixel value of the chroma component. A sample can be used as a term corresponding to a pixel or pel in a picture (or image).

[0066] The subtraction unit (10a-31) can subtract the prediction signal (predicted block, prediction samples, or prediction sample array) output from the prediction unit (10a-20) from the input image signal (original block, original samples, or original sample array) to generate a residual signal (residual block, residual samples, or residual sample array), and the generated residual signal is transmitted to the conversion unit (10a-32). The prediction unit (10a-20) can perform prediction on a block to be processed (hereinafter, referred to as a current block) and generate a predicted block including prediction samples for the current block.

[0067] The prediction unit (10a-20) can determine whether intra-prediction or inter-prediction is applied to the current block or CU unit. As described later in the description of each prediction mode, the prediction unit can generate various prediction-related information, such as prediction mode information, and transmit it to the entropy encoding unit (10a-40). The prediction-related information can be encoded by the entropy encoding unit (10a-40) and output in the form of a bitstream.

[0068] The intra prediction unit (10a-22) can predict the current block by referring to samples within the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from it, depending on the prediction mode.

[0069] In intra prediction, prediction modes can include multiple non-directional modes and multiple directional modes. Non-directional modes can include, for example, DC modes and planar modes. Directional modes can include, for example, 33 directional prediction modes or 65 directional prediction modes, depending on the granularity of the prediction direction.

[0070] However, this is only an example; depending on the settings, a greater or lesser number of directional prediction modes may be used. The intra prediction unit (10a-22) may also determine the prediction mode to be applied to the current block by utilizing the prediction mode applied to the surrounding blocks.

[0071] The inter prediction unit (10a-21) can derive a predicted block for a current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on an inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring block can include a spatial neighboring block existing in the current picture and a temporal neighboring block existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The above temporal neighboring blocks may be called collocated reference blocks, collocated CUs (colCUs), etc., and a reference picture including the temporal neighboring blocks may be called a collocated picture (colPic). For example, the inter prediction unit (10a-21) may construct a motion information candidate list based on the neighboring blocks, and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction may be performed based on various prediction modes, and for example, in the case of skip mode and merge mode, the inter prediction unit (10a-21) may use the motion information of the neighboring blocks as the motion information of the current block. In the case of skip mode, unlike the merge mode, a residual signal may not be transmitted.In the motion vector prediction (MVP) mode, the motion vector of the surrounding blocks is used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0072] The prediction unit (10a-20) can generate a prediction signal based on various prediction methods described below. For example, the prediction unit can apply intra prediction or inter prediction to predict a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). In addition, the prediction unit can perform intra block copy (IBC) to predict a block. The intra block copy can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in this document.

[0073] The prediction signal generated through the inter prediction unit (10a-21) and / or the intra prediction unit (10a-22) can be used to generate a reconstructed signal or a residual signal. The transform unit (10a-32) can generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique can include a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), a Graph-Based Transform (GBT), or a Conditionally Non-linear Transform (CNT). Here, GBT refers to a transform obtained from a graph when the relationship information between pixels is expressed as a graph. CNT refers to a transform obtained based on generating a prediction signal using all previously reconstructed pixels. In addition, the transform process can be applied to a pixel block having a square equal size, or can be applied to a block of a non-square variable size.

[0074] The quantization unit (10a-33) quantizes the transform coefficients and transmits them to the entropy encoding unit (10a-40), and the entropy encoding unit (10a-40) can encode the quantized signal (information about the quantized transform coefficients) and output it as a bitstream. The information about the quantized transform coefficients can be called residual information.

[0075] The quantization unit (10a-33) can rearrange the quantized transform coefficients in the form of a block into a one-dimensional vector based on the coefficient scan order, and can also generate information about the quantized transform coefficients based on the quantized transform coefficients in the form of the one-dimensional vector. The entropy encoding unit (10a-40) can perform various encoding methods, such as, for example, exponential Golomb, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), etc.

[0076] The entropy encoding unit (10a-40) may encode, together or separately, information necessary for video / image restoration (e.g., values ​​of syntax elements, etc.) in addition to quantized transform coefficients. The encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream in the form of a network abstraction layer (NAL) unit. The video / image information may further include information regarding various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. The signaling / transmitted information and / or syntax elements described later in this document may be encoded through the above-described encoding procedure and included in the bitstream. The bitstream may be transmitted through a network or may be stored in a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The signal output from the entropy encoding unit (10a-40) may be configured as an internal / external element of the encoding device (10a) by a transmitting unit (not shown) that transmits and / or a storing unit (not shown) that stores the signal, or the transmitting unit may be included in the entropy encoding unit (10a-40).

[0077] The quantized transform coefficients output from the quantization unit (10a-33) can be used to generate a prediction signal. For example, by applying inverse quantization and inverse transformation to the quantized transform coefficients through the inverse quantization unit (10a-34) and the inverse transform unit (10a-35), a residual signal (residual block or residual samples) can be reconstructed. The addition unit (10a-50) can add the reconstructed residual signal to the prediction signal output from the prediction unit (10a-20), thereby generating a reconstructed signal (reconstructed picture, reconstructed block, reconstructed samples, or reconstructed sample array). When there is no residual for the target block to be processed, such as when skip mode is applied, the predicted block can be used as the reconstructed block. The generated restoration signal can be used for intra prediction of the next processing target block within the current picture, and can also be used for inter prediction of the next picture after filtering as described below.

[0078] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture encoding and / or restoration process.

[0079] The filtering unit (10a-60) can improve subjective / objective picture quality by applying filtering to the restoration signal. For example, the filtering unit (10a-60) can apply various filtering methods to the restoration picture to generate a modified restoration picture, and store the modified restoration picture in the memory (10a-70), specifically, in the DPB of the memory (10a-70). The various filtering methods may include, for example, deblocking filtering, sample adaptive offset (SAO), an adaptive loop filter, a bilateral filter, etc. The filtering unit (10a-60) can generate various information regarding filtering as described below in the description of each filtering method and transmit the information to the entropy encoding unit (10a-90). The information regarding filtering may be encoded by the entropy encoding unit (10a-90) and output in the form of a bitstream.

[0080] The modified restored picture transmitted to the memory (10a-70) can be used as a reference picture in the inter prediction unit (10a-80). Through this, when inter prediction is applied, the encoding device can avoid prediction mismatch between the encoding device (10a) and the decoding device, and can also improve encoding efficiency.

[0081] The DPB of the memory (10a-70) can store the modified reconstructed picture to be used as a reference picture in the inter prediction unit (10a-21). The memory (10a-70) can store motion information of a block from which motion information in the current picture is derived (or encoded) and / or motion information of blocks in a picture that has already been reconstructed. The stored motion information can be transmitted to the inter prediction unit (10a-21) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (10a-70) can store reconstructed samples of reconstructed blocks in the current picture and transmit them to the intra prediction unit (10a-22).

[0082] Figure 3 is a drawing schematically illustrating the configuration of a video / image decoding device.

[0083] Referring to FIG. 3, the decoding device (10b) may be configured to include an entropy decoder (10b-10), a residual processor (10b-20), a predictor (10b-30), an adder (10b-40), a filter (10b-50), and a memory (10b-60). The predictor (10b-30) may include an inter-prediction unit (10b-31) and an intra-prediction unit (10b-32). The residual processor (10b-20) may include a dequantizer (10b-21) and an inverse transformer (10b-21). The entropy decoding unit (10b-10), residual processing unit (10b-20), prediction unit (10b-30), addition unit (10b-40), and filtering unit (10b-50) described above may be configured by a single hardware component (e.g., decoder chipset or processor) according to an embodiment. In addition, the memory (10b-60) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (10b-60) as an internal / external component.

[0084] When a bitstream including video / image information is input, the decoding device (10b) can restore the image corresponding to the process in which the video / image information is processed in the encoding device of FIG. 2. For example, the decoding device (10b) can derive units / blocks based on block division-related information obtained from the bitstream. The decoding device (10b) can perform decoding using a processing unit applied in the encoding device. Therefore, the processing unit of decoding may be, for example, a coding unit, and the coding unit may be divided from a coding tree unit or a maximum coding unit according to a quad tree structure, a binary tree structure, and / or a ternary tree structure. One or more transform units may be derived from the coding unit. Then, the restored image signal decoded and output by the decoding device (10b) can be reproduced through a reproduction device.

[0085] The decoding device (10b) can receive a signal output from the encoding device of FIG. 2 in the form of a bitstream, and the received signal can be decoded through the entropy decoding unit (10b-10). For example, the entropy decoding unit (10b-10) can parse the bitstream to derive information (e.g., video / image information) necessary for image restoration (or picture restoration). The video / image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information.

[0086] The decoding device can further decode the picture based on information about the parameter set and / or the general restriction information. The signaling / received information and / or syntax elements described later in this document can be decoded and obtained from the bitstream through the decoding procedure. For example, the entropy decoding unit (10b-10) can decode information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output values ​​of syntax elements required for image restoration and quantized values ​​of transform coefficients for the residual.

[0087] In more detail, the CABAC entropy decoding method receives a bin corresponding to each syntax element in a bitstream, determines a context model using information of the syntax element to be decoded and decoding information of surrounding and decoding target blocks or information of symbols / bins decoded in a previous step, and predicts the occurrence probability of the bin according to the determined context model to perform arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element. At this time, the CABAC entropy decoding method can update the context model using information of the decoded symbol / bin for the context model of the next symbol / bin after determining the context model. Among the information decoded by the entropy decoding unit (10b-10), information regarding prediction is provided to the prediction unit (10b-30), and information regarding the residual on which entropy decoding has been performed by the entropy decoding unit (10b-10), i.e., quantized transform coefficients and related parameter information, can be input to the dequantization unit (10b-21).

[0088] In addition, information regarding filtering among the information decoded by the entropy decoding unit (10b-10) may be provided to the filtering unit (10b-50). Meanwhile, a receiving unit (not shown) that receives a signal output from an encoding device may be further configured as an internal / external element of the decoding device (10b), or the receiving unit may be a component of the entropy decoding unit (10b-10). Meanwhile, the decoding device according to the present document may be called a video / video / picture decoding device, and the decoding device may be divided into an information decoder (video / video / picture information decoder) and a sample decoder (video / video / picture sample decoder). The above information decoder may include the entropy decoding unit (10b-10), and the sample decoder may include at least one of the inverse quantization unit (10b-21), the inverse transformation unit (10b-22), the prediction unit (10b-30), the addition unit (10b-40), the filtering unit (10b-50), and the memory (10b-60).

[0089] The inverse quantization unit (10b-21) can inverse quantize the quantized transform coefficients and output the transform coefficients. The inverse quantization unit (10b-21) can rearrange the quantized transform coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scanning order performed in the encoding device. The inverse quantization unit (10b-21) can perform inverse quantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain transform coefficients.

[0090] In the inverse transform unit (10b-22), the transform coefficients are inversely transformed to obtain a residual signal (residual block, residual sample array).

[0091] The prediction unit can perform a prediction for the current block and generate a predicted block including prediction samples for the current block.

[0092] The prediction unit can determine whether intra prediction or inter prediction is applied to the current block based on the information about the prediction output from the entropy decoding unit (10b-10), and can determine a specific intra / inter prediction mode.

[0093] The prediction unit can generate a prediction signal based on various prediction methods described below. For example, the prediction unit can apply intra prediction or inter prediction to predict a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). In addition, the prediction unit can perform intra block copy (IBC) to predict a block. The intra block copy can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in this document.

[0094] The intra prediction unit (10b-32) can predict the current block by referencing samples within the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from it, depending on the prediction mode.

[0095] In intra prediction, prediction modes may include multiple non-directional modes and multiple directional modes. The intra prediction unit (10b-32) may determine the prediction mode to be applied to the current block by utilizing the prediction modes applied to the surrounding blocks.

[0096] The inter prediction unit (10b-31) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on an inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.).

[0097] In the case of inter prediction, the neighboring blocks may include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit (10b-31) may construct a motion information candidate list based on the neighboring blocks, and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter prediction may be performed based on various prediction modes, and the information regarding the prediction may include information indicating the mode of inter prediction for the current block.

[0098] The addition unit (10b-40) can generate a restoration signal (restored picture, restoration block, restoration sample array) by adding the acquired residual signal to the prediction signal (predicted block, prediction sample array) output from the prediction unit (10b-30). In cases where there is no residual for the block to be processed, such as when skip mode is applied, the predicted block can be used as the restoration block.

[0099] The addition unit (10b-40) may be called a restoration unit or a restoration block generation unit.

[0100] The generated restoration signal can be used for intra prediction of the next processing target block within the current picture, can be output after filtering as described below, or can be used for inter prediction of the next picture.

[0101] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture decoding process.

[0102] The filtering unit (10b-50) can improve subjective / objective image quality by applying filtering to the restoration signal. For example, the filtering unit (10b-50) can apply various filtering methods to the restoration picture to generate a modified restoration picture, and transmit the modified restoration picture to the memory (60), specifically, the DPB of the memory (10b-60). The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0103] The (corrected) reconstructed picture stored in the DPB of the memory (10b-60) can be used as a reference picture in the inter prediction unit (10b-31). The memory (10b-60) can store motion information of a block from which motion information is derived (or decoded) within the current picture and / or motion information of blocks within a picture that has already been reconstructed. The stored motion information can be transmitted to the inter prediction unit (10b-31) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (10b-60) can store reconstructed samples of reconstructed blocks within the current picture and transmit them to the intra prediction unit (10b-32).

[0104] In this specification, the embodiments described in the prediction unit (10b-30), the inverse quantization unit (10b-21), the inverse transformation unit (10b-22), and the filtering unit (10b-50) of the decoding device (10b) can be applied to the prediction unit (10a-20), the inverse quantization unit (10a-34), the inverse transformation unit (10a-35), and the filtering unit (10a-60) of the encoding device (10a) in the same manner or correspondingly.

[0105] As described above, prediction is performed to increase compression efficiency when performing video coding. Through this, a predicted block including prediction samples for a current block, which is a coding target block, can be generated. Here, the predicted block includes prediction samples in a spatial domain (or pixel domain). The predicted block is derived identically from an encoding device and a decoding device, and the encoding device can increase video coding efficiency by signaling information (residual information) about the residual between the original block and the predicted block, rather than the original sample value of the original block itself, to a decoding device. The decoding device can derive a residual block including residual samples based on the residual information, and generate a reconstructed block including reconstructed samples by combining the residual block and the predicted block, and can generate a reconstructed picture including the reconstructed blocks.

[0106] The above residual information can be generated through transformation and quantization procedures.

[0107] For example, the encoding device can derive a residual block between the original block and the predicted block, perform a transform procedure on residual samples (a residual sample array) included in the residual block to derive transform coefficients, perform a quantization procedure on the transform coefficients to derive quantized transform coefficients, and signal related residual information to a decoding device (via a bitstream). Here, the residual information can include information such as value information, position information, transform technique, transform kernel, and quantization parameter of the quantized transform coefficients. The decoding device can perform an inverse quantization / inverse transform procedure based on the residual information to derive residual samples (or residual blocks). The decoding device can generate a reconstructed picture based on the predicted block and the residual block. The encoding device can also inverse quantize / inverse transform the quantized transform coefficients to derive a residual block for reference in inter prediction of a subsequent picture, and generate a reconstructed picture based on the residual block.

[0108] <VCM(Video coding for Machines)>

[0109] With the recent advancements in various industries such as surveillance, intelligent transportation, smart cities, intelligent industry, and intelligent content, the amount of image or feature map data consumed by machines is increasing. In contrast, traditional video compression methods currently in use were developed with human vision in mind, and therefore contain unnecessary information, making them inefficient for machine tasks. For example, the resolution of images from the viewer's perspective may be higher than that of images (e.g., feature maps) from the machine's perspective. Therefore, research on video codec technologies that efficiently compress feature maps for machine tasks is needed.

[0110] The Moving Picture Experts Group (MPEG), an international standardization group for multimedia encoding, is discussing Video Coding for Machines (VCM). VCM is an image or feature map encoding technology that targets machine vision, rather than human viewer vision. In this document, feature maps can be referred to as "feature maps," and features can be referred to as "features."

[0111] Figures 4a to 4d are exemplary diagrams showing a VCM encoder and a VCM decoder.

[0112] Referring to FIG. 4a, a VCM encoder (100a) and a VCM decoder (100b) are shown.

[0113] When a VCM encoder (100a) encodes a video and / or a feature map and transmits it as a bitstream, a VCM decoder (100b) can decode and output the bitstream. At this time, the VCM decoder (100b) can output one or more videos and / or feature maps. For example, the VCM decoder (100b) can output a first feature map for machine-based analysis and a first image for user viewing. The first image can have a higher resolution than the first feature map.

[0114] Referring to FIG. 4b, a feature extractor for extracting a feature map may be connected to the front end of the VCM encoder (100a).

[0115] The VCM encoder (100a) may include a feature encoder.

[0116] The VCM decoder (100b) may include a feature decoder and a video reconstructor. The feature decoder may decode a feature map from a bitstream and output a first feature map for machine-assisted analysis. The video reconstructor may regenerate and output a first video from the bitstream for viewing by a user.

[0117] Referring to Fig. 4c, a feature extractor for extracting a feature map is connected to the front end of the VCM encoder (100). The VCM encoder (100a) may include a feature encoder.

[0118] The VCM decoder (100b) may include a feature decoder. The feature decoder may decode a feature map from a bitstream and output a first feature map for machine-based analysis. That is, the bitstream may be encoded only as a feature map, not as an image. To elaborate, the feature map may be data containing information about features for processing a specific task of a machine based on an image.

[0119] Referring to FIG. 4d, a feature extractor may be connected to the front end of the VCM encoder (100a).

[0120] The VCM encoder (100a) may include a feature converter and a video encoder. The video encoder may be the encoding device (10a) illustrated in FIG. 2.

[0121] The VCM decoder (100b) illustrated in FIG. 4d may include a video decoder and an inverse converter. The video decoder may be the decoding device (10b) illustrated in FIG. 3.

[0122] FIG. 5 illustrates a block diagram of an encoding device according to an embodiment of the present disclosure.

[0123] In various embodiments of the present disclosure, the sub- / decoding may be performed by applying at least one of region-of-interest (ROI)-based processing, temporal resampling, and spatial resampling in the VCM. In various embodiments, the sub- / decoder may perform the ROI-based processing, temporal resampling / restoration, and spatial resampling / restoration in a predetermined order. In various embodiments, the sub- / decoder may change the order of the ROI-based processing, temporal resampling / restoration, and spatial resampling / restoration in the sub- / decoder depending on the type of data set and task. In one embodiment, the original image may be restored by explicitly or implicitly changing the order of the ROI-based processing, temporal restoration, and spatial restoration in the restoration process according to a series of units such as sequence units / GOP (group of pictures) units. In one embodiment, the sub- / decoder may change the order of each process using information such as sequence levels.

[0124] In various embodiments, the performance of the machine vision task can ultimately be improved by adaptively determining the order of region-of-interest-based processing, temporal resampling / restoration, and spatial resampling / restoration during the encoding / decoding process based on the characteristics of the image and / or the characteristics of the machine vision task.

[0125] An encoding device (10a) according to an embodiment of the present disclosure can receive an image (video) and generate a bitstream through an encoding process for a machine. The encoding device (10a) can include a temporal resampling performer (510), a spatial resampling performer (520), a region of interest-based processor (530), a bit depth converter (540), and an internal encoding performer (550).

[0126] The input image (video) to be encoded may be an original input image, and / or may be one or more feature maps extracted from the input image by a neural network. In the present disclosure, the term "image" may refer to the image (video) itself, and / or may refer to a "feature map."

[0127] The order of components included in the encoding device (10a) illustrated in FIG. 5 may be changed in various embodiments, and some components may be omitted. Depending on the embodiment, the temporal resampling performer (510) and / or the spatial resampling performer (520) may be omitted, and when the temporal resampling performer (510) and / or the spatial resampling performer (520) are omitted, an image processed based on a region of interest by a region of interest-based processor (530) may become an input to the internal encoding performer (550). For example, if the temporal resampling performer (510) and the spatial resampling performer (520) are omitted, one or more regions of interest are directly extracted from the input image by the region-of-interest-based processor (530), and the extracted regions of interest are input to the internal encoding performer (550), so that the internal encoding performer (550) can perform encoding to generate a bitstream.

[0128] For convenience of explanation in the present disclosure, the encoding device (10a) may be referred to as an encoder or encoder, and the decoding device (10b) may be referred to as a decoder or decoder.

[0129] The temporal resampling performer (510) can receive an image, perform frame-by-frame sampling, and output an image in which some frames are sampled. The temporal resampling performer (510) can change the frame rate of some frames. According to an embodiment, the temporal resampling performer (510) can perform the same degree of sampling on the entire image. That is, the temporal resampling performer (510) can resample the entire image at the same frame rate. According to another embodiment, the temporal resampling performer (510) can perform temporal resampling by applying different sampling degrees to each group of frames in the image. That is, the temporal resampling performer (510) can resample at a variable frame rate on a group-by-group basis.

[0130] In one embodiment, the temporal resampling performer (510) may perform sampling at different sampling rates for the region-of-interest-based processed image and the non-region-of-interest image. For example, the temporal resampling performer (510) may perform sampling at a first sampling rate for the region-of-interest-based processed image and at a second sampling rate for the non-region-of-interest image.

[0131] According to an embodiment, the temporal resampling performer (510) may transmit information used in the temporal resampling process (e.g., temporal sampling rate, etc.) to the decoder. After the decoder performs decoding on the region-of-interest-based processed image and the non-region-of-interest image, the decoder may parse the respective sampling rates corresponding to the region-of-interest-based processed image and the non-region-of-interest image to perform upsampling on each. At this time, the sampling rate applied to the region-of-interest-based processed image and the sampling rate applied to the non-region-of-interest image may be different.

[0132] The spatial resampling performer (520) can receive an input image or an image on which temporal resampling has been performed and output an image with a changed spatial resolution for each frame or a series of frames. The spatial resampling performer (520) can perform frame-by-frame or sequence-by-sequence sampling on the input image to output an image in which one or more frames in the sequence are spatially sampled (with a changed spatial resolution). The spatial resampling performer (520) can change the resolution for a region of interest included in each frame of the input image. In one embodiment, the spatial resampling performer (520) can change the resolution only for the region of interest. If there are multiple regions of interest in one frame, the spatial resampling performer (520) can change the resolution for each region of interest. Information on each region of interest can include changed resolution information. For example, a first scale factor can be applied to a first region of interest, and a second scale factor can be applied to a second region of interest. The scale factor can be stored using at least one of the following parameters: scale factor, scale factor nominator, scale factor, denominator, size(%), etc.

[0133] In one embodiment, the spatial resampling performer (520) can perform spatial sampling to have the same spatial resolution size through the same degree of sampling throughout the sequence. The resampling method for each input image can be any one of bilinear downsampling, bilateral downsampling, and a deep neural network including one or more convolutional layers. The spatial resampling performer (520) can select a method corresponding to each input image from among multiple resampling methods and signal it to the decoder.

[0134] In another embodiment, the spatial resampling performer (520) can perform spatial resampling at different sampling levels (variable spatial resolution) for each group of frames within an image.

[0135] The spatial resampling performer (520) can transmit information used in the spatial resampling process (e.g., a spatial sampling rate of each frame and / or a spatial sampling rate per series of frames, etc.) to the decoder. After decoding the bitstream, the decoder can parse the resolution information for the region-of-interest-based processed image and the non-region-of-interest image and change the resolution of each. At this time, the resolution applied to the region-of-interest-based processed image and the resolution applied to the non-region-of-interest may be different.

[0136] The region-of-interest-based processor (530) can receive an input image, an image on which temporal resampling has been performed, or an image on which temporal resampling and spatial resampling have been performed, extract a region of interest existing in each frame or a series of frames, and output a processed image based on the extracted region of interest. The region-of-interest-based processor (530) can transmit information used in the region-of-interest-based processing (in one example, region-of-interest ID information, region-of-interest location, region-of-interest size information, packing information, temporal sampling rate, spatial resolution, etc.) to a decoder.

[0137] The bit depth converter (540) can receive an image or an image on which at least one of the above processes has been performed, convert the bit depth of the image, and apply it as an input to the internal encoding performer (550). According to an embodiment, the bit depth converter (540) can change only the bit depth of a specific component of the input data. According to an embodiment, the bit depth converter (540) can transmit one or more of the information used in the bit depth conversion process (in one example, whether bit depth conversion has been performed, the value of the converted bit depth, the difference value between the bit depth of the original image and the converted bit depth, etc.) to the decoder.

[0138] The internal encoding performer (550) can receive an input image, or an image on which some or all of temporal / spatial resampling and region of interest processing have been performed, and perform image encoding to generate a bitstream. According to an embodiment, the internal encoding performer (550) can use a 2D video encoder (e.g., AVC / H.264, HEVC / H.265, VVC / H.266, AV1, VP9, ​​etc.) and can use a 2D video encoder including one or more convolution layers. According to an embodiment, encoding can be performed after converting the color space of the input image of the internal encoding performer (550) to a color space such as YUV420 or YUV444. At this time, according to an embodiment, the internal encoding performer (550) can convert through a conversion method defined by an agreement between the decoder and the decoder, and can transmit color space conversion information to the decoder.

[0139] FIG. 6 illustrates detailed operations of an area-of-interest-based processor according to one embodiment of the present disclosure.

[0140] According to one embodiment of the present disclosure, an area-of-interest-based processor (530) of an encoding device (10a) may include a frame analysis unit (610), an area-of-interest selection unit (620), and an area-of-interest-based image processing unit (630). In one embodiment, the frame analysis unit (610) may include an analysis area selection module (611) and an area-of-interest candidate extraction module (612). In one embodiment, the area-of-interest selection unit (620) may include an area-of-interest selection module (621) and a corresponding area-of-interest selection module (622). In various embodiments, some of the detailed components of the area-of-interest-based processor (530) may be omitted, and the order of the components in FIG. 6 may be changed.

[0141] The frame analysis unit (610) can receive an input image and output information (for example, the location of the region of interest, the size of the region of interest, the score for the region of interest, etc.) of one or more region of interest candidates existing in each frame (picture) of the image. The input image may be an original image, or may be an image to which temporal and / or spatial resampling has been applied according to various embodiments. The frame analysis unit (610) may include an analysis region selection module (611) and a region of interest candidate extraction module (612). Some of the detailed components of the frame analysis unit (610) may be omitted or the execution order may be changed.

[0142] The analysis region selection module (611) can be performed in units of one or more frames, and can select regions on which frame analysis is to be performed, and then the frame analysis module can be performed only on some regions within the image. Information on some regions can be transmitted to the decoder. Some regions can be the same region for all images. Regions can be the same within an image and a series of frame groups within the image, and can be different within a series of frame groups. Regions on which frame analysis is not performed may not be transmitted to the decoder.

[0143] The region of interest candidate extraction module (612) may utilize a deep neural network composed of one or more convolutional layers on which training has been performed, depending on the embodiment. Depending on the embodiment, the deep neural network may be a network trained for the purpose of object detection or object tracking. Depending on the embodiment, region of interest candidates may be extracted frame by frame, or may be extracted for a series of frames. Depending on the embodiment, for each frame or a series of frame groups, if the detection accuracy of a detected object (region of interest candidate) is low, it may be excluded from the region of interest candidates.

[0144] In some embodiments, when a deep neural network trained for the purpose of object tracking is used in the region of interest candidate extraction module (612), the corresponding region of interest selection module (622) may be omitted. In some embodiments, when the corresponding region of interest selection module (622) is performed in units of a series of frame groups, the region of interest candidate (detected object) of the first input frame may be selected as a candidate, and the corresponding region of interest may be selected through a correspondence relationship with the region of interest candidates of subsequent frames. In some embodiments, the correspondence relationship may be obtained based on pixel-to-pixel similarity (for example, cross-correlation and / or histogram comparison of pixel values) between the entire region or a portion of each detected object. In some embodiments, the size of the region of interest candidate in each frame may be a multiple of 4 in both height and width.

[0145] The region of interest selection unit (620) can select a region of interest from among region of interest candidates, determine a corresponding relationship among multiple regions of interest, and select a corresponding region of interest. The region of interest selection module (621) can determine a region to be encoded as a region of interest from among the region of interest candidates output from the frame analysis unit (610). The selected region of interest can be transmitted to the internal encoding performer (550) for image encoding.

[0146] In some embodiments, the non-region of interest image may be an area outside the region of interest within the frame, and the portion corresponding to the region of interest may be filled with a specific value (e.g., surrounding pixel values, median values, etc.). In some embodiments, the region of interest selected within the image may be different for each frame, or may be different for each series of frame groups. In some embodiments, when the trained deep neural network is used in the region of interest candidate extraction module (612), for each frame or series of frame groups, if the detection accuracy of the detected object (region of interest candidate) is low, the detected object may be excluded from the region of interest candidates, and the detected objects may be selected as the region of interest. In some embodiments, the threshold value for accuracy may be the same for the entire image, or may be different for each series of frame groups.

[0147] The corresponding region of interest selection module (622) may, according to an embodiment, perform downsampling or upsampling of all region of interest candidates to a specific size, and then perform the corresponding region of interest selection process. According to an embodiment, the specific size may be determined according to the resolution of the image, and / or may be determined as the size of the region of interest with the smallest size by comparing the sizes of region of interest candidates extracted for each frame. According to an embodiment, the corresponding region of interest selection module (622) may assign an ID to the selected region of interest, and this ID may be assigned for each unit in which the corresponding region of interest is selected. According to an embodiment, when the regions of region of interest candidates extracted by the region of interest candidate extraction module (612) within one frame overlap each other, the region of interest selection module (621) may classify the regions of interest as one region of interest when the overlapping region exceeds a certain threshold value. At this time, the threshold value may be determined according to the size of the frame and may be determined according to the size of the region of interest candidate. The corresponding region of interest selection module (622) may, according to an embodiment, assign a new ID to a region of interest in a current frame if there is no corresponding region of interest in a previous frame. In an embodiment, for a region of interest assigned a new ID, if there is a corresponding region of interest in a previous frame in the sequence, the existing ID of the corresponding region of interest may be assigned instead of a new ID. In an embodiment, this may be determined by a method of finding a correspondence relationship.

[0148] Depending on the embodiment, non-region-of-interest images of all frames and / or some frames may be transmitted to the internal encoding performer (550).

[0149] The region of interest-based image processing unit (630) may receive information on regions of interest selected in an image and generate a processed image based on the region of interest for each frame using the selected regions of interest according to various embodiments. In one example, the region of interest-based image processing unit (630) may pack regions of interest and generate each frame in which the regions of interest are packed. In one embodiment, the region of interest-based image processing unit (533) may fill non-regions of interest excluding the region of interest in a frame with at least one specific value (e.g., an intermediate value, etc.) and generate a frame in which the non-regions of interest are filled with the specific values. In various embodiments, the region of interest-based image processing unit (533) may determine processing for non-regions of interest in various ways. For example, the region of interest may not be encoded in whole or in part, the region of interest may be filled with a specific value determined in advance, or the non-regions of interest may be processed adaptively to the region of interest. Such embodiments are merely examples, and the region of interest-based image processing unit (533) can process the region of interest differently from the region of no interest, and can signal information about the region of interest so that the decoder can restore the region of no interest using only information about the region of interest.

[0150] According to an embodiment, the region of interest-based processor (530) may transmit information used in the region of interest extraction process (in one example, region of interest identification (ID) information, region of interest location, region of interest size information, packing information, etc.) to the decoder.

[0151] FIG. 7 illustrates detailed operations of an internal encoding performer according to an embodiment of the present disclosure.

[0152] An internal encoding performer (550) included in an encoding device (10a) according to one embodiment of the present disclosure can encode an image processed based on a region of interest and an image of a non-region of interest and encode region of interest information to generate a bitstream. In various embodiments, the internal encoding performer (550) may include a downsampling performer (710), an image encoding performer (720), and a region of interest information encoding performer (730). In one embodiment, the region of interest information encoding performer (730) may include a region of interest to be encoded selection module (810), a frame to be encoded selection module (820), and an information encoding performer module (830). In various embodiments, some of the detailed components of the internal encoding performer (550) may be omitted, and the order between the components may be changed.

[0153] The internal encoding performer (550) can perform encoding for each frame of an image processed based on a region of interest, and can perform encoding for non-region of interest images only for some frames within a series of frame groups. According to an embodiment, the image encoding performer (720) can encode information about a region of interest within an image processed based on a region of interest downsampled by the downsampling performer (710). The bitstreams each output through the image encoding performer (720) and the region of interest information encoding performer (730) can be multiplexed to become a single bitstream.

[0154] According to an embodiment, the downsampling unit (710) may perform downsampling at different sampling rates for the region-of-interest-based processed image and the non-region-of-interest image before performing image encoding. According to an embodiment, when downsampling is performed, sampling rate information may be signaled to the decoder.

[0155] The video encoding unit (720) can use a video encoder (AVC / H.264, HEVC / H.265, VVC / H.266, AV1, VP9, ​​etc.) and can use a 2D video encoder including one or more convolution layers. The video encoding unit (720) can use the same video encoder for the region-of-interest-based processed image and the non-region-of-interest image, or can use different video encoders.

[0156] The region of interest information encoding unit (730) can encode information (e.g., location, movement, mapping information, etc.) about regions of interest included in all or some frames of an input image. The region of interest information encoding unit (730) can utilize some processes (e.g., entropy coding, etc.) of the video encoder of the image encoding unit (720).

[0157] In one embodiment, the internal encoding performer (550) may apply the downsampling performer (710) and the image encoding performer (720) to the non-region of interest image. In one embodiment, the internal encoding performer (550) may apply the downsampling performer (710) and the image encoding performer (720) to the region of interest-based processed image, and separately apply the region of interest information encoding performer (730) to the information about the region of interest to generate a bitstream. In one embodiment, the internal encoding performer (550) may first perform the downsampling performer (710) to the region of interest-based processed image, then apply the region of interest information encoding performer (730) to the information about the region of interest, and then apply the image encoding performer (720) to the region of interest-based processed image to generate a bitstream.

[0158] FIG. 8 illustrates detailed operations of an area of ​​interest information encoding performing unit according to one embodiment of the present disclosure.

[0159] An area of ​​interest information encoding performing unit (730) included in an internal encoding performing unit (550) of an encoding device (10a) according to one embodiment of the present disclosure can encode area of ​​interest information for an area of ​​interest-based processed image to generate a bitstream. The area of ​​interest information encoding performing unit (730) may include an area of ​​interest selection module (810) to be encoded, a frame selection module (820) to be encoded, and an information encoding performing module (830). In various embodiments, some components of FIG. 8 may be omitted, and the execution order may be changed.

[0160] The region of interest selection module (810) to be encoded can receive an image processed based on a region of interest and select a region of interest for which information encoding is to be performed among the selected regions of interest. Depending on the embodiment, the target of information encoding may be one or more of the following: motion information of the region of interest, compensation parameters, etc., and information about these may be signaled.

[0161] The frame selection module (820) to be encoded may receive a region of interest for performing information encoding or / and a region-of-interest-based processed image, and select a frame among the frames in the image for performing information encoding. According to an embodiment, information of all regions of interest within the selected frame may be encoded. According to an embodiment, only information of regions of interest for which information encoding is to be performed, selected from a previous process, within the selected frame may be encoded.

[0162] According to an embodiment, the information encoding performing module (830) may generate a bitstream by encoding information of selected regions of interest within a frame selected from a previous process.

[0163] FIG. 9 shows a block diagram of a decoding device according to an embodiment of the present disclosure.

[0164] A decoding device (10b) according to an embodiment of the present disclosure can receive a bitstream, perform decoding, and output a restored image. In various embodiments, the decoding device (10b) can include an internal decoding performer (910), a region-of-interest-based restorer (920), a spatial restoration performer (930), a temporal restoration performer (940), a bit-depth inverse converter (950), and a post-processing filter performer (960). In various embodiments, some components of FIG. 9 can be omitted, and the execution order can be adaptively changed.

[0165] Depending on the embodiment, the order of the region-of-interest-based restoration process, the spatial / temporal restoration process, the bit depth inverse conversion process, etc., which may be performed after performing internal decoding, may be performed in the reverse order of the order in which each corresponding process (e.g., region-of-interest-based processing, spatial / temporal resampling, bit depth conversion, etc.) is performed during the image encoding process, or may be performed in an order different from the reverse order during the encoding process.

[0166] In various embodiments, the decoding device (10b) may adaptively change the order of each decoding process. In one embodiment, the order of each process may vary for each sequence, for each group of frames within a sequence, or for each slice within a frame. Depending on the embodiment, the order of execution of only some processes within each decoding process may be changed.

[0167] In some embodiments, the order information of the decoding process may be signaled by the encoding device (10a), and the decoding device (10b) may parse the corresponding information within the bitstream. In various embodiments, some fixed processes and some adaptively variable processes may differ for each specific unit in the entire decoding process. The decoding device (10b) may parse and determine information about the fixed portion and the adaptively variable portion in the entire decoding process. In one embodiment, the fixed portion and the adaptively variable portion may differ for each sequence unit in the decoding process.

[0168] According to an embodiment, the decoding device (10b) can implicitly change information for restoration within the decoding process (in one example, spatial restoration performance information, spatial restoration performance method, temporal restoration performance method, etc.) according to the order of the decoding process, and can configure candidates of available information differently and explicitly determine them.

[0169] An internal decoding performer (910) can receive a bitstream and perform image decoding to generate a restored image. At this time, the restored image may include an image of a region of interest and an image of a region of non-interest.

[0170] According to an embodiment, the internal decoding performer (910) may utilize a 2D video decoder (AVC / H.264, HEVC / H.265, VVC / H.266, AV1, VP9, ​​etc.), a 2D video decoder including one or more convolution layers, or / and a hybrid form of the two types of decoders specified above. According to an embodiment, if the color space of the reconstructed image is not RGB444 but one of YUV420, YUV444, etc., the internal decoding performer (910) may implicitly or explicitly convert the color space of the reconstructed image to RGB444 space and then perform the image decoding process thereafter. According to an embodiment, the internal decoding performer (910) may implicitly or explicitly convert the color space of the reconstructed image to another color space and then perform the image decoding process thereafter.

[0171] The region-of-interest-based restorer (920) may, according to an embodiment, restore a region-of-interest-based processed image using the decoded image and region-of-interest-based processing information (for example, region-of-interest ID information, region-of-interest size information, packing information, etc.) transmitted from the encoder.

[0172] The spatial restoration performer (930) may perform spatial restoration using, according to an embodiment, information used in the spatial resampling process transmitted from the decoded image and the encoder (for example, the spatial sampling rate of each frame and / or the spatial sampling rate of a series of frames, etc.).

[0173] The temporal restoration performer (940) may perform temporal restoration using, according to an embodiment, information (for example, temporal sampling rate, etc.) used in the temporal resampling process transmitted from the decoded image and the encoder.

[0174] The bit depth inverse converter (950) may, according to an embodiment, compensate for the bit depth of a specific component or / and all components of the restored data by using the information used in the bit depth conversion process transmitted from the decoded image and the encoder.

[0175] The post-processing filter performer (960) may, depending on the embodiment, perform filtering on an image on which the above process has been partially or fully performed. In this case, depending on the embodiment, a fixed filter may be used, or multiple filters may be defined by agreement with the encoder, and filter information may be received from the encoder to perform filtering.

[0176] FIG. 10 is an example of a decoding order definition table of a decoding device according to an embodiment of the present disclosure.

[0177] A decoding device (10b) according to an embodiment of the present disclosure can adaptively determine the decoding order of a bitstream based on the characteristics of a restored image and the characteristics of a machine vision task. In various embodiments, the decoding device (10b) can adaptively determine the order of the decoding process in units of sequences or GOPs, etc., using information such as sequence levels.

[0178] As in the example of Fig. 10, among the components of the decoding device (10b), only the execution order of the remaining components, excluding the internal decoding performer (910) and the post-processing filter performer (960), can be changed. According to an embodiment, a fixed order exists for each decoding process, and by checking the order change information, image restoration can be performed by changing the order only in some units. According to an embodiment, the order of each decoding process can be managed in the form of one or more tables defined in the agreement between the sub-decoder and the decoder for an integer N greater than or equal to 1, as in Fig. 10.

[0179] Referring to FIG. 10, the index 0 item indicates that decoding is performed in the order of a temporal restoration performer (940), a spatial restoration performer (930), a region of interest-based restorer (920), and a bit depth inverse converter (950).

[0180] Depending on the embodiment, the entries within each table may only define the order of the components that are changed among the components of the decoding device (10b) of FIG. 9. As an example, the N index entry of FIG. 10 represents a case where spatial restoration is not performed.

[0181] According to an embodiment, multiple tables such as the example of FIG. 10 may be defined by an agreement between the decoder and the decoder, and which table to use for each sequence unit, GOP unit, etc. may be parsed from the bitstream to determine. In addition, the available decoding order definition tables may be different from each other depending on the application and / or reference structure, etc. According to an embodiment, for the decoding order definition table determined in the above process, index information regarding which order of elements to use may be parsed. According to an embodiment, when the order of each decoding process of FIG. 10 is managed with a table such as the example of FIG. 10, table information (e.g., table number and index within the table, etc.) may be parsed for each order change unit, and the transmission unit may be different for multiple details within the table information.

[0182] Tables 1 to 4 illustrate an example of a parsing structure of syntax for adaptively determining a restoration order of a decoding device (10b) according to one embodiment of the present disclosure and performing image restoration. Each table includes a description of each syntax element.

[0183] Table 1 is an embodiment of determining a restoration order for each unit by parsing the index of the table by each frame within a GOP and / or by a group of frames within a GOP that includes at least one frame.

[0184] SyntaxDescriptorSemanticsAdaptive_order_information () {target_application_idxue(v)Index information about the application to be used by the decoder. Example) 0 is not specified, 1 is tracking, 2 is recognition, 3 is classification, 4 is segmentation, 5 is other applications. gop_numue(v)Number of GOPs in the sequence. Depending on the example, the number of frames in a GOP may be different for each GOP. for (i=0; i <gop_num; i++) {adaptive_ordering_flague(1)복호화 과정의 적응적 순서 변경 사용 여부를 나타내는 플래그실시 예에 따라, adaptive_ordering_flag가 0인 경우, 해당 GOP내 각 프레임 복원은 고정된 순서로 수행될 수 있다.Here, the fixed order is an order determined by the promise of the encryption / decryption period, which may be the restoration order of FIG. 9. According to an embodiment, the fixed order described above may be a form determined based on at least one of the application, the reference structure type, etc. if (adaptive_ordering_flag) {table_numue(v) Information on which table to use among the list of tables available according to the application for (j=0; j <frames; j++) {frames는 각 GOP 내의 프레임 수를 의미실시 예에 따라, 각 프레임 단위로 서로 다른 순서로 프레임이 복원될 수 있으며, 이때 프레임 단위 순서 변경이 수행되는 경우, 도 9의 시간적 복원 수행 과정은 생략되거나, 다른 과정 (일 예시로, 비트 심도 역변환, 공간적 복원, 관심 영역 기반 복원)이 모두 수행된 이후의 프레임들에 대해 시간적 복원을 수행할 수 있음table_idxue(v)적응적 순서 변경이 사용되는 경우, 어떠한 순서로 복호화가 수행될 지에 대해 나타내는 테이블의 인덱스 정보실시 예에 따라, 테이블은 다수 개가 존재할 수 있으며, 상기 명세에 작성한 바와 같이 응용 어플리케이션에 따라 서로 다른 테이블이 사용될 수 있음사용 실시 예) orderTable[target_application_idx][table_num][table_idx], 여기서 orderTable은 도 10의 예시와 같은 테이블일 수 있음…}…}.

[0185] Referring to Table 1, it may include index information for the application to be used in the decoder, the number of GOPs in the sequence, a flag indicating whether to use adaptive reordering in the decoding process, information on which table to use among the list of available tables depending on the application, and index information of a table indicating the order in which decoding will be performed if adaptive reordering is used.

[0186] Table 2 shows an embodiment of selecting a table like the example of Fig. 10 for each GOP unit without a fixed order, parsing the index of the table for each frame in the GIP and / or for each frame group unit in the GOP including at least one frame, and determining the restoration order for each unit.

[0187] SyntaxDescriptorSemanticsAdaptive_order_information () {target_application_idxue(v)Index information about the application to be used by the decoder. Example) 0 is not specified, 1 is tracking, 2 is recognition, 3 is classification, 4 is segmentation, 5 is other applications. gop_numue(v)Number of GOPs in the sequence. Depending on the example, the number of frames in a GOP may be different for each GOP. for (i=0; i <gop_num; i++) {table_numue(v)응용 어플리케이션에 따라 사용 가능한 테이블 리스트 중, 어떠한 테이블을 사용할 지에 대한 정보for (j=0; j<frames; j++) {frames는 각 GOP 내의 프레임 수를 의미실시 예에 따라, 각 프레임 단위로 서로 다른 순서로 프레임이 복원될 수 있으며, 이때 프레임 단위 순서 변경이 수행되는 경우, 도 8의 시간적 복원 수행 과정은 생략되거나, 다른 과정 (일 예시로, 비트 심도 역변환, 공간적 복원, 관심 영역 기반 복원)이 모두 수행된 이후의 프레임들에 대해 시간적 복원을 수행할 수 있음table_idxue(v)적응적 순서 변경이 사용되는 경우, 어떠한 순서로 복호화가 수행될 지에 대해 나타내는 테이블의 인덱스 정보.실시 예에 따라, 테이블은 다수 개가 존재할 수 있으며, 상기 명세에 작성한 바와 같이 응용 어플리케이션에 따라 서로 다른 테이블이 사용될 수 있음.사용 실시 예) orderTable[target_application_idx][table_num][table_idx], 여기서 orderTable은 도 10의 예시와 같은 테이블일 수 있음…}…}

[0188] Table 3 below shows an embodiment of determining decryption order information based on whether adaptive reordering is used, whether the order is the same as the restoration order of the previous GOP, and which table from the list of available tables is used depending on the application.

[0189] SyntaxDescriptorSemanticsAdaptive_order_information () {target_application_idxue(v)Index information about the application to be used by the decoder. Example) 0 is not specified, 1 is tracking, 2 is recognition, 3 is classification, 4 is segmentation, 5 is other applications. gop_numue(v)Number of GOPs in the sequence. Depending on the example, the number of frames in a GOP may be different for each GOP. for (i=0; i <gop_num; i++) {adaptive_ordering_flague(1)복호화 과정의 적응적 순서 변경 사용 여부를 나타내는 플래그실시 예에 따라, adaptive_ordering_flag가 0인 경우, 해당 GOP내 각 프레임 복원은 고정된 순서로 수행될 수 있다.Here, the fixed order may be the restoration order of FIG. 8, which is the order determined by the sub / decoder agreement. Depending on the embodiment, the fixed order described above may be a form determined based on at least one of the application, the reference structure type, etc. if (adaptive_ordering_flag) {mrg_flague(1) A flag indicating whether to use the same order as the restoration order of the previous GOP. Depending on the embodiment, if the flag is 1, restoration is performed in the same order as the restoration order of the previous GOP. At this time, if the restoration order is different for each frame or frame group in the previous GOP, if mrg_flag is 1, the restoration order of the first frame / frame group or the last frame / frame group in the previous GOP can be applied to the current GOP by the sub / decoder agreement. if (mrg_flag == 0) {table_numue(v) Information on which table to use among the list of tables available depending on the application for (j=0; j <frames; j++) {frames는 각 GOP 내의 프레임 수를 의미실시 예에 따라, 각 프레임 단위로 서로 다른 순서로 프레임이 복원될 수 있으며, 이때 프레임 단위 순서 변경이 수행되는 경우, 도 8의 시간적 복원 수행 과정은 생략되거나, 다른 과정 (일 예시로, 비트 심도 역변환, 공간적 복원, 관심 영역 기반 복원)이 모두 수행된 이후의 프레임들에 대해 시간적 복원을 수행할 수 있음table_idxue(v)적응적 순서 변경이 사용되는 경우, 어떠한 순서로 복호화가 수행될 지에 대해 나타내는 테이블의 인덱스 정보실시 예에 따라, 테이블은 다수 개가 존재할 수 있으며, 상기 명세에 작성한 바와 같이 응용 어플리케이션에 따라 서로 다른 테이블이 사용될 수 있음사용 실시 예) orderTable[target_application_idx][table_num][table_idx], 여기서 orderTable은 도 10의 예시와 같은 테이블일 수 있음…}…}.

[0190] Table 4 below shows an embodiment of determining decryption order information based on information about the application to be used in the decoder, whether to use adaptive reordering, whether to use the same order as the restoration order of the previous GOP, index information about which restoration order of the previously decrypted GOPs to use, and information about which table to use from the list of available tables depending on the application.

[0191] SyntaxDescriptorSemanticsAdaptive_order_information () {target_application_idxue(v)Index information about the application to be used by the decoder. Example) 0 is not specified, 1 is tracking, 2 is recognition, 3 is classification, 4 is segmentation, 5 is other applications. gop_numue(v)Number of GOPs in the sequence. Depending on the example, the number of frames in a GOP may be different for each GOP. for (i=0; i <gop_num; i++) {adaptive_ordering_flague(1)복호화 과정의 적응적 순서 변경 사용 여부를 나타내는 플래그실시 예에 따라, adaptive_ordering_flag가 0인 경우, 해당 GOP내 각 프레임 복원은 고정된 순서로 수행될 수 있다.Here, the fixed order may be the restoration order of FIG. 8 in the order determined by the sub / decoder period agreement. According to an embodiment, the fixed order described above may be a form determined based on at least one of the application, the reference structure type, etc. if (adaptive_ordering_flag) {mrg_flague(1) A flag indicating whether to use the same order as the restoration order of the previous GOP. According to an embodiment, if the flag is 1, restoration is performed in the same order as the restoration order of one of the previous GOPs. At this time, if the restoration order is different for each frame or frame group in the selected previous GOP, if mrg_flag is 1 and mrg_idx indicates the GOP, the restoration order of the first frame / frame group or the last frame / frame group in the previous GOP selected by the sub / decoder agreement may be applied to the current GOP. if (mrg_flag) {mrg_idxue(v) Among the previously decrypted GOPs, which Index information indicating whether to use the restoration order of GOPs. Depending on the implementation example, the restoration order information of previous GOPs can be stored and managed in a buffer. At this time, the buffer can be managed in a FIFO (First in first out) structure and can be newly initialized for each specific number of GOPs.or you can save the restoration order of all GOPs in the video} else {table_numue(v) Information on which table to use among the list of available tables depending on the application for (j=0; j <frames; j++) {frames는 각 GOP 내의 프레임 수를 의미실시 예에 따라, 각 프레임 단위로 서로 다른 순서로 프레임이 복원될 수 있으며, 이때 프레임 단위 순서 변경이 수행되는 경우, 도 8의 시간적 복원 수행 과정은 생략되거나, 다른 과정 (일 예시로, 비트 심도 역변환, 공간적 복원, 관심 영역 기반 복원)이 모두 수행된 이후의 프레임들에 대해 시간적 복원을 수행할 수 있음table_idxue(v)적응적 순서 변경이 사용되는 경우, 어떠한 순서로 복호화가 수행될 지에 대해 나타내는 테이블의 인덱스 정보실시 예에 따라, 테이블은 다수 개가 존재할 수 있으며, 상기 명세에 작성한 바와 같이 응용 어플리케이션에 따라 서로 다른 테이블이 사용될 수 있음사용 실시 예) orderTable[target_application_idx][table_num][table_idx], 여기서 orderTable은 도 10의 예시와 같은 테이블일 수 있음…}…}.

[0192] FIG. 11 illustrates detailed operations of an internal decryption performer according to an embodiment of the present disclosure.

[0193] An internal decoding performer (910) of a decoding device (10b) according to an embodiment of the present disclosure can decode a bitstream to generate a restored image and parse information about the restored image. The restored image can include a region-of-interest image and a non-region-of-interest image. Information about the restored image can include information about a restoration order (e.g., Adaptive_order_information ()), information about a region-of-interest, information about temporal restoration (e.g., temporal_restoration_data ()), information about spatial restoration, or information about a bit depth. The internal decoding performer (910) can include an image decoding performer (1101), an upsampling performer (1102), an region-of-interest information decoding performer (1103), and an image reconstruction performer (1104). Depending on the embodiment, the order of each process can be changed, and some components can be omitted.

[0194] According to an embodiment, a bitstream input from an internal decoding performer (910) may first undergo DeMUXing and then be applied as input to each decoding performer (1101, 1103) of FIG. 11. The image decoding performer (1101) may decode an image processed for a region of interest and an image of a non-region of interest from the bitstream.

[0195] According to an embodiment, the upsampling performing unit (1102) may receive a sampling rate and an upsampling method for the region-of-interest-based processed image and the non-region-of-interest image after decoding, and perform upsampling on each of them. According to an embodiment, the upsampling method for each image may use one of bilinear upsampling, bilateral upsampling, nearest-neighbor upsampling, and a deep neural network including one or more convolutional layers. According to an embodiment, the number of frames of the restored region-of-interest-based processed image and the number of frames of the restored non-region-of-interest image may be different from each other. According to an embodiment, the image reconstruction unit (1104) may receive the restored region-of-interest and the restored non-region-of-interest image and perform restoration to the original image, thereby performing restoration on the restored image.

[0196] According to an embodiment, the region of interest information decoding performing unit (1103) can decode region of interest information from a bitstream. The restored region of interest information can be used for image reconstruction together with an image restored by the image reconstruction unit (1104). If multiple regions of interest exist in one frame, the region of interest information decoding performing unit (1103) can perform decoding for each region of interest. According to an embodiment, the region of interest information can include at least one of motion information, scale information, rotation information, transformation information, etc. for each region of interest and / or all regions of interest in the frame. According to an embodiment, the region of interest information can be used by referencing information such as a previous frame and / or a previous region of interest.

[0197] FIG. 12 illustrates detailed operations of an image reconstruction unit according to an embodiment of the present disclosure.

[0198] An image reconstruction unit (1104) included in an internal decoding performer (910) of a decoding device (10b) according to one embodiment of the present disclosure can reconstruct a restored image from a restored region-of-interest-based processed image, a restored non-region-of-interest image, and restored region-of-interest information. The image reconstruction unit (1104) may include a region boundary filtering performing module (1201).

[0199] The region boundary filtering performing module (1201) can reconstruct the processed image based on the restored region of interest and the restored non-region of interest image to generate a single restored image. The region boundary filtering performing module (1201) can reconstruct the processed image based on the restored region of interest and the restored non-region of interest image using the restored region of interest information to generate a single restored image. Region boundary filtering can be performed on at least one of the region of interest including the region boundary, the entire region of the non-region of interest, and a part of the region.

[0200] In some embodiments, a reconstructed non-region of interest image may exist for each frame of the reconstructed region of interest, and one reconstructed non-region of interest image may exist for each series of frame groups. For example, a reconstructed non-region of interest image may exist only for the first Intra frame of each GOP unit. In some embodiments, when one reconstructed non-region of interest image exists for each series of frame groups, a reconstructed image may be generated using the same reconstructed non-region of interest image for each frame of the reconstructed region of interest.

[0201] According to an embodiment, the region boundary filtering performing module (1201) may perform different filtering (e.g., sharpening, deblurring, etc.) on the boundary of the region of interest and the boundary of the restored non-region of interest image at the corresponding location for each restored region of interest.

[0202] According to an embodiment, the region boundary filtering performing module (1201) may implicitly determine a filtering method for each restored region of interest based on the difference in pixel values ​​between the boundary of the region of interest and the boundary of the restored non-region of interest image at the corresponding location. According to an embodiment, the region boundary filtering performing module (1201) may receive a filtering method from an encoder and perform filtering for a specific region of interest.

[0203] In some embodiments, filtering may be applied only within a specific frame and / or for a specific region of interest, and the decoder may implicitly measure at least one specific metric, such as PSNR, SSIM, etc., for each frame and / or region of interest, and perform filtering only if the metric value is lower than a specific threshold. In this case, the threshold may be a value based on the size of the frame, the size of the region of interest, etc., determined by an agreement between the decoder and the decoder.

[0204] According to an embodiment, the region boundary filtering performing module (1201) may perform image reconstruction for the restored region of interest-based processed image and the restored non-region of interest image based on the restored region of interest information as in the example of FIG. 12, when region-of-interest-based restoration is performed after internal decoding is performed. According to an embodiment, the region boundary filtering performing module (1201) may first perform image reconstruction for the restored region of interest-based processed image and the restored non-region of interest image when spatial restoration and temporal restoration are performed before region-of-interest-based restoration after internal decoding is performed, and then, when region-of-interest-based restoration is performed, the restored region of interest information may be used to improve the accuracy of restoration for the spatial / temporal restored image.

[0205] Figures 13a and 13b illustrate detailed operations of a spatial restoration performer according to an embodiment of the present disclosure.

[0206] A spatial restoration performer (930) of a decoding device (10b) according to one embodiment of the present disclosure can perform spatial restoration on an internally decoded image. The spatial restoration performer (930) can include a spatial frame restoration performer (1301) and a region of interest spatial correction unit (1302).

[0207] According to an embodiment, the spatial restoration performer (930) can perform spatial restoration on each frame of the decoded image.

[0208] According to an embodiment, the spatial restoration performer (930) may receive a signal from the encoder regarding the degree and method of performing spatial restoration, and may perform different degrees and methods of performing spatial restoration for each frame.

[0209] According to an embodiment, the spatial restoration performer (930) may have different spatial restoration performance levels and methods for each series of frame group units. In this case, each frame group may be a unit composed of at least one frame.

[0210] According to an embodiment, the spatial restoration performer (930) may perform spatial restoration by at least one of bilinear upsampling, bicubic upsampling, a trained deep neural network composed of one or more convolutional layers, etc., and which method to use may be signaled by the encoder and determined by the decoder through parsing of the bitstream.

[0211] According to an embodiment, the spatial restoration performer (930) may perform spatial restoration by implicitly and / or explicitly determining information about a sampling rate and a restoration method when spatial restoration is performed, and in this case, the spatial resolution of the original frame and the resolution of the spatially restored frame may be different from each other. According to an embodiment, when spatial restoration is performed using the original frame, the spatial restoration performer (930) may perform spatial restoration using information about the spatial resolution of the original frame without parsing the sampling rate.

[0212] Referring to FIG. 13a, when spatial restoration is performed after performing region-of-interest-based restoration, the spatial frame restoration performing unit (1301) can perform spatial restoration based on the restored image and restored region-of-interest information.

[0213] During the spatial restoration process, the resolution of the frame, the resolution of the region of interest, and the resolution of the non-region of interest may all be changed, or / and only one of the three types of data resolutions described above may be changed. In this case, if the region of interest and the non-region of interest are changed together, or / and the resolutions of the region of interest, the non-region of interest, and the frame are all changed, information related to the scale of each region of interest among the region of interest information may be parsed and used together for spatial restoration.

[0214] Referring to FIG. 13b, when spatial restoration is performed after temporal restoration is performed, the spatial frame restoration performing unit (1301) performs spatial frame restoration on the restored image, and then the region of interest spatial correction unit (1302) can perform spatial correction on the region of interest using the restored region of interest information on the restored image on which spatial frame restoration has been performed.

[0215] During the spatial restoration process, the spatial resolution of the region of interest may not change, and only the spatial resolution of the non-region of interest and / or the spatial resolution of the current frame may change. In this case, if the spatial resolution of the region of interest does not change during the spatial restoration process, the position of the region of interest within the frame before spatial restoration (for example, the upper left position) may change to a position corresponding to the position within the frame after spatial restoration. This means, for example, that if the position of a region of interest within the frame before spatial restoration is (10, 20), when the frame size is restored to twice the width and four times the height through the spatial restoration process, the position of the region of interest may move to (20, 80). Thereafter, the region of interest spatial correction unit (1302) may parse the information of each region of interest to perform correction for each region of interest. Here, correction may mean correcting at least one of the position, size, and transformation of the region of interest within the frame.

[0216] Depending on the embodiment, the transmission unit of the spatial sampling rate of the region of interest and the transmission unit of the spatial sampling rate of the region of non-interest may be different for the region of interest and the region of non-interest. That is, as an example, the transmission unit of the spatial sampling rate of the region of interest may be a GOP unit, and the transmission unit of the spatial sampling rate of the region of non-interest may be a sequence unit.

[0217] Depending on the embodiment, the definition of a region of interest may be each non-adjacent region of interest within a frame, or may be all regions of interest existing within a frame.

[0218] Figures 14a and 14b illustrate detailed operations of a temporal restoration performer according to an embodiment of the present disclosure.

[0219] A temporal restoration performer (940) of a decoding device (10b) according to one embodiment of the present disclosure can perform temporal restoration on an internally decoded image. The temporal restoration performer (940) can include a temporal frame restoration performer (1401) and a region of interest temporal correction unit (1402).

[0220] According to an embodiment, when spatial restoration is performed after temporal restoration is performed, the temporal restoration performer (940) may use one or more frames having the same resolution to generate an intermediate frame when one or more restored frames are used to generate an intermediate frame in the temporal restoration process. In this case, the intermediate frame may be a frame in the display order among a plurality of frames input to the temporal restoration process (interpolation), or may be a frame thereafter (extrapolation).

[0221] According to an embodiment, when two or more frames with different resolutions are used in the temporal restoration process, the temporal restoration performer (940) may change the resolution of all frames to a smaller or larger resolution among the respective frames and then perform temporal restoration. According to an embodiment, after performing temporal restoration, spatial restoration may be performed by parsing a single spatial restoration information for the frames whose resolutions have been matched.

[0222] According to an embodiment, when a region-of-interest-based restoration is performed after performing temporal restoration, a temporal frame restoration performing unit (1401) may, according to an embodiment, receive information on the number of frames to be restored (for example, a temporal sampling rate, etc.), input a restored image and restored region-of-interest information, perform restoration of an intermediate frame using the restored image, and at this time, perform correction of pixel value information and position information of regions-of-interest within the restored intermediate frames using the restored region-of-interest information.

[0223] Referring to FIG. 14a, a temporal frame restoration performing unit (1401) may, according to an embodiment, receive information on the number of frames to be restored (for example, temporal sampling rate, etc.) and perform restoration of intermediate frames using a restored image.

[0224] Referring to FIG. 14b, the region of interest temporal correction unit (1402) may, according to an embodiment, receive an image on which temporal frame restoration has been performed and information on the restored region of interest, and perform correction of the regions of interest within the image on which temporal frame restoration has been performed. According to an embodiment, the temporal frame restoration performing unit (1401) may first perform temporal restoration, and then the region of interest temporal correction unit (1402) may perform temporal correction on the region of interest in the restored image.

[0225] In some embodiments, when temporal restoration of a frame is performed using two or more frames, input frames for restoration may be applied differently for each region within the frame to be restored. That is, when input frames used in the process of performing temporal restoration of an arbitrary frame are A, B, and C, temporal restoration of a part of the frame to be restored may be performed using frames A and B, and of the remaining region of the frame to be restored using frames A and C. In one embodiment, a part of the frame to be restored may be a region of interest (ROI), and the remaining region may be a non-ROI. Alternatively, both a part of the frame to be restored and the remaining region may each be a region of interest. At this time, filtering may be performed to remove discontinuities between the two regions. In various embodiments, the type of filtering may be determined adaptively. The type of filtering may be determined based on information related to temporal restoration (the ratio of the ROI) parsed from the bitstream. For example, the type of filtering may be determined based on whether the ratio of the ROI is greater than a threshold.

[0226] Depending on the embodiment, information for the temporal restoration process (for example, the temporal_restoration() information in Table 5) may be transmitted at various levels depending on the embodiment. For example, transmission is possible at various levels, such as sequence units, GOPs, frames, slices, and subsequences.

[0227] Table 5 below shows examples of syntax related to temporal restoration.

[0228] SyntaxDescriptorSemanticstemporal_restoration_data( ){temporal_restoration_flague(1)if( temporal_restoration_flag ) {Flag for whether to apply temporal resampling, if 1, temporal resampling is appliedtemporal_resampling_ratio_idxue(v)Index of the list for temporal resampling ratios, since temporal resampling is applied as temporal_restoration_flag, it does not include cases where the input signal and the output signal are the same. For example, the sampling rate can be in the form of a ratio such as 1 / 2, 1 / 4, 1 / 8, etc., and can be the number of frames per second. The encoder and decoder can agree on the list of sampling rates, and the encoder can transmit the index of the list. In some embodiments, if the resampling rate is fixed or the number of required frames is fixed for the transmitted application, the corresponding information may be omitted. same_period_flague(1) A flag for whether periodic resampling or aperiodic resampling is used when applying temporal resampling. In some embodiments, if the periodicity of temporal resampling is fixed, the corresponding information and related information may be omitted. If same_period_flag is 1, it means that temporal resampling is performed at the same period. if(!same_period_flag) {for(i=0; i <num_of_frames; i++ ) {delta_frame_idx[i]ue(v)프레임숫자에 따라 이전 프레임 인덱스와의 차분 값을 전송한다. 이전 인덱스와의 차분 값이므로 프레임율로 계산된 num_of_frames보다 1개 작은 개수를 전송하면 된다.i: 프레임 인덱스}}Remain_framesue(v)비디오 시퀀스의 시간적 복원을 수행할 마지막 주기의 잔여 프레임 수를 나타내는 값이며, 부호화기에서 복호화되야할 프레임의 전체 수가 시그널링된 경우, 생략될 수 있다.A variable that can have a value less than or equal to the temporal restoration cycle value. For example, if the cycle is 4, Remain_frames can have a value between 0 and 4.

[0229] As shown in Table 5, temporal_resampling_ratio_idx and same_period_flag can be signaled based on temporal_restoration_flag (whether to apply temporal resampling).

[0230] As shown in Table 5, when the same_period_flag value indicates 0, i.e., when periodic temporal resampling is applied (when the temporal resampling rate is fixed), information such as an index related to the amount of change in the temporal resampling rate for each frame (or picture) may not be signaled. Otherwise, information such as an index related to the amount of change in the temporal resampling rate for each frame (or picture) may be signaled.

[0231]

[0232] The examples of the present disclosure presented in this specification and drawings are intended solely to facilitate the technical content of the present disclosure and to aid understanding thereof, and are not intended to limit the scope of the present disclosure. It will be apparent to those skilled in the art that other variations are possible in addition to the examples described above.

[0233] The claims set forth in this specification may be combined in various ways. For example, the technical features of the method claims of this specification may be combined and implemented as a device, and the technical features of the device claims of this specification may be combined and implemented as a method. Furthermore, the technical features of the method claims and the technical features of the device claims of this specification may be combined and implemented as a device, and the technical features of the method claims and the technical features of the device claims of this specification may be combined and implemented as a method.

[0234] Although the above-described disclosure is described based on a series of steps or a flowchart, this does not limit the chronological order of the invention, and may be performed simultaneously or in a different order as needed. In addition, each component (e.g., unit, module, etc.) constituting the block diagram in the above-described disclosure may be implemented as a hardware device or software, or a plurality of components may be combined to be implemented as a single hardware device or software. The above-described disclosure may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., singly or in combination. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program instructions such as ROMs, RAMs, and flash memories. The hardware device may be configured to operate as one or more software modules to perform processing according to the present disclosure, and vice versa. The device according to the present disclosure may have program instructions for storing or transmitting a bitstream generated by the encoding method described above. The scope of the present disclosure may include software or machine-executable instructions (e.g., an operating system, an application, firmware, a program, etc.) that cause operations according to the methods of various embodiments to be executed on a device or a computer, and a non-transitory computer-readable medium on which such software or instructions are stored and executable on the device or computer.

[0235]

[0236] [Explanation of symbols]

[0237] 10a: Encoding device

[0238] 510: Temporal Resampling Performer

[0239] 520: Spatial Resampling Performer

[0240] 530: Region-of-interest-based processor

[0241] 540: Bit Depth Converter

[0242] 550: Internal Encoding Performer

[0243] 10b: Decoding device

[0244] 910: Internal decryption performer

[0245] 920: Region-of-interest-based restorer

[0246] 930: Spatial Restoration Performer

[0247] 940: Temporal Restoration Performer

[0248] 950: Bit Depth Inverter

[0249] 960: Post-processing filter performer

Claims

1. In a VCM (video coding for machines) decoding device, An internal decryption performer that generates a decrypted restored image from a bitstream and parses information about the restored image; A region-of-interest-based restorer that reconstructs the restored image based on information related to the region of interest included in the information about the restored image; A spatial restoration performer that restores the resolution of at least a portion of the restored image based on information related to spatial restoration included in the information about the restored image; and A temporal restoration performer that performs temporal restoration by generating at least one frame of the restored image based on information related to temporal restoration included in the information about the restored image, A VCM decoding device, wherein the execution order of at least two of the region-of-interest-based restorer, the spatial restoration performer, and the temporal restoration performer is adaptively determined based on information about the restored image.

2. In paragraph 1, The information about the above restored image includes index information for multiple decryption order definition tables, A VCM decoding device, wherein the above plurality of decoding order definition tables are shared in advance with a VCM encoding device.

3. In paragraph 1, A VCM decoding device that, when the spatial restoration or the temporal restoration is performed before the region-of-interest-based restoration, first performs image reconstruction on the restored region-of-interest-based processed image and the restored non-region-of-interest image included in the restored image in the spatial restoration or the temporal restoration process, and then improves the accuracy of the spatial restoration or the temporal restoration in the region-of-interest-based restoration process.

4. In paragraph 1, Information related to the above spatial restoration is transmitted in at least one of a group of pictures (GOP), a sequence, a subsequence, a frame, and a slice unit, A VCM decoding device, wherein the transmission unit of the spatial sampling rate for the region of interest included in the above-mentioned restored image and the transmission unit of the spatial sampling rate for the region of no interest included in the above-mentioned restored image are different from each other.

5. In paragraph 1, A VCM decoding device, wherein the spatial restoration performer, when performing resolution restoration on at least one frame included in the restored image, changes the resolution for at least one of the entire at least one frame, a region of interest included in the at least one frame, and a region of no interest included in the at least one frame.

6. In paragraph 5, A VCM decoding device, wherein the spatial restoration performer performs spatial restoration at different resolutions for the entire at least one frame, the region of interest included in the at least one frame, and the region of no interest included in the at least one frame.

7. In paragraph 1, The temporal restoration performer is a VCM decoding device, wherein a plurality of frames for generating a part of the at least one frame and a plurality of frames for generating a remaining area of ​​the at least one frame are different from each other.

8. In paragraph 1, A VCM decoding device, wherein the temporal restoration performer determines whether to perform filtering on the at least one frame and the type of filtering based on information related to the temporal restoration.

9. In paragraph 1, A VCM decoding device, wherein the temporal restoration performer determines whether to perform filtering on the at least one frame and the type of filtering based on a preset metric measurement value per generated at least one frame or per region of interest included in the at least one frame.

10. In paragraph 9, A VCM decoding device, wherein the preset metric measurement value includes at least one of a peak signal-to-noise ratio (PSNR) and a structural similarity index measure (SSIM).

11. In paragraph 9, A VCM decoding device, wherein the temporal restoration performer performs filtering to remove discontinuity between a region of interest and a region of no interest of the generated at least one frame.

12. In paragraph 1, A VCM decoding device, wherein the information regarding the above temporal restoration is transmitted in at least one of a group of pictures (GOP), a sequence, a subsequence, a frame, and a slice unit.

13. In paragraph 1, A VCM decoding device, wherein information related to the region of interest includes at least one of motion information, scale information, rotation information, and transformation information for each region of interest included in the restored image.

14. In paragraph 1, A VCM decoding device further comprising a bit depth inverse converter that compensates for the bit depth of one or more components included in the restored image based on information related to bit depth conversion included in the information about the restored image.

15. In a VCM (video coding for machines) encoding device, A temporal resampling performer that changes the frame rate of an input image; A spatial resampling performer that changes the spatial resolution of each frame or a series of frames of the input image; A region-of-interest-based processor that extracts one or more regions of interest included in each frame of the input image and generates a processed image based on the one or more regions of interest; and It includes an internal encoding performer that encodes the processed image based on the region of interest, the non-region of interest image, and information for restoration to generate a bitstream, A VCM encoding device, wherein the information for the above restoration includes index information for a plurality of decoding order definition tables, and the plurality of decoding order definition tables are shared in advance with a VCM decoding device.

16. In paragraph 15, A VCM encoding device, wherein the information for the above restoration includes at least one of region-of-interest-based processed information, temporal restoration information, and spatial restoration information.

17. In paragraph 16, The unit transmitting the information for the above restoration includes at least one of GOP, sequence, subsequence, frame and slide, A VCM encoding device in which the units for transmitting information for restoration are different for each of the above-mentioned area-of-interest-based processed information, temporal restoration information, and spatial restoration information.

18. In paragraph 15, A VCM encoding device, wherein the temporal resampling performer performs temporal resampling at different sampling rates for a series of frames within the input image.

19. In paragraph 15, A VCM encoding device in which the spatial resampling performer resamples the resolution for a region of interest in the input image differently from the resolution for a region of no interest in the input image.

20. In a non-transitory computer-readable recording medium storing a bitstream generated by a VCM (video coding for machines) encoding method, The above VCM encoding method comprises a step of performing temporal resampling to change the frame rate of an input image; A spatial resampling step for changing the spatial resolution of each frame or a series of frames of the input image; A region-of-interest-based processing step for extracting one or more regions of interest included in each frame of the input image and generating a processed image based on the one or more regions of interest; Including an internal encoding step of generating a bitstream by encoding the image processed based on the region of interest, the image of a non-region of interest, and information for restoration, A non-transitory computer-readable recording medium, wherein the information for the above restoration includes index information for a plurality of decryption order definition tables, and the plurality of decryption order definition tables are shared in advance with a VCM decoding device.

Images