Frame configuration and encoding or decoding method based on region of interest

Abstract

One example of the present disclosure presents an encoding device for performing video coding for machines (VCM). The encoding device may comprise: a region-of-interest selection unit for receiving a first video including one or more frames and selecting a region of interest for a first frame of the first video; and a packing execution unit for executing packing for reconfiguring the position and / or size of at least one region of interest included in the first frame in response to the region of interest for the first frame being selected.

Description

Region-of-interest-based frame composition and decoding method

[0001] The present disclosure relates to a region-of-interest based frame composition and decoding method.

[0002] With the continuous development of the information and communication industry, broadcasting services with HD (High Definition) resolution have spread worldwide.

[0003] Through this proliferation, many users have become accustomed to high-resolution, high-quality images and / or videos, and the demand for higher-resolution, high-quality images and / or videos, such as 4K or 8K or higher UHD (Ultra High Definition) images and videos, has increased in various fields. The technology for coding such UHD image data was completed in 2013 through the standard technology HEVC (High Efficiency Video Coding).

[0004] HEVC is a next-generation video compression technology with a higher compression ratio and lower complexity than the previous H.264 / AVC technology, and is a key technology for effectively compressing the massive data of HD and UHD video.

[0005] Like previous compression standards, HEVC performs block-level encoding. However, unlike H.264 / AVC, it has only a single profile. The core encoding technologies included in HEVC's sole profile are divided into eight areas: hierarchical encoding structure technology, transform technology, quantization technology, intra-frame prediction coding technology, inter-frame motion prediction technology, entropy coding technology, loop filter technology, and other technologies.

[0006] Since the establishment of the HEVC video codec in 2013, the Versatile Video Coding (VVC) standard, a next-generation video codec that aims to improve performance by more than twice that of HEVC, has been developed to address the expansion of realistic video and virtual reality services utilizing 4K and 8K video images. VVC is called H.266.

[0007] H.266 (VVC) was developed with the goal of being more than twice as efficient as the previous generation codec, H.265 (HEVC). VVC was initially developed with resolutions over 4K in mind, but it was also developed for ultra-high-resolution video processing at a whopping 16K level to support 360-degree videos due to the expansion of the VR market. In addition, as the HDR market is expanding due to the development of display technology, it supports 16-bit color depth as well as 10-bit color depth to respond to this, and supports brightness expressions of 1000 nits, 4000 nits, and 10000 nits. In addition, since it is being developed with the VR market and 360-degree video market in mind, it supports partial frame rates in the range of 0 to 120 FPS.

[0008] Advances in Artificial Intelligence

[0009] Artificial intelligence (AI) is also steadily developing. AI refers to the artificial imitation of human intelligence, including the ability to recognize, classify, infer, predict, and control / decision-making.

[0010] With the advancement of artificial intelligence technology and the increase in Internet of Things (IoT) devices, machine-to-machine traffic is expected to explode, and machine-dependent image analysis is expected to become widely used.

[0011] However, as the amount of images to be analyzed by machines is expected to increase exponentially, issues with server load and power consumption are expected to arise.

[0012] Accordingly, the present disclosure aims to provide a region-of-interest based frame composition and decoding method to enable effective machine-based image analysis.

[0013] In order to achieve the above-mentioned purpose, according to one disclosure of the present specification, an encoding device for performing image encoding for a machine is provided.

[0014] An encoding device for performing image encoding for a machine may include a region of interest selection unit for receiving a first video image including one or more frames, selecting a region of interest for a first frame of the first video image, and a packing unit for performing packing to reconstruct a position and / or size of at least one region of interest included in the first frame in response to the region of interest for the first frame being selected.

[0015] According to one disclosure of the present specification, a decoding device for performing image decoding for a machine includes: an internal decoding performer for receiving a bitstream and performing image decoding; and an unpacking performer for performing unpacking on the decoded image in response to receiving region-of-interest-based processing information on a first video image, wherein the unpacking performer can restore a position and / or a size of at least one region of interest included in the decoded image to an original value of the first video image based on the region-of-interest-based processing information.

[0016] According to one disclosure of the present specification, a non-volatile computer-readable storage medium having instructions recorded thereon, the instructions, when executed by one or more processors, cause the one or more processors to perform the steps of: receiving a first video image including one or more frames, selecting a region of interest for a first frame of the first video image; and, in response to the region of interest for the first frame being selected, performing packing to reconstruct a position and / or a size of at least one region of interest included in the first frame.

[0017] According to the present disclosure, image analysis by a machine can be effectively performed.

[0018] Figure 1 schematically illustrates an example of a video / image coding system.

[0019] Figure 2 is a drawing schematically illustrating the configuration of a video / image encoding device.

[0020] Figure 3 is a drawing schematically illustrating the configuration of a video / image decoding device.

[0021] Figures 4a to 4d are exemplary diagrams showing a VCM encoder and a VCM decoder.

[0022] FIG. 5 is a block diagram of components included in an encoding device that performs region-of-interest based packing according to one embodiment of the present disclosure.

[0023] FIG. 6 is a block diagram of components included in an area-of-interest-based processor of an encoding device according to an embodiment of the present disclosure.

[0024] FIG. 7 is an example of packing a region of interest according to one embodiment of the present disclosure.

[0025] FIG. 8 is an example of normalizing the size of a region of interest according to one embodiment of the present disclosure.

[0026] FIG. 9 is a flowchart illustrating a method for performing region-of-interest-based packing by an encoding device according to an embodiment of the present disclosure.

[0027] FIG. 10 is a block diagram of components included in a decoding device that performs region-of-interest-based unpacking according to one embodiment of the present disclosure.

[0028] FIG. 11 is a block diagram of components included in an unpacking unit of a decoding device according to an embodiment of the present disclosure.

[0029] FIG. 12 is an example of unpacking a region of interest according to one embodiment of the present disclosure.

[0030] FIG. 13 is a flowchart illustrating a method by which a decoding device according to an embodiment of the present disclosure performs region-of-interest based unpacking.

[0031] FIGS. 14a and 14b are examples of syntax for packing information according to one embodiment of the present disclosure.

[0032] Specific structural or step-by-step descriptions of embodiments according to the concept of the present disclosure disclosed in this specification or application are merely illustrative for the purpose of explaining embodiments according to the concept of the present disclosure, and embodiments according to the concept of the present disclosure may be implemented in various forms, and embodiments according to the concept of the present disclosure may be implemented in various forms and should not be construed as being limited to the embodiments described in this specification or application.

[0033] Embodiments according to the concept of the present disclosure may have various modifications and take various forms. Therefore, specific embodiments are illustrated in the drawings and described in detail in this specification or application. However, this is not intended to limit embodiments according to the concept of the present disclosure to specific disclosed forms, and it should be understood that all modifications, equivalents, and alternatives included within the spirit and technical scope of the present disclosure are included.

[0034] While terms such as "first" and / or "second" may be used to describe various components, these components should not be limited by these terms. These terms are only intended to distinguish one component from another; for example, without departing from the scope of the present disclosure, a first component may be referred to as a "second component," and similarly, a second component may also be referred to as a "first component."

[0035] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. Conversely, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Other expressions that describe the relationship between components, such as "between" and "directly between" or "adjacent to" and "directly adjacent to", should be interpreted similarly.

[0036] The terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit the present disclosure. The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, it should be understood that the terms "comprises" or "has" indicate the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0037] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0038] In describing the embodiments, descriptions of technical details that are well known in the technical field to which the present disclosure pertains and are not directly related to the present disclosure will be omitted. This is to avoid obscuring the gist of the present disclosure by omitting unnecessary explanations and to convey the gist more clearly.

[0039] This document relates to video / image coding. For example, the method / embodiment disclosed in this document may be related to the Versatile Video Coding (VVC) standard (ITU-T Rec. H.266), the next-generation video / image coding standard after VVC, or other video coding-related standards (e.g., the High Efficiency Video Coding (HEVC) standard (ITU-T Rec. H.265), the essential video coding (EVC) standard, the AVS2 standard, etc.).

[0040] This document presents various embodiments of video / image coding, and unless otherwise stated, the embodiments may be performed in combination with each other.

[0041] In this document, video can mean a collection of images over time. A picture generally refers to a unit representing a single image at a specific time point, and a slice / tile is a unit that constitutes part of a picture in coding. A slice / tile can include one or more CTUs (coding tree units). A picture can be composed of one or more slices / tiles. A picture can be composed of one or more tile groups. A tile group can include one or more tiles.

[0042] A pixel or pel can mean the smallest unit that constitutes a picture (or image). Additionally, the term "sample" can be used as a counterpart to a pixel. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luma component, or only the pixel / pixel value of the chroma component. Alternatively, a sample can mean a pixel value in the spatial domain, or when such a pixel value is converted to the frequency domain, it can mean a transform coefficient in the frequency domain.

[0043] A unit may represent a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to the region. One unit may include one luma block and two chroma (e.g., cb, cr) blocks. In some cases, the term "unit" may be used interchangeably with terms such as "block" or "area." In general, an MxN block may include a set (or array) of samples (or sample array) or transform coefficients consisting of M columns and N rows.

[0044] Figure 1 schematically illustrates an example of a video / image coding system.

[0045] Referring to FIG. 1, a video / image coding system may include a source device and a receive device. The source device may transmit encoded video / image information or data to the receive device via a digital storage medium or a network in the form of a file or streaming.

[0046] The source device may include a video source, an encoding apparatus, and a transmitter. The receiving device may include a receiver, a decoding apparatus, and a renderer. The encoding apparatus may be referred to as a video / image encoding apparatus, and the decoding apparatus may be referred to as a video / image decoding apparatus. The transmitter may be included in the encoding apparatus. The receiver may be included in the decoding apparatus. The renderer may include a display unit, and the display unit may be configured as a separate device or an external component.

[0047] A video source may obtain video / images through a process of capturing, synthesizing, or generating video / images. The video source may include a video / image capture device and / or a video / image generation device. A video / image capture device may include, for example, one or more cameras, a video / image archive containing previously captured video / images, etc. A video / image generation device may include, for example, a computer, a tablet, a smartphone, etc., and may (electronically) generate video / images. For example, a virtual video / image may be generated through a computer, etc., in which case the video / image capture process may be replaced by a process of generating related data.

[0048] An encoding device can encode input video / images. The encoding device can perform a series of procedures, such as prediction, transformation, and quantization, to improve compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0049] The transmission unit can transmit encoded video / image information or data output in the form of a bitstream to the receiving unit of a receiving device via a digital storage medium or network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit can include an element for generating a media file using a predetermined file format and an element for transmission via a broadcasting / communication network. The receiving unit can receive / extract the bitstream and transmit it to a decoding device.

[0050] The decoding device can decode the video / image by performing a series of procedures such as inverse quantization, inverse transformation, and prediction corresponding to the operation of the encoding device.

[0051] The renderer can render decoded video / images. The rendered video / images can be displayed through the display unit.

[0052] Figure 2 is a drawing schematically illustrating the configuration of a video / image encoding device.

[0053] The term “video encoding device” hereinafter may include a video encoding device.

[0054] Referring to FIG. 2, the encoding device (10a) may be configured to include an image partitioner (10a-10), a prediction unit (predictor) (10a-20), a residual processor (residual processor) (10a-30), an entropy encoder (entropy encoder) (10a-40), an adder (adder) (10a-50), a filter (filter) (10a-60), and a memory (10a-70). The prediction unit (10a-20) may include an inter prediction unit (10a-21) and an intra prediction unit (10a-22). The residual processing unit (10a-30) may include a transformer (10a-32), a quantizer (10a-33), a dequantizer (10a-34), and an inverse transformer (10a-35). The residual processing unit (10a-30) may further include a subtractor (10a-31). The addition unit (10a-50) may be called a reconstructor or a reconstructed block generator. The above-described image segmentation unit (10a-10), prediction unit (10a-20), residual processing unit (10a-30), entropy encoding unit (10a-40), addition unit (10a-50), and filtering unit (10a-60) may be configured by one or more hardware components (e.g., encoder chipset or processor) according to an embodiment. In addition, the memory (10a-70) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (10a-70) as an internal / external component.

[0055] The image segmentation unit (10a-10) can segment an input image (or picture, frame) input to the encoding device (10a) into one or more processing units. For example, the processing unit may be called a coding unit (CU). In this case, the coding unit may be recursively segmented from a coding tree unit (CTU) or a largest coding unit (LCU) according to a QTBTTT (Quad-tree binary-tree ternary-tree) structure. For example, one coding unit may be segmented into a plurality of coding units of deeper depth based on a quad-tree structure, a binary tree structure, and / or a ternary structure. In this case, for example, the quad-tree structure may be applied first, and the binary tree structure and / or the ternary structure may be applied later. Alternatively, the binary tree structure may be applied first. The coding procedure according to the present document may be performed based on the final coding unit that is no longer segmented. In this case, based on coding efficiency according to image characteristics, etc., the maximum coding unit can be used as the final coding unit, or, if necessary, the coding unit can be recursively divided into coding units of lower depths, and the coding unit of the optimal size can be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration described below. As another example, the processing unit may further include a prediction unit (PU) or a transformation unit (TU). In this case, the prediction unit and the transformation unit may each be divided or partitioned from the final coding unit described above.The above prediction unit may be a unit of sample prediction, and the above transformation unit may be a unit for deriving a transformation coefficient and / or a unit for deriving a residual signal from a transformation coefficient.

[0056] The term "unit" may be used interchangeably with terms such as "block" or "area" depending on the case. In general, an MxN block can represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luminance component, or only the pixel / pixel value of the chroma component. A sample can be used as a term corresponding to a pixel or pel in a picture (or image).

[0057] The subtraction unit (10a-31) can subtract the prediction signal (predicted block, prediction samples, or prediction sample array) output from the prediction unit (10a-20) from the input image signal (original block, original samples, or original sample array) to generate a residual signal (residual block, residual samples, or residual sample array), and the generated residual signal is transmitted to the conversion unit (10a-32). The prediction unit (10a-20) can perform prediction on a block to be processed (hereinafter, referred to as a current block) and generate a predicted block including prediction samples for the current block. The prediction unit (10a-20) can determine whether intra prediction or inter prediction is applied to the current block or CU unit. The prediction unit can generate various information related to prediction, such as prediction mode information, as described later in the description of each prediction mode, and transmit the information to the entropy encoding unit (10a-40). Information about the prediction can be encoded in the entropy encoding unit (10a-40) and output in the form of a bitstream.

[0058] The intra prediction unit (10a-22) can predict the current block by referring to samples within the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from it depending on the prediction mode. In intra prediction, the prediction modes may include multiple non-directional modes and multiple directional modes. The non-directional modes may include, for example, a DC mode and a planar mode. The directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes depending on the degree of detail in the prediction direction. However, this is merely an example, and a greater or lesser number of directional prediction modes may be used depending on the settings. The intra prediction unit (10a-22) may also determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.

[0059] The inter prediction unit (10a-21) can derive a predicted block for a current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on an inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring block can include a spatial neighboring block existing in the current picture and a temporal neighboring block existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The above temporal neighboring blocks may be called collocated reference blocks, collocated CUs (colCUs), etc., and a reference picture including the temporal neighboring blocks may be called a collocated picture (colPic). For example, the inter prediction unit (10a-21) may construct a motion information candidate list based on the neighboring blocks, and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction may be performed based on various prediction modes, and for example, in the case of skip mode and merge mode, the inter prediction unit (10a-21) may use the motion information of the neighboring blocks as the motion information of the current block. In the case of skip mode, unlike the merge mode, a residual signal may not be transmitted.In the motion vector prediction (MVP) mode, the motion vector of the surrounding blocks is used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0060] The prediction unit (10a-20) can generate a prediction signal based on various prediction methods described below. For example, the prediction unit can apply intra prediction or inter prediction to predict a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). In addition, the prediction unit can perform intra block copy (IBC) to predict a block. The intra block copy can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in this document.

[0061] The prediction signal generated through the inter prediction unit (10a-21) and / or the intra prediction unit (10a-22) can be used to generate a reconstructed signal or a residual signal. The transform unit (10a-32) can generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique can include a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), a Graph-Based Transform (GBT), or a Conditionally Non-linear Transform (CNT). Here, GBT refers to a transform obtained from a graph when the relationship information between pixels is expressed as a graph. CNT refers to a transform obtained based on generating a prediction signal using all previously reconstructed pixels. In addition, the transform process can be applied to a pixel block having a square equal size, or can be applied to a block of a non-square variable size.

[0062] The quantization unit (10a-33) quantizes the transform coefficients and transmits them to the entropy encoding unit (10a-40), and the entropy encoding unit (10a-40) can encode the quantized signal (information about the quantized transform coefficients) and output it as a bitstream. The information about the quantized transform coefficients may be called residual information. The quantization unit (10a-33) can rearrange the quantized transform coefficients in a block form into a one-dimensional vector form based on a coefficient scan order, and can also generate information about the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form. The entropy encoding unit (10a-40) can perform various encoding methods, such as, for example, exponential Golomb, context-adaptive variable length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC).

[0063] The entropy encoding unit (10a-40) may encode, together or separately, information necessary for video / image restoration (e.g., values ​​of syntax elements, etc.) in addition to quantized transform coefficients. The encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream in the form of a network abstraction layer (NAL) unit. The video / image information may further include information regarding various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. The signaling / transmitted information and / or syntax elements described later in this document may be encoded through the above-described encoding procedure and included in the bitstream. The bitstream may be transmitted through a network or may be stored in a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The signal output from the entropy encoding unit (10a-40) may be configured as an internal / external element of the encoding device (10a) by a transmitting unit (not shown) that transmits and / or a storing unit (not shown) that stores the signal, or the transmitting unit may be included in the entropy encoding unit (10a-40).

[0064] The quantized transform coefficients output from the quantization unit (10a-33) can be used to generate a prediction signal. For example, by applying inverse quantization and inverse transformation to the quantized transform coefficients through the inverse quantization unit (10a-34) and the inverse transform unit (10a-35), a residual signal (residual block or residual samples) can be reconstructed. The addition unit (10a-50) can add the reconstructed residual signal to the prediction signal output from the prediction unit (10a-20), thereby generating a reconstructed signal (reconstructed picture, reconstructed block, reconstructed samples, or reconstructed sample array). When there is no residual for the target block to be processed, such as when skip mode is applied, the predicted block can be used as the reconstructed block. The generated restoration signal can be used for intra prediction of the next processing target block within the current picture, and can also be used for inter prediction of the next picture after filtering as described below.

[0065] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture encoding and / or restoration process.

[0066] The filtering unit (10a-60) can improve subjective / objective picture quality by applying filtering to the restoration signal. For example, the filtering unit (10a-60) can apply various filtering methods to the restoration picture to generate a modified restoration picture, and store the modified restoration picture in the memory (10a-70), specifically, in the DPB of the memory (10a-70). The various filtering methods may include, for example, deblocking filtering, sample adaptive offset (SAO), an adaptive loop filter, a bilateral filter, etc. The filtering unit (10a-60) can generate various information regarding filtering and transmit the information to the entropy encoding unit (10a-90), as described below in the description of each filtering method. The information regarding filtering may be encoded by the entropy encoding unit (10a-90) and output in the form of a bitstream.

[0067] The modified restored picture transmitted to the memory (10a-70) can be used as a reference picture in the inter prediction unit (10a-80). Through this, when inter prediction is applied, the encoding device can avoid prediction mismatch between the encoding device (10a) and the decoding device, and can also improve encoding efficiency.

[0068] The DPB of the memory (10a-70) can store the modified restored picture to be used as a reference picture in the inter prediction unit (10a-21). The memory (10a-70) can store motion information of a block from which motion information in the current picture is derived (or encoded) and / or motion information of blocks in a picture that has already been restored. The stored motion information can be transmitted to the inter prediction unit (10a-21) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (10a-70) can store restored samples of restored blocks in the current picture and transmit them to the intra prediction unit (10a-22).

[0069] Figure 3 is a drawing schematically illustrating the configuration of a video / image decoding device.

[0070] Referring to FIG. 3, the decoding device (10b) may be configured to include an entropy decoder (10b-10), a residual processor (10b-20), a predictor (10b-30), an adder (10b-40), a filter (10b-50), and a memory (10b-60). The predictor (10b-30) may include an inter-prediction unit (10b-31) and an intra-prediction unit (10b-32). The residual processor (10b-20) may include a dequantizer (10b-21) and an inverse transformer (10b-21). The entropy decoding unit (10b-10), residual processing unit (10b-20), prediction unit (10b-30), addition unit (10b-40), and filtering unit (10b-50) described above may be configured by a single hardware component (e.g., decoder chipset or processor) according to an embodiment. In addition, the memory (10b-60) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (10b-60) as an internal / external component.

[0071] When a bitstream including video / image information is input, the decoding device (10b) can restore the image corresponding to the process in which the video / image information is processed in the encoding device of FIG. 2. For example, the decoding device (10b) can derive units / blocks based on block division-related information obtained from the bitstream. The decoding device (10b) can perform decoding using a processing unit applied in the encoding device. Therefore, the processing unit of decoding may be, for example, a coding unit, and the coding unit may be divided from a coding tree unit or a maximum coding unit according to a quad tree structure, a binary tree structure, and / or a ternary tree structure. One or more transform units may be derived from the coding unit. Then, the restored image signal decoded and output by the decoding device (10b) can be reproduced through a reproduction device.

[0072] The decoding device (10b) can receive a signal output from the encoding device of FIG. 2 in the form of a bitstream, and the received signal can be decoded through the entropy decoding unit (10b-10). For example, the entropy decoding unit (10b-10) can parse the bitstream to derive information (e.g., video / image information) necessary for image restoration (or picture restoration). The video / image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information.

[0073] The decoding device can further decode the picture based on information about the parameter set and / or the general restriction information. The signaling / received information and / or syntax elements described later in this document can be decoded and obtained from the bitstream through the decoding procedure. For example, the entropy decoding unit (10b-10) can decode information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output values ​​of syntax elements required for image restoration and quantized values ​​of transform coefficients for the residual. In more detail, the CABAC entropy decoding method receives a bin corresponding to each syntax element in a bitstream, determines a context model using information of the syntax element to be decoded and decoding information of surrounding and decoding target blocks or information of symbols / bins decoded in a previous step, and predicts the occurrence probability of the bin according to the determined context model to perform arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element. At this time, the CABAC entropy decoding method can update the context model using information of the decoded symbol / bin for the context model of the next symbol / bin after determining the context model. Among the information decoded by the entropy decoding unit (10b-10), information regarding prediction is provided to the prediction unit (10b-30), and information regarding the residual on which entropy decoding has been performed by the entropy decoding unit (10b-10), i.e., quantized transform coefficients and related parameter information, can be input to the inverse quantization unit (10b-21).

[0074] In addition, information regarding filtering among the information decoded by the entropy decoding unit (10b-10) may be provided to the filtering unit (10b-50). Meanwhile, a receiving unit (not shown) that receives a signal output from an encoding device may be further configured as an internal / external element of the decoding device (10b), or the receiving unit may be a component of the entropy decoding unit (10b-10). Meanwhile, the decoding device according to the present document may be called a video / video / picture decoding device, and the decoding device may be divided into an information decoder (video / video / picture information decoder) and a sample decoder (video / video / picture sample decoder). The above information decoder may include the entropy decoding unit (10b-10), and the sample decoder may include at least one of the inverse quantization unit (10b-21), the inverse transformation unit (10b-22), the prediction unit (10b-30), the addition unit (10b-40), the filtering unit (10b-50), and the memory (10b-60).

[0075] The inverse quantization unit (10b-21) can inverse quantize the quantized transform coefficients and output the transform coefficients. The inverse quantization unit (10b-21) can rearrange the quantized transform coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scanning order performed in the encoding device. The inverse quantization unit (10b-21) can perform inverse quantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain transform coefficients.

[0076] In the inverse transform unit (10b-22), the transform coefficients are inversely transformed to obtain a residual signal (residual block, residual sample array).

[0077] The prediction unit can perform a prediction on the current block and generate a predicted block containing prediction samples for the current block. Based on the information regarding the prediction output from the entropy decoding unit (10b-10), the prediction unit can determine whether intra-prediction or inter-prediction is applied to the current block, and can determine a specific intra / inter-prediction mode.

[0078] The prediction unit can generate a prediction signal based on various prediction methods described below. For example, the prediction unit can apply intra prediction or inter prediction to predict a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). In addition, the prediction unit can perform intra block copy (IBC) to predict a block. The intra block copy can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in this document.

[0079] The intra prediction unit (10b-32) can predict the current block by referring to samples within the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from it, depending on the prediction mode. In intra prediction, the prediction modes may include multiple non-directional modes and multiple directional modes. The intra prediction unit (10b-32) can also determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.

[0080] The inter prediction unit (10b-31) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on an inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.).

[0081] In the case of inter prediction, the neighboring blocks may include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit (10b-31) may construct a motion information candidate list based on the neighboring blocks, and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter prediction may be performed based on various prediction modes, and the information regarding the prediction may include information indicating the mode of inter prediction for the current block.

[0082] The addition unit (10b-40) can generate a restoration signal (restored picture, restoration block, restoration sample array) by adding the acquired residual signal to the prediction signal (predicted block, prediction sample array) output from the prediction unit (10b-30). In cases where there is no residual for the block to be processed, such as when skip mode is applied, the predicted block can be used as the restoration block.

[0083] The addition unit (10b-40) may be referred to as a restoration unit or restoration block generation unit. The generated restoration signal may be used for intra prediction of the next processing target block within the current picture, may be output after filtering as described below, or may be used for inter prediction of the next picture.

[0084] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture decoding process.

[0085] The filtering unit (10b-50) can improve subjective / objective image quality by applying filtering to the restoration signal. For example, the filtering unit (10b-50) can apply various filtering methods to the restoration picture to generate a modified restoration picture, and transmit the modified restoration picture to the memory (60), specifically, the DPB of the memory (10b-60). The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0086] The (corrected) reconstructed picture stored in the DPB of the memory (10b-60) can be used as a reference picture in the inter prediction unit (10b-31). The memory (10b-60) can store motion information of a block from which motion information is derived (or decoded) within the current picture and / or motion information of blocks within a picture that has already been reconstructed. The stored motion information can be transmitted to the inter prediction unit (10b-31) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (10b-60) can store reconstructed samples of reconstructed blocks within the current picture and transmit them to the intra prediction unit (10b-32).

[0087] In this specification, the embodiments described in the prediction unit (10b-30), the inverse quantization unit (10b-21), the inverse transformation unit (10b-22), and the filtering unit (10b-50) of the decoding device (10b) can be applied to the prediction unit (10a-20), the inverse quantization unit (10a-34), the inverse transformation unit (10a-35), and the filtering unit (10a-60) of the encoding device (10a) in the same manner or correspondingly.

[0088] As described above, prediction is performed to increase compression efficiency when performing video coding. Through this, a predicted block including prediction samples for a current block, which is a coding target block, can be generated. Here, the predicted block includes prediction samples in a spatial domain (or pixel domain). The predicted block is derived identically from an encoding device and a decoding device, and the encoding device can increase video coding efficiency by signaling information (residual information) about the residual between the original block and the predicted block, rather than the original sample value of the original block itself, to a decoding device. The decoding device can derive a residual block including residual samples based on the residual information, and generate a reconstructed block including reconstructed samples by combining the residual block and the predicted block, and can generate a reconstructed picture including the reconstructed blocks.

[0089] The residual information may be generated through a transformation and quantization procedure. For example, the encoding device may derive a residual block between the original block and the predicted block, perform a transformation procedure on residual samples (a residual sample array) included in the residual block to derive transform coefficients, and perform a quantization procedure on the transform coefficients to derive quantized transform coefficients, thereby signaling the related residual information to a decoding device (via a bitstream). Here, the residual information may include information such as value information, position information, a transformation technique, a transformation kernel, and quantization parameters of the quantized transform coefficients. The decoding device may perform an inverse quantization / inverse transformation procedure based on the residual information to derive residual samples (or residual blocks). The decoding device may generate a reconstructed picture based on the predicted block and the residual block. The encoding device can also inversely quantize / inversely transform the quantized transform coefficients to derive a residual block for reference in inter prediction of a subsequent picture, and generate a restored picture based on the residual block.

[0090] <VCM(Video coding for Machines)>

[0091] With the recent advancements in various industries such as surveillance, intelligent transportation, smart cities, intelligent industry, and intelligent content, the amount of image or feature map data consumed by machines is increasing. In contrast, traditional video compression methods currently in use, developed with consideration for the characteristics of human vision, contain unnecessary information and are therefore inefficient for machine tasks. Therefore, research on video codec technologies capable of efficiently compressing feature maps for machine tasks is urgently needed.

[0092] The Moving Picture Experts Group (MPEG), an international standardization group for multimedia encoding, is discussing Video Coding for Machines (VCM). VCM is an image or feature map encoding technology that targets machine vision, rather than human viewer vision.

[0093] Figures 4a to 4d are exemplary diagrams showing a VCM encoder and a VCM decoder.

[0094] Referring to FIG. 4a, a VCM encoder (100a) and a VCM decoder (100b) are shown.

[0095] When a VCM encoder (100a) encodes a video and / or a feature map and transmits it as a bitstream, a VCM decoder (100b) can decode and output the bitstream. At this time, the VCM decoder (100b) can output one or more videos and / or feature maps. For example, the VCM decoder (100b) can output a first feature map for machine-based analysis and a first image for user viewing. The first image can have a higher resolution than the first feature map.

[0096] Referring to FIG. 4b, a feature extractor for extracting a feature map may be connected to the front end of the VCM encoder (100a).

[0097] The VCM encoder (100a) may include a feature encoder.

[0098] The VCM decoder (100b) may include a feature decoder and a video reconstructor. The feature decoder may decode a feature map from a bitstream and output a first feature map for machine-assisted analysis. The video reconstructor may regenerate and output a first video from the bitstream for viewing by a user.

[0099] Referring to Fig. 4c, a feature extractor for extracting a feature map is connected to the front end of the VCM encoder (100). The VCM encoder (100a) may include a feature encoder.

[0100] The VCM decoder (100b) may include a feature decoder. The feature decoder may decode a feature map from a bitstream and output a first feature map for machine-based analysis. That is, the bitstream may be encoded only as a feature map, not as an image. To elaborate, the feature map may be data containing information about features for processing a specific task of a machine based on an image.

[0101] Referring to FIG. 4d, a feature extractor may be connected to the front end of the VCM encoder (100a).

[0102] The VCM encoder (100a) may include a feature converter and a video encoder. The video encoder may be the encoding device (10a) illustrated in FIG. 2. The VCM decoder (100b) illustrated in FIG. 4d may include a video decoder and an inverse converter. The video decoder may be the decoding device (10b) illustrated in FIG. 3.

[0103]

[0104] <Semantics의 설명>

[0105] 1) A series of frame groups: for example, a GOP (group of picture) unit

[0106] 2) n: GOP index

[0107] 3) p: frame index

[0108] 4) m: Area of ​​interest index

[0109] 5) RoI_Processing_flag: A 1-bit flag indicating whether the current sequence has undergone region-of-interest-based processing.

[0110] 6) num_of_GOPs: Information indicating the number of GOPs in a sequence

[0111] 7) num_of_frames: Information indicating the number of frames in one GOP

[0112] 8) num_of_RoIs: Information indicating the number of regions of interest in one frame.

[0113] 9) packing_flag(n): 1-bit flag indicating whether packing is performed in units of a series of frame groups (GOPs).

[0114] 10) RoI_exist_region_LT(0), RoI_exist_region_LT(1): The upper left coordinates of the region where the region of interest exists in the picture.

[0115] 11) RoI_exist_region_RB(0), RoI_exist_region_RB(1): Coordinates of the lower right corner of the region where the region of interest exists within the picture

[0116] 12) Resize_RoI_flag(n)(p)(m): 1-bit flag indicating whether normalization is performed for each region of interest.

[0117] 13) Hor_Resize_idx, Ver_Resize_idx: For regions of interest with Resize_RoI_flag set to 1, indices indicating the horizontal / vertical resizing degree of the region of interest to resize to the original size.

[0118] - For integer k greater than or equal to 1, resizing can be performed for each region of interest as (Hor_Resize_idx >> k) and (Ver_Resize_idx >> k).

[0119] 14) Rotate_RoI_flag(n)(p)(m): A 1-bit flag indicating whether to transform the region of interest. If it is 1, the region of interest is rotated by 90 degrees.

[0120] 15) upsamp_ratio_RoIs: Upsampling rate information for the region of interest, and you can obtain a processed image based on the region of interest restored with the information.

[0121] 16) upsamp_ratio_nonRoIs: Upsampling rate information for non-interest regions, and you can obtain the restored non-interest region image using that information.

[0122] 17) Filtering_RoI_flag(n)(p)(m): 1-bit flag indicating filtering information between regions of interest and regions of no interest. If 1, perform specific filtering by parsing Filter_idx, and if 0, perform fixed filtering.

[0123] - Filtering information for each region of interest can be the same in all frames.

[0124] 18) Flip_RoI_flag(n)(p)(m): 1-bit flag indicating whether to flip by region of interest. If 1, the region of interest is flipped along the horizontal axis.

[0125] 19)pos_RoI(n)(p)(m)[0], pos_RoI(n)(p)(m)[1]: Location information of regions of interest within the first frame of each series of frame groups (GOPs).

[0126] - If a non-zero value is parsed among RoI_exist_region_LT(0), RoI_exist_region_LT(1), for example, RoI_exist_region_LT(0) + pos_RoI(n)(m)[0], RoI_exist_region_LT(1) + pos_RoI(n)(m)[1] may be the final restored position of the m-th region of interest.

[0127] - For the position difference values ​​obtained through pos_diff_coding, the final restored position of the region of interest can be obtained in the same way.

[0128] 20)abs_pos_diff_greater0_flag_RoI(n)(p)(m)[0], abs_pos_diff_greater0_flag_RoI(n)(p)(m)[1]: 1-bit flag indicating whether the absolute value of the x-axis and y-axis position information difference value for the same region of interest in the 0th frame is 0.

[0129] 21)abs_pos_diff_greater1_flag_RoI(n)(p)(m)[0], abs_pos_diff_greater1_flag_RoI(n)(p)(m)[1]: 1-bit flag indicating whether the absolute value of the x-axis and y-axis position information difference value for the same region of interest in the 0th frame is 1.

[0130] 22)abs_pos_diff_minus2_RoI(i)(p)(j)[0], abs_pos_diff_minus2_RoI(i)(p)(j)[0]: Information indicating the corresponding value of -2 of the absolute value of the difference value of the x-axis and y-axis position information for the same region of interest in the 0th frame.

[0131] 23)pos_diff_sign_flag(i)(p)(j)[0], pos_diff_sign_flag(i)(p)(j)[1]: 1-bit flag indicating the x-axis and y-axis sign values ​​for the same region of interest in the 0th frame, 1 for + sign, 0 for - sign

[0132] 24) bits_rtg_image_size specifies the number of bits that rtg_image_size_x and rtg_image_size_y signal.

[0133] 25) rtg_image_size_x, rtg_image_size_y are the width and height of the frame after packing.

[0134] 26) org_image_size_x, org_image_size_y are the width and height of the original frame.

[0135] 27) flag_rtg_image_size_difference is a 1-bit flag indicating whether the original frame size and the frame size after packing are different due to packing.

[0136] 28) bits_rtg_to_org_difference specifies the number of bits that rtg_to_org_difference_x and rtg_to_org_difference_y signal.

[0137] 29) rtg_to_org_difference_x, rtg_to_org_difference_y are the x-axis and y-axis difference values ​​between the original frame size and the frame size after packing.

[0138] 30) flag_rtg_rois is a 1-bit flag indicating whether a region of interest exists. 31) bits_roi_pos is the number of bits signaled by the roi_size_x[i] and roi_size_y[i] syntax elements, which are either signaled or computed.

[0139] 32) bit_num_rois specifies the number of bits signaled by the num_rois syntax element.

[0140] 33) num_rois specifies the number of regions of interest

[0141] 34) roi_pos_x[i] is the x-axis position of the ith region of interest within the original frame.

[0142] 35) roi_pos_y[i] is the y-axis position of the ith region of interest within the original frame.

[0143] 36) roi_size_x[i] is the width of the ith region of interest within the original frame.

[0144] 37) roi_size_x[i] is the height of the i-th region of interest within the original frame.

[0145]

[0146] FIG. 5 is a block diagram of components included in an encoding device that performs region-of-interest based packing according to one embodiment of the present disclosure.

[0147] An encoding device (10a) according to an embodiment of the present disclosure can perform video image encoding for a machine to generate a bitstream. The encoding device (10a) can include a temporal resampling performer (510), a spatial resampling performer (520), a region-of-interest-based processor (530), and an internal encoding performer (540). The order of each process illustrated in FIG. 5 can be changed. According to an embodiment, the temporal resampling performer (510) and the spatial resampling performer (520) can be omitted, and when the processes are omitted, an image processed based on a region-of-interest can be applied as an input to the internal encoding performer (540).

[0148] A temporal resampling performer (510) can input an original image, perform frame-by-frame sampling, and output an image in which some frames are sampled. In one embodiment, the encoding device (10a) can transmit information used in the temporal resampling process (for example, a temporal sampling rate, etc.) to the decoding device (10b). The encoding device (10a) can optionally apply a temporal resampling performer.

[0149] A spatial resampling performer (520) can receive an original image or an image on which temporal resampling has been performed and output an image with a changed spatial resolution for each frame or a series of frames. In one embodiment, the encoding device (10a) can transmit information used in the spatial resampling process (for example, a spatial sampling rate of each frame or / and a spatial sampling rate of a series of frames, etc.) to the decoding device (10b). The encoding device (10a) can optionally apply a spatial resampling performer.

[0150] The region of interest-based processor (530) can receive an original image, an image on which temporal resampling has been performed, or an image on which temporal and spatial resampling has been performed, extract a region of interest existing in each frame or a series of frames, and output an image processed based on the region of interest. The encoding device (10a) can transmit information used in the process of extracting the region of interest (for example, region of interest ID information, region of interest size information, packing information, etc.) to the decoding device (10b). The region of interest-based processor (530) can include a frame analysis unit, a region of interest selection unit, and a packing unit.

[0151] The frame analysis unit can receive a video as input and obtain information (e.g., region of interest location, region of interest score, etc.) on one or more region of interest candidates existing in each frame of the video.

[0152] The region of interest selection unit can determine a region of interest for image encoding through an internal encoding unit among the region of interest candidates obtained from the frame analysis unit. In one embodiment, non-region of interest images of some frames can be input directly to the internal encoding unit without going through the packing unit.

[0153] The packing unit may perform packing to reconstruct each frame using regions of interest selected for each frame of the image. In one embodiment, the packing unit may not reconstruct the frame if the proportion of the region of interest selected in the frame in the entire original image is less than a specific threshold value. In other words, the packing process may be omitted if the proportion of the region of interest in the entire image is small. The specific threshold value may be set to a fixed value, and may be adaptively determined according to the size of the original image. The encoding device (10a) may determine whether to perform packing on a frame basis or a series of frame basis. The encoding device (10a) may signal whether to perform packing (packing_flag) on ​​a series of frame group basis. The specific operation of each component constituting the region of interest-based processor (530) will be described in detail in FIG. 6.

[0154] The internal encoding performer (540) can receive an image output from the region of interest-based processor (530) and perform image encoding to generate a bitstream. The internal encoding performer (540) can use a 2D video encoder (AVC / H.264, HEVC / H.265, VVC / H.266, AV1, VP9, ​​etc.) and can use a 2D video encoder including one or more convolution layers. In one embodiment, the color space of the input image of the internal encoding performer (540) can be converted into a color space such as YUV420, YUV444, and then encoding can be performed. At this time, according to the embodiment, the conversion can be performed through a conversion method defined by an agreement between the encoding / decoding (encoder / decoder), and the color space conversion information can be transmitted to the decoding device (10b).

[0155] The internal encoding performer (540) may include a down-sampling performer and an image encoding performer. The internal encoding performer (540) may perform down-sampling on an image processed based on a region of interest analyzed by the region of interest-based processor (50) and perform image encoding. The internal encoding performer (540) may perform down-sampling separately from the region of interest on an image of a non-region of interest separated into non-regions of interest by the region of interest-based processor (530) and perform image encoding. In one embodiment, the internal encoding performer (540) may perform encoding on each frame on an image of a region of interest-based processed image, and may perform encoding on only some frames within a series of frame groups on an image of a non-region of interest.

[0156] The downsampling unit may perform downsampling by applying different sampling rates to the image processed based on the region of interest and the image of the non-region of interest before image encoding. In one embodiment, when downsampling is performed, the encoding device (10a) may signal sampling rate information (upsamp_ratio_RoIs, upsamp_ratio_nonRoIs) and transmit it to the decoding device (10b).

[0157] In one embodiment, the internal encoding performer (540) can utilize a video encoder (AVC / H.264, HEVC / H.265, VVC / H.266, AV1, VP9, ​​etc.), and can utilize a 2D video encoder including one or more convolutional layers. The video encoding performer can utilize the same video encoder for the region-of-interest-based processed image and the non-region-of-interest image, or can utilize different video encoders.

[0158]

[0159] FIG. 6 is a block diagram of components included in an area-of-interest-based processor of an encoding device according to an embodiment of the present disclosure.

[0160] An encoding device (10a) according to an embodiment of the present disclosure can analyze and process a region of interest in a video image. In one embodiment, a region of interest-based processor (530) may include a frame analysis unit (610), a region of interest selection unit (620), and a packing unit (630). The region of interest-based processor (530) may extract a region of interest from an input image, perform image processing on the region of interest, and then output a region of interest-based processed image. The region of interest-based processor (530) may output non-region of interest images for the remaining regions that are not extracted as regions of interest.

[0161] The frame analysis unit (610) can analyze information (for example, region of interest location, region of interest score, etc.) of one or more region of interest candidates existing in each frame of the video. The frame analysis unit (610) can include an analysis region selection module and a region of interest candidate extraction module.

[0162] The analysis region selection module can select an analysis region in units of one or more frames. The frame analysis unit can perform frame analysis only on some regions within the image after selecting the region on which frame analysis is to be performed. The encoding device (10a) can transmit information (RoI_exist_region_LT, RoI_exist_region_RB) on some regions on which frame analysis has been performed to the decoding device (10b). The some regions may be the same for all images. Alternatively, the some regions may be different for each image and each series of frame groups within the image. The encoding device (10a) may not transmit regions on which frame analysis has not been performed to the decoding device (10b).

[0163] The region of interest candidate extraction module may utilize a deep neural network composed of one or more convolutional layers on which training has been performed. In one embodiment, the deep neural network may be a network trained for the purpose of object detection or a network trained for the purpose of object tracing. The frame analysis unit (610) may extract region of interest candidates on a frame-by-frame basis or a series of frame basis. The frame analysis unit (610) may exclude from the region of interest candidates objects (region of interest candidates) if the detection accuracy of the objects detected for each frame or a series of frame groups is low.

[0164] If the region of interest candidate extraction module uses a deep neural network trained for the purpose of object tracking, the corresponding region of interest selection module may be omitted.

[0165] When the corresponding region of interest selection module is performed in units of a series of frame groups, the frame analysis unit (610) can select a region of interest candidate (detected object) of the first input frame as a candidate and select a corresponding region of interest through a correspondence relationship with the region of interest candidates of subsequent frames. In one embodiment, the correspondence relationship can be determined based on pixel-to-pixel similarity (for example, cross-correlation or / and histogram comparison of pixel values, etc.) between the entire area and / or a partial area of ​​each detected object.

[0166] The frame analysis unit (610) can set the size of the region of interest candidate within each frame so that both the height and width are multiples of 4.

[0167] The region of interest selection unit (620) can select a region of interest from among region of interest candidates and select a corresponding region of interest. In one embodiment, the region of interest selection unit (620) can include a region of interest selection module and a corresponding region of interest selection module. In one embodiment, the region of interest can be an area to be encoded as it is determined to include a major part of the image. A non-region of interest corresponds to an area outside the region of interest within the frame, and since it corresponds to a relatively minor part, it can not be encoded, and can be filled with the surrounding pixel values ​​of the region of interest or designated as an intermediate value.

[0168] The selected region of interest may vary for each frame and for each group of frames. When using a deep neural network trained in the region of interest candidate extraction module, for each frame or a group of frames, if the detection accuracy of a detected object (a candidate region of interest) is low, the object may be excluded from the candidate region of interest, and the detected objects may be selected as the region of interest. In one embodiment, the threshold value for detection accuracy may be the same value throughout the entire image, or may be set to a different value for each group of frames.

[0169] When the region of interest selection unit (620) performs the corresponding region of interest selection module in units of a series of frame groups, it can select a region of interest candidate (detected object) of the first input frame as a candidate and select a corresponding region of interest through a correspondence relationship with the region of interest candidates of subsequent frames. In one embodiment, the correspondence relationship can be determined based on pixel-to-pixel similarity (for example, cross-correlation or / and histogram comparison of pixel values, etc.) between the entire region and / or a partial region of each detected object.

[0170] The region of interest selection unit (620) can downsample or upsample the sizes of all region of interest candidates to a specific size, and then apply the corresponding region of interest selection module. The specific size can be determined based on the resolution of the image, or can be determined as the size of the region of interest with the smallest size by comparing the sizes of region of interest candidates extracted for each frame.

[0171] The region of interest selection unit (620) can assign an ID to the region of interest, and for example, can assign an ID to each unit in which a corresponding region of interest is selected. If the regions of the region of interest candidates extracted from the region of interest candidate extraction module within a frame overlap with each other, the region of interest selection unit can classify the regions of interest into a single region of interest if the overlapping region exceeds a certain threshold value. At this time, the threshold value can be determined according to the size of the frame and the size of the region of interest candidate.

[0172] If there is no region of interest in the current frame that corresponds to the region of interest in the previous frame, the region of interest selection unit (620) may assign a new ID to the region of interest. Alternatively, if there is a corresponding region of interest in the previous frames in the order of the new ID, the region of interest selection unit (620) may assign the existing ID of the corresponding region of interest instead of the new ID. The region of interest selection unit (620) may determine whether to assign a new ID or an existing ID by using a method for obtaining a correspondence relationship.

[0173] The packing unit (630) may perform packing to reconstruct each new frame using the selected regions of interest for each frame of the image, thereby generating a processed image based on the region of interest. In one embodiment, if the area occupied by the selected region of interest within the frame is larger than a specific threshold value compared to the image size, the packing unit (630) may be omitted. In one embodiment, the specific threshold value may be a fixed threshold value, or may be adaptively determined depending on the size of the current image.

[0174] The encoding device (10a) can decide whether to omit packing on a frame-by-frame basis or on a series of frame basis.

[0175] The encoding device (10a) can normalize the sizes of selected regions of interest before performing packing. The height and width of the normalized size can be a multiple of 4. The normalized sizes can be different for each region of interest ID. The unit for performing normalization can be a region of interest unit or a series of frame group units. For a region of interest for which normalization is performed, information about the original size of the region of interest (Resize_RoI_flag, Hor_Resize_idx, Ver_Resize_idx) can be signaled.

[0176] For a normalized region of interest or a region of interest for which normalization is not performed, a transformation module may be performed. The transformation module may be a process such as rotation, and the rotation may only be performed at a specific angle (for example, 90 degrees). The region of interest on which transformation is performed may be selected by comparing the aspect ratio (height / width) of each region of interest in the first frame. If the number of regions of interest with an aspect ratio of 1 or greater is less than the number of regions of interest with an aspect ratio of less than 1, the packing unit (630) may rotate the region of interest with an aspect ratio of less than 1 by 90 degrees and perform subsequent packing. In one embodiment, if the number of two types of regions of interest is the same, the packing unit (630) may rotate the region of interest with an aspect ratio of less than 1 by 90 degrees and perform subsequent packing. The transformation process may be performed independently for each frame, or may be performed in units of a series of frames. If transformation of a region of interest is performed, the region of interest on which transformation is performed may be signaled with transformation information (Rotate_RoI_flag).

[0177] The packing unit (630) can perform packing for a region of interest. The packing unit (630) can perform packing on a frame-by-frame basis. The packing unit (630) can perform padding on an unfilled area within a packed frame with a predetermined value, wherein the predetermined value may be an intermediate value. The size of the packed frame may be the same for each frame within a series of frame groups.

[0178] The packing unit (630) may perform packing based on the ID index value for each region of interest. In one embodiment, the packing unit (630) may perform packing in the scanning order for the frame, for example, in the scanning order of Z scanning from the upper left of the frame.

[0179] The spacing between regions of interest can be a constant pixel value, and these regions can be filtered and filled with a representative value for each region of interest boundary.

[0180] Packing can be performed at the upper left position of the frame using the two smallest values ​​of the x-coordinate and y-coordinate of each region of interest, while maintaining the same spacing between each region of interest as in the original image. The size of the frame after packing may differ from the original frame size.

[0181] The packing unit (630) can select a region of interest from the original image and then newly configure only the region where the region of interest exists into a frame.

[0182] The packing unit (630) may perform packing on the region of interest with the smallest ID index value on the upper left side of the frame for each region of interest, and then sequentially / repeatedly pack the region of interest most similar to the right boundary of the packed region of interest for the remaining regions of interest. In one embodiment, the similarity between the boundaries of the regions of interest may be determined by using the difference value of the average value of n pixels for a boundary score n greater than or equal to 1 of the region based on the smaller length of the heights of the two regions of interest. The packing unit (630) may consider before / after the vertical flip of each region of interest in the process of determining the similarity. If the flip of the region of interest is performed, the packing unit (630) may signal the corresponding information (Flip_RoI_flag).

[0183] When performing packing in units of a series of frames, the packing unit (630) can apply the packing information of the first frame to all frames within the frame unit. If the selected region of interest exists from the middle frame, the packing unit (630) can perform packing for a new region of interest from that frame in units of a series of frames. If the region of interest does not exist from the middle frame within the frame group, the packing unit (630) can perform the packing process again from that frame. The packing unit (630) can signal the position value and the difference value (pos_RoI, pos_diff_coding) based on the upper left coordinate of each region of interest for the packed region of interest information. The information of the packed region of interest can signal the index of the CTU (Coding Tree Unit) to which each region of interest belongs and the position value and the difference value based on the upper left coordinate of the region of interest within the CTU. Each CTU can have the same size in the video sequence, can have the same size for each frame, and can be a region having the size MxN for governments M, N greater than or equal to 1. Each CTU can be each sub-block obtained by dividing the video frame into MxN.

[0184] After performing packing, the packing unit (630) may normalize the size of the entire region of interest or a portion of the region of interest. The height and width of the normalized size may be a multiple of 4. The normalized size may be different for each region of interest ID. The unit for performing normalization may be a region of interest unit or a series of frame units. For the region of interest on which normalization is performed, information about the original size of the region of interest may be signaled. The packing unit (630) may perform normalization if the bottom and / or left side of the region of interest after packing occupies less than or more than a certain area of ​​the CTU or / and slice. The definition of the certain area may be determined by a convention of the encoder / decoder based on the size of the input image, the size of the region of interest, etc. In one embodiment, only the height or width of the region of interest may be normalized, or both the height and width may be normalized. In this case, the normalization ratios of the height and width may be different from each other.

[0185]

[0186] FIG. 7 is an example of packing a region of interest according to one embodiment of the present disclosure.

[0187] An encoding device (10a) according to one embodiment of the present disclosure can perform packing on one or more regions of interest included in a frame. The packing unit (630) of the encoding device (10a) can perform packing to reconstruct each new frame using regions of interest selected for each frame of the image, thereby generating a region-of-interest-based processed image.

[0188] Referring to FIG. 7, (a) a frame includes a first region of interest (0), a second region of interest (1), and a third region of interest (2). The encoding device (10a) may pack the first region of interest (0), the second region of interest (1), and the third region of interest (2) in a Z-scanning order in the order of decreasing index values ​​of IDs on the upper left side of the frame. In one embodiment, the size of the frame after packing, as in (a), may be different from the size of the original frame. Alternatively, the size of the frame after packing, as in (b), may be the same as the size of the original frame. The gap between each region of interest may be a constant pixel value, and the region may be filtered and / or filled with a representative value of the boundary of each region of interest.

[0189] (c) The frame can be packed to the upper left position of the frame using the two smallest values ​​among the x-coordinate and y-coordinate of each region of interest while maintaining the spacing of each region of interest the same as the original image.

[0190] (d) The frame can be packed to the upper left position of the frame using the two smallest values ​​of the x-coordinate and y-coordinate of each region of interest while maintaining the spacing of each region of interest the same as the original image. After packing, the rectangular area where the region of interest exists can be set as a new frame. At this time, the new frame may be different in size from the original frame. This may be performed for specific units such as GOP units, depending on one embodiment.

[0191]

[0192] FIG. 8 is an example of normalizing the size of a region of interest according to one embodiment of the present disclosure.

[0193] An encoding device (10a) according to an embodiment of the present disclosure can normalize the size of the entire region of interest or a portion of the region of interest after packing is performed. The height and width of the normalized size may be a multiple of 4. The normalized size may be different for each region of interest ID. The unit for performing normalization may be a region of interest unit or a series of frame group units. For a region of interest for which normalization is performed, information about the original size of the region of interest may be signaled. The encoding device (10a) may perform normalization if the lower part and / or the left side of the region of interest after packing occupies less than or more than a certain area of ​​the CTU or / and slice. The definition of the certain area may be determined by a convention of the encoder / decoder based on the size of the input image, the size of the region of interest, etc. Normalization may be performed on only the height or width of the region of interest, or on both the height and width. In this case, the normalization ratios of the height and width may be different from each other.

[0194] Referring to FIG. 8, (a) if the bottom and left of the region of interest (820) in the frame (810) occupies less than a certain area of ​​the CTU, the size of the region of interest can be normalized to be small (821). Alternatively, (b) if the bottom and left of the region of interest (830) in the frame (810) occupies more than a certain area of ​​the CTU, the size of the region of interest can be normalized to be large (831).

[0195]

[0196] FIG. 9 is a flowchart illustrating a method for performing region-of-interest-based packing by an encoding device according to an embodiment of the present disclosure.

[0197] An encoding device (10a) according to an embodiment of the present disclosure can generate an image processed based on a region of interest by selecting a region of interest from a frame included in an image and performing packing to reconstruct each new frame using the selected regions of interest for each frame. The encoding device (10a) can include a temporal resampling performer (510), a spatial resampling performer (520), a region of interest-based processor (530), and an internal encoding performer (540). The order of each step disclosed in FIG. 9 can be changed. Depending on the embodiment, steps S910 and / or S920 can be omitted, and when the corresponding processes are omitted, the image processed based on a region of interest can be applied as an input to the internal encoding performer (540).

[0198] In step S910, the temporal resampling performer (510) of the encoding device (10a) can receive an original image, perform frame-by-frame sampling, and output an image in which some frames are sampled. In one embodiment, the encoding device (10a) can transmit information used in the temporal resampling process (for example, a temporal sampling rate, etc.) to the decoding device (10b). The encoding device (10a) can optionally apply a temporal resampling performer.

[0199] In step S920, the spatial resampling performer (520) of the encoding device (10a) can receive an original image or an image on which temporal resampling has been performed and output an image with a changed spatial resolution for each frame or a series of frames. In one embodiment, the encoding device (10a) can transmit information used in the spatial resampling process (for example, a spatial sampling rate of each frame or / and a spatial sampling rate of a series of frames, etc.) to the decoding device (10b). The encoding device (10a) can optionally apply a spatial resampling performer.

[0200] In step S930, the region of interest-based processor (530) of the encoding device (10a) can receive an original image, an image subjected to temporal resampling, or an image subjected to temporal and spatial resampling, extract a region of interest existing in each frame or a series of frames, and output an image processed based on the region of interest. The encoding device (10a) can transmit information used in the process of extracting the region of interest (for example, region of interest ID information, region of interest size information, packing information, etc.) to the decoding device (10b).

[0201] In step S931, the frame analysis unit (610) of the region-based processor (530) can receive an image and obtain information (e.g., region of interest location, region of interest score, etc.) of one or more region of interest candidates existing in each frame of the image.

[0202] In step S932, the region-of-interest selection unit (620) of the region-based processor (530) may determine a region of interest for which image encoding will be performed through an internal encoding unit among the region of interest candidates obtained from the frame analysis unit. In one embodiment, non-interest region images of some frames may be input directly to the internal encoding unit without passing through the packing unit.

[0203] In step S933, the packing unit (630) of the region-based processor (530) may perform packing to reconstruct each frame using the regions of interest selected for each frame of the image. In one embodiment, the packing unit may not reconstruct the frame if the proportion of the region of interest selected within the frame in the entire original image is smaller than a specific threshold value. In other words, the packing process may be omitted if the proportion of the region of interest in the entire image is small. The specific threshold value may be set to a fixed value, and may be adaptively determined according to the size of the original image. The encoding device (10a) may determine whether to perform packing on a frame basis or a series of frame basis. The encoding device (10a) may signal whether to perform packing (packing_flag) on ​​a series of frame group basis.

[0204] In step S940, the internal encoding performer (540) of the encoding device (10a) may receive an image output from the region of interest-based processor (530) and perform image encoding to generate a bitstream. The internal encoding performer (540) may use a 2D video encoder (AVC / H.264, HEVC / H.265, VVC / H.266, AV1, VP9, ​​etc.) and may use a 2D video encoder including one or more convolution layers. In one embodiment, encoding may be performed after converting the color space of the input image of the internal encoding performer (540) into a color space such as YUV420 or YUV444. At this time, according to the embodiment, the conversion may be performed through a conversion method defined by an agreement between the encoding / decoding (encoder / decoder), and the color space conversion information may be transmitted to the decoding device (10b).

[0205] The internal encoding performer (540) may include a down-sampling performer and an image encoding performer. The internal encoding performer (540) may perform down-sampling on an image processed based on a region of interest analyzed by the region of interest-based processor (50) and perform image encoding. The internal encoding performer (540) may perform down-sampling separately from the region of interest on an image of a non-region of interest separated into non-regions of interest by the region of interest-based processor (530) and perform image encoding. In one embodiment, the internal encoding performer (540) may perform encoding on each frame on an image of a region of interest-based processed image, and may perform encoding on only some frames within a series of frame groups on an image of a non-region of interest.

[0206] The downsampling unit may perform downsampling by applying different sampling rates to the image processed based on the region of interest and the image of the non-region of interest before image encoding. In one embodiment, when downsampling is performed, the encoding device (10a) may signal sampling rate information (upsamp_ratio_RoIs, upsamp_ratio_nonRoIs) and transmit it to the decoding device (10b).

[0207] In one embodiment, the internal encoding performer (540) can utilize a video encoder (AVC / H.264, HEVC / H.265, VVC / H.266, AV1, VP9, ​​etc.), and can utilize a 2D video encoder including one or more convolutional layers. The video encoding performer can utilize the same video encoder for the region-of-interest-based processed image and the non-region-of-interest image, or can utilize different video encoders.

[0208]

[0209] FIG. 10 is a block diagram of components included in a decoding device that performs region-of-interest-based unpacking according to one embodiment of the present disclosure.

[0210] A decoding device (10b) according to one embodiment of the present disclosure can perform image decoding for a machine. Since the decoding is the reverse process of the encoding described above, the aforementioned content is not excluded even if it is not described repeatedly.

[0211] The decoding device (10b) may include an internal decoding performer, a region-of-interest-based restorer, a spatial restorer, a temporal restorer, and a post-processing filter performer. The order of each process may be changed, and the order of the region-of-interest-based restorer, the spatial restorer, and the temporal restorer may be performed in the reverse order of the order in which each corresponding process was performed during the image encoding process.

[0212] An internal decoding performer can receive a bitstream as input and perform image decoding to generate a restored image. According to an embodiment, the image decoding can use a 2D video decoder (AVC / H.264, HEVC / H.265, VVC / H.266, AV1, VP9, ​​etc.) and can use a 2D video decoder including one or more convolutional layers. According to an embodiment, when the color space of the restored image is not RGB444 but one of YUV420, YUV444, etc., the image decoding process can be performed after implicitly and / or explicitly converting to the RGB444 space.

[0213] Depending on the embodiment, the color space of the restored image may be implicitly or / and explicitly converted to another color space, after which the image decoding process may be performed.

[0214] An internal decoding performer may include an image decoding performer and an upsampling performer. The internal decoding performer may receive a bitstream as input and perform image decoding and upsampling on an image processed based on a region of interest and an image of a non-region of interest, respectively. Depending on the embodiment, the order of each process may be changed or omitted. Depending on the embodiment, the image processed based on a region of interest and the image of a non-region of interest may receive a sampling rate (upsamp_ratio_RoIs, upsamp_ratio_nonRoIs) after decoding and perform upsampling on each of them. Depending on the embodiment, the upsampling method for each image may be fixedly used one of bilinear upsampling, bilateral upsampling, nearest-neighbor upsampling, and a deep neural network including one or more convolutional layers. Depending on the embodiment, the number of frames of the reconstructed image of the region of interest based on the processed image may be different from the number of frames of the reconstructed image of the non-region of interest.

[0215] According to an embodiment, a region-of-interest-based restorer may restore a region-of-interest-based processed image using a decoded image and region-of-interest-based processing information transmitted from an encoder (for example, region-of-interest ID information, region-of-interest size information, packing information, etc.). The region-of-interest-based restorer may include an unpacking unit and an image reconstruction unit. In the region-of-interest-based restoration process, a restored non-region-of-interest image may be input to the image reconstruction unit, a restored region-of-interest-based processed image may be input to the unpacking unit, and then the output of the unpacking unit may be input to the image reconstruction unit. The unpacking unit may receive the restored region-of-interest-based processed image as an input, and perform rotation and restoration to the original region-of-interest size and unpacking of each region-of-interest to its original position for each region-of-interest. According to an embodiment, whether or not packing is performed (packing_flag) may be transmitted in units of a series of frame groups, and if packing is performed, a restored region-of-interest may be obtained through the unpacking unit. The detailed operation of the unpacking unit will be described later with reference to FIG. 11.

[0216] The image reconstruction unit can receive the restored region of interest and the restored non-region of interest images after unpacking and perform restoration to the original image. The image reconstruction unit can include a region boundary filtering performing module. According to an embodiment, the region boundary filtering performing module can be performed on the restored region of interest and the restored non-region of interest images.

[0217] In some embodiments, a restored non-region-of-interest image may exist for each frame of the restored region of interest, and one restored non-region-of-interest image may exist for each group of frames. (For example, a restored non-region-of-interest image may exist only for the Intra frame of each GOP unit.)

[0218] In an embodiment, if there is one restored non-region-of-interest image per unit of a series of frame groups, a restored image can be generated using the same restored non-region-of-interest image for each frame of the restored region of interest.

[0219] In an embodiment, for each restored region of interest, different filtering may be performed on the boundary of the region of interest and the boundary of the restored non-region of interest image at the corresponding location. In an embodiment, for each restored region of interest, a filtering method may be implicitly determined based on the difference in pixel values ​​between the boundary of the region of interest and the boundary of the restored non-region of interest image at the corresponding location. In an embodiment, for a specific region of interest, a filtering method (Filtering_RoI_flag, Filter_idx) may be transmitted from the encoder and filtering may be performed.

[0220] The spatial restoration performer may obtain an image on which spatial restoration has been performed by using, according to an embodiment, information used in the spatial resampling process transmitted from the decoded image or the decoded and region-of-interest-based restoration image and the encoder (for example, the spatial sampling rate of each frame and / or the spatial sampling rate of a series of frames, etc.).

[0221] According to an embodiment of the temporal restoration performer, an image on which temporal restoration has been performed can be obtained by using information (for example, a temporal sampling rate, etc.) used in the temporal resampling process transmitted from an image on which the above process has been partially or fully performed.

[0222] A post-processing filter performer may, depending on the embodiment, perform filtering on an image on which part or all of the above process has been performed. At this time, depending on the embodiment, a fixed filter may be used, or multiple filters may be defined by agreement between the encoder and decoder, and filter information may be received from the encoder to perform filtering.

[0223]

[0224] FIG. 11 is a block diagram of components included in an unpacking unit of a decoding device according to an embodiment of the present disclosure.

[0225] A decoding device (10b) according to an embodiment of the present disclosure can obtain original frame information by performing unpacking on regions of interest (ROIs) packed during the encoding process. The unpacking performing unit of the decoding device (10b) may include a region of interest unpacking performing module, a region of interest inverse transformation module, and a region of interest resizing module. The order of the unpacking performing process diagrammed in FIG. 11 may be changed or omitted.

[0226] The unpacking unit can receive a processed image based on a restored region of interest, perform unpacking on the region of interest, inversely transform the region of interest, and obtain a restored region of interest through a resizing process.

[0227] The region of interest unpacking performing module may, according to an embodiment, receive information (RoI_exist_region_LT, RoI_exist_region_RB) of some regions in which regions of interest exist in the original image, and perform unpacking within the corresponding regions. The region of interest unpacking performing module may, according to an embodiment, receive position information (pos_RoI) of each region of interest, and perform unpacking of the region of interest within each frame. According to an embodiment, the position information of each region of interest may parse x, y coordinates within the frame, parse the CTU index within the frame, and parse the x, y coordinates within the CTU derived from the parsed index to derive the original position of each region of interest, and perform unpacking. According to an embodiment, position information and position information difference value information (pos_RoI, pos_diff_coding) may be received for each ID of each region of interest in units of a series of frame groups, and unpacking of the region of interest within each frame may be performed. According to an embodiment, an ID for each region of interest may be assigned to each non-overlapping region of interest within a frame, and one ID may be assigned to the entire region where the region of interest exists. According to an embodiment, the position information difference value may be transmitted to perform unpacking by receiving a CTU index difference value and x, y coordinate difference values ​​where the region of interest exists between frames. According to an embodiment, whether to flip (Flip_RoI_flag) may be transmitted for each ID of each region of interest in a series of frame groups, and unpacking of the region of interest within each frame may be performed.

[0228] The region of interest inverse transformation module, according to an embodiment, may receive transformation information (Rotate_RoI_flag) for each region of interest ID in units of a series of frame groups, and perform transformation so that each region of interest has the same direction as the original region of interest.

[0229] The region of interest resizing module, according to an embodiment, receives normalization information (Resize_RoI_flag, Hor_Resize_idx, Ver_Resize_idx) for regions of interest for which unpacking and inverse transformation have been performed, if normalization has been performed in the encoder, and performs resizing to the same size as the original region of interest for each region of interest.

[0230]

[0231] FIG. 12 is an example of unpacking a region of interest according to one embodiment of the present disclosure.

[0232] A decoding device (10b) according to one embodiment of the present disclosure can receive a processed image based on a restored region of interest and perform unpacking to obtain a restored region of interest. The processed image based on a restored region of interest is an image that has undergone packing to reconstruct each new frame using regions of interest selected for each frame of the image by the encoding device (10a).

[0233] Referring to FIG. 12, (a) the frame has a first region of interest (0) on the left, a second region of interest (1), and a third region of interest (2) packed on the upper left side, and the packing information is transmitted from the encoding device (10a) to restore the position, size, etc. of each region of interest in the original image as shown on the right. The packing information includes position information of each region of interest, and the position information may be, for example, position information / difference value information of position information by ID of each region of interest. According to an embodiment, the position information of each region of interest may be unpacked by parsing the x, y coordinates within the frame, parsing the CTU index within the frame, and parsing the x, y coordinates within the CTU derived from the parsed index to derive the original position of each region of interest. In (a), the original frame size and the frame size after packing are different due to packing, and unpacking can be performed using information on the original size of the frame. Depending on the embodiment, the spacing between each region of interest may be a constant pixel value, the regions may be filtered and / or filled with a representative value of the boundary of each region of interest.

[0234] (b) The frame has the same frame size as the original image and the frame size after packing, and can be unpacked by parsing the location information of each region of interest.

[0235] (c) The frame is packed to the upper left position of the frame using the two smallest values ​​of the x-coordinate and y-coordinate of each region of interest while maintaining the spacing of each region of interest the same as the original image, so that the positions of the second region of interest (1) and the third region of interest (2) can be unpacked together according to the original position information of the first region of interest (0). Fig. 12 proceeds in the opposite direction to the process of Fig. 7.

[0236] (d) The frame can be unpacked using information about the original frame size, which is different from the original frame size after packing, while the spacing between each region of interest is maintained the same as the original image. This can be unpacked using the original frame size and the x-coordinate and y-coordinate of the location where the region of interest exists in the original frame. Fig. 12 is performed in the opposite direction to the process of Fig. 7.

[0237]

[0238] FIG. 13 is a flowchart illustrating a method by which a decoding device according to an embodiment of the present disclosure performs region-of-interest based unpacking.

[0239] A decoding device (10b) according to an embodiment of the present disclosure can perform image decoding for a machine. The decoding device (10b) can include an internal decoding performer, a region-of-interest-based restorer, a spatial restoration performer, a temporal restoration performer, and a post-processing filter performer. The order of each process can be changed, and the order of the region-of-interest-based restorer, the spatial restoration performer, and the temporal restoration performer can be performed in the reverse order of the order in which each corresponding process was performed during the image encoding process.

[0240] In step S1310, the internal decoding performer may receive a bitstream as input and perform image decoding to generate a restored image. According to an embodiment, the image decoding may use a 2D video decoder (AVC / H.264, HEVC / H.265, VVC / H.266, AV1, VP9, ​​etc.) and may use a 2D video decoder including one or more convolutional layers. According to an embodiment, when the color space of the restored image is not RGB444 but one of YUV420, YUV444, etc., the image may be implicitly or / and explicitly converted to the RGB444 space, and then the image decoding process may be performed.

[0241] Depending on the embodiment, the color space of the restored image may be implicitly or / and explicitly converted to another color space, after which the image decoding process may be performed.

[0242] An internal decoding performer may include an image decoding performer and an upsampling performer. The internal decoding performer may receive a bitstream as input and perform image decoding and upsampling on an image processed based on a region of interest and an image of a non-region of interest, respectively. Depending on the embodiment, the order of each process may be changed or omitted. Depending on the embodiment, the image processed based on a region of interest and the image of a non-region of interest may receive a sampling rate (upsamp_ratio_RoIs, upsamp_ratio_nonRoIs) after decoding and perform upsampling on each of them. Depending on the embodiment, the upsampling method for each image may be fixedly used one of bilinear upsampling, bilateral upsampling, nearest-neighbor upsampling, and a deep neural network including one or more convolutional layers. Depending on the embodiment, the number of frames of the reconstructed image of the region of interest based on the processed image may be different from the number of frames of the reconstructed image of the non-region of interest.

[0243] In step S1320, the region-of-interest-based restorer may, according to an embodiment, restore a region-of-interest-based processed image using the decoded image and region-of-interest-based processing information (for example, region-of-interest ID information, region-of-interest size information, packing information, etc.) transmitted from the encoder. The region-of-interest-based restorer may include an unpacking unit and an image reconstruction unit. In the region-of-interest-based restoration process, the restored non-region-of-interest image may be input to the image reconstruction unit, the restored region-of-interest-based processed image may be input to the unpacking unit, and then the output of the unpacking unit may be input to the image reconstruction unit.

[0244] In step S1321, the unpacking unit can receive the processed image based on the restored region of interest, and perform rotation and restoration to the original size of the original region of interest for each region of interest, and unpack each region of interest to its original position. According to an embodiment, whether or not packing is performed (packing_flag) is transmitted in units of a series of frame groups, and if packing is performed, the restored region of interest can be obtained through the unpacking unit. The detailed operation of the unpacking unit will be described later in FIG. 11.

[0245] In step S1322, the image reconstruction unit may receive the restored region of interest and the restored non-region of interest images for which unpacking has been performed, and perform restoration to the original image. The image reconstruction unit may include a region boundary filtering performing module. According to an embodiment, the region boundary filtering performing module may be performed on the restored region of interest and the restored non-region of interest images.

[0246] In some embodiments, a restored non-region-of-interest image may exist for each frame of the restored region of interest, and one restored non-region-of-interest image may exist for each group of frames. (For example, a restored non-region-of-interest image may exist only for the Intra frame of each GOP unit.)

[0247] In an embodiment, if there is one restored non-region-of-interest image per unit of a series of frame groups, a restored image can be generated using the same restored non-region-of-interest image for each frame of the restored region of interest.

[0248] In an embodiment, for each restored region of interest, different filtering may be performed on the boundary of the region of interest and the boundary of the restored non-region of interest image at the corresponding location. In an embodiment, for each restored region of interest, a filtering method may be implicitly determined based on the difference in pixel values ​​between the boundary of the region of interest and the boundary of the restored non-region of interest image at the corresponding location. In an embodiment, for a specific region of interest, a filtering method (Filtering_RoI_flag, Filter_idx) may be transmitted from the encoder and filtering may be performed.

[0249] At step S1330, the spatial restoration performer may obtain an image on which spatial restoration has been performed by using, according to an embodiment, the decoded image or the decoded and region-of-interest-based restoration image and the information used in the spatial resampling process transmitted from the encoder (for example, the spatial sampling rate of each frame and / or the spatial sampling rate of a series of frames, etc.).

[0250] At step S1340, the temporal restoration performer may obtain an image on which temporal restoration has been performed by using information (for example, a temporal sampling rate, etc.) used in the temporal resampling process transmitted from the encoder and an image on which the above process has been partially or fully performed, depending on the embodiment.

[0251] At step S1350, the post-processing filter performer may, depending on the embodiment, perform filtering on an image on which the above process has been partially or fully performed. At this time, depending on the embodiment, a fixed filter may be used, or multiple filters may be defined by agreement between the encoder and decoder, and filter information may be received from the encoder to perform filtering.

[0252]

[0253] FIGS. 14a and 14b are examples of syntax for packing information according to one embodiment of the present disclosure.

[0254] An encoding device (10a) according to an embodiment of the present disclosure can signal packing information for a region of interest and transmit it to a decoding device (10b). FIGS. 14a and 14b are examples of packing information created according to the semantics described above. A descriptor for a syntax according to the present disclosure is identical to a descriptor used in a standard specification of a video codec (such as HEVC, VVC, etc.) (e.g., a context-adaptive arithmetic entropy-coded syntax element, an unsigned integer 0-th order Exp-Golomb-coded syntax element, etc.). For example, u(n) represents an unsigned integer using n bits, and when n is “v” in the syntax table, it represents that the number of bits varies depending on the value of another syntax element.

[0255] Referring to Fig. 14a, the RoI_Processing_flag value, which indicates whether a region of interest is processed, is set to u(1), and specifies one or more pieces of information about regions of interest that have undergone region-of-interest processing. Referring to Fig. 14b, the value for position difference can be stored and checked in detail using the pos_diff_coding() function. In various embodiments, semantics can be added, changed, or deleted by agreement between the encoder and decoder.

[0256]

[0257] FIG. 15 is an example of a syntax for packing information according to one embodiment of the present disclosure.

[0258] An encoding device (10a) according to one embodiment of the present disclosure can signal packing information for a region of interest and transmit it to a decoding device (10b). FIG. 14 is an example of packing information written according to the semantics described above.

[0259] Referring to FIG. 15, the RoI_Processing_flag value, which indicates whether a region of interest is being processed, is set to u(1) and specifies one or more pieces of information about regions of interest that have undergone region-of-interest processing. In various embodiments, semantics can be added, changed, or deleted by agreement between the decoder and the encoder.

[0260]

[0261] The examples of the present disclosure presented in this specification and drawings are intended solely to facilitate the technical content of the present disclosure and to aid understanding thereof. They are not intended to limit the scope of the present disclosure. It will be apparent to those skilled in the art that other variations are possible in addition to the examples described above.

[0262] The claims set forth in this specification may be combined in various ways. For example, the technical features of the method claims of this specification may be combined and implemented as a device, and the technical features of the device claims of this specification may be combined and implemented as a method. Furthermore, the technical features of the method claims and the technical features of the device claims of this specification may be combined and implemented as a device, and the technical features of the method claims and the technical features of the device claims of this specification may be combined and implemented as a method.

Claims

1. In an encoding device that performs image encoding for a machine, A region of interest selection unit that receives a first video image including one or more frames and selects a region of interest for a first frame of the first video image; and An encoding device comprising a packing performing unit that performs packing to reconstruct the position and / or size of at least one region of interest included in the first frame in response to a region of interest being selected for the first frame.

2. In paragraph 1, An encoding device in which the packing performing unit performs packing when the proportion occupied by the selected region of interest within the first frame is greater than or equal to a predetermined threshold value.

3. In paragraph 2, An encoding device, wherein the above predetermined threshold value is adaptively determined according to the size of the first video image.

4. In paragraph 1, An encoding device in which the packing performing unit performs packing after normalizing the size of at least one region of interest included in the first frame before performing packing for the at least one region of interest.

5. In paragraph 1, An encoding device in which the packing performing unit performs transformation on each of the at least one region of interest included in the first frame before performing packing on the at least one region of interest included in the first frame.

6. In paragraph 5, An encoding device in which the above packing performing unit performs a rotation transformation for each of the at least one regions of interest.

7. In paragraph 1, An encoding device in which the above packing performing unit performs packing in a series of frame units consecutive to the first frame.

8. In paragraph 1, An encoding device in which the packing performing unit performs packing according to the scanning order of the first frame, in descending order of the ID index values ​​of at least one area of ​​interest included in the first frame.

9. In paragraph 1, An encoding device in which the packing performing unit reconstructs the position and / or size of at least one region of interest based on the upper left corner of the first frame.

10. In paragraph 1, An encoding device in which the above packing performing unit pads an area in which at least one area of ​​interest is not located within the first frame with a predetermined value.

11. In paragraph 1, An encoding device in which the packing performing unit sequentially packs a region of interest having the smallest ID index value among at least one region of interest included in the first frame, and then sequentially packs a region of interest that is most similar to the right boundary of the packed region of interest among the remaining regions of interest.

12. In paragraph 11, The degree of similarity between the above regions of interest is determined by the difference value of the average value of n pixels about the boundary of the regions based on the smaller length of the height of the two regions of interest, - wherein n is an integer greater than or equal to 1 - encoding device.

13. In paragraph 1, An encoding device wherein the packing performing unit generates a frame of a new size including at least one region of interest.

14. In paragraph 1, An encoding device in which the above packing performing unit signals information about a position value and a difference value based on an index of a CTU (coding tree unit) to which at least one region of interest belongs and the upper left coordinate of the region of interest within the CTU.

15. In paragraph 14, An encoding device wherein the size of the CTU is the same in the first video image sequence, or is the same for each frame.

16. In paragraph 1, An encoding device in which the above packing performing unit normalizes the size of all or part of at least one of the regions of interest after performing the packing.

17. In a decoding device that performs image decoding for a machine, An internal decoding unit that receives a bitstream as input and performs image decoding; and In response to receiving region-of-interest-based processing information for a first video image, an unpacking performing unit for performing unpacking on the decrypted image is included. A decoding device in which the unpacking performing unit restores the position and / or size of at least one region of interest included in the decrypted image to the original value of the first video image based on the region of interest-based processing information.

18. In paragraph 17, A decoding device in which the unpacking performing unit unpacks at least one region of interest according to location information of the at least one region of interest included in the region of interest-based processing information.

19. In paragraph 17, A decoding device that unpacks a series of frame groups using position information and differential value information for the position of at least one region of interest by ID in units of a series of frame groups included in the first video image.

20. A non-volatile computer-readable storage medium that records commands, The above instructions, when executed by one or more processors, cause the one or more processors to: A step of inputting a first video image including one or more frames and selecting a region of interest for a first frame of the first video image; In response to a region of interest being selected for the first frame, a step of performing packing to reconstruct a position and / or size of at least one region of interest included in the first frame is performed. Non-transitory computer-readable storage medium.

Images