Method for encoding and decoding image information on basis of region of interest

Abstract

Disclosed in one embodiment of the present disclosure is a video coding for machines (VCM) encoding apparatus. The encoding apparatus comprises: a temporal resampler for changing a partial frame rate of an input image; a region-of-interest-based processor for extracting one or more regions of interest included in each of a plurality of frames of the input image, and generating a processed image on the basis of the one or more regions of interest; and an internal encoder for generating a bitstream by encoding information about the one or more regions of interest and the input image, wherein the information about the one or more regions of interest can include a temporal sampling rate.

Description

Region-of-interest-based image information encoding and decoding method

[0001] The present disclosure relates to a method for encoding and decoding image information based on a region of interest.

[0002] With the continuous development of the information and communication industry, broadcasting services with HD (High Definition) resolution have spread worldwide.

[0003] Through this proliferation, many users have become accustomed to high-resolution and high-quality images and / or videos, and the demand for higher-resolution and high-quality images / videos, such as 4K or 8K or higher UHD (Ultra High Definition) images / videos, has increased in various fields.

[0004] The technology for coding this UHD video data was completed in 2013 through HEVC (High Efficiency Video Coding), a standard technology.

[0005] HEVC is a next-generation video compression technology with a higher compression ratio and lower complexity than the previous H.264 / AVC technology, and is a key technology for effectively compressing the massive data of HD and UHD video.

[0006] HEVC performs block-by-block encoding, like previous compression standards.

[0007] However, unlike H.264 / AVC, there is only one profile. The core encoding technologies included in HEVC's sole profile are divided into eight areas: hierarchical encoding structure technology, transform technology, quantization technology, intra-frame prediction encoding technology, inter-frame motion prediction technology, entropy encoding technology, loop filter technology, and other technologies.

[0008] Since the establishment of the HEVC video codec in 2013, the Versatile Video Coding (VVC) standard, a next-generation video codec that aims to improve performance by more than twice that of HEVC, has been developed to address the expansion of realistic video and virtual reality services utilizing 4K and 8K video images. VVC is called H.266.

[0009] H.266 (VVC) was developed with the goal of being more than twice as efficient as the previous generation codec, H.265 (HEVC). VVC was initially developed with resolutions over 4K in mind, but it was also developed for ultra-high-resolution video processing at a whopping 16K level to support 360-degree videos due to the expansion of the VR market. In addition, as the HDR market is expanding due to the development of display technology, it supports 16-bit color depth as well as 10-bit color depth to respond to this, and supports brightness expressions of 1000 nits, 4000 nits, and 10000 nits. In addition, since it is being developed with the VR market and 360-degree video market in mind, it supports partial frame rates in the range of 0 to 120 FPS.

[0010] Advances in Artificial Intelligence

[0011] Artificial intelligence (AI) is also steadily developing. AI refers to the artificial imitation of human intelligence, including the ability to recognize, classify, infer, predict, and control / decision-making.

[0012] With the advancement of artificial intelligence technology and the increase in Internet of Things (IoT) devices, machine-to-machine traffic is expected to explode, and machine-dependent image analysis is expected to become widely used.

[0013] However, as the amount of images to be analyzed by machines is expected to increase exponentially, issues with server load and power consumption are expected to arise.

[0014] Accordingly, the present disclosure aims to provide a method for encoding and decoding image information based on a region of interest in order to effectively perform image analysis by a machine.

[0015] In order to achieve the above-mentioned purpose, according to one disclosure of the present specification, a VCM encoding device is provided.

[0016] A VCM encoding device includes a temporal resampling performer that changes a part of a frame rate of an input image; a region-of-interest-based processor that extracts one or more regions of interest included in each frame of a plurality of frames of the input image and generates a processed image based on the one or more regions of interest; and an internal encoding performer that encodes information about the one or more regions of interest and the input image to generate a bitstream, wherein the information about the one or more regions of interest may include a temporal sampling rate.

[0017] A VCM decoding device according to one disclosure of the present specification is provided. The VCM decoding device may include an internal decoding performer that decodes a bitstream to generate a restored image; a region-of-interest-based restorer that reconstructs a restored image based on a region of interest in the restored image using region-of-interest-based processing information parsed from the bitstream; and a temporal restoration performer that performs temporal restoration of the restored image using temporal resampling information parsed from the bitstream.

[0018] A non-volatile computer-readable storage medium having recorded thereon instructions according to one disclosure of the present specification, wherein the instructions, when executed by one or more processors, cause the one or more processors to perform the steps of: changing a portion of a frame rate of an input image; extracting one or more regions of interest included in each frame of a plurality of frames of the input image and generating a processed image based on the one or more regions of interest; and encoding information about the one or more regions of interest and the input image to generate a bitstream, wherein the information about the one or more regions of interest may include a temporal sampling rate.

[0019] According to the present disclosure, image analysis by a machine can be effectively performed.

[0020] According to the present disclosure, when a region of interest (ROI) is extracted and transmitted from an input image in a VCM encoding device, the accuracy of a deep neural network in a VCM decoding device can be improved. The VCM encoding device can signal information about the position and movement of the ROI in units of GOP or a series of frames and transmit the signal to the VCM decoding device.

[0021] According to the present disclosure, when a VCM encoding device performs temporal resampling on an input image and extracts and encodes a region of interest, encoding efficiency can be improved. Furthermore, when a VCM decoding device performs temporal restoration of a restored image based on motion information between some frames and the region of interest, the location of the region of interest can be restored more accurately. When a VCM decoding device performs a machine vision task, the task execution result can be compensated based on the motion information of the region of interest. For example, the accuracy of the result of an object task can be improved through compensation between frames where the brightness of light changes rapidly.

[0022] Figure 1 schematically illustrates an example of a video / image coding system.

[0023] Figure 2 is a drawing schematically illustrating the configuration of a video / image encoding device.

[0024] Figure 3 is a drawing schematically illustrating the configuration of a video / image decoding device.

[0025] Figures 4a to 4d are exemplary diagrams showing a VCM encoder and a VCM decoder.

[0026] FIG. 5 illustrates a block diagram of an encoding device according to an embodiment of the present disclosure.

[0027] FIG. 6 illustrates detailed operations of an area-of-interest-based processor according to one embodiment of the present disclosure.

[0028] Figures 7a, 7b and 7c illustrate detailed operations of an internal encoding performer according to an embodiment of the present disclosure.

[0029] FIG. 8 illustrates detailed operations of an area of ​​interest information encoding performing unit according to one embodiment of the present disclosure.

[0030] FIG. 9 illustrates a block diagram of a decoding device according to an embodiment of the present disclosure.

[0031] FIG. 10 illustrates detailed operations of an internal decryption performer and a region-of-interest-based restorer according to one embodiment of the present disclosure.

[0032] FIG. 11 illustrates detailed operations of a region of interest information decryption unit according to one embodiment of the present disclosure.

[0033] Figures 12a and 12b illustrate detailed operations of a temporal restoration performer according to an embodiment of the present disclosure.

[0034] FIG. 13 is an example of temporal correction for a region of interest according to one embodiment of the present disclosure.

[0035] Figures 14a and 14b illustrate detailed operations of a machine vision task performer according to an embodiment of the present disclosure.

[0036] Specific structural or step-by-step descriptions of embodiments according to the concept of the present disclosure disclosed in this specification or application are merely illustrative for the purpose of explaining embodiments according to the concept of the present disclosure, and embodiments according to the concept of the present disclosure may be implemented in various forms, and embodiments according to the concept of the present disclosure may be implemented in various forms and should not be construed as being limited to the embodiments described in this specification or application.

[0037] Embodiments according to the concept of the present disclosure may have various modifications and take various forms. Therefore, specific embodiments are illustrated in the drawings and described in detail in this specification or application. However, this is not intended to limit embodiments according to the concept of the present disclosure to specific disclosed forms, and it should be understood that all modifications, equivalents, and alternatives included within the spirit and technical scope of the present disclosure are included.

[0038] While terms such as "first" and / or "second" may be used to describe various components, these components should not be limited by these terms. These terms are only intended to distinguish one component from another; for example, without departing from the scope of the present disclosure, a first component may be referred to as a "second component," and similarly, a second component may also be referred to as a "first component."

[0039] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. Conversely, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Other expressions that describe the relationship between components, such as "between" and "directly between" or "adjacent to" and "directly adjacent to", should be interpreted similarly.

[0040] The terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit the present disclosure. The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, it should be understood that the terms "comprises" or "has" indicate the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0041] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0042] Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0043] In describing the embodiments, description of technical contents that are well known in the technical field to which the present disclosure belongs and are not directly related to the present disclosure will be omitted.

[0044] This is to convey the gist of the present disclosure more clearly without obscuring it by omitting unnecessary explanations.

[0045] This document relates to video / image coding. For example, the method / embodiment disclosed in this document may be related to the Versatile Video Coding (VVC) standard (ITU-T Rec. H.266), the next-generation video / image coding standard after VVC, or other video coding-related standards (e.g., the High Efficiency Video Coding (HEVC) standard (ITU-T Rec. H.265), the essential video coding (EVC) standard, the AVS2 standard, etc.).

[0046] This document presents various embodiments of video / image coding, and unless otherwise stated, the embodiments may be performed in combination with each other.

[0047] In this document, "video" can refer to a series of images over time. "Picture" generally refers to a unit representing a single image from a specific time period, and "slice" / "tile" are units that constitute part of a picture in coding.

[0048] A slice / tile can contain one or more coding tree units (CTUs). A picture can consist of one or more slices / tiles. A picture can consist of one or more tile groups. A tile group can contain one or more tiles.

[0049] A pixel or pel can mean the smallest unit that constitutes a picture (or image). Additionally, the term "sample" can be used as a counterpart to a pixel. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luma component, or only the pixel / pixel value of the chroma component. Alternatively, a sample can mean a pixel value in the spatial domain, or when such a pixel value is converted to the frequency domain, it can mean a transform coefficient in the frequency domain.

[0050] A unit may represent a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to that region.

[0051] A unit can contain one luma block and two chroma (e.g., cb, cr) blocks. The term "unit" may sometimes be used interchangeably with the terms "block" or "area." In general, an MxN block can contain a set (or array) of samples (or array of samples) or transform coefficients, each consisting of M columns and N rows.

[0052] Figure 1 schematically illustrates an example of a video / image coding system.

[0053] Referring to FIG. 1, a video / image coding system may include a source device and a receiving device. The source device may transmit encoded video / image information or data to the receiving device via a digital storage medium or a network in the form of a file or streaming.

[0054] The source device may include a video source, an encoding device, and a transmission unit. The receiving device may include a reception unit, a decoding device, and a renderer.

[0055] The encoding device may be referred to as a video / image encoding device, and the decoding device may be referred to as a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display unit, and the display unit may be configured as a separate device or an external component.

[0056] A video source may obtain video / images through a process of capturing, synthesizing, or generating video / images. The video source may include a video / image capture device and / or a video / image generation device. A video / image capture device may include, for example, one or more cameras, a video / image archive containing previously captured video / images, etc. A video / image generation device may include, for example, a computer, a tablet, a smartphone, etc., and may (electronically) generate video / images. For example, a virtual video / image may be generated through a computer, etc., in which case the video / image capture process may be replaced by a process of generating related data.

[0057] An encoding device can encode input video / images. The encoding device can perform a series of procedures, such as prediction, transformation, and quantization, to improve compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0058] The transmission unit can transmit encoded video / image information or data output in bitstream form to the receiving unit of the receiving device via a digital storage medium or network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit can include an element for generating a media file using a predetermined file format and an element for transmission via a broadcasting / communication network.

[0059] The receiving unit can receive / extract the bitstream and transmit it to a decoding device.

[0060] The decoding device can decode the video / image by performing a series of procedures such as inverse quantization, inverse transformation, and prediction corresponding to the operation of the encoding device.

[0061] The renderer can render decoded video / images. The rendered video / images can be displayed through the display unit.

[0062] Figure 2 is a drawing schematically illustrating the configuration of a video / image encoding device.

[0063] The term “video encoding device” hereinafter may include a video encoding device.

[0064] Referring to FIG. 2, the encoding device (10a) may be configured to include an image partitioner (10a-10), a prediction unit (predictor) (10a-20), a residual processor (residual processor) (10a-30), an entropy encoder (entropy encoder) (10a-40), an adder (adder) (10a-50), a filter (filter) (10a-60), and a memory (10a-70). The prediction unit (10a-20) may include an inter prediction unit (10a-21) and an intra prediction unit (10a-22). The residual processing unit (10a-30) may include a transformer (10a-32), a quantizer (10a-33), a dequantizer (10a-34), and an inverse transformer (10a-35). The residual processing unit (10a-30) may further include a subtractor (10a-31). The addition unit (10a-50) may be called a reconstructor or a reconstructed block generator. The above-described image segmentation unit (10a-10), prediction unit (10a-20), residual processing unit (10a-30), entropy encoding unit (10a-40), addition unit (10a-50), and filtering unit (10a-60) may be configured by one or more hardware components (e.g., encoder chipset or processor) according to an embodiment. In addition, the memory (10a-70) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (10a-70) as an internal / external component.

[0065] The image segmentation unit (10a-10) can segment an input image (or picture, frame) input to the encoding device (10a) into one or more processing units.

[0066] For example, the processing unit may be called a coding unit (CU). In this case, the coding unit may be recursively split from a coding tree unit (CTU) or a largest coding unit (LCU) according to a Quad-tree binary-tree ternary-tree (QTBTTT) structure. For example, one coding unit may be split into multiple coding units of deeper depth based on a quad-tree structure, a binary-tree structure, and / or a ternary structure. In this case, for example, the quad-tree structure may be applied first, and the binary-tree structure and / or the ternary structure may be applied later. Alternatively, the binary-tree structure may be applied first. The coding procedure according to the present document may be performed based on the final coding unit that is no longer split. In this case, based on coding efficiency according to image characteristics, etc., the maximum coding unit can be used as the final coding unit, or, if necessary, the coding unit can be recursively divided into coding units of lower depths, and the coding unit of the optimal size can be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration described below. As another example, the processing unit may further include a prediction unit (PU) or a transformation unit (TU). In this case, the prediction unit and the transformation unit may each be divided or partitioned from the final coding unit described above. The prediction unit may be a unit of sample prediction, and the transformation unit may be a unit for deriving a transform coefficient and / or a unit for deriving a residual signal from a transform coefficient.

[0067] The term "unit" may be used interchangeably with terms such as "block" or "area" depending on the case. In general, an MxN block can represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luminance component, or only the pixel / pixel value of the chroma component. A sample can be used as a term corresponding to a pixel or pel in a picture (or image).

[0068] The subtraction unit (10a-31) can subtract the prediction signal (predicted block, prediction samples, or prediction sample array) output from the prediction unit (10a-20) from the input image signal (original block, original samples, or original sample array) to generate a residual signal (residual block, residual samples, or residual sample array), and the generated residual signal is transmitted to the conversion unit (10a-32). The prediction unit (10a-20) can perform prediction on a block to be processed (hereinafter, referred to as a current block) and generate a predicted block including prediction samples for the current block.

[0069] The prediction unit (10a-20) can determine whether intra-prediction or inter-prediction is applied to the current block or CU unit. As described later in the description of each prediction mode, the prediction unit can generate various prediction-related information, such as prediction mode information, and transmit it to the entropy encoding unit (10a-40). The prediction-related information can be encoded by the entropy encoding unit (10a-40) and output in the form of a bitstream.

[0070] The intra prediction unit (10a-22) can predict the current block by referring to samples within the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from it, depending on the prediction mode.

[0071] In intra prediction, prediction modes can include multiple non-directional modes and multiple directional modes. Non-directional modes can include, for example, DC modes and planar modes. Directional modes can include, for example, 33 directional prediction modes or 65 directional prediction modes, depending on the granularity of the prediction direction.

[0072] However, this is only an example; depending on the settings, a greater or lesser number of directional prediction modes may be used. The intra prediction unit (10a-22) may also determine the prediction mode to be applied to the current block by utilizing the prediction mode applied to the surrounding blocks.

[0073] The inter prediction unit (10a-21) can derive a predicted block for a current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on an inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring block can include a spatial neighboring block existing in the current picture and a temporal neighboring block existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The above temporal neighboring blocks may be called collocated reference blocks, collocated CUs (colCUs), etc., and a reference picture including the temporal neighboring blocks may be called a collocated picture (colPic). For example, the inter prediction unit (10a-21) may construct a motion information candidate list based on the neighboring blocks, and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction may be performed based on various prediction modes, and for example, in the case of skip mode and merge mode, the inter prediction unit (10a-21) may use the motion information of the neighboring blocks as the motion information of the current block. In the case of skip mode, unlike the merge mode, a residual signal may not be transmitted.In the motion vector prediction (MVP) mode, the motion vector of the surrounding blocks is used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0074] The prediction unit (10a-20) can generate a prediction signal based on various prediction methods described below. For example, the prediction unit can apply intra prediction or inter prediction to predict a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). In addition, the prediction unit can perform intra block copy (IBC) to predict a block. The intra block copy can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in this document.

[0075] The prediction signal generated through the inter prediction unit (10a-21) and / or the intra prediction unit (10a-22) can be used to generate a reconstructed signal or a residual signal. The transform unit (10a-32) can generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique can include a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), a Graph-Based Transform (GBT), or a Conditionally Non-linear Transform (CNT). Here, GBT refers to a transform obtained from a graph when the relationship information between pixels is expressed as a graph. CNT refers to a transform obtained based on generating a prediction signal using all previously reconstructed pixels. In addition, the transform process can be applied to a pixel block having a square equal size, or can be applied to a block of a non-square variable size.

[0076] The quantization unit (10a-33) quantizes the transform coefficients and transmits them to the entropy encoding unit (10a-40), and the entropy encoding unit (10a-40) can encode the quantized signal (information about the quantized transform coefficients) and output it as a bitstream. The information about the quantized transform coefficients can be called residual information.

[0077] The quantization unit (10a-33) can rearrange the quantized transform coefficients in the form of a block into a one-dimensional vector based on the coefficient scan order, and can also generate information about the quantized transform coefficients based on the quantized transform coefficients in the form of the one-dimensional vector. The entropy encoding unit (10a-40) can perform various encoding methods, such as, for example, exponential Golomb, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), etc.

[0078] The entropy encoding unit (10a-40) may encode, together or separately, information necessary for video / image restoration (e.g., values ​​of syntax elements, etc.) in addition to quantized transform coefficients. The encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream in the form of a network abstraction layer (NAL) unit. The video / image information may further include information regarding various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. The signaling / transmitted information and / or syntax elements described later in this document may be encoded through the above-described encoding procedure and included in the bitstream. The bitstream may be transmitted through a network or may be stored in a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The signal output from the entropy encoding unit (10a-40) may be configured as an internal / external element of the encoding device (10a) by a transmitting unit (not shown) that transmits and / or a storing unit (not shown) that stores the signal, or the transmitting unit may be included in the entropy encoding unit (10a-40).

[0079] The quantized transform coefficients output from the quantization unit (10a-33) can be used to generate a prediction signal. For example, by applying inverse quantization and inverse transformation to the quantized transform coefficients through the inverse quantization unit (10a-34) and the inverse transform unit (10a-35), a residual signal (residual block or residual samples) can be reconstructed. The addition unit (10a-50) can add the reconstructed residual signal to the prediction signal output from the prediction unit (10a-20), thereby generating a reconstructed signal (reconstructed picture, reconstructed block, reconstructed samples, or reconstructed sample array). When there is no residual for the target block to be processed, such as when skip mode is applied, the predicted block can be used as the reconstructed block. The generated restoration signal can be used for intra prediction of the next processing target block within the current picture, and can also be used for inter prediction of the next picture after filtering as described below.

[0080] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture encoding and / or restoration process.

[0081] The filtering unit (10a-60) can improve subjective / objective picture quality by applying filtering to the restoration signal. For example, the filtering unit (10a-60) can apply various filtering methods to the restoration picture to generate a modified restoration picture, and store the modified restoration picture in the memory (10a-70), specifically, in the DPB of the memory (10a-70). The various filtering methods may include, for example, deblocking filtering, sample adaptive offset (SAO), an adaptive loop filter, a bilateral filter, etc. The filtering unit (10a-60) can generate various information regarding filtering and transmit the information to the entropy encoding unit (10a-90), as described below in the description of each filtering method. The information regarding filtering may be encoded by the entropy encoding unit (10a-90) and output in the form of a bitstream.

[0082] The modified restored picture transmitted to the memory (10a-70) can be used as a reference picture in the inter prediction unit (10a-80). Through this, when inter prediction is applied, the encoding device can avoid prediction mismatch between the encoding device (10a) and the decoding device, and can also improve encoding efficiency.

[0083] The DPB of the memory (10a-70) can store the modified restored picture to be used as a reference picture in the inter prediction unit (10a-21). The memory (10a-70) can store motion information of a block from which motion information in the current picture is derived (or encoded) and / or motion information of blocks in a picture that has already been restored. The stored motion information can be transmitted to the inter prediction unit (10a-21) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (10a-70) can store restored samples of restored blocks in the current picture and transmit them to the intra prediction unit (10a-22).

[0084] Figure 3 is a drawing schematically illustrating the configuration of a video / image decoding device.

[0085] Referring to FIG. 3, the decoding device (10b) may be configured to include an entropy decoder (10b-10), a residual processor (10b-20), a predictor (10b-30), an adder (10b-40), a filter (10b-50), and a memory (10b-60). The predictor (10b-30) may include an inter-prediction unit (10b-31) and an intra-prediction unit (10b-32). The residual processor (10b-20) may include a dequantizer (10b-21) and an inverse transformer (10b-21). The entropy decoding unit (10b-10), residual processing unit (10b-20), prediction unit (10b-30), addition unit (10b-40), and filtering unit (10b-50) described above may be configured by a single hardware component (e.g., decoder chipset or processor) according to an embodiment. In addition, the memory (10b-60) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (10b-60) as an internal / external component.

[0086] When a bitstream including video / image information is input, the decoding device (10b) can restore the image corresponding to the process in which the video / image information is processed in the encoding device of FIG. 2. For example, the decoding device (10b) can derive units / blocks based on block division-related information obtained from the bitstream. The decoding device (10b) can perform decoding using a processing unit applied in the encoding device. Therefore, the processing unit of decoding may be, for example, a coding unit, and the coding unit may be divided from a coding tree unit or a maximum coding unit according to a quad tree structure, a binary tree structure, and / or a ternary tree structure. One or more transform units may be derived from the coding unit. Then, the restored image signal decoded and output by the decoding device (10b) can be reproduced through a reproduction device.

[0087] The decoding device (10b) can receive a signal output from the encoding device of FIG. 2 in the form of a bitstream, and the received signal can be decoded through the entropy decoding unit (10b-10). For example, the entropy decoding unit (10b-10) can parse the bitstream to derive information (e.g., video / image information) necessary for image restoration (or picture restoration). The video / image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information.

[0088] The decoding device can further decode the picture based on information about the parameter set and / or the general restriction information. The signaling / received information and / or syntax elements described later in this document can be decoded and obtained from the bitstream through the decoding procedure. For example, the entropy decoding unit (10b-10) can decode information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output values ​​of syntax elements required for image restoration and quantized values ​​of transform coefficients for the residual.

[0089] In more detail, the CABAC entropy decoding method receives a bin corresponding to each syntax element in a bitstream, determines a context model using information of the syntax element to be decoded and decoding information of surrounding and decoding target blocks or information of symbols / bins decoded in a previous step, and predicts the occurrence probability of the bin according to the determined context model to perform arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element. At this time, the CABAC entropy decoding method can update the context model using information of the decoded symbol / bin for the context model of the next symbol / bin after determining the context model. Among the information decoded by the entropy decoding unit (10b-10), information regarding prediction is provided to the prediction unit (10b-30), and information regarding the residual on which entropy decoding has been performed by the entropy decoding unit (10b-10), i.e., quantized transform coefficients and related parameter information, can be input to the inverse quantization unit (10b-21).

[0090] In addition, information regarding filtering among the information decoded by the entropy decoding unit (10b-10) may be provided to the filtering unit (10b-50). Meanwhile, a receiving unit (not shown) that receives a signal output from an encoding device may be further configured as an internal / external element of the decoding device (10b), or the receiving unit may be a component of the entropy decoding unit (10b-10). Meanwhile, the decoding device according to the present document may be called a video / video / picture decoding device, and the decoding device may be divided into an information decoder (video / video / picture information decoder) and a sample decoder (video / video / picture sample decoder). The above information decoder may include the entropy decoding unit (10b-10), and the sample decoder may include at least one of the inverse quantization unit (10b-21), the inverse transformation unit (10b-22), the prediction unit (10b-30), the addition unit (10b-40), the filtering unit (10b-50), and the memory (10b-60).

[0091] The inverse quantization unit (10b-21) can inverse quantize the quantized transform coefficients and output the transform coefficients. The inverse quantization unit (10b-21) can rearrange the quantized transform coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scanning order performed in the encoding device. The inverse quantization unit (10b-21) can perform inverse quantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain transform coefficients.

[0092] In the inverse transform unit (10b-22), the transform coefficients are inversely transformed to obtain a residual signal (residual block, residual sample array).

[0093] The prediction unit can perform a prediction for the current block and generate a predicted block including prediction samples for the current block.

[0094] The prediction unit can determine whether intra prediction or inter prediction is applied to the current block based on the information about the prediction output from the entropy decoding unit (10b-10), and can determine a specific intra / inter prediction mode.

[0095] The prediction unit can generate a prediction signal based on various prediction methods described below. For example, the prediction unit can apply intra prediction or inter prediction to predict a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). In addition, the prediction unit can perform intra block copy (IBC) to predict a block. The intra block copy can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in this document.

[0096] The intra prediction unit (10b-32) can predict the current block by referencing samples within the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from it, depending on the prediction mode.

[0097] In intra prediction, prediction modes may include multiple non-directional modes and multiple directional modes. The intra prediction unit (10b-32) may determine the prediction mode to be applied to the current block by utilizing the prediction modes applied to the surrounding blocks.

[0098] The inter prediction unit (10b-31) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on an inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.).

[0099] In the case of inter prediction, the neighboring blocks may include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit (10b-31) may construct a motion information candidate list based on the neighboring blocks, and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter prediction may be performed based on various prediction modes, and the information regarding the prediction may include information indicating the mode of inter prediction for the current block.

[0100] The addition unit (10b-40) can generate a restoration signal (restored picture, restoration block, restoration sample array) by adding the acquired residual signal to the prediction signal (predicted block, prediction sample array) output from the prediction unit (10b-30). In cases where there is no residual for the block to be processed, such as when skip mode is applied, the predicted block can be used as the restoration block.

[0101] The addition unit (10b-40) may be called a restoration unit or a restoration block generation unit.

[0102] The generated restoration signal can be used for intra prediction of the next processing target block within the current picture, can be output after filtering as described below, or can be used for inter prediction of the next picture.

[0103] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture decoding process.

[0104] The filtering unit (10b-50) can improve subjective / objective image quality by applying filtering to the restoration signal. For example, the filtering unit (10b-50) can apply various filtering methods to the restoration picture to generate a modified restoration picture, and transmit the modified restoration picture to the memory (60), specifically, the DPB of the memory (10b-60). The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0105] The (corrected) reconstructed picture stored in the DPB of the memory (10b-60) can be used as a reference picture in the inter prediction unit (10b-31). The memory (10b-60) can store motion information of a block from which motion information is derived (or decoded) within the current picture and / or motion information of blocks within a picture that has already been reconstructed. The stored motion information can be transmitted to the inter prediction unit (10b-31) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (10b-60) can store reconstructed samples of reconstructed blocks within the current picture and transmit them to the intra prediction unit (10b-32).

[0106] In this specification, the embodiments described in the prediction unit (10b-30), the inverse quantization unit (10b-21), the inverse transformation unit (10b-22), and the filtering unit (10b-50) of the decoding device (10b) can be applied to the prediction unit (10a-20), the inverse quantization unit (10a-34), the inverse transformation unit (10a-35), and the filtering unit (10a-60) of the encoding device (10a) in the same manner or correspondingly.

[0107] As described above, prediction is performed to increase compression efficiency when performing video coding. Through this, a predicted block including prediction samples for a current block, which is a coding target block, can be generated. Here, the predicted block includes prediction samples in a spatial domain (or pixel domain). The predicted block is derived identically from an encoding device and a decoding device, and the encoding device can increase video coding efficiency by signaling information (residual information) about the residual between the original block and the predicted block, rather than the original sample value of the original block itself, to a decoding device. The decoding device can derive a residual block including residual samples based on the residual information, and generate a reconstructed block including reconstructed samples by combining the residual block and the predicted block, and can generate a reconstructed picture including the reconstructed blocks.

[0108] The above residual information can be generated through transformation and quantization procedures.

[0109] For example, the encoding device can derive a residual block between the original block and the predicted block, perform a transform procedure on residual samples (a residual sample array) included in the residual block to derive transform coefficients, perform a quantization procedure on the transform coefficients to derive quantized transform coefficients, and signal related residual information to a decoding device (via a bitstream). Here, the residual information can include information such as value information, position information, transform technique, transform kernel, and quantization parameter of the quantized transform coefficients. The decoding device can perform an inverse quantization / inverse transform procedure based on the residual information to derive residual samples (or residual blocks). The decoding device can generate a reconstructed picture based on the predicted block and the residual block. The encoding device can also inverse quantize / inverse transform the quantized transform coefficients to derive a residual block for reference in inter prediction of a subsequent picture, and generate a reconstructed picture based on the residual block.

[0110] <VCM(Video coding for Machines)>

[0111] With the recent advancements in various industries such as surveillance, intelligent transportation, smart cities, intelligent industry, and intelligent content, the amount of image or feature map data consumed by machines is increasing. In contrast, traditional video compression methods currently in use were developed with human vision in mind, and therefore contain unnecessary information, making them inefficient for machine tasks. For example, the resolution of images from the viewer's perspective may be higher than that of images (e.g., feature maps) from the machine's perspective. Therefore, research on video codec technologies that efficiently compress feature maps for machine tasks is needed.

[0112] The Moving Picture Experts Group (MPEG), an international standardization group for multimedia encoding, is discussing Video Coding for Machines (VCM). VCM is an image or feature map encoding technology that targets machine vision, rather than human viewer vision. In this document, feature maps can be referred to as "feature maps," and features can be referred to as "features."

[0113] Figures 4a to 4d are exemplary diagrams showing a VCM encoder and a VCM decoder.

[0114] Referring to FIG. 4a, a VCM encoder (100a) and a VCM decoder (100b) are shown.

[0115] When a VCM encoder (100a) encodes a video and / or a feature map and transmits it as a bitstream, a VCM decoder (100b) can decode and output the bitstream. At this time, the VCM decoder (100b) can output one or more videos and / or feature maps. For example, the VCM decoder (100b) can output a first feature map for machine-based analysis and a first image for user viewing. The first image can have a higher resolution than the first feature map.

[0116] Referring to FIG. 4b, a feature extractor for extracting a feature map may be connected to the front end of the VCM encoder (100a).

[0117] The VCM encoder (100a) may include a feature encoder.

[0118] The VCM decoder (100b) may include a feature decoder and a video reconstructor. The feature decoder may decode a feature map from a bitstream and output a first feature map for machine-assisted analysis. The video reconstructor may regenerate and output a first video from the bitstream for viewing by a user.

[0119] Referring to Fig. 4c, a feature extractor for extracting a feature map is connected to the front end of the VCM encoder (100). The VCM encoder (100a) may include a feature encoder.

[0120] The VCM decoder (100b) may include a feature decoder. The feature decoder may decode a feature map from a bitstream and output a first feature map for machine-based analysis. That is, the bitstream may be encoded only as a feature map, not as an image. To elaborate, the feature map may be data containing information about features for processing a specific task of a machine based on an image.

[0121] Referring to FIG. 4d, a feature extractor may be connected to the front end of the VCM encoder (100a).

[0122] The VCM encoder (100a) may include a feature converter and a video encoder. The video encoder may be the encoding device (10a) illustrated in FIG. 2.

[0123] The VCM decoder (100b) illustrated in FIG. 4d may include a video decoder and an inverse converter. The video decoder may be the decoding device (10b) illustrated in FIG. 3.

[0124] FIG. 5 illustrates a block diagram of an encoding device according to an embodiment of the present disclosure.

[0125] An encoding device (10a) according to an embodiment of the present disclosure can receive an image (video), perform temporal and spatial resampling, extract a region of interest, and generate and output a bit stream through internal encoding. The encoding device (10a) may include a temporal resampling performer (510), a spatial resampling performer (520), a region of interest-based processor (530), and an internal encoding performer (540). The order of components included in the encoding device (10a) illustrated in FIG. 5 may be changed in various embodiments, and some components may be omitted. According to an embodiment, the temporal resampling performer (510) and / or the spatial resampling performer (520) may be omitted, and when the temporal resampling performer (510) and / or the spatial resampling performer (520) are omitted, an image processed based on a region of interest by the region of interest-based processor (530) may become an input to the internal encoding performer (540). For example, when the temporal resampling performer (510) and the spatial resampling performer (520) are omitted, one or more regions of interest may be directly extracted from the input image by the region of interest-based processor (530), and the extracted regions of interest may be input to the internal encoding performer (540), so that the internal encoding performer (540) may perform encoding to generate a bitstream. For convenience of explanation in the present disclosure, the encoding device (10a) may be referred to as an encoder or encoder, and the decoding device (10b) may be referred to as a decoder or decoder.

[0126] The temporal resampling performer (510) can receive an image, perform frame-by-frame sampling, and output an image in which some frames are sampled. The temporal resampling performer (510) can change the frame rate of some frames. According to an embodiment, the temporal resampling performer (510) can perform the same degree of sampling on the entire image. That is, the temporal resampling performer (510) can resample the entire image at the same frame rate. According to another embodiment, the temporal resampling performer (510) can perform temporal resampling by applying different sampling degrees to each group of frames in the image. That is, the temporal resampling performer (510) can resample at a variable frame rate on a group-by-group basis.

[0127] In one embodiment, the temporal resampling performer (510) may perform sampling at different sampling rates for the region-of-interest-based processed image and the non-region-of-interest image. For example, the temporal resampling performer (510) may perform sampling at a first sampling rate for the region-of-interest-based processed image and at a second sampling rate for the non-region-of-interest image.

[0128] According to an embodiment, the temporal resampling performer (510) may transmit information used in the temporal resampling process (such as a temporal sampling rate, in one example) to the decoder. The decoder may perform decoding on the region-of-interest-based processed image and the non-region-of-interest image, and then parse the respective sampling rates corresponding to the region-of-interest-based processed image and the non-region-of-interest image to perform upsampling on each. At this time, the sampling rate applied to the region-of-interest-based processed image and the sampling rate applied to the non-region-of-interest image may be different.

[0129] The spatial resampling performer (520) can receive an input image or an image on which temporal resampling has been performed and output an image with a changed spatial resolution for each frame or a series of frames. The spatial resampling performer (520) can change the resolution for a region of interest included in each frame of the input image. In one embodiment, the spatial resampling performer (520) can change the resolution only for the region of interest. If there are multiple regions of interest in one frame, the spatial resampling performer (520) can change the resolution for each region of interest. Information about each region of interest can include changed resolution information. For example, a first scale factor can be applied to a first region of interest, and a second scale factor can be applied to a second region of interest. The scale factor can be stored using at least one of the following parameters: scale factor, scale factor nominator, scale factor, denominator, size(%).

[0130] In another embodiment, the spatial resampling performer (520) may perform sampling by applying different resolutions to the region of interest and the region of no interest, respectively. For example, the spatial resampling performer (520) may change the region of interest-based processed image to a first resolution, and change the region of no interest image to a second resolution.

[0131] According to an embodiment, the spatial resampling performer (520) may transmit information used in the spatial resampling process (e.g., a spatial sampling rate of each frame and / or a spatial sampling rate per series of frames, etc.) to the decoder. After decoding the bitstream, the decoder may parse the respective resolution information for the region-of-interest-based processed image and the non-region-of-interest image, and change the respective resolutions. At this time, the resolution applied to the region-of-interest-based processed image and the resolution applied to the non-region-of-interest may be different.

[0132] The region-of-interest-based processor (530) can receive an input image, an image on which temporal resampling has been performed, or an image on which temporal resampling and spatial resampling have been performed, extract a region of interest existing in each frame or a series of frames, and output a processed image based on the extracted region of interest. The region-of-interest-based processor (530) can include a frame analysis unit (531), a region-of-interest selection unit (532), and a region-of-interest-based image processing unit (533). The region-of-interest-based processor (530) can extract a region of interest from a plurality of frames included in the input image, and set a correspondence for the same region of interest (e.g., a detected object, etc.) based on each frame. The region-of-interest-based processor (530) can set the same region of interest for the consecutive frames when the same object (region of interest) is detected in a series of consecutive frames. The region-of-interest-based processor (530) can transmit information about the region of interest to a decoder. Information about the region of interest can be stored for each frame. The region of interest-based processor (530) can set a correspondence between multiple frames containing the same region of interest and store changes in position, size, etc. for the region of interest. For example, the region of interest-based processor (530) can signal linear motion, affine motion, etc. for the region of interest existing within a series of frame units and transmit them to the decoder. The region of interest-based processor (530) can signal a position change value for linear motion and can signal affine motion using parameters defined according to the type of motion. For example, the region of interest-based processor (530) can transmit affine motion information including changes in scale, rotation, and sheer to the decoder.The region of interest based processor (530) can transmit a flag (e.g., LinearMotion_flag) to the decoder to identify when affine motion exists.

[0133] The region-of-interest-based processor (530) can change the resolution based on the region of interest. In some embodiments, if there are multiple regions of interest in a single frame, the region-of-interest-based processor (530) can change the resolution for each region of interest. Information about each region of interest can include information about the changed resolution. For example, a first scale factor can be applied to a first region of interest, and a second scale factor can be applied to a second region of interest.

[0134] The region-of-interest-based processor (530) can transmit information used in the region-of-interest-based processing (e.g., region-of-interest ID information, region-of-interest size information, packing information, temporal sampling rate, spatial resolution, etc.) to the decoder. A detailed description of the region-of-interest-based processor (530) is described later in FIG. 6.

[0135] The internal encoding performer (540) can receive an input image, or an image on which some or all of temporal / spatial resampling and region of interest processing have been performed, and perform image encoding to generate a bitstream. The internal encoding performer (540) can include a downsampling performer (541), an image encoding performer (542), and a region of interest information encoding performer (543). According to an embodiment, the internal encoding performer (540) can use a 2D video encoder (e.g., AVC / H.264, HEVC / H.265, VVC / H.266, AV1, VP9, ​​etc.) and can use a 2D video encoder including one or more convolution layers. According to an embodiment, encoding can be performed after converting the color space of the input image of the internal encoding performer (540) to a color space such as YUV420 or YUV444. At this time, according to the embodiment, the internal encoding performer (540) can perform the conversion through a conversion method defined by the agreement between the sub-decoder and the decoder, and transmit the color space conversion information to the decoder. A detailed description of the internal encoding performer (540) will be described later in FIGS. 7A to 7C.

[0136] An encoding device (10a) according to an embodiment of the present disclosure can improve the accuracy of a deep neural network by extracting a region of interest from an input image and transmitting the extracted region. In addition, the encoding device (10a) has the effect of increasing encoding efficiency by extracting a region of interest and performing temporal resampling together to transmit the extracted region to a decoding device (10b). The encoding device (10a) can compress information such as position and movement for each region of interest for each GOP (Group of Pictures) or a series of frame bundles and transmit the compressed information to the decoding device (10b).

[0137] FIG. 6 illustrates detailed operations of an area-of-interest-based processor according to one embodiment of the present disclosure.

[0138] An area-of-interest-based processor (530) of an encoding device (10a) according to one embodiment of the present disclosure may include a frame analysis unit (531), an area-of-interest selection unit (532), and an area-of-interest-based image processing unit (533). The frame analysis unit (531) may include an analysis area selection module (531a) and an area-of-interest candidate extraction module (531b). The area-of-interest selection unit (532) may include an area-of-interest selection module (532a) and a corresponding area-of-interest selection module (532b). In various embodiments, some of the detailed components of the area-of-interest-based processor (530) may be omitted, and the order between the components may be changed.

[0139] The frame analysis unit (531) can receive an input image and output a region of interest candidate. The input image may be an original image, or, depending on various embodiments, an image that has undergone temporal and / or spatial resampling.

[0140] The frame analysis unit (531) can obtain information (e.g., number of regions of interest, location of regions of interest, score of regions of interest, etc.) of one or more region of interest candidates existing in each frame of an input image or an image that has undergone temporal / spatial resampling. The score of the region of interest can be calculated according to a predetermined standard. For example, the score of the region of interest can be calculated according to the importance or weight of the region of interest. As an example, when extracting region of interest candidates using a deep neural network model trained for the purpose of object detection or object tracking, the score of the region of interest can be calculated in proportion to the object detection accuracy included in the region of interest.

[0141] The analysis region selection module (531a) can select an analysis region for an input image in units of one or more frames. The analysis region selection module (531a) can select a region on which frame analysis is to be performed among all frames, and then the frame analysis module can be performed only on some regions within the image. The analysis region selection module (531a) can select the same region for all frames within the image. Alternatively, the analysis region selection module (531a) can select the same region in units of a series of frame groups within the image. The regions selected for each series of frame groups may be different. The analysis region selection module (531a) can transmit information on some regions selected as analysis targets to the decoder. Depending on the embodiment, the analysis region selection module (531a) may not transmit to the decoder regions on which the frame analysis module has not been performed.

[0142] The region of interest candidate extraction module (531b) can extract region of interest candidates on a frame-by-frame basis for the selected analysis target region, or can extract region of interest candidates on a series of frame basis. The size of the region of interest candidates in each frame may be a multiple of 4 in both height and width. According to an embodiment, the region of interest candidate extraction module (531b) may use a deep neural network composed of one or more convolutional layers on which training has been performed. According to an embodiment, the deep neural network may be a neural network trained for the purpose of object detection, or a neural network trained for the purpose of object tracking. The region of interest candidate extraction module (531b) can extract region of interest candidates on a frame-by-frame basis, or can extract region of interest candidates on a series of frame basis. The region of interest candidate extraction module (531b) can exclude from the region of interest candidates if the detection accuracy of a detected object (region of interest candidate) is low for each frame or a series of frame groups.

[0143] The region of interest selection unit (532) can select a region of interest from among the region of interest candidates output from the frame analysis unit (531), determine a corresponding relationship among multiple regions of interest, and select a corresponding region of interest. The selected region of interest can be transmitted to the internal encoding performer (540) for image encoding.

[0144] The region of interest selection module (532a) can determine a region of interest to be encoded among the region of interest candidates. The non-region of interest image may be an area outside the region of interest within the frame, and the portion corresponding to the non-region of interest may be filled with surrounding pixel values ​​and / or intermediate values. Depending on the embodiment, the region of interest selected from the input image may vary for each frame and may vary for each group of frames.

[0145] The corresponding region of interest selection module (532b) can identify the correspondence relationship between regions of interest included in frames within an image, and select a corresponding region of interest based on the correspondence relationship. In one embodiment, the corresponding region of interest selection module (532b) can select a corresponding region of interest for a series of frame groups when the same region of interest (e.g., a detected object) is included in consecutive frames. In one example, when the corresponding region of interest selection module (532b) is performed for a series of frame groups, the region of interest candidate (detected object) of the first input frame can be selected as a candidate, and the corresponding region of interest can be selected through the correspondence relationship with the region of interest candidate of the subsequent frame. According to an embodiment, the correspondence relationship between the region of interest candidate of the first frame and the region of interest candidate of the subsequent frame can be obtained based on the similarity between pixels between the entire region and / or a partial region of each detected object. In one example, the similarity between pixels can be calculated through cross-correlation and / or histogram comparison of pixel values.

[0146] The corresponding region of interest selection module (532b) can downsample or upsample all region of interest candidates to a specific size, and then select a corresponding region of interest. Depending on the embodiment, the specific size may be determined based on the resolution of the image, and / or the size of the region of interest with the smallest size may be determined by comparing the sizes of the region of interest candidates extracted for each frame.

[0147] The corresponding region of interest selection module (532b) can assign an ID to the selected region of interest. The corresponding region of interest selection module (532b) can assign an ID to each unit in which the corresponding region of interest is selected. If the regions of the region of interest candidates extracted by the region of interest candidate extraction module (531b) within a frame overlap with each other, and if the overlapping region exceeds a certain threshold value, the corresponding regions of interest can be classified as a single region of interest. In this case, the threshold value can be determined based on the size of the frame and / or the size of the region of interest candidate.

[0148] The corresponding region of interest selection module (532b) may assign a new ID to the region of interest in the case where there is no region of interest in the current frame that corresponds to the region of interest in the previous frame. In the case where there is a region of interest corresponding to the region of interest in the previous frame in the current frame, the corresponding region of interest selection module (532b) may assign the existing ID of the region of interest in the corresponding previous frame to the region of interest in the current frame. In various embodiments, the corresponding region of interest selection module (532b) may utilize various methods for obtaining a correspondence relationship.

[0149] According to an embodiment, when the region of interest candidate extraction module (531b) uses a neural network trained for the purpose of object tracking, the operation of the corresponding region of interest selection module (532b) may be omitted because the object to be tracked corresponds to the corresponding region of interest.

[0150] The region of interest-based image processing unit (533) may receive information on regions of interest selected in an image and generate a processed image based on the region of interest for each frame according to various embodiments. In one example, the region of interest-based image processing unit (533) may pack regions of interest and generate each frame in which the regions of interest are packed. The region of interest-based image processing unit (533) may fill non-regions of interest excluding the region of interest in a frame with a specific value (e.g., a median value, etc.) and generate a frame in which the non-regions of interest are filled with the specific values.

[0151] According to an embodiment, the region of interest-based processor (530) may transmit information used in the region of interest extraction process (in one example, region of interest identification (ID) information, region of interest size information, packing information, etc.) to the decoder.

[0152] Figures 7a, 7b and 7c illustrate detailed operations of an internal encoding performer according to an embodiment of the present disclosure.

[0153] An internal encoding performer (540) of an encoding device (10a) according to an embodiment of the present disclosure can generate a bitstream by encoding an image processed based on a region of interest, an image of a non-region of interest, and information on a region of interest. An internal encoding performer (540) according to an embodiment of the present disclosure can include a downsampling performer (541), an image encoding performer (542), and an region of interest information encoding performer (543). The region of interest information encoding performer (543) can include an region of interest selection module (543a) for performing information encoding, a frame selection module (543b) for performing information encoding, and an information encoding performer module (543c). In various embodiments, some of the detailed components of the internal encoding performer (540) may be omitted, and the order between the components may be changed.

[0154] The internal encoding performer (540) can perform encoding for each frame of an image processed based on a region of interest, and can perform encoding only for some frames within a series of frame groups for an image of a non-region of interest. The internal encoding performer (540) can perform downsampling before performing encoding, and then encode information on a region of interest within an image processed based on a region of interest. The bitstreams output through the image encoding performer (542) and the region of interest information encoding performer (543) can be combined (MUXed) to generate a single bitstream. A detailed description of the region of interest information encoding performer (543) will be described later with reference to FIG. 8.

[0155] The downsampling unit (541) can perform downsampling at different sampling rates for the region-of-interest-based processed image and the non-region-of-interest image prior to performing image encoding. When downsampling is performed, the downsampling unit (541) can signal sampling rate information to the decoder.

[0156] The video encoding unit (542) can use a video encoder (AVC / H.264, HEVC / H.265, VVC / H.266, AV1, VP9, ​​etc.) and can use a 2D video encoder including one or more convolution layers. The video encoding unit (542) can use the same video encoder for the region-of-interest-based processed image and the non-region-of-interest image, or can use different video encoders.

[0157] The region of interest information encoding unit (543) can encode information (e.g., location, movement, mapping information, etc.) about regions of interest included in all or some frames of an input image. The region of interest information encoding unit (543) can utilize some processes (e.g., entropy coding, etc.) of the video encoder of the image encoding unit (542).

[0158] Referring to FIG. 7a, the internal encoding performer (540) can apply the downsampling performer (541) and the image encoding performer (542) to the non-region of interest image. Referring to FIG. 7b, the internal encoding performer (540) can apply the downsampling performer (541) and the image encoding performer (542) to the region of interest-based processed image, and separately apply the region of interest information encoding performer (543) to the information about the region of interest to generate a bitstream. Referring to FIG. 7c, the internal encoding performer (540) can first perform the downsampling performer (541) to the region of interest-based processed image, then apply the region of interest information encoding performer (543) to the information about the region of interest, and then apply the image encoding performer (542) to the region of interest-based processed image to generate a bitstream.

[0159] FIG. 8 illustrates detailed operations of an area of ​​interest information encoding performing unit according to one embodiment of the present disclosure.

[0160] According to an embodiment of the present disclosure, an area of ​​interest information encoding performing unit (543) included in an internal encoding performing unit (540) of an encoding device (10a) may encode information about an area of ​​interest for an image processed based on an area of ​​interest to generate a bitstream. The area of ​​interest information encoding performing unit (543) according to an embodiment of the present disclosure may include an area of ​​interest selection module (543a) for performing information encoding, a frame selection module (543b) for performing information encoding, and an information encoding performing module (543c). Some of the detailed components of the area of ​​interest information encoding performing unit (543) may be omitted, and the order between the components may be changed.

[0161] The region of interest selection module (543a) for performing information encoding can receive an image processed based on a region of interest and select a region of interest for performing information encoding among the selected regions of interest. The target of information encoding can be motion information of the region of interest, compensation parameters, etc.

[0162] The frame selection module (543b) for performing information encoding can receive a region of interest for performing information encoding and / or a region of interest-based processed image, and select a frame for performing information encoding among the frames in the image.

[0163] The information encoding performing module (543c) can encode information of all regions of interest within the selected frame. The information encoding performing module (543c) can encode only information of regions of interest selected by the region of interest selection module (543a) for performing information encoding within the selected frame. The information encoding performing module (543c) can encode information of regions of interest selected by the region of interest selection module (543a) for performing information encoding within the frame selected by the frame selection module (543b) for performing information encoding, thereby generating a bitstream.

[0164] According to an embodiment, when temporal resampling is performed on an input image, the region of interest information encoding unit (543) may encode region of interest information for some frames with a period smaller than the temporal sampling rate. In one example, when temporal resampling is performed periodically using a sampling rate n (n is an integer greater than or equal to 1), the region of interest information encoding unit (543) may encode region of interest information within a corresponding frame every frame period m (m is an integer greater than or equal to 1 and less than or equal to n). In another example, when temporal resampling is performed aperiodically on the entire image, the region of interest information encoding unit (543) may encode region of interest information within a corresponding frame every k / p (p is an integer greater than or equal to 1 and less than or equal to k) periods, when the frame interval between any two adjacent frames of an image on which temporal resampling is performed is k (k is an integer greater than or equal to 1). The region of interest information encoding performing unit (543) can encode region of interest information within a frame for each u (u is an integer greater than or equal to 1) frame cycle.

[0165] Through the frame selection process, region of interest information can be encoded only for some regions of interest within the frame in which region of interest information is to be encoded.

[0166] According to an embodiment, when temporal resampling is not performed on the input image, the region of interest information encoding unit (543) may encode only information of one or more specific regions of interest. The region of interest information encoding unit (543) may encode only information of regions of interest between specific frames. According to an embodiment, when the difference in representative brightness values ​​between each frame is greater than or equal to a specific threshold value, only information of regions of interest between the corresponding frames may be encoded. When the difference in representative brightness values ​​of corresponding regions of interest within each frame between each frame is greater than or equal to a specific threshold value, only information of regions of interest between the corresponding frames may be encoded.

[0167] The region of interest information encoding unit (543) can perform region of interest information encoding on regions of interest that correspond to each other within each frame between two frames for which region of interest information encoding is to be performed. The region of interest information encoding unit (543) can encode information on the location of the region of interest and can encode a position difference value between the two regions of interest. In this case, the location may be the upper left coordinate of the region of interest. The region of interest information encoding unit (543) can encode motion parameters together with the location information (coordinates and difference values) of the region of interest. In one example, the motion parameters may be parameters for expressing affine motion. The region of interest information encoding unit (543) can encode information on translation motion as location information (coordinates and difference values) and information on scale, sheer, and rotation as motion parameters.

[0168] According to an embodiment, the affine motion can be expressed by mathematical expression 1.

[0169]

[0170] Here, a, b, c, and d represent affine parameters. Trans_x and Trans_y represent transformation motion information about the x-axis and y-axis. x and y represent the positions of pixel values ​​within the current region of interest, and x' and y' represent the positions of pixel values ​​to which affine motion is applied.

[0171] The region of interest information encoding unit (543) can transmit the position information and the motion parameters of the region of interest to the decoder. According to an embodiment, the region of interest information encoding unit (543) can encode compensation parameter information between two regions of interest when 1) the difference in the representative value of the brightness value between two frames is greater than a specific threshold value and / or 2) the difference in the representative value of the brightness value of the corresponding region of interest is greater than a specific threshold value. At this time, information on whether to apply the compensation parameter between the regions of interest can be signaled on a frame-by-frame basis or a region-by-region basis. In various embodiments, 1) the specific threshold value for the difference in the representative value of the brightness value between frames and 2) the specific threshold value for the difference in the representative value of the emission value of the corresponding region of interest may be different from each other.

[0172] According to an embodiment, the compensation parameter can be defined by two parameters (for example, α and β) and can be expressed as in mathematical expression 2 below.

[0173]

[0174] Here, RoI i (past) represents the ith region of interest of the temporally previous frame among the two frames, and RoI i (future) represents the ith region of interest in the temporally next frame.

[0175] FIG. 9 illustrates a block diagram of a decoding device according to an embodiment of the present disclosure.

[0176] A decoding device (10b) according to an embodiment of the present disclosure may receive a bitstream from an encoding device (10a), perform decoding, and output a restored image (video). The decoding device (10b) may include an internal decoding performer (910), a region-of-interest-based restorer (920), a spatial restoration performer (930), a temporal restoration performer (940), and a post-processing filter performer (950). A machine vision task performer (1000) may perform a machine vision task on an image restored by the decoding device (10b) and output a result. When the machine vision task is object detection, object tracking, or object segmentation, a metric for measuring the accuracy of the task execution result may be used. For example, the metric for measuring the accuracy of the machine vision task execution result may be mAP for an object detection result, MOTA for an object tracking result, etc. According to an embodiment, the decoding device (10b) may further include a machine vision task performer (1000). According to an embodiment, the order of the components of FIG. 9 may be changed. According to an embodiment, the order of the region-of-interest-based restorer (920), the spatial restoration performer (930), and the temporal restoration performer (940) may be the reverse order of the order in which each corresponding process (region-of-interest-based encoder (920), the spatial sampling performer (930), and the temporal sampling performer (940)) is performed in the encoding process of the original image according to the encoding device (10a) of FIG. 5.

[0177] The internal decoding performer (910) can receive a bitstream as input and perform image decoding to generate a restored image. The internal decoding performer (910) can include an image decoding performer (911), an upsampling performer (912), and a region of interest information decoding performer (913). According to an embodiment, the internal decoding performer (910) can use a 2D video decoder (AVC / H.264, HEVC / H.265, VVC / H.266, AV1, VP9, ​​etc.) and can use a 2D video decoder including one or more convolution layers. According to an embodiment, the internal decoding performer (910) can implicitly or / and explicitly convert the color space of the restored image to another color space and then perform an image decoding process thereafter. For example, if the color space of the restored image is not RGB444 but YUV420, YUV444, etc., the image decoding process can be performed after implicitly or / and explicitly converting to RGB444 space.

[0178] The region-of-interest-based restorer (920) may restore a region-of-interest-based processed image using, according to an embodiment, a decoded image and region-of-interest-based processing information (for example, region-of-interest ID information, region-of-interest size information, packing information, etc.) transmitted from the encoding device (10a). The region-of-interest-based restorer (920) may include an image reconstruction unit (921).

[0179] The spatial restoration performer (930) may perform spatial restoration on a decoded image or a decoded and region-of-interest-based restoration image using information used in the spatial resampling process transmitted from the encoder according to an embodiment (for example, a spatial sampling rate of each frame and / or a spatial sampling rate of a series of frames, etc.).

[0180] The temporal restoration performer (940) may perform temporal restoration on an image on which part or all of the preceding process of FIG. 9 has been performed, using information (for example, temporal sampling rate, etc.) used in the temporal resampling process transmitted from the encoding device (10a), according to an embodiment.

[0181] The post-processing filter performer (950) may perform filtering on an image on which part or all of the preceding steps of FIG. 9 have been performed, depending on the embodiment. Depending on the embodiment, the post-processing filter performer (950) may use a fixed filter, or may utilize a plurality of filters set by agreement between the encoding device (10a) and the decoding device (10b). The post-processing filter performer (950) may receive filter information agreed upon in advance from the encoding device (10a).

[0182] The machine vision task performer (1000) may receive a restored image or a restored image and additional information according to an embodiment and perform a process for deriving a machine vision task result (for example, mAP, MOTA, etc.). The machine vision task performer (1000) may utilize a deep neural network composed of one or more convolutional layers on which learning has been performed. The machine vision task performer (1000) may receive information on the type of machine vision task and / or the type of deep neural network from the encoding device (10a).

[0183] FIG. 10 illustrates detailed operations of an internal decryption performer and a region-of-interest-based restorer according to one embodiment of the present disclosure.

[0184] FIG. 10 illustrates each step of the internal decoding process and the region-of-interest-based restoration process by each component of the internal decoding performer (910) and the region-of-interest-based restoration unit (920) included in the decoding device (10b) of FIG. 9. According to an embodiment, the input bitstream may first undergo Demuxing and then be applied as input to each of the decoding performers (911, 913) of FIG. 9.

[0185] The image decoding unit (911) can receive a bitstream and perform decoding to restore image and region of interest information. The number of frames of the image processed based on the restored region of interest and the number of frames of the image of the restored non-region of interest may be different.

[0186] The upsampling unit (912) may receive a sampling rate and perform upsampling on the region-of-interest-based processed image and the non-region-of-interest image after decoding. The sampling rate may be different for the region-of-interest-based processed image and the non-region-of-interest image. According to an embodiment, the upsampling unit (912) may select and use one of bilinear upsampling, bilateral upsampling, nearest-neighbor upsampling, and a deep neural network including one or more convolutional layers as an upsampling method. According to an embodiment, the upsampling unit (912) may perform upsampling on the region-of-interest-based processed image and the non-region-of-interest image using different upsampling methods, and may also variably select and apply the upsampling method within the region-of-interest-based processed image. For example, the upsampling performing unit (912) can determine the upsampling method according to the sampling rate, and if the sampling rate is different, the upsampling method may also be different.

[0187] The region of interest information decoding unit (913) can receive a bitstream, perform decoding, and restore the region of interest information. If the information on the region of interest down-sampled by the encoding device (10a) is encoded, the region of interest information decoding unit (913) can perform decoding on the region of interest information and then transmit it to the upsampling unit (912) to perform upsampling. According to an embodiment, the region of interest information decoding unit (913) can perform decoding on each region of interest when multiple regions of interest exist in one frame. A detailed description of the region of interest information decoding unit (913) will be described later with reference to FIG. 11.

[0188] According to an embodiment, the image reconstruction unit (921) may receive restored region of interest and restored non-region of interest images, perform restoration to the original image, and output a restored image. According to an embodiment, restored region of interest information may be used in the image reconstruction unit (921).

[0189] According to an embodiment, the image reconstruction unit (921) may perform filtering for resolution improvement on the restored image. The image reconstruction unit (921) may further reconstruct the image using restored region of interest information to generate a restored image. In various embodiments, the image reconstruction unit (921) may perform filtering on at least one of the boundary and region of interest for the region of interest-based processed image and the non-region of interest image. In one embodiment, the image reconstruction unit (921) may perform boundary filtering of the region of interest-based processed image and the non-region of interest using a region boundary filtering performing module. In another embodiment, the image reconstruction unit (921) may perform filtering, such as deblurring, on one or more regions of interest of the region of interest-based processed image.

[0190] In various embodiments, the image reconstruction unit (921) may perform filtering for resolution improvement to restore an image to which temporal resampling has been applied. In the temporal resampling process that deletes intermediate frames, intermediate frames with relatively low quality may be restored due to quality differences between frames. If deblurring filtering is performed on a region of interest of the restored intermediate frames, the quality of the intermediate frames may be improved. For example, the image reconstruction unit (921) may perform deblurring filtering on a region of interest of two frames input during the temporal resampling process to generate a restored image. Depending on the embodiment, the image reconstruction unit (921) may perform filtering using any one of several deep neural networks trained for various functions. For example, the image reconstruction unit (921) may perform deblurring within a region of interest using a deep neural network trained for the purpose of deblurring. When deblurring filtering is applied, restoration performance may be improved.

[0191] A restored non-region of interest image may exist for each frame of the restored region of interest, and one restored non-region of interest image may exist for each series of frame groups. For example, a restored non-region of interest image may exist only for the intra frame in each GOP unit. According to an embodiment, when one restored non-region of interest image exists for each series of frame groups, the image reconstruction unit (921) may generate a restored image using the same restored non-region of interest image for each frame of the restored region of interest. According to an embodiment, for each restored region of interest, different filtering may be applied to the boundary of the region of interest and the boundary of the restored non-region of interest image at the corresponding position. According to an embodiment, the image reconstruction unit (921) may implicitly determine a filtering method based on the difference in pixel values ​​between the boundary of the region of interest and the boundary of the restored non-region of interest image at the corresponding position for each restored region of interest. According to an embodiment, the image reconstruction unit (921) may receive a filtering method for a specific region of interest from the encoding device (10a) and perform filtering.

[0192] FIG. 11 illustrates detailed operations of a region of interest information decryption unit according to one embodiment of the present disclosure.

[0193] According to an embodiment of the present disclosure, a region of interest information decoding performing unit (913) included in an internal decoding performing unit (910) of a decoding device (10b) may decode region of interest information based on a linear motion information flag (LinearMotion_flag) and a compensation parameter information flag (Comp_param_flag) parsed from a bitstream. The region of interest information decoding performing unit (913) may include a position information decoding module (913a), a motion parameter restoration performing unit (913b), and a compensation parameter restoration performing unit (913c). The region of interest information decoding performing unit (913) may perform each step of FIG. 11 on a region of interest basis and / or a frame basis. In various embodiments, the order of each step of FIG. 11 may be changed, and some configurations may be omitted.

[0194] According to an embodiment, the position information decoding module (913a) can decode position information of a specific region of interest between specific frame intervals. As an example, the position information decoding module (913a) can decode position change / difference information of a corresponding region of interest between two frames. In this case, linear position change / difference values ​​can be restored.

[0195] In some embodiments, if a particular region of interest or regions of interest within a particular frame have only linear motion in addition to additional motion, LinearMotion_flag is 0 and the decoding process of additional position and motion information may be omitted.

[0196] In an embodiment, when LiearMotion_flag is 1, it indicates that there is additional motion other than linear motion between regions of interest, and the motion parameter restoration performing unit (913b) can perform motion parameter restoration. As an example, the additional motion may be an affine motion such as scale, rotation, or sheer.

[0197] According to an embodiment, the motion parameter restoration performing unit (913b) may perform motion parameter restoration regardless of LiearMotion_flag. Affine motion information may be expressed using affine parameters (e.g., {a, b, c, d}, or {a,b,c,d, Trans_x, Trans_y}) according to mathematical expression 1. According to an embodiment, values ​​in the motion information may be independently transmitted for each region of interest. At least one of a difference value with respect to the motion information of a region of interest having an ID identical to or corresponding to a previous frame and a difference value with respect to the motion information of a region of interest already decoded in the current frame may be transmitted from the encoding device (10a). The motion information of each region of interest may be managed in units of a series of frame groups. Here, the series of frame groups may be units of a group of pictures (GOP). According to an embodiment, for each region of interest within a frame group, motion information of the region of interest within the first frame may be transmitted from the encoding device (10a), and for the region of interest within the next frame in the decoding order, only the differential value for the motion information of the region of interest of the same ID or corresponding ID of the previous frame may be transmitted. For each region of interest, the differential value of the motion information may be transmitted from an intermediate frame within the group. At this time, the differential value may be a difference in value or a difference in ratio. According to an embodiment, the differential value may be transmitted only for some of the motion information, and the remaining motion information may be transmitted on a frame-by-frame basis.

[0198] According to an embodiment, after performing decoding on position and motion information, if Comp_param_flag parsed from the bitstream is 1, the compensation parameter restoration performing module (913c) can decode compensation parameters (for example, scale parameter α, offset parameter β). The compensation parameters can be received and decoded from the encoding device (10a), and can be derived from the representative brightness values ​​between two corresponding regions of interest or two corresponding frames in the decoding device (10b).

[0199] Figures 12a and 12b illustrate detailed operations of a temporal restoration performer according to an embodiment of the present disclosure.

[0200] A temporal restoration performer (940) according to an embodiment of the present disclosure can generate an image that has been temporally restored based on a restored image and restored region of interest information. The temporal restoration performer (940) can include a temporal frame restoration performer (941) and a region of interest temporal correction unit (942). In various embodiments, the order of the detailed components of the temporal restoration performer (940) can be changed, and some components can be omitted. FIG. 12A illustrates an embodiment in which the temporal frame restoration performer (941) receives a restored image and restored region of interest information, and generates a temporally restored image. FIG. 12B illustrates an embodiment in which the temporal frame restoration performer (941) receives a restored image and performs temporal frame restoration first, and then the region of interest temporal correction unit (942) receives an image on which temporal frame restoration has been performed and restored region of interest information, and generates an image on which temporal restoration has been performed.

[0201] Referring to FIG. 12a, a temporal frame restoration performing unit (941) according to one embodiment parses information on the number of frames to be restored (for example, temporal sampling rate, etc.), receives restored images and restored region of interest information, performs restoration of intermediate frames using the restored images, and at this time, corrects pixel value information and position information of regions of interest within the restored intermediate frames using the restored region of interest information.

[0202] Referring to FIG. 12B, a temporal frame restoration performing unit (941) according to an embodiment may receive information on the number of frames to be restored (for example, temporal sampling rate, etc.) and perform restoration of intermediate frames using restored images. A region of interest temporal correction unit (942) may receive an image on which temporal frame restoration has been performed and information on the restored region of interest according to an embodiment and perform correction on regions of interest within the image on which temporal frame restoration has been performed.

[0203] FIG. 13 is an example of temporal correction for a region of interest according to one embodiment of the present disclosure.

[0204] A decoding device (10b) according to one embodiment of the present disclosure includes a temporal restoration performer (910), and the temporal restoration performer (940) can perform temporal correction for a region of interest. In one embodiment, a region of interest temporal correction unit (942) of the temporal restoration performer (940) can perform correction for a region of interest within restored intermediate frames using restored region of interest information.

[0205] According to an embodiment, for a specific region of interest having a size of W x H in a frame of a restored image, among the restored intermediate frames, based on the position of the corresponding intermediate frame, a search range nW x mH may be selected for integers n and m greater than or equal to 1, as shown in (a) of FIG. 13. According to a specific search method within the search range, the optimal corresponding region of interest in the restored intermediate frame may be found using the difference value (for example, SAD (Sum of Absolute Difference)) between the W x H block and the region of interest of the size of W x H in the frame of the restored image, and then position correction may be performed on the corresponding block according to the information of the restored region of interest, as shown in (b) of FIG. At this time, during position correction, an area where a value does not exist, that is, an area that is not included in the corresponding region of interest after correction among the corresponding regions of interest before preservation in (b) of FIG. 13, may be filled with an intermediate value, or a value may be filled in through a specific filtering process.

[0206] In an embodiment, if compensation parameter information exists in the restored region of interest information, the compensation parameter can be applied to the search process to search.

[0207] According to an embodiment, if a restored image is subject to both region-of-interest-based restoration and temporal restoration, the internal decryption performer (910) may perform temporal restoration first after performing internal decryption, and then region-of-interest-based restoration. In one embodiment, among the region-of-interest-based restoration methods, only the restoration process of position information may be performed after performing temporal restoration.

[0208] According to an embodiment, in the temporal restoration process, temporal restoration can be performed on each frame in the restored image through an internal decoding performer within a series of frame groups according to the following four methods. The temporal restoration performer (940) can perform temporal restoration using any one of the following temporal restoration methods based on the size of the region of interest included in two adjacent or non-adjacent frames.

[0209] 1) If the sizes of the regions of interest of two adjacent or non-adjacent frames are the same, the temporally earlier frame among the two frames can be copied to generate a restored frame. Thereafter, the location of the region of interest can be restored using the location information in the region-of-interest-based restoration process. The location information of the intermediate frames (frames restored using the copy method) can be restored by subsampling the location information of the regions of interest of the two frames, which are inputs to the temporal restoration process. In one embodiment, the size of the region of interest can be the size of the area in which the region of interest exists or the size of the area occupied by each region of interest.

[0210] 2) When the sizes of the regions of interest of two adjacent or non-adjacent frames are the same, the temporal frame restoration method can be determined by comparing one or more metric measurements, such as PSNR (Peak signal-to-noise ratio) and SSIM (Structural Similarity Index Measure), for the regions of interest within each frame.

[0211] 3-1) If the difference value of the size of the region of interest of two adjacent frames is less than a certain threshold value, and the number of frames for which temporal restoration is to be performed using the two frames is n, temporal restoration can be performed by copying only the preceding frame or only the following frame. Or, The second frame copies the frame that is temporally earlier than the other two frames, The second frame can perform temporal restoration by copying the frame that is later in time among the two frames. Thereafter, the location information can be used to restore the location of the region of interest in the region-of-interest-based restoration process, or the location information of the regions of interest of the two frames that are input to the temporal restoration process can be subsampled to restore the location information of the intermediate frames (frames restored through the copy method). The subsampling method may vary, and as an example, for frames generated by copying the previous frame, more weight can be given to the location information of the region of interest of the previous frame. In one embodiment, the size of the region of interest may be the size in which the region of interest exists or the size occupied by each region of interest.

[0212] 3-2) If the difference in the size of the region of interest of two adjacent frames is less than a certain threshold, one or more metric measurements, such as Peak signal-to-noise ratio (PSNR) and Structural Similarity Index Measure (SSIM), may be compared for the region of interest within the two frames to determine a temporal frame restoration method. At this time, for the metric comparison, the region of interest with a larger size among the regions of interest within the two frames may be downsampled to have the same size as the region of interest with a smaller size, and then the metric comparison may be performed. Or, conversely, the region of interest with a smaller size may be upsampled to have the same size as the region of interest with a larger size, and then the metric comparison may be performed. In some embodiments, in order to compare the regions of interest with different sizes between frames, the upper left positions of the two regions may be moved to the same extent, and then the comparison may be performed for the region of the smaller region or the region of the larger region. When the comparison is performed for the region of the larger region, the region outside the smaller region may be filled with a median value or padded using a boundary value.

[0213] 4-1) If the difference between the regions of interest within two frames exceeds a certain threshold, temporal frame restoration may not be performed.

[0214] 4-2) If the difference in the region of interest within two frames exceeds a certain threshold, temporal frame restoration can be performed by generating n frames by copying adjacent frames among the two frames according to the temporal order of the final restored image.

[0215] Figures 14a and 14b illustrate detailed operations of a machine vision task performer according to an embodiment of the present disclosure.

[0216] A machine vision task performer (1000) according to one embodiment of the present disclosure may include a deep neural network performer (1011) and an area of ​​interest information-based correction unit (1012). In various embodiments, the order of detailed components of the machine vision task performer (1000) may be changed, and some components may be omitted.

[0217] In one embodiment, the deep neural network execution unit (1011) may utilize a deep neural network composed of one or more convolutional layers on which learning is performed.

[0218] The machine vision task performer (1000) can receive information about the type of machine vision task and / or the type of deep neural network from the encoding device (10a) and perform the machine vision task based on the received information.

[0219] In one embodiment, the region of interest information-based correction unit (1012) can correct the performance results of a deep neural network (for example, location information of a region of interest in an object tracking task) using the restored region of interest information. The region of interest information-based correction unit (1012) can be included in the deep neural network execution unit (1011) and operate.

[0220] In some embodiments, when the type of machine vision task to be performed involves processes other than independent processing between image frames, such as object tracking, the position of the region of interest can be corrected as follows.

[0221] In one embodiment, for frames within an image, if a tracked object disappears at a point where object number 5 is tracked in frames 1 to 10 and object number 5 is not tracked in the 11th frame, and if information about object number 5 in the 10th frame is in the restored region of interest information, and if region of interest information exists between the 10th frame and the 11th frame in that information, the object tracking result can be corrected using that region of interest information.

[0222] In another embodiment, if the difference between the representative brightness value of the current frame and the representative brightness value of the next frame for frames in the video exceeds a certain threshold, and if information about regions of interest between the two frames exists in the region of interest information, the object tracking result may be corrected using the region of interest information as in the process of the above embodiment in addition to the object tracking result between the two frames. In another example, if the object tracked in the current frame is not tracked in the next frame, correction may be performed as in the process of the above embodiment. In yet another example, if the object tracked in the current frame is tracked in the next frame, the object tracking result may be corrected by finding an object corresponding to the tracked object of the current frame in the next frame through a search process using the tracked object information (size and position) in the current frame as shown in FIG. 13 and adding it to the object tracking result.

[0223] <Description of Semantics and Syntax Used in Embodiments of the Present Disclosure>

[0224] Semantics related to temporal restoration information are described in terms of a series of frame groups, for example, GOP units.

[0225] Ⅰ. Related to temporal_restoration_data( )

[0226] (1) temporal_restoration_flag: Flag for whether to apply temporal resampling. If 1, temporal resampling is applied.

[0227] (2)Temporal_resampling_ratio_idx: The index of the list for temporal resampling ratios, and since it is indicated by temporal_restoration_flag for applying temporal resampling, it does not include the case where the input signal and the output signal are the same. For example, the sampling rate can be in the form of a ratio such as 1 / 2, 1 / 4, 1 / 8, etc., and can be the number of frames per second. The encoder and the decoder can agree on the list of sampling rates, and the encoder can transmit the index of the list. Depending on the embodiment, if the resampling rate is fixed for the transmitted application or the number of required frames is fixed, the information may be omitted.

[0228] (3)same_period_flag: A flag indicating whether periodic resampling or aperiodic resampling is used when applying temporal resampling. Depending on the embodiment, if the periodicity of temporal resampling is fixed, this information and related information may be omitted. If same_period_flag is 1, it means that temporal resampling is performed with the same period.

[0229] (4)delta_frame_idx[i]: Transmits the difference value from the previous frame index based on the frame number. Since it is the difference value from the previous index, it is sufficient to transmit a number that is one less than num_of_frames calculated by the frame rate. Here, i represents the frame index.

[0230] (5)RoI_Processing_flag: 1-bit flag indicating whether the current sequence has undergone region-of-interest-based processing.

[0231] (6)num_of_GOPs: Information indicating the number of GOPs in a sequence

[0232] (7)num_of_frames: Information indicating the number of frames in one GOP

[0233] (8)num_of_RoIs: Information indicating the number of regions of interest in one frame

[0234] (9)upsamp_ratio_RoIs: Upsampling rate information for the region of interest, and a processed image based on the region of interest restored with the information can be obtained.

[0235] (10)upsamp_ratio_nonRoIs: Upsampling rate information for non-interest regions, and the restored non-interest region image can be obtained using the information.

[0236] (11)RoI_exist_region_LT(0), RoI_exist_region_LT(1): The upper left coordinate of the region where the region of interest exists in the picture.

[0237] (12)RoI_exist_region_RB(0), RoI_exist_region_RB(1): Coordinates of the lower right corner of the region where the region of interest exists in the picture

[0238] (13)frame_RoI_Information_flag[p]: Frame information for which region of interest information encoding is performed. If 1, encoding of region of interest information within the frame is performed. If 0, the frame is skipped. At this time, encoding of region of interest information between the two most adjacent frames with the flag set to 1 is performed.

[0239] (14)only_specific_RoI_flag and specific_RoI_flag[j]: If only_specific_RoI_flag is 1, information of all regions of interest within the frame where region of interest information encoding is performed is encoded. If only_specific_RoI_flag is 0, specific_RoI_flag[j] is parsed and information encoding is performed only for regions of interest for which the flag is 1. In this case, j represents the region of interest index.

[0240] (15) LinearMotion_flag[j]: If the flag is 1, the jth region of interest within the two frames where region of interest information encoding is performed has linear motion, and only information on the motion position and differential value is decoded. If the flag is 0, the region of interest has, for example, affine motion in addition to linear motion, and in addition to information on the motion position and differential value, affine parameters are decoded.

[0241] (16)Affine_param_a[j], Affine_param_b[j], Affine_param_c[j], Affine_param_d[j]: If LinearMotion_flag[j] is 0, it is information that can be decoded, and motion information can be obtained through the affine formula (Mathematical Formula 1) using the four affine parameters.

[0242] (17)Comp_param_flag[j]: If the flag is 1, the jth region of interest within the two frames where region of interest information encoding is performed has a compensation parameter. The compensation parameter can be explicitly signaled / parsed and derived from surrounding restored pixel information in the decoder. The compensation parameter can be applied as in the example of Equation 2.

[0243]

[0244] An example of the syntax of the above temporal restoration data is as shown in Table 1 below.

[0245] Descriptortemporal_restoration_data( ){temporal_restoration_flague(1)if( temporal_restoration_flag ) {temporal_resampling_ratio_idxue(v)same_period_flague(1)if( !same_period_flag ) {for( i=0; i <num_of_frames; i++ ) {delta_frame_idx[i]ue(v)}}}}

[0246] An example of the syntax of the above area of ​​interest information is shown in Table 2 below.

[0247] DescriptorRoI_Processing_flagu(1)if( RoI_Processing_flag ) {num_of_GOPsue(v)RoI_exist_region_LT(0)ue(v)RoI_exist_region_LT(1)ue(v)RoI_exist_region_RB(0)ue(v)RoI_exist_region_RB(1)ue(v)for ( i=0; i<num_of_GOPs; i++ ) {upsamp_ratio_RoIsue(v)upsamp_ratio_nonRoIsue(v)num_of_framesue(v)num_of_RoIsue(v)only_specific_RoI_flague(1)for ( p=0; p<num_of_frames; p++ ) {frame_RoI_Information_flag[p]ue(1)}if ( only_specific_RoI_flag ) {for ( p=0; p<num_of_frames; p++ ) {for ( j=0; j<num_of_RoIs; j++ ) {if ( frame_RoI_Information_flag[p] ) {specific_RoI_flag[j]ue(1)}}}}int a = 0;for ( p=0; p<num_of_frames; p++ ) {if ( frame_RoI_Information_flag[p]) {if ( !only_specific_RoI_flag ) {for ( j=0; j<num_of_RoIs; j++ ) {if ( a == 0 ) {pos_RoI(i)(p)(j)[0]ue(v)pos_RoI(i)(p)(j)[1]ue(v)a++;} else {pos_diff_codingLinearMotion_flague(1)Comp_param_flague(1)if ( !LinearMotion_flag ) {Affine_param_aue(v)Affine_param_bue(v)Affine_param_cue(v)Affine_param_due(v)}}}} else {for ( j=0; j<num_of_RoIs;j++ ) {if ( specific_RoI_flag[i] ) {if ( a == 0 ) {pos_RoI(i)(p)(j)[0]ue(v)pos_RoI(i)(p)(j)[1]ue(v)a++;} else {pos_diff_codingLinearMotion_flague(1)Comp_param_flague(1)if ( !LinearMotion_flag ) {Affine_param_aue(v)Affine_param_bue(v)Affine_param_cue(v)Affine_param_due(v)}}}}}}}};

[0248] In various embodiments, restoration information for positional differences can be defined as follows. The semantics related to restoration information for positional differences below are described based on a series of frame group units, e.g., GOP units. n represents a GOP index, p represents a frame index, and m represents a region of interest index.

[0249] Ⅱ. pos_diff_coding( ) related

[0250] (1)pos_RoI(n)(p)(m)[0], pos_RoI(n)(p)(m)[1]: Location information of regions of interest existing in the first frame of each series of frame groups (GOPs).

[0251] - If a non-zero value is parsed among RoI_exist_region_LT(0), RoI_exist_region_LT(1), for example, RoI_exist_region_LT(0) + pos_RoI(n)(m)[0], RoI_exist_region_LT(1) + pos_RoI(n)(m)[1] may be the final restored position of the m-th region of interest.

[0252] - For the position difference values ​​obtained through pos_diff_coding, the final restored position of the region of interest can be obtained in the same way.

[0253] (2)abs_pos_diff_greater0_flag_RoI(n)(p)(m)[0], abs_pos_diff_greater0_flag_RoI(n)(p)(m)[1]: 1-bit flag indicating whether the absolute value of the x-axis and y-axis position information difference value for the same region of interest in the 0th frame is 0.

[0254] (3)abs_pos_diff_greater1_flag_RoI(n)(p)(m)[0], abs_pos_diff_greater1_flag_RoI(n)(p)(m)[1]: 1-bit flag indicating whether the absolute value of the x-axis and y-axis position information difference value for the same region of interest in the 0th frame is 1.

[0255] (4)abs_pos_diff_minus2_RoI(i)(p)(j)[0], abs_pos_diff_minus2_RoI(i)(p)(j)[0]: Information indicating the corresponding value of -2 of the absolute value of the difference value of the x-axis and y-axis position information for the same region of interest in the 0th frame.

[0256] (5)pos_diff_sign_flag(i)(p)(j)[0], pos_diff_sign_flag(i)(p)(j)[1]: 1-bit flag indicating the x-axis and y-axis sign values ​​for the same region of interest in the 0th frame, 1 for + sign, 0 for - sign

[0257]

[0258] Table 3 below is an example syntax of restoration information for position differences according to various embodiments.

[0259] Descriptorpos_diff_coding( ){abs_pos_diff_greater0_flag_RoI(i)(p)(j)[0]ue(1)abs_pos_diff_greater0_flag_RoI(i)(p)(j)[1]ue(1)if( abs_pos_diff_greater0_flag_RoI(i)(p)(j)[0] )abs_pos_diff_greater1_flag_RoI(i)(p)(j)[0]ue(1)if( abs_pos_diff_greater0_flag_RoI(i)(p)(j)[1] )abs_pos_diff_greater1_flag_RoI(i)(p)(j)[1]ue(1)if( abs_pos_diff_greater0_flag_RoI(i)(p)(j)[0] ) {if( abs_pos_diff_greater1_flag_RoI(i)(p)(j)[0] )abs_pos_diff_minus2_RoI(i)(p)(j)[0]ue(v)pos_diff_sign_flag(i)(p)(j)[0]ue(1)}if( abs_pos_diff_greater0_flag_RoI(i)(p)(j)[1] ) {if( abs_pos_diff_greater1_flag_RoI(i)(p)(j)[1] )abs_pos_diff_minus2_RoI(i)(p)(j)[1]ue(v)pos_diff_sign_flag(i)(p)(j)[1]ue(1)}}

[0260] The examples of the present disclosure presented in this specification and drawings are intended solely to facilitate the technical content of the present disclosure and to aid understanding thereof, and are not intended to limit the scope of the present disclosure. It will be apparent to those skilled in the art that other variations are possible in addition to the examples described above.

[0261] The claims set forth in this specification may be combined in various ways. For example, the technical features of the method claims of this specification may be combined and implemented as a device, and the technical features of the device claims of this specification may be combined and implemented as a method. Furthermore, the technical features of the method claims and the technical features of the device claims of this specification may be combined and implemented as a device, and the technical features of the method claims and the technical features of the device claims of this specification may be combined and implemented as a method.

Claims

1. In a VCM (video coding for machines) encoding device, A temporal resampling performer that changes the frame rate of part of the input image; A region-of-interest-based processor that extracts one or more regions of interest included in each frame for multiple frames of the input image and generates a processed image based on the one or more regions of interest; and An internal encoding performer that encodes information about one or more regions of interest and the input image to generate a bitstream, A VCM encoding device, wherein information about said one or more regions of interest includes a temporal sampling rate.

2. In paragraph 1, The above input image is divided into an image processed based on one or more regions of interest and an image of a non-region of interest corresponding to other regions, The above internal encoding performer performs encoding on the processed image and the non-region of interest image based on the one or more regions of interest, respectively. A VCM encoding device in which the internal encoding performer muxes a bitstream encoding information about one or more regions of interest, a bitstream encoding an image processed based on the one or more regions of interest, and a bitstream encoding an image of a non-region of interest to output one bitstream.

3. In paragraph 1, The above temporal resampling performer performs temporal resampling at the same frame rate for all frames of the input image, or A VCM encoding device that performs temporal resampling at different frame rates in units of a series of frame groups within the above input image.

4. In paragraph 3, A VCM encoding device in which the internal encoding performer encodes region of interest information only for regions of interest of some frames among frames including regions of interest when temporal sampling is performed.

5. In paragraph 4, The above internal encoding performer encodes information about the region of interest for each frame period m (m is an integer less than or equal to n) for the region of interest for which temporal resampling is performed at a sampling rate n (n is an integer greater than or equal to 1) when temporal sampling is performed periodically, The above internal encoding performer is a VCM encoding device that encodes information about a region of interest at frame intervals k / p (p is an integer greater than or equal to 1 and less than k) when the interval between any two adjacent frames is k, when aperiodic temporal resampling is performed.

6. In paragraph 3, The above internal encoding performer encodes only information about one or more specific regions of interest in the input image when temporal resampling is not performed on the input image, A VCM encoding device, wherein the one or more specific regions of interest correspond to a case where the difference in representative brightness values ​​between frames included in the input image or the difference in representative brightness values ​​of regions of interest included in the frames is greater than a preset threshold value.

7. In paragraph 1, The internal encoding performer encodes information about regions of interest that correspond to each other among a plurality of frames of the input image based on the corresponding relationship, A VCM encoding device, wherein the region of interest information includes at least one of a position between two regions of interest, a motion parameter, and a compensation parameter.

8. In paragraph 7, A VCM encoding device in which the corresponding region of interest is extracted using a deep neural network trained for the purpose of object detection or object tracking.

9. In paragraph 7, The above motion parameters include one or more parameters representing affine motion including scale, sheer, and rotation, The above affine motion parameters are defined by the mathematical formula below: Here, a, b, c, d represent affine parameters, Trans_x and Trans_y represent transformation motion information for the x-axis and y-axis, x, y represent the position of the pixel value within the current region of interest, and x', y' represent the position of the pixel value to which the affine motion is applied, VCM encoding device.

10. In paragraph 7, The above compensation parameter includes one or more parameters indicating a case where the difference between the representative brightness values ​​of two frames included in the input image or the difference between the representative brightness values ​​of the corresponding region of interest is greater than a preset threshold value, The above compensation parameters are defined by the following mathematical equation: Here, RoI i (past) represents the ith region of interest of the temporally previous frame among the two frames, and RoI i (future) is a VCM encoding device that represents the ith region of interest of the temporally next frame.

11. In paragraph 1, Further comprising a spatial resampling performer that changes the spatial resolution of each frame or a series of frames of the input image, The above spatial resampling performer or region-of-interest-based processor changes the resolution only for one or more regions of interest of the input image, or changes the resolution to different resolutions for images processed based on the one or more regions of interest and images of non-regions of interest, A VCM encoding device, wherein information about one or more regions of interest includes changed resolution information.

12. In a VCM (video coding for machines) decoding device, An internal decryption performer that decodes the bitstream to generate a restored image; A region-of-interest-based restorer that reconstructs a restored image based on a region of interest in the restored image using region-of-interest-based processing information parsed from the bitstream; A VCM decoding device comprising a temporal restoration performer that performs temporal restoration of the restored image using temporal resampling information parsed from the bitstream.

13. In paragraph 12, A VCM decoding device, wherein the above region-of-interest-based processing information includes at least one of region-of-interest ID information, region-of-interest size information, region-of-interest packing information, corresponding region-of-interest information, position information between two regions-of-interest among corresponding regions-of-interest, motion parameters of the corresponding region-of-interest, and compensation parameters of the corresponding region-of-interest.

14. In paragraph 12, The above internal decoding performer demuxes the bitstream to decode an image processed based on a region of interest, information about the region of interest, and an image of a non-region of interest, respectively. Based on the information about the region of interest, the image processed based on the region of interest is restored, A VCM decoding device, wherein the information about the region of interest includes at least one of position information of the region of interest, linear motion parameters, affine motion parameters, and compensation parameters.

15. In paragraph 14, The above affine motion parameters represent affine motions including scale, sheer, and rotation, The above affine motion parameters are defined by the mathematical formula below: Here, a, b, c, d represent affine parameters, Trans_x and Trans_y represent transformation motion information for the x-axis and y-axis, x, y represent the position of the pixel value within the current region of interest, and x', y' represent the position of the pixel value to which the affine motion is applied, VCM decoding device.

16. In paragraph 12, A VCM decoding device, wherein the temporal restoration performer first performs temporal restoration after performing decoding on the bitstream, and then generates a restored image based on one or more regions of interest.

17. In paragraph 12, The temporal restoration performer corrects the position of the region of interest within the restored intermediate frames according to one or more temporal frame restoration methods based on information about the decoded region of interest, A VCM decoding device, wherein the one or more temporal frame restoration methods are selected based on whether the frames include regions of interest of the same size, whether the difference in the sizes of the regions of interest of two adjacent frames is less than or equal to a preset threshold, or whether the difference in the sizes of the regions of interest of two adjacent frames exceeds a preset threshold.

18. In paragraph 12, Including a machine vision task performer, The above machine vision task performer is a VCM decoding device that performs a machine vision task using a deep neural network on which learning has been performed, and corrects the machine vision task performance result based on the restored region of interest-based processing information.

19. In paragraph 18, The above machine vision task performer tracks an object in a restored image using a deep neural network trained for the purpose of object tracking, A VCM decoding device that corrects the position of a region of interest based on restored region-of-interest-based processing information for a region of interest including a tracked object.

20. A non-volatile computer-readable storage medium that records commands, The above instructions, when executed by one or more processors, cause the one or more processors to: Step of changing the frame rate of part of the input video; A step of extracting one or more regions of interest included in each frame for a plurality of frames of the input image and generating a processed image based on the one or more regions of interest; and Performing a step of generating a bitstream by encoding information about one or more regions of interest and the input image, A non-transitory computer-readable storage medium comprising information about one or more regions of interest, the information including a temporal sampling rate.

Images