Encoding and decoding method through feature-based adaptive mapping of region of interest in vcm

Abstract

One example of the present disclosure presents an encoding apparatus for video coding for machines (VCM). The encoding apparatus may include: a region-of-interest extractor for extracting at least one region of interest for each frame of an input image; a region-of-interest feature-based processor for deriving a mapping relationship for the one or more regions of interest and performing mapping on the basis of features of the one or more regions of interest; and an internal encoder for generating a bitstream by encoding the input image by using information on the one or more regions of interest.

Description

Encoding and decoding method using feature-based adaptive mapping of regions of interest in VCM

[0001] The present disclosure relates to a method for encoding and decoding through feature-based adaptive mapping of a region of interest in VCM.

[0002] With the continuous development of the information and communication industry, broadcasting services with HD (High Definition) resolution have spread worldwide.

[0003] Through this proliferation, many users have become accustomed to high-resolution and high-quality images and / or videos, and the demand for higher-resolution and high-quality images / videos, such as 4K or 8K or higher UHD (Ultra High Definition) images / videos, has increased in various fields.

[0004] The technology for coding this UHD video data was completed in 2013 through HEVC (High Efficiency Video Coding), a standard technology.

[0005] HEVC is a next-generation video compression technology with a higher compression ratio and lower complexity than the previous H.264 / AVC technology, and is a key technology for effectively compressing the massive data of HD and UHD video.

[0006] HEVC performs block-by-block encoding, like previous compression standards.

[0007] However, unlike H.264 / AVC, there is only one profile. The core encoding technologies included in HEVC's sole profile are divided into eight areas: hierarchical encoding structure technology, transform technology, quantization technology, intra-frame prediction encoding technology, inter-frame motion prediction technology, entropy encoding technology, loop filter technology, and other technologies.

[0008] Since the establishment of the HEVC video codec in 2013, the Versatile Video Coding (VVC) standard, a next-generation video codec that aims to improve performance by more than twice that of HEVC, has been developed to address the expansion of realistic video and virtual reality services utilizing 4K and 8K video images. VVC is called H.266.

[0009] H.266 (VVC) was developed with the goal of being more than twice as efficient as the previous generation codec, H.265 (HEVC). VVC was initially developed with resolutions over 4K in mind, but it was also developed for ultra-high-resolution video processing at a whopping 16K level to support 360-degree videos due to the expansion of the VR market. In addition, as the HDR market is expanding due to the development of display technology, it supports 16-bit color depth as well as 10-bit color depth to respond to this, and supports brightness expressions of 1000 nits, 4000 nits, and 10000 nits. In addition, since it is being developed with the VR market and 360-degree video market in mind, it supports partial frame rates in the range of 0 to 120 FPS.

[0010] Advances in Artificial Intelligence

[0011] Artificial intelligence (AI) is also steadily developing. AI refers to the artificial imitation of human intelligence, including the ability to recognize, classify, infer, predict, and control / decision-making.

[0012] With the advancement of artificial intelligence technology and the increase in Internet of Things (IoT) devices, machine-to-machine traffic is expected to explode, and machine-dependent image analysis is expected to become widely used.

[0013] However, as the amount of images to be analyzed by machines is expected to increase exponentially, issues with server load and power consumption are expected to arise.

[0014] Accordingly, the present disclosure aims to provide a method of encoding and decoding through feature-based adaptive mapping of a region of interest in VCM in order to enable effective machine-based image analysis.

[0015] To achieve the above-mentioned purpose, according to one disclosure of the present specification, a method of encoding and decoding through feature-based adaptive mapping of a region of interest in VCM is proposed.

[0016] A VCM encoding device may include a region of interest extractor that extracts one or more regions of interest for each frame of an input image; a region of interest feature-based processor that derives a mapping relationship for the one or more regions of interest and performs the mapping based on features of the one or more regions of interest; and an internal encoder that encodes the input image using information about the one or more regions of interest to generate a bitstream.

[0017] A VCM decoding device according to one disclosure of the present specification is provided. The VCM decoding device may include an internal decoder that decodes a bitstream to generate a restored image and extracts information on a region of interest in the restored image; and a region of interest restorer that performs restoration for one or more regions of interest included in the restored image based on the information on the region of interest.

[0018] A non-volatile computer-readable storage medium having recorded thereon instructions according to one disclosure of the present specification, wherein the instructions, when executed by one or more processors, may cause the one or more processors to: extract one or more regions of interest for each frame of an input image; derive a mapping relationship for the one or more regions of interest and perform mapping based on characteristics of the one or more regions of interest; and encode the input image using information about the one or more regions of interest to generate a bitstream.

[0019] According to the present disclosure, image analysis by a machine can be effectively performed.

[0020] Figure 1 schematically illustrates an example of a video / image coding system.

[0021] Figure 2 is a drawing schematically illustrating the configuration of a video / image encoding device.

[0022] Figure 3 is a drawing schematically illustrating the configuration of a video / image decoding device.

[0023] Figures 4a to 4d are exemplary diagrams showing a VCM encoder and a VCM decoder.

[0024] FIG. 5 is a block diagram of an encoding device according to an embodiment of the present disclosure.

[0025] FIG. 6 illustrates a process of extracting a region of interest and processing a region of interest feature based on an encoding device according to an embodiment of the present disclosure.

[0026] FIG. 7 is an example of (a) object extraction, (b) region of interest selection, and (c) region of interest classification according to one embodiment of the present disclosure.

[0027] FIG. 8 is an example of a histogram for chrominance components of a region of interest according to one embodiment of the present disclosure.

[0028] FIG. 9 is an example of mapping information according to one embodiment of the present disclosure.

[0029] FIG. 10 is an example of a histogram before mapping (a) and histograms after mapping (b, c, d) according to one embodiment of the present disclosure.

[0030] FIG. 11 is a block diagram of a decoding device according to an embodiment of the present disclosure.

[0031] FIG. 12 illustrates a process of restoring a region of interest of a decoding device according to an embodiment of the present disclosure.

[0032] Specific structural or step-by-step descriptions of embodiments according to the concept of the present disclosure disclosed in this specification or application are merely illustrative for the purpose of explaining embodiments according to the concept of the present disclosure, and embodiments according to the concept of the present disclosure may be implemented in various forms, and embodiments according to the concept of the present disclosure may be implemented in various forms and should not be construed as being limited to the embodiments described in this specification or application.

[0033] Embodiments according to the concept of the present disclosure may have various modifications and take various forms. Therefore, specific embodiments are illustrated in the drawings and described in detail in this specification or application. However, this is not intended to limit embodiments according to the concept of the present disclosure to specific disclosed forms, and it should be understood that all modifications, equivalents, and alternatives included within the spirit and technical scope of the present disclosure are included.

[0034] While terms such as "first" and / or "second" may be used to describe various components, these components should not be limited by these terms. These terms are only intended to distinguish one component from another; for example, without departing from the scope of the present disclosure, a first component may be referred to as a "second component," and similarly, a second component may also be referred to as a "first component."

[0035] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. Conversely, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Other expressions that describe the relationship between components, such as "between" and "directly between" or "adjacent to" and "directly adjacent to", should be interpreted similarly.

[0036] The terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit the present disclosure. The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, it should be understood that the terms "comprises" or "has" indicate the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0037] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0038] Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0039] In describing the embodiments, description of technical contents that are well known in the technical field to which the present disclosure belongs and are not directly related to the present disclosure will be omitted.

[0040] This is to convey the gist of the present disclosure more clearly without obscuring it by omitting unnecessary explanations.

[0041] This document relates to video / image coding. For example, the method / embodiment disclosed in this document may be related to the Versatile Video Coding (VVC) standard (ITU-T Rec. H.266), the next-generation video / image coding standard after VVC, or other video coding-related standards (e.g., the High Efficiency Video Coding (HEVC) standard (ITU-T Rec. H.265), the essential video coding (EVC) standard, the AVS2 standard, etc.).

[0042] This document presents various embodiments of video / image coding, and unless otherwise stated, the embodiments may be performed in combination with each other.

[0043] In this document, "video" can refer to a series of images over time. "Picture" generally refers to a unit representing a single image from a specific time period, and "slice" / "tile" are units that constitute part of a picture in coding.

[0044] A slice / tile can contain one or more coding tree units (CTUs). A picture can consist of one or more slices / tiles. A picture can consist of one or more tile groups. A tile group can contain one or more tiles.

[0045] A pixel or pel can mean the smallest unit that constitutes a picture (or image). Additionally, the term "sample" can be used as a counterpart to a pixel. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luma component, or only the pixel / pixel value of the chroma component. Alternatively, a sample can mean a pixel value in the spatial domain, or when such a pixel value is converted to the frequency domain, it can mean a transform coefficient in the frequency domain.

[0046] A unit may represent a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to that region.

[0047] A unit can contain one luma block and two chroma (e.g., cb, cr) blocks. The term "unit" may sometimes be used interchangeably with the terms "block" or "area." In general, an MxN block can contain a set (or array) of samples (or array of samples) or transform coefficients, each consisting of M columns and N rows.

[0048] Figure 1 schematically illustrates an example of a video / image coding system.

[0049] Referring to FIG. 1, a video / image coding system may include a source device and a receiving device. The source device may transmit encoded video / image information or data to the receiving device via a digital storage medium or a network in the form of a file or streaming.

[0050] The source device may include a video source, an encoding device, and a transmission unit. The receiving device may include a reception unit, a decoding device, and a renderer.

[0051] The encoding device may be referred to as a video / image encoding device, and the decoding device may be referred to as a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display unit, and the display unit may be configured as a separate device or an external component.

[0052] A video source may obtain video / images through a process of capturing, synthesizing, or generating video / images. The video source may include a video / image capture device and / or a video / image generation device. A video / image capture device may include, for example, one or more cameras, a video / image archive containing previously captured video / images, etc. A video / image generation device may include, for example, a computer, a tablet, a smartphone, etc., and may (electronically) generate video / images. For example, a virtual video / image may be generated through a computer, etc., in which case the video / image capture process may be replaced by a process of generating related data.

[0053] An encoding device can encode input video / images. The encoding device can perform a series of procedures, such as prediction, transformation, and quantization, to improve compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0054] The transmission unit can transmit encoded video / image information or data output in bitstream form to the receiving unit of the receiving device via a digital storage medium or network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit can include an element for generating a media file using a predetermined file format and an element for transmission via a broadcasting / communication network.

[0055] The receiving unit can receive / extract the bitstream and transmit it to a decoding device.

[0056] The decoding device can decode the video / image by performing a series of procedures such as inverse quantization, inverse transformation, and prediction corresponding to the operation of the encoding device.

[0057] The renderer can render decoded video / images. The rendered video / images can be displayed through the display unit.

[0058] Figure 2 is a drawing schematically illustrating the configuration of a video / image encoding device.

[0059] The term “video encoding device” hereinafter may include a video encoding device.

[0060] Referring to FIG. 2, the encoding device (10a) may be configured to include an image partitioner (10a-10), a prediction unit (predictor) (10a-20), a residual processor (residual processor) (10a-30), an entropy encoder (entropy encoder) (10a-40), an adder (adder) (10a-50), a filter (filter) (10a-60), and a memory (10a-70). The prediction unit (10a-20) may include an inter prediction unit (10a-21) and an intra prediction unit (10a-22). The residual processing unit (10a-30) may include a transformer (10a-32), a quantizer (10a-33), a dequantizer (10a-34), and an inverse transformer (10a-35). The residual processing unit (10a-30) may further include a subtractor (10a-31). The addition unit (10a-50) may be called a reconstructor or a reconstructed block generator. The above-described image segmentation unit (10a-10), prediction unit (10a-20), residual processing unit (10a-30), entropy encoding unit (10a-40), addition unit (10a-50), and filtering unit (10a-60) may be configured by one or more hardware components (e.g., encoder chipset or processor) according to an embodiment. In addition, the memory (10a-70) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (10a-70) as an internal / external component.

[0061] The image segmentation unit (10a-10) can segment an input image (or picture, frame) input to the encoding device (10a) into one or more processing units.

[0062] For example, the processing unit may be called a coding unit (CU). In this case, the coding unit may be recursively split from a coding tree unit (CTU) or a largest coding unit (LCU) according to a Quad-tree binary-tree ternary-tree (QTBTTT) structure. For example, one coding unit may be split into multiple coding units of deeper depth based on a quad-tree structure, a binary-tree structure, and / or a ternary structure. In this case, for example, the quad-tree structure may be applied first, and the binary-tree structure and / or the ternary structure may be applied later. Alternatively, the binary-tree structure may be applied first. The coding procedure according to the present document may be performed based on the final coding unit that is no longer split. In this case, based on coding efficiency according to image characteristics, etc., the maximum coding unit can be used as the final coding unit, or, if necessary, the coding unit can be recursively divided into coding units of lower depths, and the coding unit of the optimal size can be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration described below. As another example, the processing unit may further include a prediction unit (PU) or a transformation unit (TU). In this case, the prediction unit and the transformation unit may each be divided or partitioned from the final coding unit described above. The prediction unit may be a unit of sample prediction, and the transformation unit may be a unit for deriving a transform coefficient and / or a unit for deriving a residual signal from a transform coefficient.

[0063] The term "unit" may be used interchangeably with terms such as "block" or "area" depending on the case. In general, an MxN block can represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luminance component, or only the pixel / pixel value of the chroma component. A sample can be used as a term corresponding to a pixel or pel in a picture (or image).

[0064] The subtraction unit (10a-31) can subtract the prediction signal (predicted block, prediction samples, or prediction sample array) output from the prediction unit (10a-20) from the input image signal (original block, original samples, or original sample array) to generate a residual signal (residual block, residual samples, or residual sample array), and the generated residual signal is transmitted to the conversion unit (10a-32). The prediction unit (10a-20) can perform prediction on a block to be processed (hereinafter, referred to as a current block) and generate a predicted block including prediction samples for the current block.

[0065] The prediction unit (10a-20) can determine whether intra-prediction or inter-prediction is applied to the current block or CU unit. As described later in the description of each prediction mode, the prediction unit can generate various prediction-related information, such as prediction mode information, and transmit it to the entropy encoding unit (10a-40). The prediction-related information can be encoded by the entropy encoding unit (10a-40) and output in the form of a bitstream.

[0066] The intra prediction unit (10a-22) can predict the current block by referring to samples within the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from it, depending on the prediction mode.

[0067] In intra prediction, prediction modes can include multiple non-directional modes and multiple directional modes. Non-directional modes can include, for example, DC modes and planar modes. Directional modes can include, for example, 33 directional prediction modes or 65 directional prediction modes, depending on the granularity of the prediction direction.

[0068] However, this is only an example; depending on the settings, a greater or lesser number of directional prediction modes may be used. The intra prediction unit (10a-22) may also determine the prediction mode to be applied to the current block by utilizing the prediction mode applied to the surrounding blocks.

[0069] The inter prediction unit (10a-21) can derive a predicted block for a current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on an inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring block can include a spatial neighboring block existing in the current picture and a temporal neighboring block existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The above temporal neighboring blocks may be called collocated reference blocks, collocated CUs (colCUs), etc., and a reference picture including the temporal neighboring blocks may be called a collocated picture (colPic). For example, the inter prediction unit (10a-21) may construct a motion information candidate list based on the neighboring blocks, and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction may be performed based on various prediction modes, and for example, in the case of skip mode and merge mode, the inter prediction unit (10a-21) may use the motion information of the neighboring blocks as the motion information of the current block. In the case of skip mode, unlike the merge mode, a residual signal may not be transmitted.In the motion vector prediction (MVP) mode, the motion vector of the surrounding blocks is used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0070] The prediction unit (10a-20) can generate a prediction signal based on various prediction methods described below. For example, the prediction unit can apply intra prediction or inter prediction to predict a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). In addition, the prediction unit can perform intra block copy (IBC) to predict a block. The intra block copy can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in this document.

[0071] The prediction signal generated through the inter prediction unit (10a-21) and / or the intra prediction unit (10a-22) can be used to generate a reconstructed signal or a residual signal. The transform unit (10a-32) can apply a transform technique to the residual signal to generate transform coefficients. For example, the transform technique can include a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), a Graph-Based Transform (GBT), or a Conditionally Non-linear Transform (CNT). Here, GBT refers to a transform obtained from a graph when the relationship information between pixels is expressed as a graph. CNT refers to a transform obtained based on generating a prediction signal using all previously reconstructed pixels. In addition, the transform process can be applied to a pixel block having a square equal size, or can be applied to a block of a non-square variable size.

[0072] The quantization unit (10a-33) quantizes the transform coefficients and transmits them to the entropy encoding unit (10a-40), and the entropy encoding unit (10a-40) can encode the quantized signal (information about the quantized transform coefficients) and output it as a bitstream. The information about the quantized transform coefficients can be called residual information.

[0073] The quantization unit (10a-33) can rearrange the quantized transform coefficients in the form of a block into a one-dimensional vector based on the coefficient scan order, and can also generate information about the quantized transform coefficients based on the quantized transform coefficients in the form of the one-dimensional vector. The entropy encoding unit (10a-40) can perform various encoding methods, such as, for example, exponential Golomb, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), etc.

[0074] The entropy encoding unit (10a-40) may encode, together or separately, information necessary for video / image restoration (e.g., values ​​of syntax elements, etc.) in addition to quantized transform coefficients. The encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream in the form of a network abstraction layer (NAL) unit. The video / image information may further include information regarding various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. The signaling / transmitted information and / or syntax elements described later in this document may be encoded through the above-described encoding procedure and included in the bitstream. The bitstream may be transmitted through a network or may be stored in a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The signal output from the entropy encoding unit (10a-40) may be configured as an internal / external element of the encoding device (10a) by a transmitting unit (not shown) that transmits and / or a storing unit (not shown) that stores the signal, or the transmitting unit may be included in the entropy encoding unit (10a-40).

[0075] The quantized transform coefficients output from the quantization unit (10a-33) can be used to generate a prediction signal. For example, by applying inverse quantization and inverse transformation to the quantized transform coefficients through the inverse quantization unit (10a-34) and the inverse transform unit (10a-35), a residual signal (residual block or residual samples) can be reconstructed. The addition unit (10a-50) can add the reconstructed residual signal to the prediction signal output from the prediction unit (10a-20), thereby generating a reconstructed signal (reconstructed picture, reconstructed block, reconstructed samples, or reconstructed sample array). When there is no residual for the target block to be processed, such as when skip mode is applied, the predicted block can be used as the reconstructed block. The generated restoration signal can be used for intra prediction of the next processing target block within the current picture, and can also be used for inter prediction of the next picture after filtering as described below.

[0076] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture encoding and / or restoration process.

[0077] The filtering unit (10a-60) can improve subjective / objective picture quality by applying filtering to the restoration signal. For example, the filtering unit (10a-60) can apply various filtering methods to the restoration picture to generate a modified restoration picture, and store the modified restoration picture in the memory (10a-70), specifically, in the DPB of the memory (10a-70). The various filtering methods may include, for example, deblocking filtering, sample adaptive offset (SAO), an adaptive loop filter, a bilateral filter, etc. The filtering unit (10a-60) can generate various information regarding filtering as described below in the description of each filtering method and transmit the information to the entropy encoding unit (10a-90). The information regarding filtering may be encoded by the entropy encoding unit (10a-90) and output in the form of a bitstream.

[0078] The modified restored picture transmitted to the memory (10a-70) can be used as a reference picture in the inter prediction unit (10a-80). Through this, when inter prediction is applied, the encoding device can avoid prediction mismatch between the encoding device (10a) and the decoding device, and can also improve encoding efficiency.

[0079] The DPB of the memory (10a-70) can store the modified restored picture to be used as a reference picture in the inter prediction unit (10a-21). The memory (10a-70) can store motion information of a block from which motion information in the current picture is derived (or encoded) and / or motion information of blocks in a picture that has already been restored. The stored motion information can be transmitted to the inter prediction unit (10a-21) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (10a-70) can store restored samples of restored blocks in the current picture and transmit them to the intra prediction unit (10a-22).

[0080] Figure 3 is a drawing schematically illustrating the configuration of a video / image decoding device.

[0081] Referring to FIG. 3, the decoding device (10b) may be configured to include an entropy decoder (10b-10), a residual processor (10b-20), a predictor (10b-30), an adder (10b-40), a filter (10b-50), and a memory (10b-60). The predictor (10b-30) may include an inter-prediction unit (10b-31) and an intra-prediction unit (10b-32). The residual processor (10b-20) may include a dequantizer (10b-21) and an inverse transformer (10b-21). The entropy decoding unit (10b-10), residual processing unit (10b-20), prediction unit (10b-30), addition unit (10b-40), and filtering unit (10b-50) described above may be configured by a single hardware component (e.g., decoder chipset or processor) according to an embodiment. In addition, the memory (10b-60) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (10b-60) as an internal / external component.

[0082] When a bitstream including video / image information is input, the decoding device (10b) can restore the image corresponding to the process in which the video / image information is processed in the encoding device of FIG. 2. For example, the decoding device (10b) can derive units / blocks based on block division-related information obtained from the bitstream. The decoding device (10b) can perform decoding using a processing unit applied in the encoding device. Therefore, the processing unit of decoding may be, for example, a coding unit, and the coding unit may be divided from a coding tree unit or a maximum coding unit according to a quad tree structure, a binary tree structure, and / or a ternary tree structure. One or more transform units may be derived from the coding unit. Then, the restored image signal decoded and output by the decoding device (10b) can be reproduced through a reproduction device.

[0083] The decoding device (10b) can receive a signal output from the encoding device of FIG. 2 in the form of a bitstream, and the received signal can be decoded through the entropy decoding unit (10b-10). For example, the entropy decoding unit (10b-10) can parse the bitstream to derive information (e.g., video / image information) necessary for image restoration (or picture restoration). The video / image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information.

[0084] The decoding device can further decode the picture based on information about the parameter set and / or the general restriction information. The signaling / received information and / or syntax elements described later in this document can be decoded and obtained from the bitstream through the decoding procedure. For example, the entropy decoding unit (10b-10) can decode information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output values ​​of syntax elements required for image restoration and quantized values ​​of transform coefficients for the residual.

[0085] In more detail, the CABAC entropy decoding method receives a bin corresponding to each syntax element in a bitstream, determines a context model using information of the syntax element to be decoded and decoding information of surrounding and decoding target blocks or information of symbols / bins decoded in a previous step, and predicts the occurrence probability of the bin according to the determined context model to perform arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element. At this time, the CABAC entropy decoding method can update the context model using information of the decoded symbol / bin for the context model of the next symbol / bin after determining the context model. Among the information decoded by the entropy decoding unit (10b-10), information regarding prediction is provided to the prediction unit (10b-30), and information regarding the residual on which entropy decoding has been performed by the entropy decoding unit (10b-10), i.e., quantized transform coefficients and related parameter information, can be input to the dequantization unit (10b-21).

[0086] In addition, information regarding filtering among the information decoded by the entropy decoding unit (10b-10) may be provided to the filtering unit (10b-50). Meanwhile, a receiving unit (not shown) that receives a signal output from an encoding device may be further configured as an internal / external element of the decoding device (10b), or the receiving unit may be a component of the entropy decoding unit (10b-10). Meanwhile, the decoding device according to the present document may be called a video / video / picture decoding device, and the decoding device may be divided into an information decoder (video / video / picture information decoder) and a sample decoder (video / video / picture sample decoder). The above information decoder may include the entropy decoding unit (10b-10), and the sample decoder may include at least one of the inverse quantization unit (10b-21), the inverse transformation unit (10b-22), the prediction unit (10b-30), the addition unit (10b-40), the filtering unit (10b-50), and the memory (10b-60).

[0087] The inverse quantization unit (10b-21) can inverse quantize the quantized transform coefficients and output the transform coefficients. The inverse quantization unit (10b-21) can rearrange the quantized transform coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scanning order performed in the encoding device. The inverse quantization unit (10b-21) can perform inverse quantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain transform coefficients.

[0088] In the inverse transform unit (10b-22), the transform coefficients are inversely transformed to obtain a residual signal (residual block, residual sample array).

[0089] The prediction unit can perform a prediction for the current block and generate a predicted block including prediction samples for the current block.

[0090] The prediction unit can determine whether intra prediction or inter prediction is applied to the current block based on the information about the prediction output from the entropy decoding unit (10b-10), and can determine a specific intra / inter prediction mode.

[0091] The prediction unit can generate a prediction signal based on various prediction methods described below. For example, the prediction unit can apply intra prediction or inter prediction to predict a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). In addition, the prediction unit can perform intra block copy (IBC) to predict a block. The intra block copy can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in this document.

[0092] The intra prediction unit (10b-32) can predict the current block by referencing samples within the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from it, depending on the prediction mode.

[0093] In intra prediction, prediction modes may include multiple non-directional modes and multiple directional modes. The intra prediction unit (10b-32) may determine the prediction mode to be applied to the current block by utilizing the prediction modes applied to the surrounding blocks.

[0094] The inter prediction unit (10b-31) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on an inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.).

[0095] In the case of inter prediction, the neighboring blocks may include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit (10b-31) may construct a motion information candidate list based on the neighboring blocks, and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter prediction may be performed based on various prediction modes, and the information regarding the prediction may include information indicating the mode of inter prediction for the current block.

[0096] The addition unit (10b-40) can generate a restoration signal (restored picture, restoration block, restoration sample array) by adding the acquired residual signal to the prediction signal (predicted block, prediction sample array) output from the prediction unit (10b-30). In cases where there is no residual for the block to be processed, such as when skip mode is applied, the predicted block can be used as the restoration block.

[0097] The addition unit (10b-40) may be called a restoration unit or a restoration block generation unit.

[0098] The generated restoration signal can be used for intra prediction of the next processing target block within the current picture, can be output after filtering as described below, or can be used for inter prediction of the next picture.

[0099] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture decoding process.

[0100] The filtering unit (10b-50) can improve subjective / objective image quality by applying filtering to the restoration signal. For example, the filtering unit (10b-50) can apply various filtering methods to the restoration picture to generate a modified restoration picture, and transmit the modified restoration picture to the memory (60), specifically, the DPB of the memory (10b-60). The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0101] The (corrected) reconstructed picture stored in the DPB of the memory (10b-60) can be used as a reference picture in the inter prediction unit (10b-31). The memory (10b-60) can store motion information of a block from which motion information is derived (or decoded) within the current picture and / or motion information of blocks within a picture that has already been reconstructed. The stored motion information can be transmitted to the inter prediction unit (10b-31) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (10b-60) can store reconstructed samples of reconstructed blocks within the current picture and transmit them to the intra prediction unit (10b-32).

[0102] In this specification, the embodiments described in the prediction unit (10b-30), the inverse quantization unit (10b-21), the inverse transformation unit (10b-22), and the filtering unit (10b-50) of the decoding device (10b) can be applied to the prediction unit (10a-20), the inverse quantization unit (10a-34), the inverse transformation unit (10a-35), and the filtering unit (10a-60) of the encoding device (10a) in the same manner or correspondingly.

[0103] As described above, prediction is performed to increase compression efficiency when performing video coding. Through this, a predicted block including prediction samples for a current block, which is a coding target block, can be generated. Here, the predicted block includes prediction samples in a spatial domain (or pixel domain). The predicted block is derived identically from an encoding device and a decoding device, and the encoding device can increase video coding efficiency by signaling information (residual information) about the residual between the original block and the predicted block, rather than the original sample value of the original block itself, to a decoding device. The decoding device can derive a residual block including residual samples based on the residual information, and generate a reconstructed block including reconstructed samples by combining the residual block and the predicted block, and can generate a reconstructed picture including the reconstructed blocks.

[0104] The above residual information can be generated through transformation and quantization procedures.

[0105] For example, the encoding device can derive a residual block between the original block and the predicted block, perform a transform procedure on residual samples (a residual sample array) included in the residual block to derive transform coefficients, perform a quantization procedure on the transform coefficients to derive quantized transform coefficients, and signal related residual information to a decoding device (via a bitstream). Here, the residual information can include information such as value information, position information, transform technique, transform kernel, and quantization parameter of the quantized transform coefficients. The decoding device can perform an inverse quantization / inverse transform procedure based on the residual information to derive residual samples (or residual blocks). The decoding device can generate a reconstructed picture based on the predicted block and the residual block. The encoding device can also inverse quantize / inverse transform the quantized transform coefficients to derive a residual block for reference in inter prediction of a subsequent picture, and generate a reconstructed picture based on the residual block.

[0106] <VCM(Video coding for Machines)>

[0107] With the recent advancements in various industries such as surveillance, intelligent transportation, smart cities, intelligent industry, and intelligent content, the amount of image or feature map data consumed by machines is increasing. In contrast, traditional video compression methods currently in use were developed with human vision in mind, and therefore contain unnecessary information, making them inefficient for machine tasks. For example, the resolution of images from the viewer's perspective may be higher than that of images (e.g., feature maps) from the machine's perspective. Therefore, research on video codec technologies that efficiently compress feature maps for machine tasks is needed.

[0108] The Moving Picture Experts Group (MPEG), an international standardization group for multimedia encoding, is discussing Video Coding for Machines (VCM). VCM is an image or feature map encoding technology that targets machine vision, rather than human viewer vision. In this document, feature maps can be referred to as "feature maps," and features can be referred to as "features."

[0109] Figures 4a to 4d are exemplary diagrams showing a VCM encoder and a VCM decoder.

[0110] Referring to FIG. 4a, a VCM encoder (100a) and a VCM decoder (100b) are shown.

[0111] When a VCM encoder (100a) encodes a video and / or a feature map and transmits it as a bitstream, a VCM decoder (100b) can decode and output the bitstream. At this time, the VCM decoder (100b) can output one or more videos and / or feature maps. For example, the VCM decoder (100b) can output a first feature map for machine-based analysis and a first image for user viewing. The first image can have a higher resolution than the first feature map.

[0112] Referring to FIG. 4b, a feature extractor for extracting a feature map may be connected to the front end of the VCM encoder (100a).

[0113] The VCM encoder (100a) may include a feature encoder.

[0114] The VCM decoder (100b) may include a feature decoder and a video reconstructor. The feature decoder may decode a feature map from a bitstream and output a first feature map for machine-assisted analysis. The video reconstructor may regenerate and output a first video from the bitstream for viewing by a user.

[0115] Referring to Fig. 4c, a feature extractor for extracting a feature map is connected to the front end of the VCM encoder (100). The VCM encoder (100a) may include a feature encoder.

[0116] The VCM decoder (100b) may include a feature decoder. The feature decoder may decode a feature map from a bitstream and output a first feature map for machine-based analysis. That is, the bitstream may be encoded only as a feature map, not as an image. To elaborate, the feature map may be data containing information about features for processing a specific task of a machine based on an image.

[0117] Referring to FIG. 4d, a feature extractor may be connected to the front end of the VCM encoder (100a).

[0118] The VCM encoder (100a) may include a feature converter and a video encoder. The video encoder may be the encoding device (10a) illustrated in FIG. 2.

[0119] The VCM decoder (100b) illustrated in FIG. 4d may include a video decoder and an inverse converter. The video decoder may be the decoding device (10b) illustrated in FIG. 3.

[0120] FIG. 5 is a block diagram of an encoding device according to an embodiment of the present disclosure.

[0121] An encoding device (10a) according to an embodiment of the present disclosure can encode an input signal (input image) through adaptive mapping based on region-of-interest features for VCM. Machine vision applications mainly perform tracking, recognition, classification, and separation centered on objects existing within an image, and the importance of the background is relatively low. In human vision, the size, color, and position of an actual object within an image are important information along with the background, but in machine vision, the accuracy of machine vision applications (e.g., object recognition, tracking, classification, and separation) is important regardless of information such as the color or size of the original. An encoding / decoding device according to various embodiments of the present disclosure can improve decoding efficiency without degrading the performance of machine vision applications through region-of-interest-based decoding centered on an object for decoding efficiency. Decoding efficiency can be improved by mapping regions of interest based on features of regions of interest within an input image and performing decoding and encoding on the mapped signal. The encoding device (10a) can transmit information about the region of interest to the decoding device (10b), thereby enabling the decoding device (10b) to restore the original image and the region of interest when necessary.

[0122] An encoding device (10a) according to various embodiments may receive an image (video) and perform encoding through region-of-interest-based adaptive mapping to generate a bitstream. The encoding device (10a) may include a temporal resampler (510), a spatial resampler (520), a region-of-interest extractor (530), a region-of-interest feature-based processor (540), and an internal encoder (550). The order of components included in the encoding device (10a) illustrated in FIG. 5 may be changed in various embodiments, and some components may be omitted. Referring to FIG. 5, the encoding device (10a) may first perform temporal / spatial resampling, and then extract and process a region of interest. Unlike FIG. 5, the encoding device (10a) according to one embodiment may first extract a region of interest, and then perform encoding in the order of a region of interest extractor (530), a temporal resampler (510), a spatial resampler (520), a region of interest feature-based processor (540), and an internal encoder (550) to perform spatiotemporal resampling based on the region of interest information. Alternatively, the encoding device (10a) in one embodiment may perform encoding in the order of a region of interest extractor (530), a region of interest feature-based processor (540), a temporal resampler (510), a spatial resampler (520), and an internal encoder (550).

[0123] For convenience of explanation in the present disclosure, the encoding device (10a) may be referred to as an encoder or encoder, and the decoding device (10b) may be referred to as a decoder or decoder.

[0124] The input image (video) is an input signal of an encoding device (10a) for machine vision, and can be an image acquired by various sensors having the characteristics of a video, such as a general optical camera image, a thermal camera, or a lidar sensor image. The composition of the input image can include a single component or multiple components depending on the characteristics of the sensor. For example, an image acquired by an optical camera can be composed of multiple color difference components in the RGB and YUV domains.

[0125] The temporal resampler (510) is a step for changing the frame rate of the input image, and can perform upsampling or downsampling on a frame-by-frame basis. The frame rate can be determined according to the application purpose of machine vision, and in one embodiment, temporal resampling can be performed according to a predetermined frame rate. Alternatively, the frame rate can be variable in units of a series of frame groups or sequences. When the frame rate is applied variably, the encoding device (10a) can transmit information related to temporal resampling (e.g., frame rate, resampling unit, etc.) to the decoding device (10b).

[0126] The spatial resampler (520) is a step for changing the resolution of each frame of the input image, and may perform upsampling or downsampling on a frame-by-frame basis. Depending on the embodiment, the spatial resampler (520) may perform upsampling or downsampling with the same resolution for the entire frame of the input image. Depending on the embodiment, the spatial resampler (520) may perform upsampling or downsampling by applying different resolutions on a frame-by-frame basis of the input image. The encoding device (10a) may transmit information related to spatial resampling (e.g., target resolution, resampling unit, etc.) to the decoding device (10b).

[0127] In various embodiments, the encoding device (10a) may extract objects from each frame of the input image for special purposes required in machine vision applications for the input image or for general machine vision purposes. For example, machine vision applications may be object tracking, recognition, classification, separation, etc. In various embodiments, the objects may be, for example, people, objects, animals, robots, drones, mobile electronic devices, or automobiles, and may be defined in various forms for various applications.

[0128] The region of interest extractor (530) is a step for distinguishing a region expected to be an object from a background corresponding to a part other than the object in an input image, and can extract information on the location and size of the object using a learning or image processing technique within the image. According to an embodiment, the region of interest extractor (530) can extract multiple regions of interest from the input image, and information on the size and location of the regions of interest can be transmitted to the internal encoder (550) for effective encoding. The internal encoder (550) can perform encoding by changing the image quality by performing different quantizations for the object and the background, respectively.

[0129] The region of interest feature-based processor (540) is a step that analyzes the features of the region of interest extracted through the previous region of interest extractor (530) and performs specific processing on values ​​inside and outside the region of interest using the features. The detailed steps for the specific processing may be as shown in FIG. 6.

[0130] According to an embodiment, the region of interest feature-based processor (540) may analyze and process features by dividing the entire region of interest and the entire non-interest region extracted from the image, or may analyze and process features for each region of interest among multiple regions of interest obtained from the region of interest extractor (530). Alternatively, there may be a method of analyzing features for the entire region of interest and applying processing for the features equally to the region of interest and the non-interest region.

[0131] According to an embodiment, the region of interest feature-based processor (540) may analyze and process features of each component of the entire region of interest, or may analyze and process features of each component of a plurality of regions of interest obtained from the region of interest extractor (530).

[0132] Depending on the embodiment, the region of interest to be analyzed and processed for features may be some of the multiple regions of interest extracted through the region of interest extractor (530). For example, if the number of regions of interest extracted through the region of interest extractor (530) is K, only some of the regions of interest, not all of the K, may be subject to feature analysis and processing, and this may be determined by the size, location, shape, etc. of the region of interest. For example, there may be a method of defining characteristics for regions of interest to be processed by the region of interest feature-based processor (540), such as regions of interest having a size greater than / less than a specific size, regions of interest confirmed to have occurred continuously in multiple frames, regions of interest appearing within a specific location of the image, regions of interest extracted as rectangles / squares, etc., and limiting only regions of interest satisfying the characteristics to be subject to the region of interest feature-based processor (540). Alternatively, there may be a method of limiting the target of the region of interest feature-based processor (540) to only regions of interest that satisfy a specific rank by sorting the region of interest in order of size, or limiting the area ratio of the entire region of interest in the image to below a certain ratio and limiting the area of ​​the region of interest to only regions within a specific rank as targets of the region of interest feature-based processor (540) so that the area of ​​the region of interest is limited within the limited ratio. When the target of the region of interest feature-based processor (540) is limited, depending on the embodiment, if there is no region of interest that satisfies the corresponding feature, the steps of the processor after the region of interest extractor (530) may be omitted.

[0133] Depending on the embodiment, the components of the region of interest whose features are analyzed and processed may be some, but not all, of the components. For example, when the input image is an image composed of three YUV components, feature analysis may be performed and processed using one or more of the YUV components. In this case, the number of components for which features are analyzed and the number of components for which processing is performed through feature analysis may not be the same. For example, feature analysis may be performed on the Y component, and processing based on the feature may be applied to all YUV components. Alternatively, processing may be performed only on components for which feature analysis has been performed, and processing may not be performed on components for which feature analysis has not been performed.

[0134] According to an embodiment, if the invention includes a feature map extractor as in FIG. 4b, the component may be a feature map.

[0135]

[0136] The internal encoder (550) can perform encoding on the input image. The internal encoder (550) can utilize temporal resampling information from previously performed steps for the reference structure. The internal encoder (550) can determine the reference method for objects and backgrounds based on information about the region of interest, or can perform effective encoding by changing quantization parameters.

[0137] Depending on the embodiment, the execution order of the region of interest extractor (530), the region of interest feature-based processor (540), the temporal resampler (510), and the spatial resampler (520) may be changed. However, when the information of the region of interest extracted from the region of interest extractor (530) in the encoding device (10a) is used in the temporal resampler (510) and / or the spatial resampler (520), the region of interest extractor (530) may be executed prior to the temporal resampler (510) and the spatial resampler (520).

[0138] FIG. 6 illustrates a process of extracting a region of interest and processing a region of interest feature based on an encoding device according to an embodiment of the present disclosure.

[0139] An encoding device (10a) according to various embodiments can extract a region of interest for an input image and perform one or more processes based on the region of interest characteristics. With reference to FIGS. 6 and 7, the detailed operations of the region of interest extractor (530) and the region of interest feature-based processor (540) of the encoding device (10a) will be described. FIG. 7 is an example of (a) object extraction, (b) region of interest selection, and (c) region of interest classification according to one embodiment.

[0140] In the object region extraction step (610), the region of interest extractor (530) can extract one or more object regions from the input image. The region of interest extractor (530) can extract six object regions (obj1, obj2, obj3, obj4, obj5, obj6) from one frame, as shown in (a) of FIG. 7. The region of interest extractor (530) can extract objects using a learning-based neural network model. The region of interest extractor (530) can extract objects for each frame of the entire frame of the input image, and the number of objects extracted for each frame may be different. In some cases, there may be frames from which no objects are extracted. Multiple objects may be extracted within one frame, and the same object may be extracted from multiple frames. Multiple frames containing the same object may be continuous frames, but may also be discontinuous. The object extraction results may also be different depending on various input images. Frames from which no objects are extracted can be encoded directly.

[0141] In the region of interest selection step (620), the region of interest extractor (530) can select one or more regions of interest from among the extracted object regions. The region of interest extractor (530) can select three object regions (obj1, obj2, obj4) from among the six object regions of (a) as regions of interest (RoI1, RoI2, RoI3) as shown in (b) of FIG. 7. The regions of interest can be targets of encoding. The region of interest extractor (530) can vary the method of selecting the region of interest from among the object regions according to various embodiments. In one embodiment, the region of interest extractor (530) can select the region of interest based on size information of the object region (e.g., number of horizontal pixels, number of vertical pixels, area of ​​the region, etc.). In another embodiment, the region of interest extractor (530) can select an object region having a predetermined specific size or larger as the region of interest. In another embodiment, the region of interest extractor (530) may select an object region smaller than a predetermined size as a region of interest. In one embodiment, the region of interest extractor (530) may select a region of interest based on the accuracy of the object extracted using a learning-based neural network model. Depending on the embodiment, the region of interest extractor (530) may apply multiple methods for selecting a region of interest among object regions. The number of regions of interest may be 0 or more and n or less, and the number of regions of interest may be equal to the number of object regions. If the region of interest is 0, the corresponding frame may be encoded directly.

[0142] In the region of interest feature analysis step (630), the region of interest feature-based processor (540) analyzes the features of each region of interest for one or more selected regions of interest. The features of the region of interest may be a histogram of the chrominance components of the region of interest, an average, a variance, an edge of an object, a classification of an object, etc., and one or more features may be used depending on the embodiment. Fig. 8 is an example of a histogram of the chrominance components of the region of interest according to an embodiment of the present disclosure. If an analysis is performed using the histogram of the chrominance components of the region of interest as a feature according to the embodiment, features for each RGB component may appear as shown in Fig. 8.

[0143] In the region of interest classification and chrominance sampling step (640), the region of interest feature-based processor (540) can classify one or more regions of interest into M based on the characteristics of the analyzed region of interest. For example, if the histogram of each chrominance component is analyzed as a feature for the region of interest, it can be divided into M classifications based on the similarity of the histogram. For example, if the characteristics of the region of interest appear as in FIG. 8, since (a) and (b) strongly show the characteristics of the chrominance component of R, and (c) strongly shows the characteristics of the chrominance component of G, (a) and (b) can be classified as one, and (c) can be distinguished as another. Accordingly, as in FIG. 7 (c), RoI1 and RoI2 can be classified as GRoI1, and RoI3 can be classified as GRoI2. Classifying multiple regions of interest based on the chrominance components of the region of interest is just one example, and can be classified in various ways based on various criteria depending on the embodiment. Depending on the purpose of the machine vision application, the classification items can be subdivided or simplified. In some embodiments, object sampling can be performed during the process of classifying a region of interest. Chromatic sampling refers to the removal of some chrominance signals from an image signal (input image). For example, it refers to the removal of one or more of the R, G, and B components from an image composed of RGB. Typically, one component can be selected and removed.

[0144] In the mapping relationship extraction and mapping step (650), the region of interest feature-based processor (540) can derive a mapping relationship for each chrominance component and perform mapping on signals for which chrominance sampling has been performed among the classified M regions of interest. Fig. 9 is an example of mapping information according to an embodiment of the present disclosure.

[0145] The region of interest feature-based processor (540) can extract a mapping relationship for a single component and perform mapping if one component remains through chrominance sampling or if the input image is an image signal consisting of one component. The region of interest feature-based processor (540) can perform the corresponding process according to the number of chrominance signals if the image is composed of multiple chrominance signals because chrominance sampling is not performed according to an embodiment. The mapping relationship can be set to various relationships for encoding efficiency.

[0146] FIG. 10 is an example of a histogram (a) before mapping and histograms (b, c, d) after mapping according to an embodiment of the present disclosure. (b) represents a change in a histogram that appears in a mapping relationship in which min_value, which corresponds to the minimum value of the chrominance component determined by mapping, is mapped to 0 and min_value is subtracted from values ​​above that value, so that the overall value is mapped to a value that is smaller on average by min_value. (c) and (d) represent a change in a histogram when the minimum value is mapped to 0 and the size of the interval is adaptively changed for the intervals before and after mapping for values ​​above the minimum value. When the slope of the interval before mapping (delta_value) and the interval after mapping (delta_mapping_value) is less than ½, the interval before mapping is wider and the interval after mapping is narrower, which may result in signal loss. However, if the mapping interval is appropriately adjusted for cases where there is no signal in the histogram or where the signal size is very small, effective expression through mapping is possible without significant signal loss. In various embodiments, the region of interest feature-based processor (540) can adaptively set the mapping interval and effectively transmit the corresponding information through syntax such as VCM_Colour mapping_SP(), mapping_fun_list_data(), VCM_Colour mapping_FP(), and ROI_Ioc().

[0147] The semantics and syntax used in the embodiments of the present disclosure are as follows.

[0148] In various embodiments, the encoding device (10a) can encode information about a mapping relationship for a region of interest as shown in Tables 1 to 6.

[0149] [Table 1]

[0150]

[0151] [Table 2]

[0152]

[0153] [Table 3]

[0154]

[0155] [Table 4]

[0156]

[0157] In an embodiment, a feature-based processor (540) of an area of ​​interest may transmit mapping information as shown in Table 5 below, omitting the corresponding information when the number of sections and the size of the sections, delta, are fixed by agreement between the encoders and decoders.

[0158] [Table 5]

[0159]

[0160] According to Table 5, the mapping ratio of delta_value[k] and mapping_delta_value[k] can be fixed values, and for the efficiency of mapping, it is 2. n It can be mapped in the form of n. n is fixed to a specific value by agreement between the encoder and decoder and may not be transmitted. Here, n can be a positive number, 0, or a negative number. Depending on the embodiment, the mapping ratio n of delta_value[k] and mapping_delta_value[k] can be transmitted from the encoder to the decoder. The syntax for transmitting the mapping ratio n to the decoder can be defined as shown in Table 6 below.

[0161] [Table 6]

[0162]

[0163] In Tables 2, 5, and 6, min_value[i] denotes the minimum value of the input signal (input image), and may be transmitted in the form of a value obtained by taking log2 for transmission efficiency. In some embodiments, if the value of min_value[i] is 0 and the minimum value before and after mapping is the same as 0, the value may be omitted. min_value[i] may be transmitted in the form of a difference value from min_value[i-1] transmitted previously for transmission efficiency.

[0164] Mapping relationships are fundamentally extracted by region of interest (ROI) or by category of ROI, and the corresponding information can be transmitted to the decoder. Depending on the embodiment, for transmission efficiency, a list of mapping relationships can be transmitted on a sequence or GOP basis, and only the index of the list can be transmitted for each frame's ROI, thereby improving the encoding efficiency of the transmission of mapping relationships.

[0165] In the non-region of interest processing step (660), the region of interest feature-based processor (540) processes regions other than the region of interest. The region of interest feature-based processor (540) performs the same processing using information corresponding to the widest region among the region of interest classifications for internal encoding. Depending on the embodiment, for encoding efficiency, a method used in a specific region of interest classification may be used and the corresponding information may be transmitted, which may be transmitted on a frame-by-frame basis.

[0166] The internal encoder (550) performs encoding on the mapped signal. The signal input to the internal encoder (550) is encoded by transmitting the chrominance signal selected in the mapping step. Generally, the internal encoder (550) supports various formats of the input chrominance signal, but depending on the embodiment, the number of chrominance components to be encoded may be fixed to a specific number. For example, in the chrominance sampling process, only one chrominance signal is left and sampled, and processing based on region of interest features is performed and the result of the processing is transmitted to the internal encoder (550). The encoder may be an encoder that requires three components. In that case, before the encoding step, the sampled signal may be copied into three components so that the required number of signals is satisfied, and the signal may be transmitted as an input to the internal encoder (550).

[0167] FIG. 11 is a block diagram of a decoding device according to an embodiment of the present disclosure.

[0168] A decoding device (10b) according to an embodiment of the present disclosure can input a bitstream, decode it, and output a restored image. The decoding device (10b) can include an internal decoder (1110), a region-of-interest restorer (1120), a spatial restorer (1130), and a temporal restorer (1140). In Fig. 11, the decoding device (10b) shows the structure of a decoder for machine vision, and depending on the embodiment, steps of detailed configuration may be omitted and some orders may be changed. The order of the decoder generally proceeds in the reverse order of the encoder, but if there is no dependency among the region-of-interest restorer (1120), the temporal restorer (1140), and the spatial restorer (1130), the order of the decoder may not correspond to the order of the encoder. However, if information about the region of interest in the encoder is used in the spatial processor (spatial resampler (520) or region of interest feature-based processor (540)) and the temporal processor (temporal resampler (510) or region of interest feature-based processor (540)), and the information is used in the spatial restorer (1130) and the temporal restorer (1140), the order of the decoder's progress may be as shown in FIG. 11.

[0169] The internal decoder (1110) receives a bitstream and performs decoding. Depending on the embodiment, various decoders are used, but a decoder corresponding to the internal encoder used in the encoder is used.

[0170] The region of interest restorer (1120) performs restoration by region of interest classification on the video signal decoded by the internal decoder (1110). If resolution restoration is requested from the encoding device (10a) that performed encoding on the input video, resolution restoration can be performed on the bitstream. For example, the restoration process can be performed only when color_restoration_flag is 1 in the transmitted bitstream. The region of interest restorer (1120) performs reverse mapping based on the position information for each region of interest received from the encoder to restore, and additionally, if sampling for the chrominance component has been performed, restores the component sampled using the received method.

[0171] FIG. 12 illustrates a process of restoring a region of interest of a decoding device according to an embodiment of the present disclosure.

[0172] The internal decoder (1110) can decode the bitstream and transmit it to the region of interest restorer (1120).

[0173] In the reverse mapping information extraction step (1201), the region of interest restorer (1120) can decode the information transmitted through the syntax defined in Tables 1 to 6 and derive the reverse mapping relationship. The leaf mapping relationship is the same relationship as the mapping function of FIG. 9, and the reverse mapping relationship can be derived by calculating the value corresponding to each section using the values ​​transmitted as differential values, such as delta_mapping_value and delta_value. The region of interest restorer (1120) can decode the corresponding reverse mapping information for each region of interest transmitted from the encoder and derive the reverse mapping relationship.

[0174] In the region of interest location information extraction step (1202), the region of interest restorer (1120) can decode the region of interest location information from the syntax transmitted through FIGS. 1 to 6 and derive location and size information in an actual frame for each region of interest. The region of interest restorer (1120) can apply the information to an image decoded through an internal decoder (1110) to specify an area for which restoration is to be performed through reverse mapping for the internally decoded image.

[0175] In the region of interest reverse mapping step (1203), the region of interest restorer (1120) can perform reverse mapping using reverse mapping information for a specific image location through the location information of the region of interest.

[0176] In the component restoration step (1204), the region of interest restorer (1120) can restore the chrominance component if the num_of_component is 1 and the sensor_info is 0 for the image decoded through internal decoding, i.e., if it is an optical camera image. The chrominance component can be restored based on the information transmitted through Tables 1 to 6. The image signal with the chrominance component restored in this way can be transmitted to the next step. For example, the image signal with the chrominance component restored can be transmitted to a spatial restorer (1130) or a temporal restorer (1140).

[0177] The spatial restorer (1130) can perform spatial restoration using spatial restoration information transmitted from the encoder. Spatial restoration refers to restoration of resolution for each frame, and based on the information transmitted from the encoder, a signal with a changed resolution can be output for a signal input to the spatial restorer. Spatial restoration by the spatial restorer (1130) refers to restoration using spatial restoration-related information transmitted from the encoder, and depending on the embodiment, the resolution restored through the spatial restorer (1130) may not be the same as the resolution input to the spatial resampler (520) in the encoder.

[0178] The temporal restorer (1140) can perform temporal restoration using temporal restoration information transmitted from the encoder. Temporal restoration refers to restoration of the frame rate per second, and based on the information transmitted from the encoder, a signal with a changed frame rate can be output for a signal input to the temporal restorer (1140). Temporal restoration by the temporal restorer (1140) refers to restoration using temporal restoration-related information transmitted from the encoder, and according to an embodiment, the frame rate restored by the temporal restorer (1140) may not be the same as the frame rate input to the temporal resampler (510) in the encoder.

[0179] The output signal is a video signal that has undergone temporal restoration by decoding the bitstream. Filtering may be performed on the restored signals during the spatial / temporal restoration process due to reasons such as image quality degradation. Depending on the application purpose of machine vision, the output signal can be input to deep neural networks, etc.

[0180] The examples of the present disclosure presented in this specification and drawings are intended solely to facilitate the technical content of the present disclosure and to aid understanding thereof, and are not intended to limit the scope of the present disclosure. It will be apparent to those skilled in the art that other variations are possible in addition to the examples described above.

[0181] The claims set forth in this specification may be combined in various ways. For example, the technical features of the method claims of this specification may be combined and implemented as a device, and the technical features of the device claims of this specification may be combined and implemented as a method. Furthermore, the technical features of the method claims and the technical features of the device claims of this specification may be combined and implemented as a device, and the technical features of the method claims and the technical features of the device claims of this specification may be combined and implemented as a method.

Claims

1. In a VCM (video coding for machines) encoding device, A region of interest extractor that extracts one or more regions of interest for each frame of an input image; A region of interest feature-based processor that derives a mapping relationship for one or more regions of interest and performs mapping based on the features of the one or more regions of interest; and An encoding device comprising an internal encoder that encodes the input image using information about one or more regions of interest to generate a bitstream.

2. In paragraph 1, The above region of interest extractor is, An encoding device that extracts one or more object regions from the input image and selects one or more regions of interest among the one or more object regions according to preset criteria.

3. In paragraph 2, An encoding device wherein the above preset standard is set according to size information of an object area, and the size information of the object area includes at least one of the width, height, and area of ​​the object area.

4. In paragraph 1, An encoding device in which the region of interest extractor extracts one or more objects from the input image using a learning-based neural network model and selects a region of interest based on the detection accuracy of the extracted objects.

5. In paragraph 1, An encoding device, wherein the features of the region of interest include at least one of a histogram for a chrominance component, a mean, a variance, an edge of an object, and a classification of an object.

6. In paragraph 5, An encoding device wherein the above region of interest feature-based processor classifies one or more regions of interest based on the degree of similarity of color difference components for the regions of interest.

7. In paragraph 6, An encoding device in which the above region of interest feature-based processor performs chrominance sampling to remove some chrominance signals of each region of interest in the process of classifying one or more regions of interest.

8. In paragraph 6, An encoding device in which the above-mentioned region of interest feature-based processor determines the classification degree for one or more regions of interest according to the purpose of machine vision application for the input image.

9. In paragraph 1, An encoding device that performs mapping for each color difference component for each region of interest based on the above region of interest feature-based processor.

10. In paragraph 1, An encoding device in which the above-mentioned region-of-interest feature-based processor adaptively sets a mapping section by considering the loss of signal according to the slope of the section before and after mapping between each region of interest.

11. In paragraph 1, An encoding device wherein a mapping relationship between one or more regions of interest is extracted by region of interest or by classification of region of interest.

12. In paragraph 11, An encoding device in which a list of mapping relationships between one or more regions of interest is included in the bitstream in units of sequences or GOPs (group of pictures) of the input image, and information on the region of interest for each frame includes an index of the list of mapping relationships.

13. In paragraph 1, An encoding device wherein the internal encoder encodes information about the one or more regions of interest and information about a mapping relationship between the one or more regions of interest.

14. In paragraph 1, Further comprising a temporal resampler that changes the frame rate of the input image, An encoding device wherein the temporal resampler performs upsampling or downsampling on a frame-by-frame basis based on information about the one or more regions of interest.

15. In paragraph 1, Further comprising a spatial resampler that changes the resolution of the input image, An encoding device wherein the spatial resampler performs upsampling or downsampling on a frame-by-frame basis based on information about one or more regions of interest.

16. In a VCM (video coding for machines) decoding device, An internal decoder that decodes a bitstream to generate a restored image and extracts information about a region of interest in the restored image; and A decoding device including a region of interest restorer that performs restoration for one or more regions of interest included in the restored image based on information about the region of interest.

17. In paragraph 16, A decoding device that performs reverse mapping for one or more regions of interest based on location information for the one or more regions of interest, wherein the region of interest restorer is a device.

18. In paragraph 17, The above region of interest restorer is a decoding device that restores the color difference component for the region of interest based on information about the region of interest.

19. In paragraph 16, A spatial restorer that restores the resolution of the restored image using spatial restoration information extracted from the bit stream; and Further comprising a temporal restorer that restores the frame rate for the restored image using temporal restoration information extracted from the bit stream, The above spatial restorer performs spatial restoration based on information about the region of interest, The above temporal restorer is a decoding device that performs temporal restoration based on information about the region of interest.

20. A non-volatile computer-readable storage medium that records commands, The above instructions, when executed by one or more processors, cause the one or more processors to: A step of extracting one or more regions of interest for each frame of an input image; A step of deriving a mapping relationship for one or more regions of interest and performing mapping based on the characteristics of the one or more regions of interest; and A non-transitory computer-readable storage medium comprising a step of encoding the input image using information about the one or more regions of interest to generate a bitstream.

Images