Method for encoding and decoding image information based on region of interest

Abstract

In one embodiment of the present disclosure, a machine-oriented video coding (VCM) encoding apparatus is disclosed. The encoding device comprises: a time resampler for changing a partial frame rate of an input image; a region-of-interest-based processor for extracting one or more regions of interest included in each of the plurality of frames of the input image, and generating an image processed based on the one or more regions of interest; and an internal encoder for generating a bitstream by encoding information about the one or more regions of interest and an input image, where the information about the one or more regions of interest may include a time sampling rate.

Description

Technical Field

[0001] This disclosure relates to methods for encoding and decoding image information based on regions of interest. Background Technology

[0002] With the continuous development of the information and communication industry, broadcasting services with HD (high definition) resolution have spread globally.

[0003] This spread has made many users familiar with high-resolution and high-definition images and / or videos, and has increased the demand for high-resolution and high-quality images and videos, such as UHD (Ultra-High Definition) images / videos with higher definition, i.e., 4K or 8K or even higher definition, in various fields.

[0004] The technology used to encode this UHD image data was completed in 2013 using the standard technology HEVC (High Efficiency Video Coding).

[0005] HEVC is a next-generation image compression technology with higher compression ratio and lower complexity than the previous H.264 / AVC technology, and it is a core technology for effectively compressing large amounts of data in HD and UHD images.

[0006] Like previous compression standards, HEVC performs encoding on a block-by-block basis.

[0007] However, unlike H.264 / AVC, it has only one configuration file. The core coding techniques included in HEVC's single configuration file include layered coding structure techniques, transform techniques, quantization techniques, intra-frame predictive coding techniques, inter-frame motion prediction techniques, entropy coding techniques, loop filtering techniques, and other techniques, totaling eight areas.

[0008] Since the establishment of the HEVC video codec in 2013, with the expansion of services such as realistic images using 4K and 8K video images and virtual reality, a new standard, VVC (Variety Video Coding), has been developed as the next-generation video codec, aiming to improve performance by at least twice that of HEVC. VVC is also known as H.266.

[0009] H.266 (VVC) was developed with the goal of achieving twice the efficiency of the previous generation codec H.265 (HEVC). Initially developed with 4K resolution or higher in mind, VVC has been expanded to handle massive 16K ultra-high resolution images due to the growing VR market and its responsiveness to 360-degree imaging. Furthermore, as the HDR market has expanded along with display technology advancements, VVC supports not only 10-bit color depth but also 16-bit color depth to accommodate the HDR market, and supports brightness levels of 1000 nits, 4000 nits, and 10000 nits. Additionally, because VVC is being developed with the VR and 360-degree imaging markets in mind, it supports partial frame rates from 0 FPS to 120 FPS.

[0010] The Development of Artificial Intelligence

[0011] Artificial intelligence (AI) is also gradually developing. AI refers to the artificial imitation of human intelligence, that is, the intelligence that can recognize, classify, reason, predict, control / make decisions, etc.

[0012] With the development of artificial intelligence technology and the increase in Internet of Things (IoT) devices, machine-to-machine traffic is expected to increase dramatically, and machine-based image analytics is expected to be widely used.

[0013] However, server load and power consumption issues are expected as the number of images to be analyzed by machines is anticipated to increase exponentially. Summary of the Invention

[0014] Technical issues

[0015] Therefore, this disclosure provides methods for encoding and decoding image information based on regions of interest to efficiently perform machine-based image analysis.

[0016] Technical solutions

[0017] To achieve the above objectives, a VCM encoding device is provided according to a disclosure in this specification.

[0018] The VCM encoding apparatus includes: a temporal resampling actuator for changing a portion of the frame rate of an input image; a region-of-interest (ROI)-based processor for extracting one or more ROIs from each of multiple frames of the input image and generating an image based on the one or more ROIs; and an internal encoding actuator for generating a bitstream by encoding information about the one or more ROIs and the input image, wherein the information about the one or more ROIs may include the temporal sampling rate.

[0019] A VCM decoding apparatus according to one disclosure of this specification is provided. The VCM decoding apparatus may include: an internal decoding executor for decoding a bitstream to generate a restored image; a region-of-interest (ROI)-based restorer for reconfiguring the restored image based on the ROI in the restored image using ROI-based processing information parsed from the bitstream; and a time restoration executor for performing time restoration of the restored image using time resampling information parsed from the bitstream.

[0020] As a non-volatile computer-readable storage medium for recording commands according to a disclosure of this specification, the commands, when executed by at least one processor, can cause at least one processor to: change a portion of the frame rate of an input image; extract one or more regions of interest (ROIs) included in each of multiple frames of the input image and generate an image based on the processing of the one or more ROIs; and generate a bitstream by encoding information about the one or more ROIs and the input image, wherein the information about the one or more ROIs may include a temporal sampling rate.

[0021] Technical effect

[0022] Based on this disclosure, machine-based image analysis can be performed efficiently.

[0023] According to this disclosure, when the VCM encoding device extracts and transmits the region of interest (ROI) of the input image, the accuracy of the deep neural network in the VCM decoding device can be improved. The VCM encoding device can signal information about the position and motion of the ROI in units of GOPs or a series of frames and transmit it to the VCM decoding device.

[0024] According to this disclosure, when a VCM encoding device performs temporal resampling on an input image, extracts regions of interest (ROIs), and encodes the ROIs, encoding efficiency can be improved. Furthermore, when a VCM decoding device performs temporal reconstruction of the reconstructed image based on motion information between the ROIs and partial frames, the location of the ROIs can be recovered more accurately. When the VCM decoding device performs machine vision tasks, the task execution results can be modified based on the ROI motion information. For example, the accuracy of object-oriented tasks can be improved by modifying the results between frames where the brightness of light changes rapidly. Attached Figure Description

[0025] Figure 1 An example illustrating a video / image coding system.

[0026] Figure 2 This is a diagram schematically illustrating the configuration of a video / image encoding device.

[0027] Figure 3This is a diagram schematically illustrating the configuration of a video / image decoding device.

[0028] Figures 4a to 4d This is an exemplary diagram representing a VCM encoder and a VCM decoder.

[0029] Figure 5 A block diagram of an encoding apparatus according to an embodiment of the present disclosure is shown.

[0030] Figure 6 This describes the detailed operation of a region-of-interest-based processor according to an embodiment of this disclosure.

[0031] Figure 7a , Figure 7b and Figure 7c This describes the detailed operation of the internal coding executor according to the embodiments of this disclosure.

[0032] Figure 8 This describes the detailed operation of the region of interest information encoding execution unit according to the embodiments of this disclosure.

[0033] Figure 9 A block diagram of a decoding apparatus according to an embodiment of the present disclosure is shown.

[0034] Figure 10 This describes the detailed operation of the internal decoding executor and the region-of-interest-based restorer according to embodiments of this disclosure.

[0035] Figure 11 This describes the detailed operation of the region of interest information decoding execution unit according to the embodiments of this disclosure.

[0036] Figure 12a and Figure 12b This describes the detailed operation of the time recovery actuator according to the embodiments of this disclosure.

[0037] Figure 13 This is an example of time modification of the region of interest according to the implementation of this disclosure.

[0038] Figure 14a and Figure 14b This describes the detailed operation of a machine vision task executor according to an embodiment of the present disclosure. Detailed Implementation

[0039] The specific structural or phased descriptions of embodiments of the concept of this disclosure disclosed in this specification or application are shown only for the purpose of describing embodiments of the concept of this disclosure, and embodiments of the concept of this disclosure may be implemented in various forms and should not be construed as limited to the embodiments described in this specification or application.

[0040] Because various modifications and forms can be made to the embodiments based on the concept of this disclosure, specific embodiments are shown in the accompanying drawings and described in detail in this specification or application. However, it is not intended to limit the embodiments based on the concept of this disclosure to the specific forms disclosed, and it should be understood that they include all changes, equivalents, or substitutions included within the spirit and scope of this disclosure.

[0041] Various components may be described using terms such as first, second, etc., but components should not be limited by these terms. These terms are used only for the purpose of distinguishing one component from other components, and, for example, without departing from the scope of the claims based on the concept of this disclosure, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component.

[0042] When a component is referred to as "linked" or "connected" to other components, it should be understood that the component can be directly linked or connected to other components, but other components may exist in between. On the other hand, when a component is referred to as "directly linked" or "directly connected" to other components, it should be understood that no other components exist in between. Other expressions describing the relationship between components, namely "between" and "directly between," or "adjacent to" and "directly adjacent to," should also be interpreted similarly.

[0043] Since the terminology used in this specification is for describing particular embodiments only, it is not intended to limit the scope of this disclosure. Singular expressions include plural expressions unless the singular expression clearly has a different meaning depending on the context. In this specification, it should be understood that terms such as "comprising" or "having" refer to the presence of the described features, numbers, steps, movements, components, portions, or combinations thereof, but do not preclude the presence or prior addition of one or more other features, numbers, steps, movements, components, portions, or combinations thereof.

[0044] Unless otherwise defined, all terms used herein, including technical or scientific terms, shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0045] Terms defined in commonly used dictionaries should be interpreted as having the same meaning as in the context of the relevant technology, and should not be interpreted in an ideal or overly formalized sense unless explicitly defined in this specification.

[0046] In describing the implementation methods, technical details that are well-known in the technical field to which this disclosure pertains and are not directly related to this disclosure have been omitted.

[0047] This is to convey the key points of this disclosure more clearly, without obscuring them by omitting unnecessary descriptions.

[0048] This document relates to video / image coding. For example, the methods / implementations disclosed in this document may relate to the Multifunctional Video Coding (VVC) standard (ITU-T Rec.H.266), next-generation video / image coding standards that comply with VVC, or other video coding-related standards (e.g., High Efficiency Video Coding (HEVC) standard (ITU-T Rec.H.265), Basic Video Coding (EVC) standard, AVS2 standard, etc.).

[0049] This document presents various implementations related to video / image coding, and unless otherwise stated, these implementations can be combined with each other.

[0050] In this document, video can refer to a set of images over time. An image is generally a unit representing an image at a specific time interval, and a slice / tile is a unit that configures a portion of an image in the encoding.

[0051] A slice / tile may include one or more coding tree units (CTUs). An image may consist of one or more slices / tiles. An image may consist of one or more tile groups. A tile group may include one or more tiles.

[0052] A pixel, or pel, can refer to the smallest unit that configures a picture (or image). Alternatively, a "sample" can be used as the term corresponding to a pixel. A sample can typically represent a pixel or pixel value, or it can represent only the pixel / pixel value of the luminance component, or it can represent only the pixel / pixel value of the chrominance component. Alternatively, a sample can refer to a pixel value in the spatial domain, or, when such a pixel value is transformed to the frequency domain, it can refer to the transform coefficients in the frequency domain.

[0053] A unit can represent a basic unit used for image processing. A unit may include a specific region of an image and at least one of the information associated with that region.

[0054] A unit may include one luminance block and two chrominance (e.g., cb, cr) blocks. In some cases, the term "unit" can be used interchangeably with terms such as "block" and "region." Typically, A block may include a sample (or sample array) consisting of M columns and N rows or a set (or array) of transform coefficients.

[0055] Figure 1 An example illustrating a video / image coding system.

[0056] Reference Figure 1 A video / image encoding system may include a source device and a receiving device. The source device may transmit encoded video / image information or data to the receiving device in the form of a file or stream via a digital storage medium or network.

[0057] Source devices may include video sources, encoding devices, and transmitters. Receiving devices may include receivers, decoding devices, and renderers.

[0058] The encoding device can be referred to as a video / image encoding device, and the decoding device can be referred to as a video / image decoding device. A transmitter can be included in the encoding device. A receiver can be included in the decoding device. The renderer can include a display unit, and the display unit can include a separate device or an external component.

[0059] Video sources can acquire video / images through processes such as video / image capture, compositing, and generation. Video sources may include video / image capture devices and / or video / image generation devices. Video / image capture devices may include, for example, one or more camera devices, video / image files including previously captured video / images, etc. Video / image generation devices may include, for example, computers, tablets, smartphones, etc., and can (electronically) generate video / images. For example, virtual video / images can be generated by computers, etc., and in this case, video / image capture processing can be replaced by processing that generates related data.

[0060] An encoding device can encode input video / images. The encoding device can perform a series of processes such as prediction, transformation, and quantization for compression and encoding efficiency. The encoded data (encoded video / image information) can be output as a bitstream.

[0061] A transmitter can transmit encoded video / image information or data, output in bitstream form, to a receiver of a receiving device via a digital storage medium or network, either as a file or a stream. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitter can include elements for generating media files according to a predetermined file format and may include elements for transmission over a broadcast / communication network.

[0062] The receiver can receive / extract bit streams and transmit them to the decoding device.

[0063] Decoding devices can decode video / images by performing a series of processes corresponding to the operations of encoding devices, such as dequantization, inverse transform, and prediction.

[0064] The renderer can render decoded video / images. The rendered video / images can then be displayed using a display unit.

[0065] Figure 2 This is a diagram schematically illustrating the configuration of a video / image encoding device.

[0066] In the following text, a video encoding device may include an image encoding device.

[0067] Reference Figure 2 The encoding device 10a can be configured to include an image segmenter 10a-10, a predictor 10a-20, a residual processor 10a-30, an entropy encoder 10a-40, an adder 10a-50, a filter 10a-60, and a memory 10a-70. The predictor 10a-20 may include an inter-frame predictor 10a-21 and an intra-frame predictor 10a-22. The residual processor 10a-30 may include a transformer 10a-32, a quantizer 10a-33, a dequantizer 10a-34, and an inverse transformer 10a-35. The residual processor 10a-30 may also include a subtractor 10a-31. The adder 10a-50 may be referred to as a reconstructor or a reconstructed block generator. According to the implementation, the image segmenter 10a-10, predictor 10a-20, residual processor 10a-30, entropy encoder 10a-40, adder 10a-50, and filter 10a-60 described above can be configured by one or more hardware components (e.g., encoder chipsets or processors). Furthermore, the memory 10a-70 may include a decoded image buffer (DPB) and can be configured by a digital storage medium. The hardware components may also include the memory 10a-70 as an internal / external component.

[0068] Image segmenters 10a-10 can split an input image (or picture, frame) input to encoding device 10a into one or more processing units.

[0069] As an example, a processing unit may be referred to as a coding unit (CU). In this case, the coding unit can be recursively split from a coding tree unit (CTU) or a maximum coding unit (LCU) according to a quadtree-binary-tritree (QTBTTT) structure. For example, a coding unit can be split into multiple coding units of greater depth based on a quadtree structure, a binary tree structure, and / or a ternary tree structure. In this case, for example, a quadtree structure can be applied first, and a binary tree structure and / or a ternary tree structure can be applied later. Alternatively, a binary tree structure can be applied first. The coding process according to this document can be performed based on the final coding unit that is no longer split. In this case, based on coding efficiency and other factors, and according to image characteristics, the maximum coding unit can be directly used as the final coding unit, or, if necessary, the coding unit can be recursively split into deeper coding units, and the optimally sized coding unit can be used as the final coding unit. Here, the coding process may include processes such as prediction, transformation, reconstruction, etc., as described later. As another example, the processing unit may also include a prediction unit (PU) or a transformation unit (TU). In this case, the prediction unit and the transform unit can be split or divided from the final coding unit described above. The prediction unit can be a unit for sample prediction, and the transform unit can be a unit for obtaining transform coefficients and / or a unit for obtaining residual signals from the transform coefficients.

[0070] In some cases, the term "unit" can be used interchangeably with terms such as "block" or "region." Generally, A block can represent a set of transform coefficients or samples consisting of M columns and N rows. Samples can typically represent pixels or pixel values, or they can represent only pixel / pixel values ​​of the luminance component, or only pixel / pixel values ​​of the chrominance component. Samples can be used as a term corresponding to pixels or pels in a picture (or image).

[0071] Subtractor 10a-31 generates a residual signal (residual block, residual sample, or residual sample array) by subtracting the prediction signal (prediction block, prediction sample, or prediction sample array) output from predictor 10a-20 from the input image signal (original block, original sample, or original sample array), and the generated residual signal is transmitted to transformer 10a-32. Predictor 10a-20 can perform prediction on the block to be processed (hereinafter referred to as the current block) and generate a prediction block including the prediction sample for the current block.

[0072] Predictors 10a-20 can determine whether to apply intra-frame prediction or inter-frame prediction on a block or CU basis. As described below in the description of each prediction mode, the predictor can generate various prediction-related information, such as prediction mode information, and pass it to entropy encoder 10a-40. The prediction-related information can be encoded by entropy encoder 10a-40 and output as a bitstream.

[0073] The intra-frame predictor 10a-22 can predict the current block by referring to samples within the current image. Depending on the prediction mode, the reference sample can be located near or far from the current block.

[0074] In intra-frame prediction, the prediction modes can include multiple non-directional modes and multiple directional modes. Non-directional modes can include, for example, DC modes and planar modes. Depending on the granularity of the prediction direction, the directional modes can include, for example, 33 directional prediction modes or 65 directional prediction modes.

[0075] However, this is just an example, and more or fewer directional prediction modes can be used depending on the settings. The intra-frame predictor 10a-22 can also determine the prediction mode to be applied to the current block by using the prediction modes applied to neighboring blocks.

[0076] Inter-frame predictors 10a-21 can obtain the predicted block of the current block based on a reference block (reference sample array) specified by motion vectors on a reference image. In this case, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted on a block, sub-block, or sample basis based on the correlation of motion information between neighboring blocks and the current block. Motion information may include motion vectors and reference image indices. Motion information may also include information about the inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). For inter-frame prediction, neighboring blocks may include spatially adjacent blocks existing in the current image and temporally adjacent blocks existing in the reference image. The reference image including the reference block and the reference image including the temporally adjacent block may be the same or different. The temporally adjacent block may be referred to as a juxtaposed reference block, juxtaposed CU (colCU), etc., and the reference image including the temporally adjacent block may be referred to as a juxtaposed image (colPic). For example, inter-frame predictors 10a-21 can construct a motion information candidate list based on neighboring blocks and generate information indicating which candidate is used to obtain the motion vector and / or reference image index of the current block. Inter-frame prediction can be performed based on various prediction modes, and for example, for skip and merge modes, the inter-frame predictor 10a-21 can use motion information from neighboring blocks as motion information for the current block. In skip mode, unlike merge mode, residual signals may not be transmitted. In motion vector prediction (MVP) mode, motion vectors from neighboring blocks can be used as motion vector predictors, and the motion vector difference can be signaled to indicate the motion vector of the current block.

[0077] Predictors 10a-20 can generate prediction signals based on various prediction methods described below. For example, the predictor can not only apply intra-frame prediction or inter-frame prediction to predict a block, but can also apply intra-frame prediction and inter-frame prediction simultaneously. This can be referred to as combined intra-frame and inter-frame prediction (CIIP). Additionally, the predictor can perform intra-frame block copying (IBC) to predict blocks. Intra-frame block copying can be used, for example, for content image / video coding including games, such as Screen Content Coding (SCC). IBC essentially performs prediction within the current frame, but can be performed in a similar manner to inter-frame prediction because it obtains a reference block within the current frame. In other words, IBC can use at least one of the inter-frame prediction techniques described in this document.

[0078] The predicted signals generated by the inter-frame predictors 10a-21 and / or the intra-frame predictors 10a-22 can be used to generate the reconstructed signal or the residual signal. The transformer 10a-32 can apply transformation techniques to the residual signal to generate transform coefficients. For example, transform techniques may include Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), Graph-Based Transform (GBT), Conditional Nonlinear Transform (CNT), etc. Here, GBT refers to the transform obtained from a graph when the relationship information between pixels is represented in the graph. CNT refers to the transform obtained by generating the predicted signal using all previously reconstructed pixels and based on it. Furthermore, the transform process can be applied to square pixel blocks of the same size or to non-square blocks of variable size.

[0079] The quantizer 10a-33 quantizes the transform coefficients and transmits them to the entropy encoder 10a-40, which encodes the quantized signal (information about the quantized transform coefficients) and outputs it as a bitstream. This information about the quantized transform coefficients can be referred to as residual information.

[0080] The quantizer 10a-33 can reorder the block-shaped quantized transform coefficients into a one-dimensional vector shape based on the coefficient scan order, and can also generate information about the quantized transform coefficients based on the one-dimensional vector shape of the quantized transform coefficients. The entropy encoder 10a-40 can perform various encoding methods, such as exponential Columbus, context-adaptive variable-length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC).

[0081] The entropy encoder 10a-40 can also encode information required for video / image reconstruction, excluding quantized transform coefficients (e.g., values ​​of syntax elements, etc.), either together or separately. The encoded information (e.g., encoded video / image information) can be transmitted as a bitstream or stored in units of the network abstraction layer. The video / image information can also include information about various parameter sets such as Adaptive Parameter Set (APS), Picture Parameter Set (PPS), Sequence Parameter Set (SPS), Video Parameter Set (VPS), etc. Additionally, the video / image information can also include general constraint information. Information and / or syntax elements signaled / transmitted later in this document can be encoded and included in the bitstream using the encoding process described above. The bitstream can be transmitted over a network or stored on a digital storage medium. Here, the network can include broadcast networks and / or communication networks, etc., and the digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmitter (not shown) for transmitting signals output from the entropy encoder 10a-40 and / or a storage device (not shown) for storing signals output from the entropy encoder 10a-40 may be configured as an internal / external element of the encoding device 10a, or the transmitter may be included in the entropy encoder 10a-40.

[0082] The quantized transform coefficients output from quantizer 10a-33 can be used to generate a prediction signal. For example, the residual signal (residual block or residual sample) can be reconstructed by applying dequantization and inverse transform to the quantized transform coefficients via dequantizer 10a-34 and inverse transform 10a-35. Adder 10a-50 can generate a reconstructed signal (reconstructed image, reconstructed block, reconstructed sample, or reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from predictor 10a-20. When there is no residual for the block to be processed, such as when a skip mode is applied, the prediction block can be used as the reconstructed block. The generated reconstructed signal can be used for intra-frame prediction of the next block to be processed within the current image, and can also be used for inter-frame prediction of the next image by filtering, as described below.

[0083] Simultaneously, during image encoding and / or reconstruction processing, luminance mapping with chroma scaling (LMCS) can be applied.

[0084] Filters 10a-60 can apply filtering to the reconstructed signal to improve subjective / objective image quality. For example, filters 10a-60 can apply various filtering methods to the reconstructed image to generate a modified reconstructed image, and the modified reconstructed image can be stored in memory 10a-70, specifically in the DPB of memory 10a-70. Various filtering methods can include, for example, deblocking filtering, sample adaptive offset (SAO), adaptive loop filtering, bilateral filtering, etc. Filters 10a-60 can generate various filter-related information and transmit it to entropy encoder 10a-90, as described later in the description of each filtering method. The filter-related information can be encoded by entropy encoder 10a-90 and output as a bitstream.

[0085] The modified reconstructed images transferred to the memories 10a-70 can be used as reference images in the inter-frame predictors 10a-80. When inter-frame prediction is applied in this way, the encoding device can avoid prediction mismatch between the encoding device 10a and the decoding device, and can also improve encoding efficiency.

[0086] The DPB of memory 10a-70 can store a modified reconstructed image that will be used as a reference image in inter-frame predictor 10a-21. Memory 10a-70 can store motion information of blocks from which motion information is derived (or encoded) within the current image, and / or motion information of blocks within previously reconstructed images. The stored motion information can be transmitted to inter-frame predictor 10a-21 as motion information for spatially adjacent blocks or temporally adjacent blocks. Memory 10a-70 can store reconstructed samples of reconstructed blocks within the current image and transmit them to intra-frame predictor 10a-22.

[0087] Figure 3 This is a diagram schematically illustrating the configuration of a video / image decoding device.

[0088] Reference Figure 3The decoding device 10b can be configured including an entropy decoder 10b-10, a residual processor 10b-20, a predictor 10b-30, an adder 10b-40, a filter 10b-50, and a memory 10b-60. The predictor 10b-30 may include an inter-frame predictor 10b-31 and an intra-frame predictor 10b-32. The residual processor 10b-20 may include a dequantizer 10b-21 and an inverse transformer 10b-21. According to an embodiment, the entropy decoder 10b-10, residual processor 10b-20, predictor 10b-30, adder 10b-40, and filter 10b-50 described above can be configured by a single hardware component (e.g., a decoder chipset or processor). Furthermore, the memory 10b-60 may include a decoded picture buffer (DPB) and can be configured by a digital storage medium. The hardware component may also include the memory 10b-60 as an internal / external component.

[0089] When a bitstream including video / image information is input, the decoding device 10b can respond to the video / image information in it. Figure 2 The image is reconstructed through processing within the encoding apparatus. For example, the decoding apparatus 10b can obtain units / blocks based on block splitting information obtained from the bitstream. The decoding apparatus 10b can perform decoding using processing units applied in the encoding apparatus. Therefore, the processing unit used for decoding can be, for example, an encoding unit, and the encoding unit can be split from encoding tree units or maximally encoded units according to a quadtree structure, binary tree structure, and / or ternary tree structure. One or more transform units can be obtained from the encoding unit. Furthermore, the reconstructed image signal decoded and output by the decoding apparatus 10b can be played back by a playback device.

[0090] Decoding device 10b can receive data in bitstream form from... Figure 2 The signal output by the encoding device, and the received signal, can be decoded by the entropy decoder 10b-10. For example, the entropy decoder 10b-10 can parse the bitstream to obtain the information required for image reconstruction (or picture reconstruction) (e.g., video / image information). The video / image information may also include information about various parameter sets such as adaptive parameter sets (APS), picture parameter sets (PPS), sequence parameter sets (SPS), video parameter sets (VPS), etc. In addition, the video / image information may also include general constraint information.

[0091] The decoding device can additionally decode the image based on information about the parameter set and / or general constraint information. Information and / or syntax elements notified / received by signals, as described later in this document, can be decoded and obtained from the bitstream through the decoding process. For example, the entropy decoder 10b-10 can decode the information within the bitstream based on encoding methods such as exponential Golomb coding, CAVLC, CABAC, etc., and output the values ​​of the syntax elements required for image reconstruction, as well as the values ​​of the quantized transform coefficients with respect to the residuals.

[0092] More specifically, the CABAC entropy decoding method can receive bins corresponding to each syntax element from the bitstream, determine a context model by using information about the decoded target syntax element and decoding information of adjacent target blocks and decoded target blocks, or information about symbols / bins decoded in previous steps, and generate symbols corresponding to the values ​​of each syntax element by performing arithmetic decoding of bins to predict the occurrence probability of bins based on the determined context model. In this case, the CABAC entropy decoding method can update the context model after determining the context model by using information about the decoded symbols / bins for the context model of the next symbol / bin. The prediction information in the information decoded by the entropy decoder 10b-10 can be provided to the predictor 10b-30, and the residual information, i.e., the quantized transform coefficients and related parameter information, obtained by the entropy decoder 10b-10, can be input to the dequantizer 10b-21.

[0093] Additionally, filtering information from the information decoded by the entropy decoder 10b-10 can be provided to the filter 10b-50. Meanwhile, the receiver (not shown) receiving the signal output from the encoding device can be configured as an internal / external element of the decoding device 10b, or the receiver can be a component of the entropy decoder 10b-10. Furthermore, the decoding device according to this document can be referred to as a video / video / image decoding device, and this decoding device can be divided into an information decoder (video / video / image information decoder) and a sample decoder (video / video / image sample decoder). The information decoder can include the entropy decoder 10b-10, and the sample decoder can include at least one of a dequantizer 10b-21, an inverse transformer 10b-22, a predictor 10b-30, an adder 10b-40, a filter 10b-50, and a memory 10b-60.

[0094] Dequantizer 10b-21 can dequantize quantized transform coefficients to output transform coefficients. Dequantizer 10b-21 can reorder the quantized transform coefficients in the form of two-dimensional blocks. In this case, the reordering can be performed based on the coefficient scan order executed by the encoding device. Dequantizer 10b-21 can dequantize the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain the transform coefficients.

[0095] The inverse transformer 10b-22 performs an inverse transformation on the transformation coefficients to obtain the residual signal (residual block, residual sample array).

[0096] The predictor can perform predictions on the current block and generate a prediction block for the current block, including the predicted samples.

[0097] The predictor can determine whether to apply intra-frame prediction or inter-frame prediction to the current block based on prediction-related information output from the entropy decoder 10b-10, and can determine a specific intra-frame prediction mode / inter-frame prediction mode.

[0098] The predictor can generate a predicted signal based on various prediction methods described below. For example, the predictor can not only apply intra-frame prediction or inter-frame prediction to predict a block, but also apply both intra-frame prediction and inter-frame prediction simultaneously. This can be referred to as combined intra-frame and inter-frame prediction (CIIP). Additionally, the predictor can perform intra-frame block copying (IBC) to predict blocks. Intra-frame block copying can be used, for example, for content image / video coding including games, such as Screen Content Coding (SCC). IBC essentially performs prediction within the current frame, but can be performed in a similar manner to inter-frame prediction because it obtains a reference block within the current frame. In other words, IBC can use at least one of the inter-frame prediction techniques described in this document.

[0099] The intra-frame predictor 10b-32 can predict the current block by referring to samples within the current image. Depending on the prediction mode, the reference sample can be located near or far from the current block.

[0100] In intra-frame prediction, the prediction mode can include multiple non-directional modes and multiple directional modes. The intra-frame predictor 10b-32 can also determine the prediction mode applied to the current block by using the prediction modes applied to neighboring blocks.

[0101] The inter-frame predictor 10b-31 can obtain the predicted block for the current block based on a reference block (reference sample array) specified by motion vectors on a reference image. In this case, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted on a block, sub-block, or sample basis based on the correlation of motion information between neighboring blocks and the current block. Motion information may include motion vectors and reference image indices. Motion information may also include information about the inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.).

[0102] For inter-frame prediction, neighboring blocks can include spatially adjacent blocks existing in the current image and temporally adjacent blocks existing in the reference image. For example, inter-frame predictor 10b-31 can construct a motion information candidate list based on neighboring blocks and obtain the motion vector of the current block and / or the reference image index based on the received candidate selection information. Inter-frame prediction can be performed based on various prediction modes, and prediction-related information can include information indicating the inter-frame prediction mode of the current block.

[0103] Adder 10b-40 generates the reconstructed signal (reconstructed image, reconstructed block, reconstructed sample array) by adding the obtained residual signal to the prediction signal (predicted block, predicted sample array) output from predictor 10b-30. When there is no residual for the block to be processed, such as when a skip mode is applied, the predicted block can be used as the reconstructed block.

[0104] Adder 10b-40 can be referred to as a rebuilder or rebuild block generator.

[0105] The generated reconstructed signal can be used for intra-frame prediction of the next block to be processed within the current image, and as described below, it can also be output by filtering or used for inter-frame prediction of the next image.

[0106] Meanwhile, luminance mapping with chroma scaling (LMCS) can be applied during image decoding processing.

[0107] Filters 10b-50 can apply filtering to the reconstructed signal to improve subjective / objective image quality. For example, filters 10b-50 can apply various filtering methods to the reconstructed image to generate a modified reconstructed image, and the modified reconstructed image can be transferred to memory 60, specifically to the DPB of memory 10b-60. Various filtering methods may include, for example, deblocking filtering, adaptive sample shifting, adaptive loop filtering, bilateral filtering, etc.

[0108] The (modified) reconstructed image stored in the DPB of memory 10b-60 can be used as a reference image in inter-frame predictor 10b-31. Memory 10b-60 can store motion information of blocks from which motion information is derived (or decoded) within the current image, and / or motion information of blocks within previously reconstructed images. The stored motion information can be transmitted to inter-frame predictor 10b-31 as motion information for spatially adjacent blocks or temporally adjacent blocks. Memory 10b-60 can store reconstructed samples of reconstructed blocks within the current image and transmit them to intra-frame predictor 10b-32.

[0109] In this specification, the embodiments described in the predictor 10b-30, dequantizer 10b-21, inverse transformer 10b-22 and filter 10b-50 of the decoding device 10b can also be applied in the same or corresponding manner to the predictor 10a-20, dequantizer 10a-34, inverse transformer 10a-35 and filter 10a-60 of the encoding device 10a.

[0110] As described above, prediction is performed during video encoding to improve compression efficiency. This allows the generation of a prediction block, including predicted samples, for the current block, which is the target block for encoding. Here, the prediction block includes predicted samples in the spatial domain (or pixel domain). The prediction block is obtained by both the encoding and decoding units in the same manner, and the encoding unit can improve image encoding efficiency by signaling information about the residual between the original block and the prediction block (residual information) to the decoding unit, rather than the original sample values ​​of the original block themselves. The decoding unit can obtain a residual block including residual samples based on the residual information, generate a reconstruction block including reconstructed samples by combining the residual block and the prediction block, and generate a reconstructed image including the reconstruction block.

[0111] Residual information can be generated through transformation and quantization processes.

[0112] For example, the encoding device can obtain a residual block between the original block and the prediction block, perform a transform process on the residual samples (residual sample array) included in the residual block to obtain transform coefficients, and signal the relevant residual information to the decoding device (via a bitstream) by obtaining the transform coefficients quantized through a quantization process on the transform coefficients. Here, the residual information may include information such as the value information of the quantized transform coefficients, position information, transform technique, transform kernel, quantization parameters, etc. The decoding device can perform a dequantization / inverse transform process based on the residual information and obtain residual samples (or residual blocks). The decoding device can generate a reconstructed image based on the prediction block and the residual block. The encoding device can also obtain residual blocks by performing dequantization / inverse transform on the quantized transform coefficients for inter-frame prediction reference of subsequent images, and generate a reconstructed image based on this.

[0113] <Video Coding for Machines (VCM)>

[0114] With recent developments in various industrial fields such as surveillance, intelligent transportation, smart cities, smart industries, and smart content, the amount of image or feature map data used by machines is increasing. In contrast, since the conventional image compression methods currently in use are technologies developed by considering the human visual characteristics recognized by viewers, which include unnecessary information, performing machine tasks is inefficient. For example, an image based on the vision of a viewer may have a higher resolution than an image based on the data usage vision of a machine (e.g., a feature map). Therefore, research on video codec technologies for efficiently compressing feature maps is required to perform machine tasks.

[0115] Video Coding for Machines (VCM) technology is being discussed by the Moving Picture Experts Group (MPEG), an international standardization organization for multimedia coding. VCM is an image or feature map coding technology based on the data usage vision of machines (machine vision), rather than the vision of human viewers. In this document, a feature map may be referred to as a feature map, and a feature may be referred to as a feature.

[0116] Figures 4a to 4d is an exemplary diagram showing a VCM encoder and a VCM decoder.

[0117] Refer to <000027—>, a VCM encoder 100a and a VCM decoder 100b are shown.

[0118] When the VCM encoder 100a encodes a video and / or a feature map and transmits it as a bitstream, the VCM decoder 100b can decode the bitstream and output it. In this case, the VCM decoder 100b can output one or more videos and / or feature maps. For example, the VCM decoder 100b can output a first feature map for analysis using a machine, or output a first image for a user to view. The first image may have a higher resolution than the first feature map.

[0119] Refer to Figure 4b , a feature extractor for extracting a feature map may be connected to the front end of the VCM encoder 100a.

[0120] The VCM encoder 100a may include a feature encoder.

[0121] The VCM decoder 100b may include a feature decoder and a video reconstructor. The feature decoder can decode a feature map from the bitstream and output a first feature map for analysis using a machine. The video reconstructor can reconstruct a first image from the bitstream and output the first image for a user to view.

[0122] Reference Figure 4c A feature extractor for extracting feature maps can be connected to the front end of the VCM encoder 100a. The VCM encoder 100a may include a feature encoder.

[0123] VCM decoder 100b may include a feature decoder. The feature decoder can decode feature maps from a bitstream and output a first feature map for machine analysis. In other words, the bitstream may be encoded only as feature maps, rather than images. To explain further, a feature map may be data that includes information about features used for a specific task of image-based machine processing.

[0124] Reference Figure 4d The feature extractor can be connected to the front end of the VCM encoder 100a.

[0125] The VCM encoder 100a may include a feature converter and a video encoder. The video encoder may be... Figure 2 The encoding device 10a shown is shown.

[0126] Figure 4d The VCM decoder 100b shown may include a video decoder and an inverse converter. The video decoder may be... Figure 3 The decoding device 10b shown is illustrated.

[0127] Figure 5 A block diagram of an encoding apparatus according to an embodiment of the present disclosure is shown.

[0128] The encoding apparatus 10a according to embodiments of this disclosure can receive an image (video) as input to perform temporal and spatial resampling, extract regions of interest (ROIs), and generate and output a bitstream through internal encoding. The encoding apparatus 10a may include a temporal resampling executor 510, a spatial resampling executor 520, a ROI-based processor 530, and an internal encoding executor 540. In various embodiments, the specific details may be varied. Figure 5The order of components included in the encoding apparatus 10a shown is illustrated, and some components may be omitted. According to an embodiment, the temporal resampling executor 510 and / or the spatial resampling executor 520 may be omitted, and when the temporal resampling executor 510 and / or the spatial resampling executor 520 are omitted, the image processed by the region-of-interest (ROI)-based processor 530 can become the input to the internal encoding executor 540. For example, when the temporal resampling executor 510 and the spatial resampling executor 520 are omitted, one or more ROIs can be directly extracted from the ROI-based processor 530, and the extracted ROIs can be input to the internal encoding executor 540, which can then perform encoding to generate a bitstream. In this disclosure, for ease of explanation, the encoding apparatus 10a may be referred to as an encoder or encoder, and the decoding apparatus 10b may be referred to as a decoder or decoder.

[0129] The temporal resampling executor 510 can receive an image as input, perform sampling frame by frame, and output an image containing some of the sampled frames. The temporal resampling executor 510 can change the frame rate of some frames. According to one embodiment, the temporal resampling executor 510 can perform the same level of sampling on the entire image. In other words, the temporal resampling executor 510 can resample the entire image at the same frame rate. According to another embodiment, the temporal resampling executor 510 can perform temporal resampling by applying different levels of sampling to a series of frame groups for each unit within the image. In other words, the temporal resampling executor 510 can perform resampling at a variable frame rate on a series of frame groups.

[0130] In this implementation, the temporal resampling executor 510 can perform sampling at different sampling rates on the image based on region of interest (ROI) processing and the image on the non-ROI processing. For example, the temporal resampling executor 510 can perform sampling on the image based on ROI processing at a first sampling rate and can perform sampling on the image on the non-ROI processing at a second sampling rate.

[0131] According to the implementation, the temporal resampling unit 510 can transmit information used in the temporal resampling process (in this example, the temporal sampling rate, etc.) to the decoder. The decoder can perform decoding on the image and the non-region of interest (ROI) images based on ROI processing, and then perform upsampling separately by parsing each sampling rate corresponding to the ROI-based image and the non-ROI image. In this case, the sampling rate applied to the ROI-based image and the sampling rate applied to the non-ROI image can be different.

[0132] The spatial resampling executor 520 can receive an input image or an image resampled at execution time as input, and output an image whose spatial resolution is varied on a per-frame or series of frames basis. The spatial resampling executor 520 can change the resolution of the regions of interest (ROIs) included in each frame of the input image. In one implementation, the spatial resampling executor 520 can change only the resolution of the ROIs. When multiple ROIs exist in a frame, the spatial resampling executor 520 can change the resolution of each ROI. Information about each ROI can include the changed resolution information. For example, a first scaling factor can be applied to a first ROI, and a second scaling factor can be applied to a second ROI. The scaling factor can be stored using at least one of the following: scaling factor, scaling factor numerator, scaling factor denominator, size (%) parameter, etc.

[0133] In another embodiment, the spatial resampling executor 520 can perform sampling by applying different resolutions to the region of interest (ROI) and non-ROI regions respectively. For example, the spatial resampling executor 520 can modify the image processed based on the ROI at a first resolution and can modify the image of the non-ROI region at a second resolution.

[0134] According to the implementation, the spatial resampling executor 520 can transmit information used in the spatial resampling process (in this example, the spatial sampling rate per frame and / or the spatial sampling rate per series of frames, etc.) to the decoder. After decoding the bitstream, the decoder can separately parse the resolution information of the image based on the region of interest (ROI) processing and the non-ROI images, and change the resolution. In this case, the resolution applied to the image based on the ROI processing and the resolution applied to the non-ROI can be different.

[0135] The region-of-interest (ROI)-based processor 530 can receive an input image, an image to which it has undergone temporal resampling, or an image to which it has undergone both temporal and spatial resampling as input. It extracts ROIs within a series of frames or units within each frame and outputs an image processed based on the extracted ROIs. The ROI-based processor 530 may include a frame analysis unit 531, an ROI selection unit 532, and an ROI-based image processing unit 533. The ROI-based processor 530 can extract ROIs from multiple frames included in the input image and establish a correspondence for the same ROIs (e.g., detected objects) for each frame. When the same object (ROI) is detected in a series of consecutive frames, the ROI-based processor 530 can set the same ROIs for the consecutive frames. The ROI-based processor 530 can transmit information about the ROIs to the decoder. Information about the ROIs can be stored for each frame. The ROI-based processor 530 can establish a correspondence for multiple frames including the same ROIs and store changes in the position and size of the ROIs. For example, the ROI-based processor 530 can signal linear motion, affine motion, etc., of the ROIs existing within units of a series of frames and transmit this signal to the decoder. The region-of-interest (ROI)-based processor 530 can signal position change values ​​for linear motion and signal affine motion using parameters defined according to the motion type. For example, the ROI-based processor 530 can transmit affine motion information, including scaling, rotation, and shearing changes, to the decoder. The ROI-based processor 530 can also transmit a flag (e.g., LinearMotion_flag) to the decoder to identify the presence of affine motion.

[0136] The region-of-interest (ROI)-based processor 530 can change the resolution by taking the ROI into account. According to an implementation, when multiple ROIs exist in a frame, the ROI-based processor 530 can change the resolution of each ROI. Information about each ROI may include the changed resolution information. For example, a first scaling factor may be applied to a first ROI, and a second scaling factor may be applied to a second ROI.

[0137] The region-of-interest (ROI)-based processor 530 can transmit information used during ROI-based processing (in this example, ROI ID information, ROI size information, packing information, temporal sampling rate, spatial resolution, etc.) to the decoder. (More on this later...) Figure 6 The text describes a detailed description of a region-of-interest (ROI) based processor 530.

[0138] The internal encoding executor 540 can receive an input image, or a portion or all of an image to which region of interest processing and temporal / spatial resampling are performed, as input, and perform image encoding to generate a bitstream. The internal encoding executor 540 may include a downsampling execution unit 541, an image encoding execution unit 542, and a region of interest information encoding execution unit 543. According to embodiments, the internal encoding executor 540 may use a 2D video encoder (e.g., AVC / H.264, HEVC / H.265, VVC / H.266, AV1, VP9, ​​etc.), and may use a 2D video encoder including one or more convolutional layers. According to embodiments, encoding may be performed after the color space of the input image to the internal encoding executor 540 is transformed to a color space such as YUV420, YUV444, etc. In this case, according to embodiments, the internal encoding executor 540 may perform the transformation using a transformation method defined by an agreement between the encoder and decoder, and transmit the color space transformation information to the decoder. (Later in...) Figures 7a to 7c The document describes a detailed description of the internal code executor 540.

[0139] The encoding apparatus 10a according to embodiments of this disclosure can improve the accuracy of deep neural networks by extracting and transmitting regions of interest (ROIs) from input images. Furthermore, the encoding apparatus 10a improves the encoding efficiency transmitted to the decoding apparatus 10b by extracting ROIs and performing temporal resampling together. The encoding apparatus 10a can compress information such as the position, motion, etc., of each ROI in each group of pictures (GOP) or a series of frames and transmit it to the decoding apparatus 10b.

[0140] Figure 6 This describes the detailed operation of a region-of-interest-based processor according to an embodiment of this disclosure.

[0141] The region-of-interest (ROI) based processor 530 of the encoding apparatus 10a according to embodiments of this disclosure may include a frame analysis unit 531, an ROI selection unit 532, and an ROI-based image processing unit 533. The frame analysis unit 531 may include an analysis region selection module 531a and an ROI candidate extraction module 531b. The ROI selection unit 532 may include an ROI selection module 532a and a corresponding ROI selection module 532b. In various embodiments, some detailed components of the ROI-based processor 530 may be omitted, and the order of the components may be changed.

[0142] The frame analysis unit 531 can receive an input image as input and output region of interest candidates. The input image can be the original image or an image that has been temporally and / or spatially resampled according to various embodiments.

[0143] The frame analysis unit 531 can obtain information about one or more region-of-interest (ROI) candidates (e.g., the number of RIOs, their locations, scores, etc.) within each frame of the input image or an image to which temporal / spatial resampling has been performed. The scores of the RIOs can be calculated according to predetermined criteria. For example, the scores can be calculated based on the importance or weight of the RIOs. As an example, when extracting RIO candidates using a deep neural network model learned for object detection or object tracking, the scores of the RIOs can be calculated proportionally to the object detection accuracy included in the RIOs.

[0144] The analysis region selection module 531a can select analysis regions for the input image on a unit of one or more frames. The analysis region selection module 531a can select regions from all frames where frame analysis will be performed, and can perform frame analysis only on some regions within the image. The analysis region selection module 531a can select the same regions for all frames within the image. Alternatively, the analysis region selection module 531a can select the same regions within a series of frame groups within the image. The regions selected for each of the series of frame groups can be different. The analysis region selection module 531a can transmit information about some regions selected as analysis targets to the decoder. According to an embodiment, the analysis region selection module 531a may not transmit regions where frame analysis will not be performed to the decoder.

[0145] The region of interest (ROI) candidate extraction module 531b can extract ROI candidates for the selected target region of analysis on a frame-by-frame basis, or on a series of frames. The size of the ROI candidate within each frame can be a multiple of 4 in both height and width. According to an embodiment, the ROI candidate extraction module 531b can use a deep neural network including one or more learned convolutional layers. According to an embodiment, the deep neural network can be a neural network learned for object detection or a neural network learned for object tracking. The ROI candidate extraction module 531b can extract ROI candidates on a frame-by-frame basis or on a series of frames. When the detection accuracy of the detected object (ROI candidate) for each frame or series of frames is low, the ROI candidate extraction module 531b can exclude it from the ROI candidates.

[0146] The region of interest (ROI) selection unit 532 can select a ROI from the ROI candidates output from the frame analysis unit 531, identify the correspondence between multiple ROIs, and select the corresponding ROI. The selected ROI can be transmitted to the internal encoding executor 540 for image encoding.

[0147] The region of interest (ROI) selection module 532a can identify the region to be encoded from the ROI candidate as the ROI. The non-ROI image can be a region within a frame other than the ROI, and the portion corresponding to the non-ROI can be filled with neighboring pixel values ​​and / or intermediate values. According to the implementation, the ROI selected from the input image can be different for each frame, and can be different for each in a series of frame groups.

[0148] The corresponding region of interest (ROI) selection module 532b can identify the correspondence between ROIs included in frames within an image and select the corresponding ROI based on the correspondence. In an implementation, when consecutive frames include the same ROI (e.g., a detected object), the corresponding ROI selection module 532b can select the corresponding ROI on a series of frame groups. In an example, when the corresponding ROI selection module 532b is executed on a series of frame groups, the ROI candidate (detected object) of the first input frame is used as a candidate to select the corresponding ROI based on the correspondence with the ROI candidate of subsequent frames. According to an implementation, the correspondence between the ROI candidate of the first frame and the ROI candidate of subsequent frames can be calculated based on the pixel similarity between the entire region and / or some regions of each detected object. In an example, the pixel similarity can be calculated through histogram comparison and / or cross-correlation of pixel values, etc.

[0149] The region of interest (ROI) selection module 532b can select the corresponding ROI after downsampling or upsampling the sizes of all ROI candidates to a specific size. According to the implementation, the specific size can be determined based on the image resolution, and / or the specific size can be determined as the minimum ROI size by comparing the sizes of the ROI candidates extracted for each frame.

[0150] The corresponding region of interest (ROI) selection module 532b can assign an ID to the selected ROI. The corresponding ROI selection module 532b can assign an ID to each unit within which a corresponding ROI is selected. When the regions of ROI candidates extracted from the ROI candidate extraction module 531b overlap within a frame, the corresponding ROI can be classified as a single ROI if the overlapping area exceeds a specific threshold. In this case, the threshold can be determined based on the frame size and / or the size of the ROI candidates.

[0151] When no region corresponding to a region of interest in a previous frame exists in the region of interest of the current frame, the region of interest selection module 532b can assign a new ID to the corresponding region of interest. When a corresponding region of interest from a previous frame exists in the region of interest of the current frame in sequence, the region of interest selection module 532b can assign the existing ID of the corresponding region of interest from the previous frame to the region of interest of the current frame. In various embodiments, the region of interest selection module 532b can use various methods to calculate the correspondence.

[0152] According to the implementation method, when the region of interest candidate extraction module 531b uses a neural network for object tracking learning, the operation of the corresponding region of interest selection module 532b can be omitted because the object to be tracked corresponds to the corresponding region of interest.

[0153] According to various embodiments, the region-of-interest (ROI)-based image processing unit 533 can receive selected ROI information within an image as input and use the ROI selected for each frame to generate an image based on ROI processing. In an example, the ROI-based image processing unit 533 can pack ROIs and generate each frame in which ROI packing is performed. The ROI-based image processing unit 533 can fill non-ROIs within a frame with specific values ​​(e.g., intermediate values) and generate frames in which non-ROIs are filled with specific values.

[0154] According to the implementation, the region-of-interest (ROI) based processor 530 can transmit information used in the ROI extraction process (in the example, ROI identification (ID) information, ROI size information, packing information, etc.) to the decoder.

[0155] Figure 7a , Figure 7b and Figure 7c This describes the detailed operation of the internal coding executor according to the embodiments of this disclosure.

[0156] The internal encoding executor 540 of the encoding apparatus 10a according to embodiments of the present disclosure can encode images based on region of interest (ROI) processing, non-ROI images, and ROI information to generate a bitstream. The internal encoding executor 540 according to embodiments of the present disclosure may include a downsampling execution unit 541, an image encoding execution unit 542, and an ROI information encoding execution unit 543. The ROI information encoding execution unit 543 may include an ROI selection module 543a for performing information encoding, a frame selection module 543b for performing information encoding, and an information encoding execution module 543c. In various embodiments, some detailed components of the internal encoding executor 540 may be omitted, and the order of the components may be changed.

[0157] The internal encoding executor 540 can perform encoding on each frame of the image based on region of interest (ROI) processing, and can also perform encoding only on some frames within a series of frames of non-ROI images. The internal encoding executor 540 can perform downsampling before encoding, and then encode information about the ROI within the image based on ROI processing. The bitstream output by the image encoding execution unit 542 and the ROI information encoding execution unit 543 can be multiplexed (multiplexed) to generate a single bitstream. (More details to follow...) Figure 8 The text describes a detailed description of the region of interest information encoding execution unit 543.

[0158] The downsampling execution unit 541 can perform downsampling on the image based on region of interest processing and the non-region of interest image at different sampling rates before performing image encoding. When performing downsampling, the downsampling execution unit 541 can send a signal to the decoder to notify it of the sampling rate information.

[0159] The image encoding execution unit 542 may use a video encoder (e.g., AVC / H.264, HEVC / H.265, VVC / H.266, AV1, VP9, ​​etc.) and may use a 2D video encoder including one or more convolutional layers. The image encoding execution unit 542 may use the same video encoder for images based on region of interest (ROI) processing and images not based on ROI, or it may use different video encoders.

[0160] The region of interest (ROI) information encoding execution unit 543 can encode information (e.g., position, motion, mapping information, etc.) about the regions of interest included in all or some frames of the input image. The ROI information encoding execution unit 543 can use some of the processing (e.g., entropy coding, etc.) of the video encoder of the image encoding execution unit 542.

[0161] Reference Figure 7a The internal encoding executor 540 can apply the downsampling execution unit 541 and the image encoding execution unit 542 to the image of the non-region of interest. (See reference...) Figure 7b The internal encoding executor 540 can apply the downsampling execution unit 541 and the image encoding execution unit 542 to the image based on region of interest processing, and apply the region of interest information encoding execution unit 543 separately to information about the region of interest to generate a bitstream. (See reference...) Figure 7c The internal encoding executor 540 can first perform downsampling execution unit 541 on the image based on region of interest processing, apply region of interest information encoding execution unit 543 to the information about the region of interest, and apply image encoding execution unit 542 to the image based on region of interest processing to generate a bitstream.

[0162] Figure 8 This describes the detailed operation of the region of interest information encoding execution unit according to the embodiments of this disclosure.

[0163] The region of interest (ROI) information encoding execution unit 543 included in the internal encoding executor 540 of the encoding apparatus 10a according to embodiments of the present disclosure can encode information about the region of interest (ROI) of an image based on ROI processing to generate a bitstream. The ROI information encoding execution unit 543 according to embodiments of the present disclosure may include an ROI selection module 543a for performing information encoding, a frame selection module 543b for performing information encoding, and an information encoding execution module 543c. Some detailed components of the ROI information encoding execution unit 543 may be omitted, and the order of the components may be changed.

[0164] The region of interest (ROI) selection module 543a, used for performing information encoding, can receive an image based on ROI processing as input, and select a region of interest (ROI) for information encoding from the selected ROIs. The information encoding target may include motion information of the ROI, compensation parameters, etc.

[0165] The frame selection module 543b for performing information encoding can receive an image based on region of interest processing and / or a region of interest for performing information encoding as input, and select a frame for performing information encoding within the frame of the image.

[0166] The information encoding execution module 543c can encode information from all regions of interest (ROIs) within a selected frame. Alternatively, the information encoding execution module 543c can encode only the information from the ROIs selected within the selected frame by the ROI selection module 543a. The information encoding execution module 543c can also generate a bitstream by encoding the information from the ROIs selected by the ROI selection module 543a within the frame selected by the frame selection module 543b.

[0167] According to the implementation, when performing temporal resampling on the input image, the region of interest (ROI) information encoding execution unit 543 can encode ROI information for some frames at a period less than the temporal sampling rate. In an example, when performing temporal resampling periodically using a sampling rate n (n is an integer greater than or equal to 1), the ROI information encoding execution unit 543 can encode information about the ROI within the corresponding frame at every m (m is an integer greater than or equal to 1 and less than or equal to n) frame periods. In another example, when performing temporal resampling non-periodically across the entire image, the ROI information encoding execution unit 543 can encode ROI information within the corresponding frame at every k / p (p is an integer greater than or equal to 1 and less than k) periods when the frame period between any two adjacent frames of the image for which temporal resampling is performed is k (k is an integer greater than or equal to 1). The ROI information encoding execution unit 543 can also encode ROI information within the corresponding frame at every u (u is an integer greater than or equal to 1) frame periods.

[0168] Frame selection processing allows encoding of region of interest information only for certain regions within a frame that will be encoded.

[0169] According to the implementation, when temporal resampling of the input image is not performed, the region of interest (ROI) information encoding execution unit 543 may encode only the information of one or more specific ROIs. The ROI information encoding execution unit 543 may encode only the information of ROIs between specific frames. According to the implementation, when the difference in representative brightness values ​​between each frame is greater than or equal to a specific threshold, only the information of the ROIs between corresponding frames may be encoded. When the difference in representative brightness values ​​of corresponding ROIs within each frame between each frame is greater than or equal to a specific threshold, only the information of the ROIs between corresponding frames may be encoded.

[0170] The region of interest (ROI) information encoding execution unit 543 can perform ROI information encoding on the corresponding ROI within each frame between two frames to which ROI information encoding will be performed. The ROI information encoding execution unit 543 can encode information about the position of the ROI and the position difference between two ROIs. In this case, the position can be the upper-left coordinate of the ROI. The ROI information encoding execution unit 543 can encode the position information (coordinates and difference) of the ROI as well as motion parameters. In the example, the motion parameters can be parameters used to express affine motion. The ROI information encoding execution unit 543 can use the position information (coordinates and difference) to encode information about translational motion and use the motion parameters to encode information about scaling, shearing, and rotation.

[0171] According to the implementation method, affine motion can be represented by Equation 1.

[0172] [Equation 1]

[0173]

[0174] Here, a, b, c, and d represent affine parameters. Trans_x and Trans_y represent translational motion information about the x-axis and y-axis, respectively. x and y represent the position of the pixel value within the current region of interest, and x' and y' represent the position of the pixel value to which the affine motion is applied.

[0175] The region of interest (ROI) information encoding execution unit 543 can transmit the position information and motion parameters of the ROI to the decoder. According to an embodiment, the ROI information encoding execution unit 543 can encode compensation parameter information between two ROIs in the following situations: 1) when the difference between the representative values ​​of the luminance values ​​between two frames is greater than a specific threshold and / or 2) when the difference between the representative values ​​of the luminance values ​​of the corresponding ROIs is greater than a specific threshold. In this case, information regarding whether to apply compensation parameters between ROIs can be signaled on a frame-by-frame or region-by-ROI basis. In various embodiments, the specific threshold for the difference between the representative values ​​of the luminance values ​​between frames and the specific threshold for the difference between the representative values ​​of the luminance values ​​of the corresponding ROIs can be different.

[0176] According to the implementation, the compensation parameter can be defined by two parameters (e.g., α and β) and can be expressed as Equation 2 below.

[0177] [Equation 2]

[0178]

[0179] Here, RoI i (Past) represents the i-th region of interest in the previous frame in time between two frames, and RoI i (Future) represents the i-th region of interest in the next frame in time.

[0180] Figure 9 A block diagram of a decoding apparatus according to an embodiment of the present disclosure is shown.

[0181] According to embodiments of this disclosure, the decoding apparatus 10b can receive a bitstream from the encoding apparatus 10a and perform decoding to output a recovered image (video). The decoding apparatus 10b may include an internal decoding executor 910, a region-of-interest (ROI) based restorer 920, a spatial restorer 930, a temporal restorer 940, and a post-processing filter executor 950. The machine vision task executor 1000 can perform a machine vision task on the image recovered by the decoding apparatus 10b to output a result. When the machine vision task is object detection, object tracking, or object segmentation, a metric for measuring the accuracy of the task execution result can be used. For example, the metric for measuring the accuracy of the machine vision task execution result may be mAP for object detection results, MOTA for object tracking results, etc. According to embodiments, the decoding apparatus 10b may also include the machine vision task executor 1000. According to embodiments, Figure 9 The order of the components can be changed. According to an embodiment, the order of the region-of-interest restorer 920, the spatial restorer 930, and the temporal restorer 940 can be based on… Figure 5 The encoding device 10a performs each corresponding process (the encoder 920 based on the region of interest, the spatial sampling actuator 930, and the temporal sampling actuator 940) in the reverse order during the encoding processing of the original image.

[0182] The internal decoding executor 910 can receive a bitstream as input, perform image decoding, and generate a restored image. The internal decoding executor 910 may include an image decoding execution unit 911, an upsampling execution unit 912, and a region of interest (ROI) information decoding execution unit 913. According to embodiments, the internal decoding executor 910 can use a 2D video decoder (AVC / H.264, HEVC / H.265, VVC / H.266, AVI, VP9, ​​etc.), and can use a 2D video decoder including one or more convolutional layers. According to embodiments, the internal decoding executor 910 can perform subsequent image decoding processing after implicitly and / or explicitly transforming the color space of the restored image to another color space. For example, when the color space of the restored image is one of YUV420, YUV444, etc., instead of RGB444, subsequent image decoding processing can be performed after implicitly and / or explicitly transforming it to the RGB444 space.

[0183] The region-of-interest (ROI)-based restorer 920 can restore an image processed based on ROI by using ROI-based processing information (e.g., ROI ID information, ROI size information, packing information, etc.) transmitted from the encoding device 10a and the decoded image. The ROI-based restorer 920 may include an image reconfiguration unit 921.

[0184] According to the implementation, the spatial restoration executor 930 can perform spatial restoration on the decoded image or the decoded and region-of-interest-based restored image by using information transmitted from the encoder that is used in the spatial resampling process (e.g., spatial sampling rate per frame and / or spatial sampling rate per series of frames).

[0185] According to an implementation, the time recovery executor 940 can perform execution using information (e.g., time sampling rate, etc.) transmitted from the encoding device 10a that is used in the time resampling process. Figure 9 The image execution time is restored for some or all of the aforementioned processing.

[0186] According to the implementation method, the post-processing filter actuator 950 can perform... Figure 9 The post-processing filter executor 950 performs filtering on some or all of the images processed described above. According to an embodiment, the post-processing filter executor 950 may use a fixed filter or multiple filters agreed upon between the encoding device 10a and the decoding device 10b. The post-processing filter executor 950 may receive pre-agreed filter information from the encoding device 10a.

[0187] According to an embodiment, the machine vision task executor 1000 can receive a recovered image or a recovered image and additional information to perform processing for obtaining machine vision task results (e.g., mAP, MOTA, etc.). The machine vision task executor 1000 can use a deep neural network consisting of one or more learned convolutional layers. The machine vision task executor 1000 can receive information from the encoding device 10a regarding the type of machine vision task and / or the type of deep neural network, etc.

[0188] Figure 10 This describes the detailed operation of the internal decoding executor and the region-of-interest-based restorer according to embodiments of this disclosure.

[0189] Figure 10 It shows the way Figure 9 The decoding apparatus 10b includes an internal decoding executor 910 and a region-of-interest (ROI)-based restorer 920, each component of which performs internal decoding execution processing and ROI-based restore processing steps. According to the embodiment, the input bitstream can first be demultiplexed and allowed to be used separately as... Figure 9 The input to each decoding execution unit 911 and 913 in the process.

[0190] The image decoding execution unit 911 can receive a bitstream as input and perform decoding to recover the image and region of interest information. The number of frames in the recovered region-of-interest-based image may differ from the number of frames in the recovered non-region-of-interest image.

[0191] The upsampling execution unit 912 can receive a sampling rate to perform upsampling after decoding the image based on region-of-interest (ROI) processing and the non-ROI image, respectively. The sampling rate can be different for the ROI-based image and the non-ROI image, respectively. According to an embodiment, the upsampling execution unit 912 can select and use one of bilinear upsampling, bilateral upsampling, nearest neighbor upsampling, and a deep neural network including one or more convolutional layers as the upsampling method. According to an embodiment, the upsampling execution unit 912 can perform upsampling by using different upsampling methods for the ROI-based image and the non-ROI image, and can also variably select and apply the upsampling method within the ROI-based image. For example, the upsampling execution unit 912 can determine the upsampling method based on the sampling rate, and the upsampling method can also be different when the sampling rate is different.

[0192] The Region of Interest (ROI) information decoding execution unit 913 can receive a bitstream as input and perform decoding to recover information about the ROI. The ROI information decoding execution unit 913 can decode the ROI information, transmit it to the upsampling execution unit 912, and perform upsampling when encoding information about the ROI downsampled in the encoding device 10a. According to an embodiment, when multiple ROIs exist within a frame, the ROI information decoding execution unit 913 can perform decoding on each ROI. (More details to follow...) Figure 11 The text describes a detailed description of the region of interest information decoding execution unit 913.

[0193] According to an embodiment, the image reconfiguration unit 921 can receive the restored region of interest (ROI) and the restored non-ROI images as input, and perform restoration on the original image to output the restored image. According to an embodiment, the restored ROI information can be used in the image reconfiguration unit 921.

[0194] According to various embodiments, the image reconfiguration unit 921 can perform filtering on the recovered image to improve resolution. The image reconfiguration unit 921 can generate a recovered image by further reconfiguring the image using the recovered region of interest (ROI) information. In various embodiments, the image reconfiguration unit 921 can perform filtering on at least one of the non-ROI image and the ROI and boundaries of the ROI-based image. In one embodiment, the image reconfiguration unit 921 can perform boundary filtering on the ROI-based image and the non-ROI image using a region boundary filtering execution module. In another embodiment, the image reconfiguration unit 921 can perform filtering on one or more ROIs of the ROI-based image, such as deblurring.

[0195] In various embodiments, the image reconfiguration unit 921 can perform filtering for resolution improvement to recover the image that has been temporally resampled. Intermediate frames with relatively low quality due to quality differences between frames can be recovered during the temporal resampling process that removes intermediate frames, and the quality of the intermediate frames can be improved when deblurring filtering is performed on the regions of interest (ROIs) of the recovered intermediate frames. For example, the image reconfiguration unit 921 can generate the recovered image by performing deblurring filtering on the ROIs of two frames input during the temporal resampling process. According to embodiments, the image reconfiguration unit 921 can perform filtering by using any of a plurality of deep neural networks learned for various functions. For example, the image reconfiguration unit 921 can perform deblurring on the ROI using a deep neural network learned for deblurring. When deblurring filtering is applied, recovery performance can be improved.

[0196] The recovered non-region of interest (NRO) image can exist in each frame of the recovered NRO, and a recovered NRO image can exist relative to a series of frame groups per unit. As an example, the recovered NRO image can exist only per GOP relative to intra-coded frames. According to an embodiment, when a recovered NRO image exists relative to a series of frame groups per unit, the image reconfiguration unit 921 can generate a recovered image by using the same recovered NRO image for each frame of the recovered NRO. According to an embodiment, for each recovered NRO, different filters can be applied to the boundary of the recovered NRO image at the corresponding location and the boundary of the corresponding NRO. According to an embodiment, for each recovered NRO, the image reconfiguration unit 921 can implicitly determine the filtering method based on the difference in pixel values ​​between the boundary of the corresponding NRO and the boundary of the recovered NRO image at the corresponding location. According to an embodiment, the image reconfiguration unit 921 can perform filtering by receiving a filtering method for a specific NRO from the coding device 10a.

[0197] Figure 11 This describes the detailed operation of the region of interest information decoding execution unit according to the embodiments of this disclosure.

[0198] According to embodiments of the present disclosure, the region of interest (ROI) information decoding execution unit 913 included in the internal decoding executor 910 of the decoding apparatus 10b can decode ROI information based on the linear motion information flag (LinearMotion_flag) and compensation parameter information flag (Comp_param_flag) parsed from the bitstream. The ROI information decoding execution unit 913 may include a position information decoding module 913a, a motion parameter recovery execution unit 913b, and a compensation parameter recovery execution unit 913c. The ROI information decoding execution unit 913 can execute on a region-of-interest (ROI) basis and / or on a frame basis. Figure 11 Each step. In various implementations, the methods can be modified. Figure 11 The order of each step, and some configurations can be omitted.

[0199] According to the implementation, the location information decoding module 913a can decode the location information of a specific region of interest between specific frame intervals. As an example, the location information decoding module 913a can decode the location change / difference information of corresponding regions of interest between two frames. In this case, linear location change / difference can be recovered.

[0200] According to the implementation method, when a specific region of interest within a specific frame has only linear motion and not additional motion, LinearMotion_flag can be 0, and the decoding process for additional position and motion information can be omitted.

[0201] According to the implementation, when LiearMotion_flag is 1, it indicates that there is additional motion in addition to linear motion between regions of interest, and the motion parameter recovery execution unit 913b can perform motion parameter recovery. As an example, the additional motion can be affine motion, such as scaling, rotation, and shearing.

[0202] According to an embodiment, the motion parameter recovery execution unit 913b can perform motion parameter recovery regardless of the LiearMotion_flag. According to Equation 1, affine motion information can be represented using affine parameters (e.g., {a, b, c, d} or {a, b, c, d, Trans_x, Trans_y}). According to an embodiment, for each region of interest (ROI), values ​​within the motion information can be transmitted independently. At least one of the difference between motion information from ROIs having the same or corresponding ID as a previous frame and the difference between motion information from ROIs already decoded in the current frame can be transmitted from the encoding device 10a. The motion information of each ROI can be managed in units of a series of frame groups. Here, the unit of a series of frame groups can be a GOP (Group of Pictures) unit. According to an embodiment, the motion information of the ROI in the first frame can be transmitted from the encoding device 10a for each ROI within a frame group, and the ROIs in the next frame in the decoding order can be transformed only by the difference between the motion information of ROIs with the same or corresponding ID as those in the previous frame. For each ROI, the difference in motion information can be transmitted from intermediate frames within the group. In this case, the difference can be a difference in values ​​or a difference in ratios. According to the implementation, the difference can be transmitted only for a portion of the motion information, and other motion information can be transmitted in frames.

[0203] According to the implementation, after decoding the position and motion information, when the Comp_param_flag parsed from the bitstream is 1, the compensation parameter recovery execution module 913c can decode the compensation parameters (e.g., the scaling parameter α, the offset parameter β). The compensation parameters can be received and decoded from the encoding device 10a, and can be obtained from the representative brightness values ​​between two corresponding regions of interest or two corresponding frames in the decoding device 10b.

[0204] Figure 12a and Figure 12b This describes the detailed operation of the time recovery actuator according to the embodiments of this disclosure.

[0205] The time recovery executor 940 according to embodiments of this disclosure can generate an image for which time recovery is performed based on the recovered image and the recovered region of interest information. The time recovery executor 940 may include a time frame recovery execution unit 941 and a region of interest time modification unit 942. In various embodiments, the order of the detailed components of the time recovery executor 940 may be changed, and some configurations may be omitted. Figure 12a An embodiment is shown in which the time frame recovery execution unit 941 receives the recovered image and the recovered region of interest information to generate a time-recovered image. Figure 12bIn this process, the time frame recovery execution unit 941 can receive the recovered image to first perform time frame recovery, and then the region of interest time modification unit 942 can receive the image on which the time frame recovery is performed and the recovered region of interest information to generate an image on which the time recovery is performed.

[0206] Reference Figure 12a According to the embodiment, the time frame recovery execution unit 941 can parse information about the number of frames to be recovered (e.g., time sampling rate, etc.), receive the recovered image and the recovered region of interest information as input, and use the recovered image to perform the recovery of intermediate frames. In this case, the pixel value information and position information of the region of interest in the intermediate frame recovered by using the recovered region of interest information are modified.

[0207] Reference Figure 12b According to the embodiment, the time frame recovery execution unit 941 can receive information about the number of frames to be recovered (e.g., time sampling rate, etc.) to perform intermediate frame recovery using the recovered image. According to the embodiment, the region of interest time modification unit 942 can receive the image on which time frame recovery is performed and the recovered region of interest information as input to perform modification on the region of interest within the image on which time frame recovery is performed.

[0208] Figure 13 This is an example of time modification of the region of interest according to an implementation of this disclosure.

[0209] The decoding apparatus 10b according to an embodiment of the present disclosure includes a time recovery executor 910, and the time recovery executor 940 can perform time modification on the region of interest. In an embodiment, the region of interest time modification unit 942 of the time recovery executor 940 can perform modification on the region of interest within the recovered intermediate frame using the recovered region of interest information.

[0210] According to the implementation method, by means of frames within the recovered image having The region of interest (ROI) information is recovered from the intermediate frames based on the size of the ROI. The search range can be selected based on the positions of the corresponding intermediate frames as integers n and m greater than or equal to 1. , such as in Figure 13 (a) In. Based on the specific search method within the search range, by using Blocks and frames within the recovered image After finding the best corresponding region of interest within the recovered intermediate frame by using the difference between regions of interest of different sizes (e.g., SAD (sum of absolute differences)), the corresponding block position can be modified based on the information of the recovered region of interest. Figure 13 In (b), the region where the value does not exist is... Figure 13 In (b), the regions in the region of interest before the modification that were not included in the region of interest after the modification can be filled with intermediate values ​​when the position is modified, or the values ​​can be filled with specific filtering processing.

[0211] According to the implementation method, when the compensation parameter information exists in the recovered region of interest information, the search can be performed by applying the compensation parameter to the search process.

[0212] According to the implementation, when region-of-interest (ROI)-based restoration and temporal restoration are performed together on the restored image, the internal decoding executor 910 can perform temporal restoration first after performing internal decoding, and then perform ROI-based restoration. In the implementation, in the ROI-based restoration method, only the restoration processing of location information can be performed after temporal restoration.

[0213] According to the implementation, during the time recovery execution process, time recovery can be performed on each frame within the recovered image using an internal decoding executor within a unit of a series of frame groups, according to the following four methods. The time recovery executor 940 can perform time recovery based on the size of the region of interest included in two adjacent or non-adjacent frames, using any of the following time recovery methods.

[0214] 1) When the regions of interest (ROIs) of two adjacent or non-adjacent frames are the same size, a restored frame can be generated by copying the earlier frame from the two frames. Then, during the ROI-based restoration process, location information can be used to restore the location of the ROIs. The location information of the intermediate frame (the frame restored by the copying method) can be restored by subsampling the location information of the ROIs of the two frames, which are inputs to the temporal restoration process. In this implementation, the size of the ROI can be the size of the region where the ROI exists or the size of the region occupied by each ROI.

[0215] 2) When the size of the region of interest is the same in two adjacent or non-adjacent frames, the temporal frame restoration method can be determined by comparing one or more of the metrics such as peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) of the region of interest in each frame.

[0216] 3-1) When the difference in the size of the region of interest between two adjacent frames is less than a certain threshold, and when the number of frames for which time recovery is performed using two frames is n, time recovery can be performed either by copying only the previous frame or by copying only the subsequent frame. Alternatively, the... The first frame can copy the earlier of the two frames, while the second frame can copy the earlier frame. A frame can be copied from the later of two frames to perform temporal restoration. The position of the region of interest (ROI) can then be restored by using position information during the region-of-interest (ROI) restoration process, or by subsampling the position information of the ROIs of the two frames that are input to the temporal restoration process to restore the position information of the intermediate frame (the frame restored by the copying method). Various subsampling methods can exist in this case, and as an example, for a frame generated by copying a previous frame, weights can be further assigned to the position information of the ROI of the previous frame. In an implementation, the size of the ROI can be the size of the ROI itself or the size occupied by each ROI.

[0217] 3-2) When the difference in size between the regions of interest (ROIs) of two adjacent frames is less than or equal to a specific threshold, the temporal frame restoration method can be determined by comparing one or more metric measurements such as the Peak Signal-to-Noise Ratio (PSNR) and the Structural Similarity Index (SSIM) of the ROIs within the two frames. In this case, the larger ROI in the two frames can be downsampled to the same size as the smaller ROI for metric comparison, and then the metric comparison can be performed. Alternatively, the smaller ROI can be upsampled to the same size as the larger ROI, and then the metric comparison can be performed. According to the implementation, to compare ROIs of different sizes between frames, the upper-left positions of the two regions can be moved in the same way, and then the comparison can be performed on the region of the smaller region or the region of the larger region. In this case, when performing a comparison on the larger region, padding can be performed by filling the region outside the smaller region with intermediate values ​​or by using boundary values.

[0218] 4-1) When the difference between two regions of interest in one frame exceeds a certain threshold, time frame recovery may not be performed.

[0219] 4-2) When the difference between the regions of interest in two frames exceeds a certain threshold, temporal frame restoration can be performed by copying the adjacent frames of the two frames in the temporal order of the finally restored image and generating n frames.

[0220] Figure 14a and Figure 14b This describes the detailed operation of a machine vision task executor according to an embodiment of the present disclosure.

[0221] The machine vision task executor 1000 according to embodiments of this disclosure may include a deep neural network execution unit 1011 and a modification unit 1012 based on region of interest information. In various embodiments, the order of the detailed components of the machine vision task executor 1000 may be changed, and some configurations may be omitted.

[0222] In an implementation, the deep neural network execution unit 1011 may use a deep neural network consisting of one or more learning convolutional layers.

[0223] The machine vision task executor 1000 can receive information from the encoding device 10a regarding the type of machine vision task and / or the type of deep neural network, etc., to perform the machine vision task based on the received information.

[0224] In one implementation, the modification unit 1012 based on region of interest information can modify the execution result of the deep neural network (e.g., the location information of the region of interest in an object tracking task) by using the recovered region of interest information. The modification unit 1012 based on region of interest information can be operated by being included in the deep neural network execution unit 1011.

[0225] According to the implementation method, when there are processes other than independent processing between image frames, and the type of machine vision task to be performed is object tracking, the position modification of the region of interest can be performed as follows.

[0226] In an implementation, for points in the image where the object being tracked in a frame disappears, for example, when object 5 is tracked from frame 1 to frame 10 and there is no object 5 being tracked in frame 11, when information about object 5 in frame 10 is present in the recovered region of interest information, and when region of interest information between frame 10 and frame 11 is present in the corresponding information, the object tracking result can be modified by using the corresponding region of interest information.

[0227] In another embodiment, when the difference between the representative brightness value of the current frame and the representative brightness value of the next frame exceeds a specific threshold within the image, the object tracking result can be modified by using the region of interest information as described in the embodiment, in addition to the object tracking result between the two frames when information about the region of interest between the two frames exists within the region of interest information. In another example, when an object tracked in the current frame is not tracked in the next frame, modification can be performed as described in the embodiment. In another example, when an object tracked in the current frame is tracked in the next frame, modification can be performed as described in the embodiment. Figure 13 In the process, the information of the tracked object (size and position) in the current frame can be used to find the object corresponding to the tracked object in the current frame in the next frame through search processing, and add it to the object tracking result, thereby modifying the object tracking result.

[0228] <Description of the semantics and syntax of the implementation methods used in this disclosure>

[0229] The semantics related to time recovery data are described based on a series of frame groups, such as GOPs.

[0230] Ⅰ. temporal_restoration_data()

[0231] (1) temporal_restore_flag: A flag indicating whether temporal resampling is applied. When it is 1, temporal resampling is applied.

[0232] (2) Temporal_sampling_ratio_idx: The index of the list of temporal resampling rates. Since it is indicated as the tempotal_restoration_flag for temporal resampling applications, it does not include cases where the input and output signals are the same. For example, the sampling rate can be a ratio such as 1 / 2, 1 / 4, 1 / 8, or it can be the number of frames per second. The encoder and decoder can agree on the corresponding sampling rate, and the encoder can transmit the index of the list. According to the implementation, when the number of frames required for the transmitted application is fixed or the resampling rate is fixed, the corresponding information can be omitted.

[0233] (3) same_period_flag: This flag indicates whether periodic or aperiodic resampling is used when applying time resampling. Depending on the implementation, when the periodicity of time resampling is fixed, the corresponding information and related information can be omitted. When same_period_flag is 1, it means that time resampling is performed with the same period.

[0234] (4) delta_frame_idx[i]: Transmits the difference between the frame number and the previous frame index. Since it is the difference from the previous index, it transmits one less than num_of_frames calculated in terms of frame rate. In this case, i represents the frame index.

[0235] (5)RoI_Processing_flag: A 1-bit flag indicating whether the current sequence is undergoing region-of-interest processing.

[0236] (6) num_of_GOPs: Information representing the number of GOPs in a sequence.

[0237] (7) num_of_frames: Information indicating the number of frames in a GOP.

[0238] (8) num_of_RoIs: Information indicating the number of regions of interest in a frame.

[0239] (9) upsamp_ratio_RoIs: Upsampling rate information for the region of interest. This allows you to obtain an image based on the region of interest that has been recovered using this information.

[0240] (10) upsamp_ratio_nonRoIs: Upsampling rate information for non-ROIs. This allows you to obtain the non-ROI image recovered from the corresponding information.

[0241] (11) RoI_exist_region_LT(0), RoI_exist_region_LT(1): The top-left coordinates of the region of interest in the image.

[0242] (12) RoI_exist_region_RB(0), RoI_exist_region_RB(1): The lower right coordinates of the region of interest in the image.

[0243] (13) frame_RoI_Information_flag[p]: Frame information for which region of interest (ROI) information encoding is performed. When it is 1, the ROI information within the corresponding frame is encoded, and when it is 0, the corresponding frame is skipped. In this case, the ROI information between the two closest frames with the corresponding flag set to 1 is encoded.

[0244] (14) only_specific_RoI_flag and specific_RoI_flag[j]: When only_specific_RoI_flag is 1, information of all regions of interest (ROIs) within the frame for which region of interest information encoding is performed is encoded. When only_specific_ROI_flag is 0, information encoding is performed only on ROIs whose flag is 1 after specific_ROI_flag[j] is parsed. In this case, j represents the index of the ROI.

[0245] (15)LinearMotion_flag[j]: When the corresponding flag is 1, the j-th region of interest in the two frames that perform region of interest information encoding has linear motion, and only the motion position and difference information is decoded. When the corresponding flag is 0, the region of interest has, for example, affine motion other than linear motion, and the affine parameters are decoded in addition to the motion position and difference information.

[0246] (16) Affine_param_a[j], Affine_param_b[j], Affine_param_c[j], Affine_param_d[j]: When LinearMotion_flag[j] is 0, the information can be decoded and the motion information can be obtained by using the affine formula of the four affine parameters (Equation 1).

[0247] (17) Comp_param_flag[j]: When the corresponding flag is 1, the j-th region of interest within the two frames where region of interest encoding is performed has a compensation parameter. In this case, the compensation parameter can be explicitly signaled / resolved, or it can be derived by the decoder from the information of the neighboring recovered pixels. The compensation parameter can be applied as in the example in Equation 2.

[0248] The syntax for time recovery data is the same as that in Table 1 below.

[0249] [Table 1]

[0250]

[0251] The syntax example for region of interest information is the same as that in Table 2 below.

[0252] [Table 2]

[0253]

[0254] In various implementations, the recovery information regarding positional differences can be defined as follows. Semantics related to the recovery information regarding subsequent positional differences are described based on a series of frame groups, such as Group of Pictures (GOPs). n represents the GOP index, p represents the frame index, and m represents the region of interest index.

[0255] II. About pos_diff_coding()

[0256] (1)pos_RoI(n)(p)(m)[0],pos_RoI(n)(p)(m)[1]: Location information of the region of interest in the first frame within each series of frame groups (GOP) units.

[0257] - When parsing non-zero values ​​of RoI_exist_region_LT(0) or RoI_exist_region_LT(1), as an example, RoI_exist_region_LT(0) + pos_RoI(n)(m)[0] and RoI_exist_region_LT(1) + pos_RoI(n)(m)[1] can be the final recovered position of the m-th region of interest.

[0258] - The final recovered position of the region of interest can be obtained by using the same method for the position difference obtained through pos_diff_coding.

[0259] (2) abs_pos_diff_greater0_flag_RoI(n)(p)(m)[0], abs_pos_diff_greater0_flag_RoI(n)(p)(m)[1]: a 1-bit flag indicating whether the absolute value of the difference between the x-axis and y-axis position information of the same region of interest in frame 0 is 0.

[0260] (3)abs_pos_diff_greater1_flag_RoI(n)(p)(m)[0],abs_pos_diff_greater1_flag_RoI(n)(p)(m)[1]: a 1-bit flag indicating whether the absolute value of the difference between the x-axis and y-axis position information of the same region of interest in frame 0 is 1.

[0261] (4) abs_pos_diff_minus2_RoI(i)(p)(j)[0], abs_pos_diff_minus2_RoI(i)(p)(j)[0]: represents the value corresponding to -2 of the absolute value of the difference between the x-axis and y-axis position information of the same region of interest as the 0th frame.

[0262] (5) pos_diff_sign_flag(i)(p)(j)[0], pos_diff_sign_flag(i)(p)(j)[1]: A 1-bit flag representing the x-axis and y-axis sign values ​​of the same region of interest in frame 0. If it is 1, it is a + sign, and if it is 0, it is a - sign.

[0263] Table 3 below provides examples of the syntax for restoring positional differences according to various implementation methods.

[0264] [Table 3]

[0265]

[0266] The examples and figures presented in this specification are merely specific examples used to readily explain the technical content of this disclosure and to aid in understanding it, and are not intended to limit the scope of this specification. It will be apparent to those skilled in the art that other variations besides the examples described above may be feasible.

[0267] The claims set forth in this specification can be combined in various ways. For example, the technical features of the method claims in this specification can be combined and implemented in an apparatus, or the technical features of the apparatus claims in this specification can be combined and implemented in a method. Furthermore, the technical features of the method claims and the apparatus claims in this specification can be combined and implemented in an apparatus, or the technical features of the method claims and the apparatus claims in this specification can be combined and implemented in a method.

Claims

1. In a machine-oriented video coding (VCM) encoding apparatus, the VCM encoding apparatus includes: A time resampling actuator is used to change a portion of the frame rate of the input image; A region-of-interest (ROI) processor is used to extract one or more ROIs from each of multiple frames of the input image and generate an image based on the one or more ROIs. as well as An internal encoding executor is used to generate a bitstream by encoding information about the one or more regions of interest and the input image. The information regarding the one or more regions of interest includes the temporal sampling rate.

2. The apparatus according to claim 1, wherein: The input image is divided into images processed based on one or more regions of interest and images of non-regions of interest corresponding to other regions. The internal encoding executor performs encoding on the image and non-region of interest images processed based on the one or more regions of interest, respectively. The internal encoding executor outputs a bitstream by multiplexing the following: a bitstream encoding information about the one or more regions of interest, a bitstream encoding an image processed based on the one or more regions of interest, and a bitstream encoding an image of the non-regions of interest.

3. The apparatus according to claim 1, wherein, The time resampling actuator performs: All frames of the input image are resampled at the same frame rate, or Temporal resampling is performed on a series of frames within the input image at different frame rates.

4. The apparatus according to claim 3, wherein: When performing temporal resampling, the internal encoding executor only encodes the region of interest information of a portion of the frame that includes the region of interest.

5. The apparatus according to claim 4, wherein: When the temporal resampling is performed periodically, the internal encoding executor encodes information about the region of interest (ROI) for each frame period m (m is an integer less than or equal to n) for the region of interest that is temporally resampled at a sampling rate n (n is an integer greater than or equal to 1). When the temporal resampling is performed non-periodically, when the interval between two arbitrary adjacent frames is k, the internal encoding executor encodes the information about the region of interest per frame period k / p (p is an integer greater than or equal to 1 and less than k).

6. The apparatus according to claim 3, wherein: When the temporal resampling of the input image is not performed, the internal encoding executor encodes only information about one or more specific regions of interest in the input image, and The one or more specific regions of interest correspond to the difference in representative brightness values ​​between frames included in the input image or the difference in representative brightness values ​​of regions of interest included in frames that is greater than or equal to a preset threshold.

7. The apparatus according to claim 1, wherein: For the corresponding regions of interest (ROIs) in the plurality of frames of the input image, the internal encoder executor encodes the ROI information based on the correspondence relationship, and The region of interest information includes at least one of the following: the location between two regions of interest, motion parameters, and compensation parameters.

8. The apparatus according to claim 7, wherein: The corresponding region of interest is extracted by using a deep neural network learned for object detection or object tracking.

9. The apparatus according to claim 7, wherein: The motion parameters include one or more parameters representing affine motion, which includes scaling, shearing, and rotation. The parameters of affine motion are defined by the following equation: Where a, b, c, and d represent affine parameters, Trans_x and Trans_y represent translational motion information about the x and y axes, x and y represent the position of the pixel value within the current region of interest, and x' and y' represent the position of the pixel value to which the affine motion is applied.

10. The apparatus according to claim 7, wherein: The compensation parameters include one or more parameters indicating that the difference in representative brightness values ​​between two frames in the input image or the difference in representative brightness values ​​of the corresponding regions of interest is greater than or equal to a preset threshold. The compensation parameter is defined by the following equation. Among them, RoI i (Past) represents the i-th region of interest (ROI) in the previous frame in time between the two frames, and RoI. i (Future) represents the i-th region of interest in the next frame in time.

11. The apparatus according to claim 1, wherein: The apparatus further includes a spatial resampling actuator for changing the spatial resolution in units of each frame or a series of frames of the input image. The spatial resampling actuator or the region-of-interest (ROI) based processor may change the resolution only for one or more ROIs of the input image, or change the resolution to different resolutions for images processed based on the one or more ROIs and non-ROI images. The information regarding the one or more regions of interest includes changed resolution information.

12. In a machine-oriented video coding (VCM) decoding apparatus, the VCM decoding apparatus includes: An internal decoding executor is used to decode the bitstream to generate the recovered image; A region-of-interest (ROI)-based restorer is used to reconfigure the restored image based on the ROI in the restored image by using ROI-based processing information parsed from the bitstream; as well as A time recovery executor is used to perform time recovery of the recovered image by using time resampling information parsed from the bitstream.

13. The apparatus according to claim 12, wherein: The processing information based on the region of interest includes at least one of the following: region of interest ID information, region of interest size information, region of interest packing information, corresponding region of interest information, position information between two regions of interest of the corresponding region of interest, motion parameters of the corresponding region of interest, and compensation parameters of the corresponding region of interest.

14. The apparatus according to claim 12, wherein: The internal decoding executor demultiplexes the bitstream to decode the image based on the region of interest, information about the region of interest, and images of non-regions of interest, respectively. The internal decoding executor recovers the region-of-interest-based image based on the information about the region of interest, and The information about the region of interest includes at least one of the following: the location information of the region of interest, linear motion parameters, affine motion parameters, and compensation parameters.

15. The apparatus according to claim 14, wherein: The affine motion parameters represent affine motions including scaling, shearing, and rotation, and The affine motion parameters are defined by the following equation. Where a, b, c, and d represent affine parameters, Trans_x and Trans_y represent translational motion information about the x and y axes, x and y represent the position of the pixel value within the current region of interest, and x' and y' represent the position of the pixel value to which the affine motion is applied.

16. The apparatus according to claim 12, wherein: After decoding the bitstream, the time recovery executor first performs time recovery and then generates the recovered image based on one or more regions of interest.

17. The apparatus according to claim 12, wherein: The time recovery actuator modifies the position of the region of interest within an intermediate frame recovered according to one or more time frame recovery methods based on information about the decoded region of interest. The method for selecting one or more time frames is based on whether the frames contain regions of interest of the same size, whether the difference in the size of the regions of interest of two adjacent frames is less than or equal to a preset threshold, or whether the difference in the size of the regions of interest of the two adjacent frames is greater than the preset threshold.

18. The apparatus according to claim 12, wherein: The device also includes a machine vision task executor, and The machine vision task executor performs machine vision tasks by using a deep neural network learned therein, and modifies the results of performing the machine vision tasks based on recovered region-of-interest processing information.

19. The apparatus according to claim 18, wherein, The machine vision task executor: Objects in the recovered image are tracked using a deep neural network learned for object tracking, and The position of the region of interest is modified based on the recovered region-of-interest processing information.

20. As a non-volatile computer-readable storage medium for recording commands, wherein, The command, when executed by at least one processor, causes the at least one processor to: Change the frame rate of a portion of the input image; Extract one or more regions of interest from each frame of the input image, and generate an image based on the one or more regions of interest. as well as A bitstream is generated by encoding information about the one or more regions of interest and the input image. The information regarding the one or more regions of interest includes the temporal sampling rate.

Images