Encoding and decoding method based on adaptive spatiotemporal resampling in vcm

Abstract

An example of the present disclosure presents an encoding apparatus for performing video coding for a machine (VCM). The encoding apparatus comprises: a temporal resampler that changes a frame rate of an input image; a spatial resampler that changes the resolution of the input image; and an internal encoder that encodes the input image, wherein temporal restoration data including the frame rate of the input image changed by the temporal resampler and spatial restoration data including the resolution of the input image changed by the spatial resampler may be transmitted to a decoding apparatus together with the encoded image.

Description

Coding and decoding methods based on adaptive spatiotemporal resampling in VCM

[0001] The present disclosure relates to an adaptive spatiotemporal resampling-based encoding and decoding method in VCM.

[0002] With the continuous development of the information and communication industry, broadcasting services with HD (High Definition) resolution have spread worldwide.

[0003] Through this proliferation, many users have become accustomed to high-resolution and high-quality images and / or videos, and the demand for higher-resolution and high-quality images / videos, such as 4K or 8K or higher UHD (Ultra High Definition) images / videos, has increased in various fields.

[0004] The technology for coding this UHD video data was completed in 2013 through the standard technology HEVC (High Efficiency Video Coding).

[0005] HEVC is a next-generation video compression technology with a higher compression ratio and lower complexity than the previous H.264 / AVC technology, and is a key technology for effectively compressing the massive data of HD and UHD video.

[0006] Like previous compression standards, HEVC performs block-level encoding. However, unlike H.264 / AVC, it has only a single profile. The core encoding technologies included in HEVC's sole profile are divided into eight areas: hierarchical encoding structure technology, transform technology, quantization technology, intra-frame prediction coding technology, inter-frame motion prediction technology, entropy coding technology, loop filter technology, and other technologies.

[0007] Since the establishment of the HEVC video codec in 2013, the Versatile Video Coding (VVC) standard, a next-generation video codec that aims to improve performance by more than twice that of HEVC, has been developed to address the expansion of realistic video and virtual reality services utilizing 4K and 8K video images. VVC is called H.266.

[0008] H.266 (VVC) was developed with the goal of being more than twice as efficient as the previous generation codec, H.265 (HEVC). VVC was initially developed with resolutions over 4K in mind, but it was also developed for ultra-high-resolution video processing at a whopping 16K level to support 360-degree videos due to the expansion of the VR market. In addition, as the HDR market is expanding due to the development of display technology, it supports 16-bit color depth as well as 10-bit color depth to respond to this, and supports brightness expressions of 1000 nits, 4000 nits, and 10000 nits. In addition, since it is being developed with the VR market and 360-degree video market in mind, it supports partial frame rates in the range of 0 to 120 FPS.

[0009] Advances in Artificial Intelligence

[0010] Artificial intelligence (AI) is also steadily developing. AI refers to the artificial imitation of human intelligence, including the ability to recognize, classify, infer, predict, and control / decision-making.

[0011] With the advancement of artificial intelligence technology and the increase in Internet of Things (IoT) devices, machine-to-machine traffic is expected to explode, and machine-dependent image analysis is expected to become widely used.

[0012] However, as the amount of images to be analyzed by machines is expected to increase exponentially, issues with server load and power consumption are expected to arise.

[0013] Accordingly, the present disclosure aims to provide an adaptive spatiotemporal resampling-based encoding and decoding method in VCM to enable effective machine-based image analysis.

[0014] One embodiment of the present disclosure can provide an encoding and decoding method that adaptively changes a frame rate and resolution according to an application purpose.

[0015] In order to achieve the above-mentioned purpose, according to one disclosure of the present specification, an encoding device for performing image encoding for a machine is provided.

[0016] An encoding device for performing image encoding for a machine comprises: a temporal resampler for changing a frame rate for an input image; a spatial resampler for changing a resolution for the input image; and an internal encoder for performing encoding for the input image, and can transmit temporal restoration data including a frame rate of the input image changed by the temporal resampler and spatial restoration data including a resolution of the input image changed by the spatial resampler together with the encoded image to a decoding device.

[0017] According to one disclosure of the present specification, a decoding device for performing image decoding for a machine may include an internal decoder for receiving an encoded input image from an encoding device and performing decoding; a spatial restorer for changing a resolution of the decoded image based on spatial restoration data for the input image received from the encoding device and restoring the resolution to the resolution of the original image of the input image; and a temporal restorer for changing a frame rate of the decoded image based on temporal restoration data for the input image received from the encoding device and restoring the frame rate to the frame rate of the original image of the input image.

[0018] According to one disclosure of the present specification, a non-volatile computer-readable storage medium having instructions recorded thereon, wherein the instructions, when executed by one or more processors, cause the one or more processors to perform the steps of: changing a frame rate for an input image; changing a resolution for the input image; performing encoding for the input image; and transmitting temporal restoration data including a frame rate of the input image changed according to the temporal resampling and spatial restoration data including a resolution of the input image changed according to the spatial resampling together with the encoded image to a decoding device.

[0019] According to the present disclosure, image analysis by a machine can be effectively performed.

[0020] According to one embodiment of the present disclosure, the frame rate and resolution of the original image can be adaptively changed according to the application purpose, thereby increasing not only the decoding efficiency but also the effectiveness of the machine vision application.

[0021] Figure 1 schematically illustrates an example of a video / image coding system.

[0022] Figure 2 is a drawing schematically illustrating the configuration of a video / image encoding device.

[0023] Figure 3 is a drawing schematically illustrating the configuration of a video / image decoding device.

[0024] Figures 4a to 4d are exemplary diagrams showing a VCM encoder and a VCM decoder.

[0025] FIG. 5 is a block diagram including components of an encoding device for machine vision according to one embodiment of the present disclosure.

[0026] FIG. 6 is an example of changing the resolution according to temporal resampling in one embodiment of the present disclosure.

[0027] FIGS. 7a, 7b and 7c are examples of syntax and semantics for temporal resampling according to one embodiment of the present disclosure.

[0028] FIG. 8 is an example of changing the resolution according to spatial resampling in one embodiment of the present disclosure.

[0029] FIG. 9 is an example of rearranging frames by resolution after spatial resampling according to one embodiment of the present disclosure.

[0030] FIG. 10 is another example of changing resolution according to spatial resampling according to one embodiment of the present disclosure.

[0031] Figures 11a, 11b and 11c are examples of syntax and semantics for spatial resampling according to one embodiment of the present disclosure.

[0032] FIG. 12 is a block diagram including components of an encoding device for machine vision according to one embodiment of the present disclosure.

[0033] FIG. 13 is a block diagram including components of a decoding device for machine vision according to one embodiment of the present disclosure.

[0034] FIG. 14 is a flowchart illustrating a temporal resampling method according to an example of the present disclosure.

[0035] FIG. 15 is a flowchart illustrating a temporal resampling method according to another example of the present disclosure.

[0036] FIGS. 16A and 16B are examples of syntax and semantics for temporal resampling according to one embodiment of the present disclosure.

[0037] Specific structural or step-by-step descriptions of embodiments according to the concept of the present disclosure disclosed in this specification or application are merely illustrative for the purpose of explaining embodiments according to the concept of the present disclosure, and embodiments according to the concept of the present disclosure may be implemented in various forms, and embodiments according to the concept of the present disclosure may be implemented in various forms and should not be construed as being limited to the embodiments described in this specification or application.

[0038] Embodiments according to the concept of the present disclosure may have various modifications and take various forms. Therefore, specific embodiments are illustrated in the drawings and described in detail in this specification or application. However, this is not intended to limit embodiments according to the concept of the present disclosure to specific disclosed forms, and it should be understood that all modifications, equivalents, and alternatives included within the spirit and technical scope of the present disclosure are included.

[0039] While terms such as "first" and / or "second" may be used to describe various components, these components should not be limited by these terms. These terms are only intended to distinguish one component from another; for example, without departing from the scope of the present disclosure, a first component may be referred to as a "second component," and similarly, a second component may also be referred to as a "first component."

[0040] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. Conversely, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Other expressions that describe the relationship between components, such as "between" and "directly between" or "adjacent to" and "directly adjacent to", should be interpreted similarly.

[0041] The terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit the present disclosure. The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, it should be understood that the terms "comprises" or "has" indicate the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0042] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0043] In describing the embodiments, descriptions of technical details that are well known in the technical field to which the present disclosure pertains and are not directly related to the present disclosure will be omitted. This is to more clearly convey the gist of the present disclosure without obscuring it by omitting unnecessary explanations.

[0044] This document relates to video / image coding. For example, the method / embodiment disclosed in this document may be related to the Versatile Video Coding (VVC) standard (ITU-T Rec. H.266), the next-generation video / image coding standard after VVC, or other video coding-related standards (e.g., the High Efficiency Video Coding (HEVC) standard (ITU-T Rec. H.265), the essential video coding (EVC) standard, the AVS2 standard, etc.).

[0045] This document presents various embodiments of video / image coding, and unless otherwise stated, the embodiments may be performed in combination with each other.

[0046] In this document, video can mean a collection of images over time. A picture generally refers to a unit representing a single image at a specific time point, and a slice / tile is a unit that constitutes part of a picture in coding. A slice / tile can include one or more CTUs (coding tree units). A picture can be composed of one or more slices / tiles. A picture can be composed of one or more tile groups. A tile group can include one or more tiles.

[0047] A pixel or pel can mean the smallest unit that constitutes a picture (or image). Additionally, the term "sample" can be used as a counterpart to a pixel. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luma component, or only the pixel / pixel value of the chroma component. Alternatively, a sample can mean a pixel value in the spatial domain, or when such a pixel value is converted to the frequency domain, it can mean a transform coefficient in the frequency domain.

[0048] A unit may represent a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to the region. One unit may include one luma block and two chroma (e.g., cb, cr) blocks. In some cases, the term "unit" may be used interchangeably with terms such as "block" or "area." In general, an MxN block may include a set (or array) of samples (or sample array) or transform coefficients consisting of M columns and N rows.

[0049] Figure 1 schematically illustrates an example of a video / image coding system.

[0050] Referring to FIG. 1, a video / image coding system may include a source device and a receive device. The source device may transmit encoded video / image information or data to the receive device via a digital storage medium or a network in the form of a file or streaming.

[0051] The source device may include a video source, an encoding apparatus, and a transmitter. The receiving device may include a receiver, a decoding apparatus, and a renderer. The encoding apparatus may be referred to as a video / image encoding apparatus, and the decoding apparatus may be referred to as a video / image decoding apparatus. The transmitter may be included in the encoding apparatus. The receiver may be included in the decoding apparatus. The renderer may include a display unit, and the display unit may be configured as a separate device or an external component.

[0052] A video source may obtain video / images through a process of capturing, synthesizing, or generating video / images. The video source may include a video / image capture device and / or a video / image generation device. A video / image capture device may include, for example, one or more cameras, a video / image archive containing previously captured video / images, etc. A video / image generation device may include, for example, a computer, a tablet, a smartphone, etc., and may (electronically) generate video / images. For example, a virtual video / image may be generated through a computer, etc., in which case the video / image capture process may be replaced by a process of generating related data.

[0053] An encoding device can encode input video / images. The encoding device can perform a series of procedures, such as prediction, transformation, and quantization, to improve compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0054] The transmission unit can transmit encoded video / image information or data output in the form of a bitstream to the receiving unit of a receiving device via a digital storage medium or network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit can include an element for generating a media file using a predetermined file format and an element for transmission via a broadcasting / communication network. The receiving unit can receive / extract the bitstream and transmit it to a decoding device.

[0055] The decoding device can decode the video / image by performing a series of procedures such as inverse quantization, inverse transformation, and prediction corresponding to the operation of the encoding device.

[0056] The renderer can render decoded video / images. The rendered video / images can be displayed through the display unit.

[0057] Figure 2 is a drawing schematically illustrating the configuration of a video / image encoding device.

[0058] The term “video encoding device” hereinafter may include a video encoding device.

[0059] Referring to FIG. 2, the encoding device (10a) may be configured to include an image partitioner (10a-10), a prediction unit (predictor) (10a-20), a residual processor (residual processor) (10a-30), an entropy encoder (entropy encoder) (10a-40), an adder (adder) (10a-50), a filter (filter) (10a-60), and a memory (10a-70). The prediction unit (10a-20) may include an inter prediction unit (10a-21) and an intra prediction unit (10a-22). The residual processing unit (10a-30) may include a transformer (10a-32), a quantizer (10a-33), a dequantizer (10a-34), and an inverse transformer (10a-35). The residual processing unit (10a-30) may further include a subtractor (10a-31). The addition unit (10a-50) may be called a reconstructor or a reconstructed block generator. The above-described image segmentation unit (10a-10), prediction unit (10a-20), residual processing unit (10a-30), entropy encoding unit (10a-40), addition unit (10a-50), and filtering unit (10a-60) may be configured by one or more hardware components (e.g., encoder chipset or processor) according to an embodiment. In addition, the memory (10a-70) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (10a-70) as an internal / external component.

[0060] The image segmentation unit (10a-10) can segment an input image (or picture, frame) input to the encoding device (10a) into one or more processing units. For example, the processing unit may be called a coding unit (CU). In this case, the coding unit may be recursively segmented from a coding tree unit (CTU) or a largest coding unit (LCU) according to a QTBTTT (Quad-tree binary-tree ternary-tree) structure. For example, one coding unit may be segmented into a plurality of coding units of deeper depth based on a quad-tree structure, a binary tree structure, and / or a ternary structure. In this case, for example, the quad-tree structure may be applied first, and the binary tree structure and / or the ternary structure may be applied later. Alternatively, the binary tree structure may be applied first. The coding procedure according to the present document may be performed based on the final coding unit that is no longer segmented. In this case, based on coding efficiency according to image characteristics, etc., the maximum coding unit can be used as the final coding unit, or, if necessary, the coding unit can be recursively divided into coding units of lower depths, and the coding unit of the optimal size can be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration described below. As another example, the processing unit may further include a prediction unit (PU) or a transformation unit (TU). In this case, the prediction unit and the transformation unit may each be divided or partitioned from the final coding unit described above.The above prediction unit may be a unit of sample prediction, and the above transformation unit may be a unit for deriving a transformation coefficient and / or a unit for deriving a residual signal from a transformation coefficient.

[0061] The term "unit" may be used interchangeably with terms such as "block" or "area" depending on the case. In general, an MxN block can represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luminance component, or only the pixel / pixel value of the chroma component. A sample can be used as a term corresponding to a pixel or pel in a picture (or image).

[0062] The subtraction unit (10a-31) can subtract the prediction signal (predicted block, prediction samples, or prediction sample array) output from the prediction unit (10a-20) from the input image signal (original block, original samples, or original sample array) to generate a residual signal (residual block, residual samples, or residual sample array), and the generated residual signal is transmitted to the transformation unit (10a-32). The prediction unit (10a-20) can perform prediction on a block to be processed (hereinafter, referred to as a current block) and generate a predicted block including prediction samples for the current block. The prediction unit (10a-20) can determine whether intra prediction or inter prediction is applied to the current block or CU unit. The prediction unit can generate various information related to prediction, such as prediction mode information, as described later in the description of each prediction mode, and transmit the information to the entropy encoding unit (10a-40). Information about the prediction can be encoded in the entropy encoding unit (10a-40) and output in the form of a bitstream.

[0063] The intra prediction unit (10a-22) can predict the current block by referring to samples within the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from it depending on the prediction mode. In intra prediction, the prediction modes may include multiple non-directional modes and multiple directional modes. The non-directional modes may include, for example, a DC mode and a planar mode. The directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes depending on the degree of detail in the prediction direction. However, this is merely an example, and a greater or lesser number of directional prediction modes may be used depending on the settings. The intra prediction unit (10a-22) may also determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.

[0064] The inter prediction unit (10a-21) can derive a predicted block for a current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on an inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring block can include a spatial neighboring block existing in the current picture and a temporal neighboring block existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The above temporal neighboring blocks may be called collocated reference blocks, collocated CUs (colCUs), etc., and a reference picture including the temporal neighboring blocks may be called a collocated picture (colPic). For example, the inter prediction unit (10a-21) may construct a motion information candidate list based on the neighboring blocks, and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction may be performed based on various prediction modes, and for example, in the case of skip mode and merge mode, the inter prediction unit (10a-21) may use the motion information of the neighboring blocks as the motion information of the current block. In the case of skip mode, unlike the merge mode, a residual signal may not be transmitted.In the motion vector prediction (MVP) mode, the motion vector of the surrounding blocks is used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0065] The prediction unit (10a-20) can generate a prediction signal based on various prediction methods described below. For example, the prediction unit can apply intra prediction or inter prediction to predict a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). In addition, the prediction unit can perform intra block copy (IBC) to predict a block. The intra block copy can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in this document.

[0066] The prediction signal generated through the inter prediction unit (10a-21) and / or the intra prediction unit (10a-22) can be used to generate a reconstructed signal or a residual signal. The transform unit (10a-32) can generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique can include a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), a Graph-Based Transform (GBT), or a Conditionally Non-linear Transform (CNT). Here, GBT refers to a transform obtained from a graph when the relationship information between pixels is expressed as a graph. CNT refers to a transform obtained based on generating a prediction signal using all previously reconstructed pixels. In addition, the transform process can be applied to a pixel block having a square equal size, or can be applied to a block of a non-square variable size.

[0067] The quantization unit (10a-33) quantizes the transform coefficients and transmits them to the entropy encoding unit (10a-40), and the entropy encoding unit (10a-40) can encode the quantized signal (information about the quantized transform coefficients) and output it as a bitstream. The information about the quantized transform coefficients may be called residual information. The quantization unit (10a-33) can rearrange the quantized transform coefficients in a block form into a one-dimensional vector form based on a coefficient scan order, and can also generate information about the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form. The entropy encoding unit (10a-40) can perform various encoding methods, such as, for example, exponential Golomb, context-adaptive variable length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC).

[0068] The entropy encoding unit (10a-40) may encode, together or separately, information necessary for video / image restoration (e.g., values ​​of syntax elements, etc.) in addition to quantized transform coefficients. The encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream in the form of a network abstraction layer (NAL) unit. The video / image information may further include information regarding various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. The signaling / transmitted information and / or syntax elements described later in this document may be encoded through the above-described encoding procedure and included in the bitstream. The bitstream may be transmitted through a network or may be stored in a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The signal output from the entropy encoding unit (10a-40) may be configured as an internal / external element of the encoding device (10a) by a transmitting unit (not shown) that transmits and / or a storing unit (not shown) that stores the signal, or the transmitting unit may be included in the entropy encoding unit (10a-40).

[0069] The quantized transform coefficients output from the quantization unit (10a-33) can be used to generate a prediction signal. For example, by applying inverse quantization and inverse transformation to the quantized transform coefficients through the inverse quantization unit (10a-34) and the inverse transform unit (10a-35), a residual signal (residual block or residual samples) can be reconstructed. The addition unit (10a-50) can add the reconstructed residual signal to the prediction signal output from the prediction unit (10a-20), thereby generating a reconstructed signal (reconstructed picture, reconstructed block, reconstructed samples, or reconstructed sample array). When there is no residual for the target block to be processed, such as when skip mode is applied, the predicted block can be used as the reconstructed block. The generated restoration signal can be used for intra prediction of the next processing target block within the current picture, and can also be used for inter prediction of the next picture after filtering as described below.

[0070] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture encoding and / or restoration process.

[0071] The filtering unit (10a-60) can improve subjective / objective picture quality by applying filtering to the restoration signal. For example, the filtering unit (10a-60) can apply various filtering methods to the restoration picture to generate a modified restoration picture, and store the modified restoration picture in the memory (10a-70), specifically, in the DPB of the memory (10a-70). The various filtering methods may include, for example, deblocking filtering, sample adaptive offset (SAO), an adaptive loop filter, a bilateral filter, etc. The filtering unit (10a-60) can generate various information regarding filtering and transmit the information to the entropy encoding unit (10a-90), as described below in the description of each filtering method. The information regarding filtering may be encoded by the entropy encoding unit (10a-90) and output in the form of a bitstream.

[0072] The modified restored picture transmitted to the memory (10a-70) can be used as a reference picture in the inter prediction unit (10a-80). Through this, when inter prediction is applied, the encoding device can avoid prediction mismatch between the encoding device (10a) and the decoding device, and can also improve encoding efficiency.

[0073] The DPB of the memory (10a-70) can store the modified reconstructed picture to be used as a reference picture in the inter prediction unit (10a-21). The memory (10a-70) can store motion information of a block from which motion information in the current picture is derived (or encoded) and / or motion information of blocks in a picture that has already been reconstructed. The stored motion information can be transmitted to the inter prediction unit (10a-21) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (10a-70) can store reconstructed samples of reconstructed blocks in the current picture and transmit them to the intra prediction unit (10a-22).

[0074] Figure 3 is a drawing schematically illustrating the configuration of a video / image decoding device.

[0075] Referring to FIG. 3, the decoding device (10b) may be configured to include an entropy decoder (10b-10), a residual processor (10b-20), a predictor (10b-30), an adder (10b-40), a filter (10b-50), and a memory (10b-60). The predictor (10b-30) may include an inter-prediction unit (10b-31) and an intra-prediction unit (10b-32). The residual processor (10b-20) may include a dequantizer (10b-21) and an inverse transformer (10b-21). The entropy decoding unit (10b-10), residual processing unit (10b-20), prediction unit (10b-30), addition unit (10b-40), and filtering unit (10b-50) described above may be configured by a single hardware component (e.g., decoder chipset or processor) according to an embodiment. In addition, the memory (10b-60) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (10b-60) as an internal / external component.

[0076] When a bitstream including video / image information is input, the decoding device (10b) can restore the image corresponding to the process in which the video / image information is processed in the encoding device of FIG. 2. For example, the decoding device (10b) can derive units / blocks based on block division-related information obtained from the bitstream. The decoding device (10b) can perform decoding using a processing unit applied in the encoding device. Therefore, the processing unit of decoding may be, for example, a coding unit, and the coding unit may be divided from a coding tree unit or a maximum coding unit according to a quad tree structure, a binary tree structure, and / or a ternary tree structure. One or more transform units may be derived from the coding unit. Then, the restored image signal decoded and output by the decoding device (10b) can be reproduced through a reproduction device.

[0077] The decoding device (10b) can receive a signal output from the encoding device of FIG. 2 in the form of a bitstream, and the received signal can be decoded through the entropy decoding unit (10b-10). For example, the entropy decoding unit (10b-10) can parse the bitstream to derive information (e.g., video / image information) necessary for image restoration (or picture restoration). The video / image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information.

[0078] The decoding device can further decode the picture based on information about the parameter set and / or the general restriction information. The signaling / received information and / or syntax elements described later in this document can be decoded and obtained from the bitstream through the decoding procedure. For example, the entropy decoding unit (10b-10) can decode information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output values ​​of syntax elements required for image restoration and quantized values ​​of transform coefficients for the residual. In more detail, the CABAC entropy decoding method receives a bin corresponding to each syntax element in a bitstream, determines a context model using information of the syntax element to be decoded and decoding information of surrounding and decoding target blocks or information of symbols / bins decoded in a previous step, and predicts the occurrence probability of the bin according to the determined context model to perform arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element. At this time, the CABAC entropy decoding method can update the context model using information of the decoded symbol / bin for the context model of the next symbol / bin after determining the context model. Among the information decoded by the entropy decoding unit (10b-10), information regarding prediction is provided to the prediction unit (10b-30), and information regarding the residual on which entropy decoding has been performed by the entropy decoding unit (10b-10), i.e., quantized transform coefficients and related parameter information, can be input to the dequantization unit (10b-21).

[0079] In addition, information regarding filtering among the information decoded by the entropy decoding unit (10b-10) may be provided to the filtering unit (10b-50). Meanwhile, a receiving unit (not shown) that receives a signal output from an encoding device may be further configured as an internal / external element of the decoding device (10b), or the receiving unit may be a component of the entropy decoding unit (10b-10). Meanwhile, the decoding device according to the present document may be called a video / video / picture decoding device, and the decoding device may be divided into an information decoder (video / video / picture information decoder) and a sample decoder (video / video / picture sample decoder). The above information decoder may include the entropy decoding unit (10b-10), and the sample decoder may include at least one of the inverse quantization unit (10b-21), the inverse transformation unit (10b-22), the prediction unit (10b-30), the addition unit (10b-40), the filtering unit (10b-50), and the memory (10b-60).

[0080] The inverse quantization unit (10b-21) can inverse quantize the quantized transform coefficients and output the transform coefficients. The inverse quantization unit (10b-21) can rearrange the quantized transform coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scanning order performed in the encoding device. The inverse quantization unit (10b-21) can perform inverse quantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain transform coefficients.

[0081] In the inverse transform unit (10b-22), the transform coefficients are inversely transformed to obtain a residual signal (residual block, residual sample array).

[0082] The prediction unit can perform a prediction on the current block and generate a predicted block containing prediction samples for the current block. Based on the information regarding the prediction output from the entropy decoding unit (10b-10), the prediction unit can determine whether intra-prediction or inter-prediction is applied to the current block, and can determine a specific intra / inter-prediction mode.

[0083] The prediction unit can generate a prediction signal based on various prediction methods described below. For example, the prediction unit can apply intra prediction or inter prediction to predict a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). In addition, the prediction unit can perform intra block copy (IBC) to predict a block. The intra block copy can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in this document.

[0084] The intra prediction unit (10b-32) can predict the current block by referring to samples within the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from it, depending on the prediction mode. In intra prediction, the prediction modes may include multiple non-directional modes and multiple directional modes. The intra prediction unit (10b-32) can also determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.

[0085] The inter prediction unit (10b-31) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on an inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.).

[0086] In the case of inter prediction, the neighboring blocks may include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit (10b-31) may construct a motion information candidate list based on the neighboring blocks, and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter prediction may be performed based on various prediction modes, and the information regarding the prediction may include information indicating the mode of inter prediction for the current block.

[0087] The addition unit (10b-40) can generate a restoration signal (restored picture, restoration block, restoration sample array) by adding the acquired residual signal to the prediction signal (predicted block, prediction sample array) output from the prediction unit (10b-30). In cases where there is no residual for the block to be processed, such as when skip mode is applied, the predicted block can be used as the restoration block.

[0088] The addition unit (10b-40) may be referred to as a restoration unit or restoration block generation unit. The generated restoration signal may be used for intra prediction of the next processing target block within the current picture, may be output after filtering as described below, or may be used for inter prediction of the next picture.

[0089] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture decoding process.

[0090] The filtering unit (10b-50) can improve subjective / objective image quality by applying filtering to the restoration signal. For example, the filtering unit (10b-50) can apply various filtering methods to the restoration picture to generate a modified restoration picture, and transmit the modified restoration picture to the memory (60), specifically, the DPB of the memory (10b-60). The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0091] The (corrected) reconstructed picture stored in the DPB of the memory (10b-60) can be used as a reference picture in the inter prediction unit (10b-31). The memory (10b-60) can store motion information of a block from which motion information is derived (or decoded) within the current picture and / or motion information of blocks within a picture that has already been reconstructed. The stored motion information can be transmitted to the inter prediction unit (10b-31) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (10b-60) can store reconstructed samples of reconstructed blocks within the current picture and transmit them to the intra prediction unit (10b-32).

[0092] In this specification, the embodiments described in the prediction unit (10b-30), the inverse quantization unit (10b-21), the inverse transformation unit (10b-22), and the filtering unit (10b-50) of the decoding device (10b) can be applied to the prediction unit (10a-20), the inverse quantization unit (10a-34), the inverse transformation unit (10a-35), and the filtering unit (10a-60) of the encoding device (10a) in the same manner or correspondingly.

[0093] As described above, prediction is performed to increase compression efficiency when performing video coding. Through this, a predicted block including prediction samples for a current block, which is a coding target block, can be generated. Here, the predicted block includes prediction samples in a spatial domain (or pixel domain). The predicted block is derived identically from an encoding device and a decoding device, and the encoding device can increase video coding efficiency by signaling information (residual information) about the residual between the original block and the predicted block, rather than the original sample value of the original block itself, to a decoding device. The decoding device can derive a residual block including residual samples based on the residual information, and generate a reconstructed block including reconstructed samples by combining the residual block and the predicted block, and can generate a reconstructed picture including the reconstructed blocks.

[0094] The residual information may be generated through a transformation and quantization procedure. For example, the encoding device may derive a residual block between the original block and the predicted block, perform a transformation procedure on residual samples (a residual sample array) included in the residual block to derive transform coefficients, and perform a quantization procedure on the transform coefficients to derive quantized transform coefficients, thereby signaling the related residual information to a decoding device (via a bitstream). Here, the residual information may include information such as value information, position information, a transformation technique, a transformation kernel, and quantization parameters of the quantized transform coefficients. The decoding device may perform an inverse quantization / inverse transformation procedure based on the residual information to derive residual samples (or residual blocks). The decoding device may generate a reconstructed picture based on the predicted block and the residual block. The encoding device can also inversely quantize / inversely transform the quantized transform coefficients to derive a residual block for reference in inter prediction of a subsequent picture, and generate a restored picture based on the residual block.

[0095] <VCM(Video coding for Machines)>

[0096] With the recent advancements in various industries such as surveillance, intelligent transportation, smart cities, intelligent industry, and intelligent content, the amount of image or feature map data consumed by machines is increasing. In contrast, traditional video compression methods currently in use, developed with consideration for the characteristics of human vision, contain unnecessary information and are therefore inefficient for machine tasks. Therefore, research on video codec technologies capable of efficiently compressing feature maps for machine tasks is urgently needed.

[0097] The Moving Picture Experts Group (MPEG), an international standardization group for multimedia encoding, is discussing Video Coding for Machines (VCM). VCM is an image or feature map encoding technology that targets machine vision, rather than human viewer vision.

[0098] Figures 4a to 4d are exemplary diagrams showing a VCM encoder and a VCM decoder.

[0099] Referring to FIG. 4a, a VCM encoder (100a) and a VCM decoder (100b) are shown.

[0100] When a VCM encoder (100a) encodes a video and / or a feature map and transmits it as a bitstream, a VCM decoder (100b) can decode and output the bitstream. At this time, the VCM decoder (100b) can output one or more videos and / or feature maps. For example, the VCM decoder (100b) can output a first feature map for machine-based analysis and a first image for user viewing. The first image can have a higher resolution than the first feature map.

[0101] Referring to FIG. 4b, a feature extractor for extracting a feature map may be connected to the front end of the VCM encoder (100a).

[0102] The VCM encoder (100a) may include a feature encoder.

[0103] The VCM decoder (100b) may include a feature decoder and a video reconstructor. The feature decoder may decode a feature map from a bitstream and output a first feature map for machine-assisted analysis. The video reconstructor may regenerate and output a first video from the bitstream for viewing by a user.

[0104] Referring to Fig. 4c, a feature extractor for extracting a feature map is connected to the front end of the VCM encoder (100). The VCM encoder (100a) may include a feature encoder.

[0105] The VCM decoder (100b) may include a feature decoder. The feature decoder may decode a feature map from a bitstream and output a first feature map for machine-based analysis. That is, the bitstream may be encoded only as a feature map, not as an image. To elaborate, the feature map may be data containing information about features for processing a specific task of a machine based on an image.

[0106] Referring to FIG. 4d, a feature extractor may be connected to the front end of the VCM encoder (100a).

[0107] The VCM encoder (100a) may include a feature converter and a video encoder. The video encoder may be the encoding device (10a) illustrated in FIG. 2. The VCM decoder (100b) illustrated in FIG. 4d may include a video decoder and an inverse converter. The video decoder may be the decoding device (10b) illustrated in FIG. 3.

[0108] FIG. 5 is a block diagram including components of an encoding device for machine vision according to one embodiment of the present disclosure.

[0109] The encoding device (10a) may include a temporal resampler (510), a spatial resampler (520), a region of interest extractor (530), and an internal encoder (540). The structure of the encoding device of FIG. 5 may have some of its components omitted in various embodiments, and the order of the components may be changed. For convenience of explanation in the present disclosure, the encoding device may be referred to as an encoder or encoder, and the decoding device may be referred to as a decoder or decoder.

[0110] In VCM, the resolution and frame rate required by human vision are not necessary for the actual application purposes of machine vision. The application purposes of the input image may be, but are not limited to, tracking, recognition, classification, or segmentation, and the purposes that can be utilized in machine vision can be defined in various ways. The required frame rate and resolution of the input image can be determined according to the application purpose of machine vision. The encoder and decoder can resample the input image based on the frame rate and resolution determined according to the application purpose, and, if necessary, can restore the resampled image to its original state. The encoder and decoder can transmit and receive resampling information (e.g., temporal_restoration_data and spatial_restoration_data) written according to a mutually defined format.

[0111] Referring to FIG. 5, the encoding device (10a) can change the frame rate and resolution of an input signal through a temporal resampling and spatial resampling process in a stage prior to internal encoding for encoding efficiency for machine vision, and perform encoding on the changed signal in an internal encoder.

[0112] Adaptively changing the frame rate and resolution according to the application purpose can increase the effectiveness of machine vision applications as well as decoding and efficiency. The encoding device (10a) can transmit information related to the frame rate and resolution change to the decoding device (10b) according to the application purpose and embodiment.

[0113] The input signal of the encoding device (10a) for machine vision can be an image acquired by various sensors having video characteristics, such as a general optical camera image, a thermal camera, or a lidar sensor image. The composition of the image can be composed of a single component or multiple components depending on the characteristics of the sensor. For example, an image acquired by an optical camera can be composed of multiple color difference components in the RGB and YUV domains.

[0114] The temporal resampler (510) can upsample or downsample on a frame-by-frame basis, which is a step for changing the frame rate of an input image (or input signal). The frame rate can be fixed depending on the application purpose of machine vision, and when used adaptively, the corresponding frame rate can be transmitted to the decoder. Depending on the embodiment, the frame rate can be variable, and for example, for an image with an input signal of 30 frames per second, some signals can be sampled at 5 frames per second, and some signals can be sampled at 1 frame per second.

[0115] FIG. 6 is an example of changing the resolution according to temporal resampling in one embodiment of the present disclosure.

[0116] The temporal resampler (510) can determine the sampling unit fixedly or variably. The temporal resampler (510) can sample the entire input signal at a fixed cycle, or can sample a portion of the input signal at a fixed cycle. Alternatively, the temporal resampler (510) can sample by variably determining the sampling cycle for the input signal. For example, sampling can be performed at a fixed cycle as in (a) of FIG. 6, or adaptively sampling can be performed as a frame required in machine vision aperiodically as in (b). The encoding device (10a) can transmit temporal resampling information (e.g., temporal_resoration_data of FIGS. 7a and 7b) to the decoder. In an embodiment, if necessary, the cycle or frame number (index) of the sampled frame can be transmitted to the decoder. The frame index can be an index sequentially assigned to all frames included in the input image. The temporal resampler (510) may not separately transmit period information to the decoder if the frame period is a fixed value determined in advance by the decoder. The temporal resampler (510) may separately transmit period information if the frame period is variable. Alternatively, the temporal resampler (510) may transmit the index number of the frame if the frame period is variable. In some embodiments, if the frame rate is fixed according to the application purpose and application purpose of the video decoded by the decoder, information about the frame rate may be omitted. Information about the syntax transmitted from the encoder to the decoder is as shown in FIGS. 7a and 7b, which may be used for temporal restoration corresponding to temporal resampling in the decoder, and in some embodiments, if the decoder does not require a temporal restoration process, the encoder may omit transmission of information related to temporal restoration.

[0117] FIGS. 7A, 7B, and 7C are examples of syntax and semantics for temporal resampling according to an embodiment of the present disclosure. FIG. 7A illustrates information (temporal_restoraion_data) related to temporal resampling. Target_application_idx represents an information index (idx) for a machine vision target application to be used by the decoder. If not specified, information about the corresponding index in the target application list, such as tracking, recognition, classification, or segmentation, may be transmitted. In an embodiment, an idx value of 0 may represent unspecified, 1 may represent tracking, 2 may represent recognition, 3 may represent classification, and 4 may represent segmentation.

[0118] Temporal_restoration_flag is a flag indicating whether temporal resampling is applied.

[0119] The Same_period_flag flag indicates whether periodic or aperiodic resampling is used when applying temporal resampling. Depending on the embodiment, if the periodicity of temporal resampling is fixed, this information and related information may be omitted. A Same_period_flag of 1 indicates resampling at the same period.

[0120] Temporal_resampling_rete_idx is an index into a list of resampling rates, and since temporal resampling is applied as indicated by temporal_restoration_flag, it does not include cases where the input signal and the output signal are the same. For example, the sampling rate can be in the form of a ratio such as 1 / 2, 1 / 4, 1 / 8, etc., and can be the number of frames per second. The encoder and decoder can agree on the list in advance, and the encoder can transmit the index of the list. Depending on the embodiment, if the resampling rate is fixed or the number of required frames is fixed for the transmitted application, the information may be omitted.

[0121] When aperiodic resampling is performed, the index information of the resampled frame can be transmitted (Same_period_flag). When aperiodic resampling is applied, the number of frames can be calculated through temporal_resampling_rate_idx. For example, if the frame number is 30 and the resampling rate is 1 / 2, frame_num becomes 15. Depending on the embodiment, if the frame number is fixed for the application, frame_num can be determined by the fixed number. The difference value (delta_frame_idx[i]) from the previous frame index is transmitted according to the frame number. Since it is the difference value from the previous index, it is sufficient to transmit a number that is 1 less than the frame_num calculated by the frame rate.

[0122] According to an embodiment, temporal resampling period information can be transmitted through the difference value between the previous reference frame index and the current reference frame index without transmitting a flag for the aperiodicity or periodicity of the temporal resampling period. For example, if temporal resampling is performed periodically, delta_frame_idx[i] will have the same value. FIG. 7b is an example of a syntax for transmitting the difference value between frame indices.

[0123] Information related to temporal restoration (temporal_restoration) may be transmitted at various levels depending on the embodiment. For example, the temporal resampler (510) may transmit temporal restoration information at the GOP level, frame level, slice level, sequence level, subsequence level, etc. Although the embodiment published through Fig. 7a is based on transmission at the sequence level, the actual detailed information of temporal_restoration, excluding target_application_idx, may be transmitted at various levels such as GOP, frame, slice, subsequence, etc. Figs. 7a and 7b are examples of information that must be transmitted from the encoder to the decoder for temporal_restoration, and do not necessarily mean that the information is transmitted all at once at the same level. Depending on the embodiment, information related to temporal_restoration may be transmitted in a distributed form at multiple stages among sequence, subsequence, GOP, frame, and slice. For example, target_application_idx may be transmitted at the sequence level, and all information after temporal_restoration_flag may be transmitted at the GOP level. Alternatively, target_application_idx can be transmitted at the sequence level, temporal_restoration_flag, same_period_flag, temporal_resampling_rate_idx can be transmitted at the GOP level, and delta_frame_idx[i] can be transmitted at the frame or slice level.

[0124] Referring to FIG. 7c, the temporal resampler (510) transmits a resampling flag on a slice-by-slice basis, and can transmit temporal resampling-related information by not transmitting internal slice encoding information when resampled through the flag. From an HRD perspective, trailing bits are transmitted to prevent underflow issues, so that information of 1+a bits is actually transmitted.

[0125] Referring to FIG. 5, the spatial resampler (520) can upsample or downsample the resolution on a frame-by-frame basis, which is a step for changing the resolution of the input image. For example, the spatial resampler (520) can downsample the resolution of the input image by half. Alternatively, according to an embodiment, the spatial resampler (520) can upsample or downsample the input frame to different resolutions on a frame-by-frame basis.

[0126] FIG. 8 is an example of changing the resolution according to spatial resampling of one embodiment of the present disclosure. FIG. 8 (a) is an example of resampling the resolution of each frame of an input signal to be the same. For example, an input signal having a 4K resolution can be changed to have a 2K resolution by downsampling it equally by 1 / 2 for all frames. As another example, FIG. 8 (b) is an example of resampling the resolution of each frame of an input signal to be different. The resolution of the input signal can be downsampled by 1 / 2 for frames corresponding to a certain cycle (such as the first, fourth, seventh, ...), and the resolution of the input signal can be downsampled by 1 / 4 for the remaining frames (such as the second, third, fifth, sixth, eighth, ninth, ...). The encoding device (10a) can transmit spatial resampling information (spatial_restoration_data) to the decoder.

[0127] FIG. 9 illustrates an example of reordering frames by resolution after spatial resampling, according to an embodiment of the present disclosure. According to the embodiment, for efficiency and internal encoding in the subsequent steps after spatial resampling, signals resampled with different resolutions can be reordered by resolution as shown in FIG. 9 and then proceed to the next step.

[0128] As shown in FIG. 8, the spatial resampler (520) can upsample or downsample the entire input signal. Alternatively, as another example, the spatial resampler (520) can upsample or downsample a portion of the input image.

[0129] FIG. 10 is another example of changing the resolution by spatial resampling according to an embodiment of the present disclosure. Referring to (a) of FIG. 10, the spatial resampler (520) can downsample the resolution of the entire input signal. This is the same as FIG. 8. Referring to (b) of FIG. 10, the spatial resampler (520) can change the resolution or size of an image by cropping only a portion of the input image to fit the required resolution. Referring to (c) of FIG. 10, the spatial resampler (520) can crop a portion of the input image and then downsample the cropped image. Referring to (d) of FIG. 10, the spatial resampler (520) can crop a portion of the input image and then upsample the cropped image. Information regarding the syntax transmitted from the encoder to the decoder is as shown in FIG. 11a, and can be used for spatial restoration corresponding to spatial resampling in the decoder. In some embodiments, if the decoder does not require a spatial restoration process, the encoder may omit the transmission of information related to spatial restoration.

[0130] Figures 11a, 11b, and 11c illustrate examples of syntax and semantics for spatial resampling according to one embodiment of the present disclosure. Figure 11a includes syntax related to spatial resampling for the entire input signal.

[0131] Target_application_idx represents the information index (idx) for the machine vision target application to be used by the decoder. If not specified, information about the corresponding index in the target application list, such as tracking, recognition, classification, or segmentation, can be transmitted. In an embodiment, the idx value may indicate unspecified if 0, tracking if 1, recognition if 2, classification if 3, and segmentation if 4.

[0132] spatial_restoration_flag is a flag indicating whether spatial resampling is applied.

[0133] Same_ratio_flag indicates whether each frame in the sequence has been resampled to the same resolution. A Same_ration_flag of 1 indicates that the frames have been resampled to unequal resolutions.

[0134] num_spatial_ratio_minus2 is the number of resampled heterogeneous resolutions, and does not include the cases of 0 and 1, so if the number transmitted is 0, it could mean 2 heterogeneous resolutions.

[0135] num_spatial_ration can be calculated by adding 2 to the transmitted num_spatial_ratio_minus2. If Same_ratio_flag is 0 and num_spatial_ratio_minus2 is not transmitted, num_spatial_ratio will be 1.

[0136] Spatial_resampling_rate_idx[i] represents the index of the list for the resampling rate for each resolution according to the type of heterogeneous resolution. For example, it can be a ratio such as 2, 4, 1 / 2, 1 / 4, etc., and can be direct size information for horizontal and vertical resolution. Since whether resampling is applied is transmitted by spatial_restoraion_flag, if num_spatial_ratio is 1, the case where the input signal and the output signal have the same resolution is not included. Depending on the embodiment, if num_spatial_ratio is 2 or more, it can have heterogeneous resolution after resampling, so the case where the input signal and the output signal have the same resolution can be included.

[0137] resampling_method_idx[i] represents an index into a list of resolution-specific resampling methods. It may contain information about methods such as downsampling, upsampling, and cropping, as well as, in the case of upsampling and downsampling, the filter used in the resampling.

[0138] In case of having different resolutions, the indexes of frames having the same resolution can be transmitted at a higher level, and at this time, the differential value of the frame index corresponding to each resolution can be transmitted as delta_frame_idx[i]. Alternatively, depending on the embodiment, resolution information can be transmitted to the decoder for each frame at the frame level. This may mean directly transmitting information about the resolution of each frame from the encoder to the decoder, or including indirect information that can determine the resolution. For example, there may be a method of transmitting direct numbers of the horizontal and vertical resolutions of each frame in the header of each frame, or transmitting an index of a list for resolution information. Alternatively, there may be a direct value for the ratio with respect to the resolution of a previously transmitted frame, or an index of a list for ratio information, a direct value for the ratio with respect to the resolution information of the previously transmitted original, or an index of a list for ratio information.

[0139] The spatial resampler (520) can transmit information about the location of the cropped rectangle to the decoder when cropping is used as a resampling method. FIG. 11b includes syntax (crop_location) related to spatial resampling when cropping is applied.

[0140] Location[x] represents the x-coordinate of the left corner of the cropped rectangle. The encoder can take the logarithm and transmit it for transmission efficiency, and the decoder can extract the actual coordinate within the image based on the value transmitted from the encoder in a predetermined way.

[0141] Location[y] represents the y-coordinate of the left corner of the cropped rectangle. The encoder can take a logarithm and transmit it for transmission efficiency, and the decoder can extract the actual coordinate within the image based on the value transmitted from the encoder in a predetermined manner.

[0142] Restoration_ref_idx indicates the reference index of the image referenced to restore the region removed by the encoder. Depending on the embodiment, there may be one or more. The decoder can copy the reference image and use it to restore the surrounding region that was cropped and removed.

[0143] The decoder can use the location information when decoding the cropped image during the restoration process and restoring the cropped and removed area. In some embodiments, the location information may not be transmitted if the cropped and removed area is not restored separately, even though the image is resampled in a cropped form. Figure 11c includes syntax (crop_location) related to spatial resampling in cases where the location of the cropped rectangle changes based on the location of the region of interest.

[0144] Location[x] represents the x-coordinate of the left corner of the cropped rectangle. The encoder can take the logarithm and transmit it for transmission efficiency, and the decoder can extract the actual coordinate within the image based on the value transmitted from the encoder in a predetermined way.

[0145] Location[y] represents the y-coordinate of the left corner of the cropped rectangle. The encoder can take a logarithm and transmit it for transmission efficiency, and the decoder can extract the actual coordinate within the image based on the value transmitted from the encoder in a predetermined manner.

[0146] Restoration_ref_idx indicates the reference index of the image referenced to restore the region removed by the encoder. Depending on the embodiment, there may be one or more. The decoder can copy the reference image and use it to restore the surrounding region that was cropped and removed.

[0147] loc_delta_flag is a flag for cases where the position of the cropped rectangle changes for each frame, requiring the transmission of additional information about the changed position. The additional information can be calculated through resampling_mehod_idx[i] and delta_frame_idx[i] transmitted via Figure 11a.

[0148] delta_location[x] represents the location of the left corner of the cropped rectangle. It is the x-axis difference from the cropped location of the most recent frame that was resampled to the same size through cropping in the time sequence of the image.

[0149] delta_location[y] represents the location of the left corner of the cropped rectangle. It is the y-axis difference from the cropped location of the most recent frame that was resampled to the same size through cropping in the time sequence of the image.

[0150] The spatial resampler (530) can change the position of the cropped rectangle based on the position of the region of interest when cropping is used in the resampling process, and can transmit information about the position of the cropped rectangle to the decoder for each cropped frame through FIG. 11C. Transmitting for each frame includes not only a method of directly transmitting the syntax corresponding to the header of the cropped frame (FIG. 11C), but also an indirect method of transmitting at a higher level but matching it with the cropped frame to identify the cropped position in the corresponding frame. For example, the index of the frame resampled by cropping and the syntax information of FIG. 11C mapped to the index can be transmitted as a single sequence or GOP unit.

[0151] Referring back to FIG. 5, the region of interest extractor (530) can distinguish a region expected to be an object from the background in the input signal for special or general purposes required in machine vision applications. The region of interest extractor (530) can extract location and size information of a region expected to be an object from within an image through learning or an image processing technique. According to an embodiment, multiple regions of interest can be extracted from the input signal. The region of interest extractor (530) can transmit size and location information for each of the multiple regions of interest to the internal encoder (540). The internal encoder (540) can perform encoding by changing the image quality by performing different quantizations for the object and the background.

[0152] The internal encoder (540) can perform encoding on the input signal. The internal encoder (540) can utilize the temporal resampling information of the previously performed step for the reference structure, or can change the reference method of the object and background and the quantization parameter based on the information on the region of interest to perform effective encoding. When an input image including frames with various types of resolutions is input as shown in FIG. 9, the internal encoder (540) can support encoding by supporting multiple subsequences within a single sequence in the form of subsequences.

[0153] Fig. 12 is a block diagram including components of an encoding device for machine vision according to an embodiment of the present disclosure. Fig. 12 includes all components of Fig. 5, except that the order of the components is different. Each component of Fig. 12 corresponds to the components of Fig. 5, and thus, descriptions of each component are omitted to avoid redundancy.

[0154] Referring to FIG. 12, the encoding device (10a) can first extract a region of interest for an input signal, and then sequentially perform temporal resampling, spatial resampling, and internal encoding. The encoding device (10a) according to the encoding structure of FIG. 12 can adaptively perform temporal resampling and / or spatial resampling based on region of interest information extracted from the input signal. For example, the encoding device (10a) can perform aperiodic temporal resampling centered on a frame from which a region of interest suitable for the application purpose is extracted, or change the resolution through cropping. The procedure performed at each stage of the encoding device (10a) is the same as FIG. 5, but since the region of interest is extracted first, it has the effect that information on the region of interest can be used by the temporal resampler (510) and the spatial resampler (520).

[0155] An encoding device (10a) for performing image encoding for a machine according to one embodiment of the present disclosure includes a temporal resampler (510) for changing a frame rate for an input image, a spatial resampler (520) for changing a resolution for the input image, and an internal encoder (540) for performing encoding for the input image, and can transmit at least one of temporal restoration data including a frame rate of the input image changed by the temporal resampler together with the encoded image and spatial restoration data including a resolution of the input image changed by the spatial resampler to a decoding device (10b).

[0156] The above temporal resampler (510) upsamples or downsamples the entire frame included in the input image on a frame-by-frame basis, and the sampling period can be determined according to the application purpose of the input image.

[0157] The above temporal resampler (510) can transmit the sampling period to the decoding device (10b) when periodically resampling the input image.

[0158] The above temporal resampler (510) can transmit the index number of the resampled frame when resampling aperiodically.

[0159] The above temporal resampler (510) transmits the difference value between the previous frame index and the current frame index as information on the frame rate according to resampling to the decoding device (10b), and the difference value can be calculated as the difference between corresponding position pixels between two frames or as the MSE (mean square error) through motion prediction between two frames.

[0160] The above temporal restoration information may be transmitted in whole or in part as at least one unit among a sequence unit, a GOP unit, a frame unit, a slice unit, and a subsequence unit.

[0161] The above spatial resampler (520) upsamples or downsamples the resolution of the input image in units of frames, and the target resolution to be resampled may be the same for the entire frame or may be determined differently in units of frames depending on the application purpose.

[0162] The spatial resampler (520) may resample the resolution of all or part of each frame of the input image. Some parts of the frame of the input image may be acquired by cropping a part of the original frame. The spatial resampler (520) may upsample or downsample the resolution of a part of the input image cropped by the cropping method.

[0163] The above spatial resampler (520) can change the target resolution according to the application purpose by cropping a portion of the input image.

[0164] The above spatial resampler (520) can reorder the resampled input image by resolution when the resolution of each frame of the input image is resampled differently.

[0165] The above temporal resampler (510) and the above spatial resampler (520) resample the frame rate and resolution based on the application purpose of the input image, respectively, and the application purpose of the input image may correspond to any one of tracking, recognition, classification, and segmentation.

[0166] The encoding device (10a) further includes a region of interest extractor (530) that extracts one or more regions of interest from the input image, and the region of interest extractor (530) can distinguish between objects and backgrounds in the input image and extract one or more regions of interest corresponding to one or more objects. The temporal resampler (510) and the spatial resampler (520) can perform temporal resampling and spatial resampling based on information about one or more regions of interest for the input image.

[0167] The above region of interest extractor (530) can extract the region of interest for an input image for which at least one of temporal resampling and spatial resampling has been performed on the input image.

[0168] FIG. 13 is a block diagram including components of a decoding device for machine vision according to one embodiment of the present disclosure.

[0169] The decoding device (10b) may include an internal decoder (1310), a spatial restorer (1320), and a temporal restorer (1330). In various embodiments, some components of the decoding device (10b) may be omitted, or the execution order between components may be changed.

[0170] The internal decoder (1310) can receive a bitstream and perform decoding. Depending on the embodiment, various decoders may be used, and in this case, the internal decoder (1310) is determined to correspond to the internal encoder (540) used in the encoder.

[0171] The spatial restorer (1320) can perform spatial restoration when necessary using spatial restoration information (e.g., spatial_restoration_data) transmitted from the encoder. If this is not necessary, it may be the case that spatial restoration does not need to be performed for the application purpose of machine vision. For example, if the spatial resampler (520) of the encoder performs downsampling, encodes it, and transmits it to the decoder, but the decoder determines that upsampling at the downsampled resolution is not necessary for the application purpose of the decoder, the decoder may proceed to the next step without performing the restoration process using the spatial restoration information transmitted from the encoder. Alternatively, if the encoder determines that the application does not require spatial restoration, that is, if the spatial_restoration_flag is 0, the encoder does not transmit information about spatial restoration, and thus the decoder does not perform spatial restoration. If the decoder determines that spatial restoration is necessary or spartial_restoration_flag is 1, the decoder can extract information for spatial restoration of the decoded image through the internal decoder.

[0172] The spatial restorer (1320) can first check whether all frames have been resampled to the same resolution, and extract information related to the resolution and resampling method as many times as the number of different resolutions. If resampled to a single resolution, the spatial restorer (1320) can extract information for restoring the corresponding resolution. The spatial restorer (1320) can determine the restoration method through the resampling method. For example, if upsampling or downsampling is performed at a specific ratio, upsampling or downsampling can be performed on the decoded image at the specific ratio. If sampling filter information is transmitted at this time, the spatial restorer (1320) can apply restoration filtering based on the information.

[0173] The spatial restorer (1320) can, in another embodiment, when resampling through cropping, position the decoded image with the internal decoder through the position information of the cropped rectangle (Fig. 11b), and restore the surrounding area that has been removed from the encoder and not decoded with the internal decoder of the decoder. At this time, the spatial restorer (1320) can determine a frame to be referenced for restoration through restoration_ref_idx, and perform restoration through the corresponding image. For example, an area other than the cropped area can be used by identically copying the signal of the frame corresponding to the corresponding index. Alternatively, restoration can be performed by generating an image through deep learning. In an embodiment, when the position is changed in a cropped frame, restoration can be performed through the information of Fig. 11c.

[0174] The temporal restorer (1330) can perform temporal restoration when necessary using the temporal restoration information transmitted from the encoder. If not necessary, it may be the case that temporal restoration is not necessary for the application purpose transmitted according to FIGS. 7a and 7b. Alternatively, if the encoder determines that the application does not require spatial restoration, that is, if temporal_restoration_flag is 0, the encoder does not transmit information regarding temporal restoration, and thus the decoder may not perform temporal restoration. If the decoder determines that temporal restoration is necessary, and if temporal_restoration_flag is 1, the temporal restorer (1330) can extract necessary information from the information (temporal_restoration_data) transmitted from the encoder for temporal restoration. First, the temporal restorer (1330) can determine whether sampling was performed at the same period through same_period_flag. In the case of sampling at the same period, since the signal decoded through the internal decoder (1310) and output through the spatial restorer (1320) is sampled at the same period, the temporal restorer (1330) can restore the image corresponding to the temporal intermediate frame to restore the frame rate before sampling. For example, the temporal restorer (1330) can restore through interpolation of the input signals or through a method of restoration through a method of restoration through a copying method. Alternatively, restoration through deep learning is also possible. In the case of sampling at different periods (non-identical periods), since there is a difference in the sampling interval, the temporal restorer (1330) can extract information about the temporal sampling period through delta_frame_idx[i] and perform restoration according to the interval between indices (sampling period). As in the case of the same period, the temporal restorer (1330) can restore through methods such as interpolation, copying, and deep learning.

[0175] The decoder (10b) processes the video signal through the temporal restoration process. Filtering may be performed on the restored signals during the spatial and temporal restoration processes to address reasons such as image quality degradation. The output signal may be used as an input for an application, depending on the intended machine vision application. For example, the output signal may be used as an input value for a neural network model.

[0176] A decoding device (10b) for performing image decoding for a machine according to another embodiment of the present disclosure may include an internal decoder that receives an encoded input image from an encoding device (10a) and performs decoding, a spatial restorer that changes the resolution of the decoded image based on spatial restoration information (spatial restoration data) for the input image received from the encoding device and restores it to the resolution of the original image of the input image, and a temporal restorer that changes the frame rate of the decoded image based on temporal restoration information (temporal restoration data) for the input image received from the encoding device and restores it to the frame rate of the original image of the input image.

[0177] The above spatial restorer can upsample or downsample the resolution of the input image frame by frame based on the above spatial restoration information.

[0178] The above spatial restoration information may include at least one of information about whether spatial resampling is applied, an index of a list of resampling methods by resolution, information about the resampling method, information about a filter used in the resampler in case of upsampling and downsampling, location information of a cropped area, and image information referenced by an encoding device to restore an area removed according to spatial resampling.

[0179] The above temporal restorer can restore the input image to a frame rate of the original image of the input image based on the above temporal restoration information.

[0180] The above temporal restoration information may include at least one of information on whether temporal resampling is applied, whether periodic resampling or aperiodic resampling is applied, and in the case of aperiodic resampling, information on the frame period or the index of the resampled frame.

[0181] Another embodiment of the present disclosure provides a non-volatile computer-readable storage medium having recorded thereon instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of: changing a frame rate for an input image; changing a resolution for the input image; performing encoding for the input image; and transmitting temporal restoration data including a frame rate of the input image changed according to the temporal resampling and spatial restoration data including a resolution of the input image changed according to the spatial resampling, together with the encoded image, to a decoding device.

[0182]

[0183] FIG. 14 is a flowchart illustrating a temporal resampling method according to an example of the present disclosure.

[0184] In step S1410, the encoding device (10a) or the temporal resampler (510) can calculate an optimal resampling interval for decoding efficiency and transmit it to the decoder. The temporal resampler (510) can calculate delta_frame_idx[i] corresponding to the temporal resampling interval. The temporal resampler (510) can calculate the frame variation between the input images. The differential variation can be calculated through the difference between corresponding position pixels between the two images or the mean square error (MSE) through motion estimation (ME) between the two images.

[0185] In step S1420, the temporal resampler (510) can select a reference frame. Through step S1410, the temporal resampler (510) can calculate the frame variation between each frame and determine the reference frame based on information about a section in which the frame variation exceeds a certain size or a section in which the accumulated frame variation from the previous reference frame exceeds a certain size.

[0186] At step S1430, the temporal resampler (510) can transmit the interval between the current reference frame and the previous reference frame to the decoder via delta_frame_idx[i].

[0187]

[0188] FIG. 15 is a flowchart illustrating a temporal resampling method according to another example of the present disclosure.

[0189] In step S1510, the encoding device (10a) or the temporal resampler (510) can calculate an optimal resampling interval for decoding efficiency and transmit it to the decoder. The encoder can calculate delta_frame_idx[i] corresponding to the temporal resampling interval. The temporal resampler (510) calculates the frame variation between the input images. The differential variation can be calculated through the difference between corresponding position pixels between the two images or the MSE through motion prediction between the two images.

[0190] In step S1520, the temporal resampler (510) determines a candidate for a reference frame based on information about a section in which the frame change amount exceeds a certain size or a section in which the accumulated frame change amount from the previous reference frame exceeds a certain size.

[0191] In steps S1530 to S1550, the temporal resampler (510) can simulate various restoration methods that can be performed in the decoder using the reference frame determined as a candidate in the above step to select an optimal restoration method. The temporal resampler (510) can determine the optimal restoration method by calculating an error between a frame restored through restoration and a frame resampled through temporal resampling. Information related to the optimal restoration method can be transmitted to the decoder according to FIGS. 16A and 16B. FIGS. 16A and 16B are examples of syntax and semantics for temporal resampling according to an embodiment of the present disclosure.

[0192] At step S1550, the temporal resampler (510) predicts a restoration error of a frame to be restored through restoration in a decoder through temporal restoration, and if the reference frame candidate is not suitable, repeatedly performs verification through temporal restoration with the reference frame candidate to select an optimal reference frame.

[0193] At step S1560, the temporal resampler (510) can transmit information about the interval between reference frames determined at the above step, i.e., the number of temporally sampled frames between reference frames, to the decoder as delta_frame_idx[i]. According to an embodiment, an optimal frame rate for an application can be determined, and an optimal delta_frame_idx[i] can be calculated by reflecting the frame rate, the frame variation between reference frames, and the restoration error for temporal restoration.

[0194] At step S1570, the temporal resampler (510) can transmit delta_frame_idx[i] to the decoder according to FIGS. 16a and 16b. FIGS. 16a and 16b are syntax and semantics representing information that the temporal resampler (510) performed temporal resampling to correspond to the optimal restoration selected by simulating various restorations.

[0195]

[0196] FIGS. 16A and 16B are examples of syntax and semantics for temporal resampling according to one embodiment of the present disclosure.

[0197] Figure 16a includes the following syntax.

[0198] target_application_idx represents the information index (idx) for the machine vision target application to be used by the decoder. If not specified, information about the corresponding index in the target application list, such as tracking, recognition, classification, or segmentation, can be transmitted. In an embodiment, the idx value may indicate unspecified if 0, tracking if 1, recognition if 2, classification if 3, and segmentation if 4.

[0199] temporal_restoration_flag is a flag indicating whether temporal resampling is applied.

[0200] When aperiodic resampling is applied, the number of frames can be calculated through temporal_resampling_rate_idx. For example, if the frame number is 30 and the resampling rate is 1 / 2, frame_num becomes 15. In some embodiments, if the frame number is fixed for the application, frame_num can be determined by the fixed number. The difference value (delta_frame_idx[i]) from the previous frame index is transmitted according to the frame number. Since it is the difference value from the previous index, it is sufficient to transmit a number that is one less than the frame_num calculated by the frame rate.

[0201] restoration_method[i] represents the index of the list of restoration methods agreed upon between the decoder and the encoder. For example, 0 represents temporal bilinear interpolation, 1 represents copying the previous reference frame in display time, 2 represents copying the previous restored image in decoding order, and 3 represents deep learning-based image restoration. This transmits the index of the list of restoration methods agreed upon between the decoder and the encoder, and performs restoration in the same way on the frame temporally restored through delta_frame_idx[i] using the method of the corresponding index.

[0202] Figure 16b includes the following syntax.

[0203] target_application_idx represents the information index (idx) for the machine vision target application to be used by the decoder. If not specified, information about the corresponding index in the target application list, such as tracking, recognition, classification, or segmentation, can be transmitted. In an embodiment, the idx value may indicate unspecified if 0, tracking if 1, recognition if 2, classification if 3, and segmentation if 4.

[0204] temporal_restoration_flag is a flag indicating whether temporal resampling is applied.

[0205] When aperiodic resampling is applied, the number of frames can be calculated through temporal_resampling_rate_idx. For example, if the frame number is 30 and the resampling rate is 1 / 2, frame_num becomes 15. In some embodiments, if the frame number is fixed for the application, frame_num can be determined by the fixed number. The difference value (delta_frame_idx[i]) from the previous frame index is transmitted according to the frame number. Since it is the difference value from the previous index, it is sufficient to transmit a number that is one less than the frame_num calculated by the frame rate.

[0206] restoration_method[i][j] represents the index of the list of restoration methods agreed upon between the decoder and the encoder. For example, 0 represents temporal bilinear interpolation, 1 represents copying the previous reference frame in display time, 2 represents copying the previous restored image in decoding order, and 3 represents deep learning-based image restoration. At this time, the index of the restoration method is separately transmitted for each frame to be temporally restored through delta_frame_idx[i], and the decoder can perform temporal restoration using the restoration method transmitted for each frame.

[0207]

[0208] The examples of the present disclosure presented in this specification and drawings are intended solely to facilitate the technical content of the present disclosure and to aid understanding thereof. They are not intended to limit the scope of the present disclosure. It will be apparent to those skilled in the art that other variations are possible in addition to the examples described above.

[0209] The claims set forth in this specification may be combined in various ways. For example, the technical features of the method claims of this specification may be combined and implemented as a device, and the technical features of the device claims of this specification may be combined and implemented as a method. Furthermore, the technical features of the method claims and the technical features of the device claims of this specification may be combined and implemented as a device, and the technical features of the method claims and the technical features of the device claims of this specification may be combined and implemented as a method.

Claims

1. In an encoding device that performs image encoding for a machine, A temporal resampler that changes the frame rate of the input video; A spatial resampler that changes the resolution of the input image; and It includes an internal encoder that performs encoding on the input image, An encoding device that transmits to a decoding device at least one of temporal restoration data including a frame rate of an input image changed by the temporal resampler together with the encoded image and spatial restoration data including a resolution of an input image changed by the spatial resampler.

2. In paragraph 1, The above temporal resampler upsamples or downsamples the entire frame included in the input image on a frame-by-frame basis, An encoding device in which the above sampling period is determined according to the application purpose of the input image.

3. In paragraph 1, An encoding device, wherein the temporal resampler transmits the sampling period to the decoding device when periodically resampling the input image.

4. In paragraph 1, An encoding device wherein the temporal resampler transmits an index number of a resampled frame when resampling aperiodically.

5. In paragraph 1, The above temporal resampler transmits the difference value between the previous frame index and the current frame index as information about the frame rate according to resampling to the decoding device, An encoding device wherein the above difference value is calculated as the difference between corresponding position pixels between two frames or as the mean square error (MSE) through motion prediction between two frames.

6. In paragraph 1, An encoding device that transmits the above temporal restoration information in whole or in part as at least one unit among a sequence unit, a GOP unit, a frame unit, a slice unit, and a subsequence unit.

7. In paragraph 1, The above spatial resampler upsamples or downsamples the resolution of the input image in frame units, The target resolution to be resampled is determined by the encoding device, either the same for the entire frame or different on a frame-by-frame basis, depending on the application purpose.

8. In paragraph 1, An encoding device wherein the spatial resampler resamples the resolution of all or part of each frame of the input image.

9. In paragraph 8, An encoding device that obtains a portion of each frame of the input image by cropping a portion of the original frame.

10. In paragraph 9, The above spatial resampler is an encoding device that upsamples or downsamples the resolution of a portion of an original frame cut out in a cropping manner.

11. In paragraph 1, The above spatial resampler is an encoding device that changes the target resolution according to the application purpose by cropping a portion of the input image.

12. In paragraph 1, The above spatial resampler is an encoding device that reorders the resampled input image by resolution when the resolution of each frame of the input image is resampled differently.

13. In paragraph 1, The above temporal resampler and the above spatial resampler resample the frame rate and resolution based on the application purpose of the input image, respectively. An encoding device for which the application purpose of the above input image corresponds to any one of tracking, recognition, classification, and segmentation.

14. In paragraph 1, Further comprising a region of interest extractor for extracting one or more regions of interest from the input image, The above region of interest extractor is an encoding device that distinguishes objects and backgrounds from the input image and extracts one or more regions of interest corresponding to one or more objects.

15. In paragraph 14, An encoding device wherein the temporal resampler and the spatial resampler perform temporal resampling and spatial resampling based on information about one or more regions of interest of the input image.

16. In paragraph 14, An encoding device, wherein the region of interest extractor extracts the region of interest from an input image for which at least one of temporal resampling and spatial resampling has been performed on the input image.

17. In a decoding device that performs image decoding for a machine, An internal decoder that receives an encoded input image from an encoding device and performs decoding; A spatial restorer that changes the resolution of the decoded image based on spatial restoration data for the input image received from the encoding device and restores the resolution of the original image of the input image; and A decoding device including a temporal restorer that changes the frame rate of the decoded image based on temporal restoration data for the input image received from the encoding device and restores the frame rate of the original image of the input image.

18. In paragraph 17, The above spatial restorer upsamples or downsamples the resolution of the input image frame by frame based on the above spatial restoration information, A decoding device, wherein the spatial restoration information includes at least one of: whether spatial resampling is applied, an index of a list of resampling methods by resolution, information about the resampling method, information about a filter used in the resampler in case of upsampling and downsampling, location information of a cropped area, and image information referenced to restore an area removed according to spatial resampling in an encoding device.

19. In paragraph 17, The temporal restorer restores the input image to the frame rate of the original image of the input image based on the temporal restoration information, A decoding device, wherein the temporal restoration information includes at least one of information on whether temporal resampling is applied, whether periodic resampling or aperiodic resampling is applied, and in the case of aperiodic resampling, information on the frame period or the index of the resampled frame.

20. A non-volatile computer-readable storage medium that records commands, The above instructions, when executed by one or more processors, cause the one or more processors to: A temporal resampling step that changes the frame rate of the input image; A spatial resampling step for changing the resolution of the above input image; A step of performing encoding on the above input image; and A step of transmitting temporal restoration information including a frame rate of an input image changed according to the temporal resampling together with the encoded image and spatial restoration information including a resolution of an input image changed according to the spatial resampling to a decoding device, Non-transitory computer-readable storage medium.

Images