Encoding / decoding method, device, and recording medium using scaling parameter

Abstract

An encoding / decoding method, device, and recording medium using a scaling parameter according to an embodiment of the present disclosure may: perform at least one of spatial sampling, feature map dimension reduction, and feature map truncation on an input image including one or more feature maps; extract one or more scaling parameters related to a scale change of the one or more feature maps used in the spatial sampling process, the feature map dimension reduction process, or the feature map truncation process for the one or more feature maps; perform quantization of the one or more feature maps into an integer form; rearrange the one or more feature maps according to a predetermined unit to perform packing; and encode the one or more feature maps, encoding information extracted in the process, and the one or more scaling parameters so as to generate a bitstream.

Description

Encoding / decoding method, device and recording medium using scaling parameters

[0001] The present disclosure may relate to an image encoding / decoding method, device and recording medium for a machine, and an encoding / decoding method, device and recording medium using a scaling parameter.

[0002] With the continuous development of the information and communication industry, broadcasting services with HD (High Definition) resolution have spread worldwide.

[0003] Through this proliferation, many users have become accustomed to high-resolution and high-quality images and / or videos, and the demand for higher-resolution and high-quality images / videos, such as 4K or 8K or higher UHD (Ultra High Definition) images / videos, has increased in various fields.

[0004] The technology for coding this UHD video data was completed in 2013 through the standard technology HEVC (High Efficiency Video Coding).

[0005] HEVC is a next-generation video compression technology with a higher compression ratio and lower complexity than the previous H.264 / AVC technology, and is a key technology for effectively compressing the massive data of HD and UHD video.

[0006] HEVC performs block-by-block encoding, like previous compression standards.

[0007] However, unlike H.264 / AVC, there is only one profile. The core encoding technologies included in HEVC's sole profile are divided into eight areas: hierarchical encoding structure technology, transform technology, quantization technology, intra-frame prediction encoding technology, inter-frame motion prediction technology, entropy encoding technology, loop filter technology, and other technologies.

[0008] Since the establishment of the HEVC video codec in 2013, the Versatile Video Coding (VVC) standard, a next-generation video codec that aims to improve performance by more than twice that of HEVC, has been developed to address the expansion of realistic video and virtual reality services utilizing 4K and 8K video images. VVC is called H.266.

[0009] H.266 (VVC) was developed with the goal of being more than twice as efficient as the previous generation codec, H.265 (HEVC). VVC was initially developed with resolutions over 4K in mind, but it was also developed for ultra-high-resolution video processing at a whopping 16K level to support 360-degree videos due to the expansion of the VR market. In addition, as the HDR market is expanding due to the development of display technology, it supports 16-bit color depth as well as 10-bit color depth to respond to this, and supports brightness expressions of 1000 nits, 4000 nits, and 10000 nits. In addition, since it is being developed with the VR market and 360-degree video market in mind, it supports partial frame rates in the range of 0 to 120 FPS.

[0010] Advances in Artificial Intelligence

[0011] Artificial intelligence (AI) is also steadily developing. AI refers to the artificial imitation of human intelligence, including the ability to recognize, classify, infer, predict, and control / decision-making.

[0012] With the advancement of artificial intelligence technology and the increase in Internet of Things (IoT) devices, machine-to-machine traffic is expected to explode, and machine-dependent image analysis is expected to become widely used.

[0013] However, as the amount of images to be analyzed by machines is expected to increase exponentially, issues with server load and power consumption are expected to arise.

[0014] Accordingly, the present disclosure aims to provide a method, device, and recording medium for encoding / decoding a feature map using a scaling parameter so as to enable effective machine-based image analysis.

[0015] An encoding / decoding method, device, and recording medium using a scaling parameter according to one embodiment of the present disclosure may perform at least one of spatial sampling, feature map dimensionality reduction, or feature map truncation on an input image including one or more feature maps, extract one or more scaling parameters related to a scale change of the one or more feature maps used in the spatial sampling process for the one or more feature maps, the feature map dimensionality reduction process, or the feature map truncation process, perform quantization of the one or more feature maps into integer form, perform packing by rearranging the one or more feature maps according to a predetermined unit, and encode the one or more feature maps, encoding information extracted in the process, and the one or more scaling parameters to generate a bitstream.

[0016] In an encoding / decoding method, device, and recording medium using a scaling parameter according to one embodiment of the present disclosure, the order in which a spatial sampling operation, a feature map dimensionality reduction operation, and a feature map truncation operation are performed for the one or more feature maps is variable, and the one or more feature maps are input values ​​of an operation performed first, and an output value of each operation can be an input value of a next operation.

[0017] In an encoding / decoding method, device and recording medium using a scaling parameter according to one embodiment of the present disclosure, the operation of performing spatial sampling on the one or more feature maps may perform at least one of spatial upsampling, downsampling or truncation on the one or more feature maps.

[0018] In an encoding / decoding method, device, and recording medium using a scaling parameter according to one embodiment of the present disclosure, the operation of performing dimensionality reduction on the one or more feature maps may include extracting redundant components included in the one or more feature maps, removing the redundant components, and performing mapping on feature maps from which redundant components have been removed among the one or more feature maps.

[0019] In an encoding / decoding method, device and recording medium using a scaling parameter according to one embodiment of the present disclosure, the operation of performing truncation on the one or more feature maps may remove at least a portion of the one or more feature maps.

[0020] In an encoding / decoding method, device, and recording medium using a scaling parameter according to one embodiment of the present disclosure, the operation of extracting the one or more scaling parameters may include extracting one or more pieces of information about the one or more feature maps, and deriving a scaling parameter based on the one or more pieces of information, or deriving and outputting a differential value for an already derived scaling parameter.

[0021] In an encoding / decoding method, device, and recording medium using a scaling parameter according to one embodiment of the present disclosure, one or more pieces of information extracted from the one or more feature maps may include at least some of a unit-wise average, a maximum value, a minimum value, a median value, a variance, a structural similarity index measure (SSIM), an explained variability ratio (EVR), a basis vector, or a basis vector coefficient.

[0022] In an encoding / decoding method, device and recording medium using a scaling parameter according to one embodiment of the present disclosure, a unit for extracting the one or more scaling parameters from the one or more feature maps includes a sequence, a frame, a frame group, a feature level, a feature, a layer or a channel, and a certain range for extracting the one or more pieces of information can be designated within each unit.

[0023] In an encoding / decoding method, device and recording medium using a scaling parameter according to one embodiment of the present disclosure, the scaling parameter can be transmitted to a decoding device in units of at least one of a task, a sequence, a frame, a frame group, a feature level or a channel.

[0024] In an encoding / decoding method, device, and recording medium using a scaling parameter according to one embodiment of the present disclosure, the encoding information may include at least some of a scanning order, a packing method, a quantization method, a quantization range, a truncation method, a truncation degree, an original data size, a dimensionality reduction method, channel grouping information, a coding method for each of the grouped channels, or a spatial sampling method, a scaling method, and a scaling parameter for the one or more feature maps.

[0025] According to the present disclosure, image analysis by a machine can be effectively performed.

[0026] Figure 1 schematically illustrates an example of a video / image coding system.

[0027] Figure 2 is a drawing schematically illustrating the configuration of a video / image encoding device.

[0028] Figure 3 is a drawing schematically illustrating the configuration of a video / image decoding device.

[0029] Figures 4a to 4d are exemplary diagrams showing a VCM encoder and a VCM decoder.

[0030] FIG. 5 illustrates a block diagram of an encoding device according to one embodiment of the present disclosure.

[0031] FIG. 6 is a block diagram including components of a spatial sampling performer according to one embodiment of the present disclosure.

[0032] FIG. 7 is a block diagram including components of a scaling parameter extractor according to one embodiment of the present disclosure.

[0033] FIG. 8 is a block diagram including components of a dimensionality reduction performer according to one embodiment of the present disclosure.

[0034] FIG. 9 is a block diagram including components of a cutting performer according to one embodiment of the present disclosure.

[0035] FIG. 10 is a block diagram including components of a quantization performer according to one embodiment of the present disclosure.

[0036] FIG. 11 is a block diagram including components of a packing performer according to one embodiment of the present disclosure.

[0037] FIG. 12 is a block diagram including components of an internal encoding performer according to one embodiment of the present disclosure.

[0038] FIG. 13 illustrates a block diagram of a decoding device according to one embodiment of the present disclosure.

[0039] FIG. 14 is a block diagram including components of an internal decryption performer according to one embodiment of the present disclosure.

[0040] FIG. 15 is a block diagram including components of a feature map scaling performer according to one embodiment of the present disclosure.

[0041] Specific structural or step-by-step descriptions of embodiments according to the concept of the present disclosure disclosed in this specification or application are merely illustrative for the purpose of explaining embodiments according to the concept of the present disclosure, and embodiments according to the concept of the present disclosure may be implemented in various forms, and embodiments according to the concept of the present disclosure may be implemented in various forms and should not be construed as being limited to the embodiments described in this specification or application.

[0042] Embodiments according to the concept of the present disclosure may be subject to various modifications and take various forms. Therefore, specific embodiments are illustrated in the drawings and described in detail in this specification or application. However, this is not intended to limit embodiments according to the concept of the present disclosure to specific disclosed forms, and it should be understood that all modifications, equivalents, and alternatives included within the spirit and technical scope of the present disclosure are included.

[0043] While terms such as "first" and / or "second" may be used to describe various components, these components should not be limited by these terms. These terms are only intended to distinguish one component from another; for example, without departing from the scope of the present disclosure, a first component may be referred to as a "second component," and similarly, a second component may also be referred to as a "first component."

[0044] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. Conversely, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Other expressions that describe the relationship between components, such as "between" and "directly between" or "adjacent to" and "directly adjacent to", should be interpreted similarly.

[0045] The terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit the present disclosure. The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, it should be understood that the terms "comprises" or "has" indicate the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0046] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0047] Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0048] In describing the embodiments, descriptions of technical contents that are well known in the technical field to which the present disclosure belongs and are not directly related to the present disclosure are omitted.

[0049] This is to convey the gist of the present disclosure more clearly without obscuring it by omitting unnecessary explanations.

[0050] This document relates to video / image coding. For example, the method / embodiment disclosed in this document may be related to the Versatile Video Coding (VVC) standard (ITU-T Rec. H.266), the next-generation video / image coding standard after VVC, or other video coding-related standards (e.g., the High Efficiency Video Coding (HEVC) standard (ITU-T Rec. H.265), the essential video coding (EVC) standard, the AVS2 standard, etc.).

[0051] This document presents various embodiments of video / image coding, and unless otherwise stated, the embodiments may be performed in combination with each other.

[0052] In this document, "video" can refer to a series of images over time. "Picture" generally refers to a unit representing a single image from a specific time period, and "slice" / "tile" are units that constitute part of a picture in coding.

[0053] A slice / tile can contain one or more coding tree units (CTUs). A picture can consist of one or more slices / tiles. A picture can consist of one or more tile groups. A tile group can contain one or more tiles.

[0054] A pixel or pel can mean the smallest unit that constitutes a picture (or image). Additionally, the term "sample" can be used as a counterpart to a pixel. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luma component, or only the pixel / pixel value of the chroma component. Alternatively, a sample can mean a pixel value in the spatial domain, or when such a pixel value is converted to the frequency domain, it can mean a transform coefficient in the frequency domain.

[0055] A unit may represent a basic unit of image processing. A unit may include at least one of a specific region of a picture or information related to that region.

[0056] A unit can contain one luma block and two chroma (e.g., cb, cr) blocks. The term "unit" may sometimes be used interchangeably with the terms "block" or "area." In general, an MxN block can contain a set (or array) of samples (or array of samples) or transform coefficients, each consisting of M columns and N rows.

[0057] Figure 1 schematically illustrates an example of a video / image coding system.

[0058] Referring to FIG. 1, a video / image coding system may include a source device and a receiving device. The source device may transmit encoded video / image information or data to the receiving device via a digital storage medium or a network in the form of a file or streaming.

[0059] The source device may include a video source, an encoding device, and a transmission unit. The receiving device may include a reception unit, a decoding device, and a renderer.

[0060] The encoding device may be referred to as a video / image encoding device, and the decoding device may be referred to as a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display unit, and the display unit may be configured as a separate device or an external component.

[0061] A video source may obtain video / images through a process of capturing, synthesizing, or generating video / images. The video source may include a video / image capture device and / or a video / image generation device. A video / image capture device may include, for example, one or more cameras, a video / image archive containing previously captured video / images, etc. A video / image generation device may include, for example, a computer, a tablet, a smartphone, etc., and may (electronically) generate video / images. For example, a virtual video / image may be generated through a computer, etc., in which case the video / image capture process may be replaced by a process of generating related data.

[0062] An encoding device can encode input video / images. The encoding device can perform a series of procedures, such as prediction, transformation, and quantization, to improve compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0063] The transmission unit can transmit encoded video / image information or data output in bitstream form to the receiving unit of the receiving device via a digital storage medium or network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit can include an element for generating a media file using a predetermined file format and an element for transmission via a broadcasting / communication network.

[0064] The receiving unit can receive / extract the bitstream and transmit it to a decoding device.

[0065] The decoding device can decode the video / image by performing a series of procedures such as inverse quantization, inverse transformation, and prediction corresponding to the operation of the encoding device.

[0066] The renderer can render decoded video / images. The rendered video / images can be displayed through the display unit.

[0067] Figure 2 is a drawing schematically illustrating the configuration of a video / image encoding device.

[0068] The term “video encoding device” hereinafter may include a video encoding device.

[0069] Referring to FIG. 2, the encoding device (10a) may be configured to include an image partitioner (10a-10), a prediction unit (predictor) (10a-20), a residual processor (residual processor) (10a-30), an entropy encoder (entropy encoder) (10a-40), an adder (adder) (10a-50), a filter (filter) (10a-60), and a memory (10a-70). The prediction unit (10a-20) may include an inter prediction unit (10a-21) and an intra prediction unit (10a-22). The residual processing unit (10a-30) may include a transformer (10a-32), a quantizer (10a-33), a dequantizer (10a-34), and an inverse transformer (10a-35). The residual processing unit (10a-30) may further include a subtractor (10a-31). The addition unit (10a-50) may be called a reconstructor or a reconstructed block generator. The above-described image segmentation unit (10a-10), prediction unit (10a-20), residual processing unit (10a-30), entropy encoding unit (10a-40), addition unit (10a-50), and filtering unit (10a-60) may be configured by one or more hardware components (e.g., encoder chipset or processor) according to an embodiment. In addition, the memory (10a-70) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (10a-70) as an internal / external component.

[0070] The image segmentation unit (10a-10) can segment an input image (or picture, frame) input to the encoding device (10a) into one or more processing units.

[0071] For example, the processing unit may be called a coding unit (CU). In this case, the coding unit may be recursively split from a coding tree unit (CTU) or a largest coding unit (LCU) according to a Quad-tree binary-tree ternary-tree (QTBTTT) structure. For example, one coding unit may be split into multiple coding units of deeper depth based on a quad-tree structure, a binary-tree structure, and / or a ternary structure. In this case, for example, the quad-tree structure may be applied first, and the binary-tree structure and / or the ternary structure may be applied later. Alternatively, the binary-tree structure may be applied first. The coding procedure according to the present document may be performed based on the final coding unit that is no longer split. In this case, based on coding efficiency according to image characteristics, etc., the maximum coding unit can be used as the final coding unit, or, if necessary, the coding unit can be recursively divided into coding units of lower depths, and the coding unit of the optimal size can be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration described below. As another example, the processing unit may further include a prediction unit (PU) or a transformation unit (TU). In this case, the prediction unit and the transformation unit may each be divided or partitioned from the final coding unit described above. The prediction unit may be a unit of sample prediction, and the transformation unit may be a unit for deriving a transform coefficient and / or a unit for deriving a residual signal from a transform coefficient.

[0072] The term "unit" may be used interchangeably with terms such as "block" or "area" depending on the case. In general, an MxN block can represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luminance component, or only the pixel / pixel value of the chroma component. A sample can be used as a term corresponding to a pixel or pel in a picture (or image).

[0073] The subtraction unit (10a-31) can subtract the prediction signal (predicted block, prediction samples, or prediction sample array) output from the prediction unit (10a-20) from the input image signal (original block, original samples, or original sample array) to generate a residual signal (residual block, residual samples, or residual sample array), and the generated residual signal is transmitted to the conversion unit (10a-32). The prediction unit (10a-20) can perform prediction on a block to be processed (hereinafter, referred to as a current block) and generate a predicted block including prediction samples for the current block.

[0074] The prediction unit (10a-20) can determine whether intra-prediction or inter-prediction is applied to the current block or CU unit. As described later in the description of each prediction mode, the prediction unit can generate various prediction-related information, such as prediction mode information, and transmit it to the entropy encoding unit (10a-40). The prediction-related information can be encoded by the entropy encoding unit (10a-40) and output in the form of a bitstream.

[0075] The intra prediction unit (10a-22) can predict the current block by referring to samples within the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from it, depending on the prediction mode.

[0076] In intra prediction, prediction modes can include multiple non-directional modes and multiple directional modes. Non-directional modes can include, for example, DC modes and planar modes. Directional modes can include, for example, 33 directional prediction modes or 65 directional prediction modes, depending on the granularity of the prediction direction.

[0077] However, this is only an example; depending on the settings, a greater or lesser number of directional prediction modes may be used. The intra prediction unit (10a-22) may also determine the prediction mode to be applied to the current block by utilizing the prediction mode applied to the surrounding blocks.

[0078] The inter prediction unit (10a-21) can derive a predicted block for a current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on an inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring block can include a spatial neighboring block existing in the current picture and a temporal neighboring block existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The above temporal neighboring blocks may be called collocated reference blocks, collocated CUs (colCUs), etc., and a reference picture including the temporal neighboring blocks may be called a collocated picture (colPic). For example, the inter prediction unit (10a-21) may construct a motion information candidate list based on the neighboring blocks, and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction may be performed based on various prediction modes, and for example, in the case of skip mode and merge mode, the inter prediction unit (10a-21) may use the motion information of the neighboring blocks as the motion information of the current block. In the case of skip mode, unlike the merge mode, a residual signal may not be transmitted.In the motion vector prediction (MVP) mode, the motion vector of the surrounding blocks is used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0079] The prediction unit (10a-20) can generate a prediction signal based on various prediction methods described below. For example, the prediction unit can apply intra prediction or inter prediction to predict a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). In addition, the prediction unit can perform intra block copy (IBC) to predict a block. The intra block copy can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in this document.

[0080] The prediction signal generated through the inter prediction unit (10a-21) and / or the intra prediction unit (10a-22) can be used to generate a reconstructed signal or a residual signal. The transform unit (10a-32) can apply a transform technique to the residual signal to generate transform coefficients. For example, the transform technique can include a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), a Graph-Based Transform (GBT), or a Conditionally Non-linear Transform (CNT). Here, GBT refers to a transform obtained from a graph when the relationship information between pixels is expressed as a graph. CNT refers to a transform obtained based on generating a prediction signal using all previously reconstructed pixels. In addition, the transform process can be applied to a pixel block having a square equal size, or can be applied to a block of a non-square variable size.

[0081] The quantization unit (10a-33) quantizes the transform coefficients and transmits them to the entropy encoding unit (10a-40), and the entropy encoding unit (10a-40) can encode the quantized signal (information about the quantized transform coefficients) and output it as a bitstream. The information about the quantized transform coefficients can be called residual information.

[0082] The quantization unit (10a-33) can rearrange the quantized transform coefficients in the form of a block into a one-dimensional vector based on the coefficient scan order, and can also generate information about the quantized transform coefficients based on the quantized transform coefficients in the form of the one-dimensional vector. The entropy encoding unit (10a-40) can perform various encoding methods, such as, for example, exponential Golomb, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), etc.

[0083] The entropy encoding unit (10a-40) may encode, together or separately, information necessary for video / image restoration (e.g., values ​​of syntax elements, etc.) in addition to quantized transform coefficients. The encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream in the form of a network abstraction layer (NAL) unit. The video / image information may further include information regarding various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. The signaling / transmitted information and / or syntax elements described later in this document may be encoded through the above-described encoding procedure and included in the bitstream. The bitstream may be transmitted through a network or may be stored in a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The signal output from the entropy encoding unit (10a-40) may be configured as an internal / external element of the encoding device (10a) by a transmitting unit (not shown) that transmits and / or a storing unit (not shown) that stores the signal, or the transmitting unit may be included in the entropy encoding unit (10a-40).

[0084] The quantized transform coefficients output from the quantization unit (10a-33) can be used to generate a prediction signal. For example, by applying inverse quantization and inverse transformation to the quantized transform coefficients through the inverse quantization unit (10a-34) and the inverse transform unit (10a-35), a residual signal (residual block or residual samples) can be reconstructed. The addition unit (10a-50) can add the reconstructed residual signal to the prediction signal output from the prediction unit (10a-20), thereby generating a reconstructed signal (reconstructed picture, reconstructed block, reconstructed samples, or reconstructed sample array). When there is no residual for the target block to be processed, such as when skip mode is applied, the predicted block can be used as the reconstructed block. The generated restoration signal can be used for intra prediction of the next processing target block within the current picture, and can also be used for inter prediction of the next picture after filtering as described below.

[0085] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture encoding and / or restoration process.

[0086] The filtering unit (10a-60) can improve subjective / objective picture quality by applying filtering to the restoration signal. For example, the filtering unit (10a-60) can apply various filtering methods to the restoration picture to generate a modified restoration picture, and store the modified restoration picture in the memory (10a-70), specifically, in the DPB of the memory (10a-70). The various filtering methods may include, for example, deblocking filtering, sample adaptive offset (SAO), an adaptive loop filter, a bilateral filter, etc. The filtering unit (10a-60) can generate various information regarding filtering and transmit the information to the entropy encoding unit (10a-90), as described below in the description of each filtering method. The information regarding filtering may be encoded by the entropy encoding unit (10a-90) and output in the form of a bitstream.

[0087] The modified restored picture transmitted to the memory (10a-70) can be used as a reference picture in the inter prediction unit (10a-80). Through this, when inter prediction is applied, the encoding device can avoid prediction mismatch between the encoding device (10a) and the decoding device, and can also improve encoding efficiency.

[0088] The DPB of the memory (10a-70) can store the modified restored picture to be used as a reference picture in the inter prediction unit (10a-21). The memory (10a-70) can store motion information of a block from which motion information in the current picture is derived (or encoded) and / or motion information of blocks in a picture that has already been restored. The stored motion information can be transmitted to the inter prediction unit (10a-21) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (10a-70) can store restored samples of restored blocks in the current picture and transmit them to the intra prediction unit (10a-22).

[0089] Figure 3 is a drawing schematically illustrating the configuration of a video / image decoding device.

[0090] Referring to FIG. 3, the decoding device (10b) may be configured to include an entropy decoder (10b-10), a residual processor (10b-20), a predictor (10b-30), an adder (10b-40), a filter (10b-50), and a memory (10b-60). The predictor (10b-30) may include an inter-prediction unit (10b-31) and an intra-prediction unit (10b-32). The residual processor (10b-20) may include a dequantizer (10b-21) and an inverse transformer (10b-21). The entropy decoding unit (10b-10), residual processing unit (10b-20), prediction unit (10b-30), addition unit (10b-40), and filtering unit (10b-50) described above may be configured by a single hardware component (e.g., decoder chipset or processor) according to an embodiment. In addition, the memory (10b-60) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (10b-60) as an internal / external component.

[0091] When a bitstream including video / image information is input, the decoding device (10b) can restore the image corresponding to the process in which the video / image information is processed in the encoding device of FIG. 2. For example, the decoding device (10b) can derive units / blocks based on block division-related information obtained from the bitstream. The decoding device (10b) can perform decoding using a processing unit applied in the encoding device. Therefore, the processing unit of decoding may be, for example, a coding unit, and the coding unit may be divided from a coding tree unit or a maximum coding unit according to a quad tree structure, a binary tree structure, and / or a ternary tree structure. One or more transform units may be derived from the coding unit. Then, the restored image signal decoded and output by the decoding device (10b) can be reproduced through a reproduction device.

[0092] The decoding device (10b) can receive a signal output from the encoding device of FIG. 2 in the form of a bitstream, and the received signal can be decoded through the entropy decoding unit (10b-10). For example, the entropy decoding unit (10b-10) can parse the bitstream to derive information (e.g., video / image information) necessary for image restoration (or picture restoration). The video / image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information.

[0093] The decoding device can further decode the picture based on information about the parameter set and / or the general restriction information. The signaling / received information and / or syntax elements described later in this document can be decoded and obtained from the bitstream through the decoding procedure. For example, the entropy decoding unit (10b-10) can decode information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output values ​​of syntax elements required for image restoration and quantized values ​​of transform coefficients for the residual.

[0094] In more detail, the CABAC entropy decoding method receives a bin corresponding to each syntax element in a bitstream, determines a context model using information of the syntax element to be decoded and decoding information of surrounding and decoding target blocks or information of symbols / bins decoded in a previous step, and predicts the occurrence probability of the bin according to the determined context model to perform arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element. At this time, the CABAC entropy decoding method can update the context model using information of the decoded symbol / bin for the context model of the next symbol / bin after determining the context model. Among the information decoded by the entropy decoding unit (10b-10), information regarding prediction is provided to the prediction unit (10b-30), and information regarding the residual on which entropy decoding has been performed by the entropy decoding unit (10b-10), i.e., quantized transform coefficients and related parameter information, can be input to the inverse quantization unit (10b-21).

[0095] In addition, information regarding filtering among the information decoded by the entropy decoding unit (10b-10) may be provided to the filtering unit (10b-50). Meanwhile, a receiving unit (not shown) that receives a signal output from an encoding device may be further configured as an internal / external element of the decoding device (10b), or the receiving unit may be a component of the entropy decoding unit (10b-10). Meanwhile, the decoding device according to the present document may be called a video / video / picture decoding device, and the decoding device may be divided into an information decoder (video / video / picture information decoder) and a sample decoder (video / video / picture sample decoder). The above information decoder may include the entropy decoding unit (10b-10), and the sample decoder may include at least one of the inverse quantization unit (10b-21), the inverse transformation unit (10b-22), the prediction unit (10b-30), the addition unit (10b-40), the filtering unit (10b-50), or the memory (10b-60).

[0096] The inverse quantization unit (10b-21) can inverse quantize the quantized transform coefficients and output the transform coefficients. The inverse quantization unit (10b-21) can rearrange the quantized transform coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scanning order performed in the encoding device. The inverse quantization unit (10b-21) can perform inverse quantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain transform coefficients.

[0097] In the inverse transform unit (10b-22), the transform coefficients are inversely transformed to obtain a residual signal (residual block, residual sample array).

[0098] The prediction unit can perform a prediction for the current block and generate a predicted block including prediction samples for the current block.

[0099] The prediction unit can determine whether intra prediction or inter prediction is applied to the current block based on the information about the prediction output from the entropy decoding unit (10b-10), and can determine a specific intra / inter prediction mode.

[0100] The prediction unit can generate a prediction signal based on various prediction methods described below. For example, the prediction unit can apply intra prediction or inter prediction to predict a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). In addition, the prediction unit can perform intra block copy (IBC) to predict a block. The intra block copy can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in this document.

[0101] The intra prediction unit (10b-32) can predict the current block by referencing samples within the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from it, depending on the prediction mode.

[0102] In intra prediction, prediction modes may include multiple non-directional modes and multiple directional modes. The intra prediction unit (10b-32) may determine the prediction mode to be applied to the current block by utilizing the prediction modes applied to the surrounding blocks.

[0103] The inter prediction unit (10b-31) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on an inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.).

[0104] In the case of inter prediction, the neighboring blocks may include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit (10b-31) may construct a motion information candidate list based on the neighboring blocks, and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter prediction may be performed based on various prediction modes, and the information regarding the prediction may include information indicating the mode of inter prediction for the current block.

[0105] The addition unit (10b-40) can generate a restoration signal (restored picture, restoration block, restoration sample array) by adding the acquired residual signal to the prediction signal (predicted block, prediction sample array) output from the prediction unit (10b-30). In cases where there is no residual for the block to be processed, such as when skip mode is applied, the predicted block can be used as the restoration block.

[0106] The addition unit (10b-40) may be called a restoration unit or a restoration block generation unit.

[0107] The generated restoration signal can be used for intra prediction of the next processing target block within the current picture, can be output after filtering as described below, or can be used for inter prediction of the next picture.

[0108] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture decoding process.

[0109] The filtering unit (10b-50) can improve subjective / objective image quality by applying filtering to the restoration signal. For example, the filtering unit (10b-50) can apply various filtering methods to the restoration picture to generate a modified restoration picture, and transmit the modified restoration picture to the memory (60), specifically, the DPB of the memory (10b-60). The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0110] The (corrected) reconstructed picture stored in the DPB of the memory (10b-60) can be used as a reference picture in the inter prediction unit (10b-31). The memory (10b-60) can store motion information of a block from which motion information is derived (or decoded) within the current picture and / or motion information of blocks within a picture that has already been reconstructed. The stored motion information can be transmitted to the inter prediction unit (10b-31) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (10b-60) can store reconstructed samples of reconstructed blocks within the current picture and transmit them to the intra prediction unit (10b-32).

[0111] In this specification, the embodiments described in the prediction unit (10b-30), the inverse quantization unit (10b-21), the inverse transformation unit (10b-22), and the filtering unit (10b-50) of the decoding device (10b) can be applied to the prediction unit (10a-20), the inverse quantization unit (10a-34), the inverse transformation unit (10a-35), and the filtering unit (10a-60) of the encoding device (10a) in the same manner or correspondingly.

[0112] As described above, prediction is performed to increase compression efficiency when performing video coding. Through this, a predicted block including prediction samples for a current block, which is a coding target block, can be generated. Here, the predicted block includes prediction samples in a spatial domain (or pixel domain). The predicted block is derived identically from an encoding device and a decoding device, and the encoding device can increase video coding efficiency by signaling information (residual information) about the residual between the original block and the predicted block, rather than the original sample value of the original block itself, to a decoding device. The decoding device can derive a residual block including residual samples based on the residual information, and generate a reconstructed block including reconstructed samples by combining the residual block and the predicted block, and can generate a reconstructed picture including the reconstructed blocks.

[0113] The above residual information can be generated through transformation and quantization procedures.

[0114] For example, the encoding device can derive a residual block between the original block and the predicted block, perform a transform procedure on residual samples (a residual sample array) included in the residual block to derive transform coefficients, perform a quantization procedure on the transform coefficients to derive quantized transform coefficients, and signal related residual information to a decoding device (via a bitstream). Here, the residual information can include information such as value information, position information, transform technique, transform kernel, and quantization parameter of the quantized transform coefficients. The decoding device can perform an inverse quantization / inverse transform procedure based on the residual information to derive residual samples (or residual blocks). The decoding device can generate a reconstructed picture based on the predicted block and the residual block. The encoding device can also inverse quantize / inverse transform the quantized transform coefficients to derive a residual block for reference in inter prediction of a subsequent picture, and generate a reconstructed picture based on the residual block.

[0115] <VCM(Video coding for Machines)>

[0116] With the recent advancements in various industries such as surveillance, intelligent transportation, smart cities, intelligent industry, and intelligent content, the amount of image or feature map data consumed by machines is increasing. In contrast, traditional video compression methods currently in use were developed with human vision in mind, and therefore contain unnecessary information, making them inefficient for machine tasks. For example, the resolution of images from the viewer's perspective may be higher than that of images (e.g., feature maps) from the machine's perspective. Therefore, research on video codec technologies that efficiently compress feature maps for machine tasks is needed.

[0117] The Moving Picture Experts Group (MPEG), an international standardization group for multimedia encoding, is discussing Video Coding for Machines (VCM). VCM is an image or feature map encoding technology that targets machine vision, rather than human viewer vision. In this document, the feature map can be referred to as a "feature map," and the features can be referred to as "features."

[0118] Figures 4a to 4d are exemplary diagrams showing a VCM encoder and a VCM decoder.

[0119] Referring to FIG. 4a, a VCM encoder (100a) and a VCM decoder (100b) are shown.

[0120] When a VCM encoder (100a) encodes a video and / or a feature map and transmits it as a bitstream, a VCM decoder (100b) can decode and output the bitstream. At this time, the VCM decoder (100b) can output one or more videos and / or feature maps. For example, the VCM decoder (100b) can output a first feature map for machine-based analysis and a first image for user viewing. The first image can have a higher resolution than the first feature map.

[0121] Referring to FIG. 4b, a feature extractor for extracting a feature map may be connected to the front end of the VCM encoder (100a).

[0122] The VCM encoder (100a) may include a feature encoder.

[0123] The VCM decoder (100b) may include a feature decoder and a video reconstructor. The feature decoder may decode a feature map from a bitstream and output a first feature map for machine-assisted analysis. The video reconstructor may regenerate and output a first video from the bitstream for viewing by a user.

[0124] Referring to Fig. 4c, a feature extractor for extracting a feature map is connected to the front end of the VCM encoder (100). The VCM encoder (100a) may include a feature encoder.

[0125] The VCM decoder (100b) may include a feature decoder. The feature decoder may decode a feature map from a bitstream and output a first feature map for machine-based analysis. That is, the bitstream may be encoded only as a feature map, not as an image. To elaborate, the feature map may be data containing information about features for processing a specific task of a machine based on an image.

[0126] Referring to FIG. 4d, a feature extractor may be connected to the front end of the VCM encoder (100a).

[0127] The VCM encoder (100a) may include a feature converter and a video encoder. The video encoder may be the encoding device (10a) illustrated in FIG. 2.

[0128] The VCM decoder (100b) illustrated in FIG. 4d may include a video decoder and an inverse converter. The video decoder may be the decoding device (10b) illustrated in FIG. 3.

[0129] To efficiently compress high feature dimensions in feature coding techniques for machines, a feature reduction process that reduces feature dimensionality can be included in a feature coding test model (FCTM). FCTM is a test model that evaluates how well a specific machine learning or deep learning model performs based on features extracted from a video. This allows for verification that the model learns appropriate features and efficiently processes them while retaining information during encoding and decoding. However, the feature reduction process can alter the overall feature values, significantly reducing the accuracy of the machine learning task. In embodiments of the present disclosure, scaling based on the maximum value of a feature can be performed to maintain the accuracy of the machine learning task. The embodiments may include various methods for compensating for and restoring changes caused by feature reduction, as well as methods for transmitting parameters associated with this process. In one embodiment, the method may include a method for deriving scaling parameters to mitigate the decrease in accuracy of restored features. In another embodiment, the method may include a method for applying the scaling parameters. In one embodiment, the method may include a method for predicting a scaling parameter.

[0130] FIG. 5 illustrates a block diagram of an encoding device according to one embodiment of the present disclosure.

[0131] According to one embodiment, an encoding device (10a) may receive a feature map (image), perform encoding preprocessing, and then generate and output a bitstream through an internal encoding process. The encoding device (10a) may include a spatial sampling performer (510) for the feature map, a scaling parameter extractor (520), a dimensionality reduction performer (530), a truncation performer (540), a quantization performer (550), a packing performer (560), and an internal encoding performer (570). Depending on the embodiment, the order of each process may be changed or some components may be omitted.

[0132] In one embodiment, an image input to the encoding device (10a) may be a feature map having a size of W×H×C for integers H, W, and C greater than or equal to 1. The feature map may be one or more feature maps extracted from an intermediate layer of a deep neural network composed of one or more convolutions.

[0133] In one embodiment, a spatial sampling performer (510) can receive a feature map as input and perform spatial sampling on the feature map. The spatial sampling performer (510) can perform spatial sampling on a dimensionally reduced feature map output from a dimensionality reduction performer (530). Alternatively, the spatial sampling performer (510) can perform spatial sampling on a truncated feature map output from a feature map truncation performer (540). In one embodiment, sampling can indicate spatial (size of one channel) sampling, and can mean, for example, upsampling, downsampling, or cropping. The sampled feature map can be used as an input to the dimensionality reduction performer (530) or the scaling parameter extractor (520). Depending on the embodiment, spatial sampling on the feature map can be selectively performed.

[0134] According to one embodiment, the scaling parameter extractor (520) may receive a sampled feature map or feature map as input and extract a scaling parameter related to the characteristics of the feature map. Alternatively, the scaling parameter extractor (520) may extract a scaling parameter related to the characteristics of a dimensionally reduced feature map output from a dimensionality reduction performer (530). Alternatively, the scaling parameter extractor (520) may extract a scaling parameter related to the characteristics of a truncated feature map output from a truncation performer (540). The extracted scaling parameter may be used as an input of the internal encoding performer (570). Information generated in the process of extracting scaling parameters for a feature map (e.g., mean, maximum, minimum, median, variance, structural similarity index measure (SSIM), explained variability ratio (EVR), basis vector, or basis vector coefficient) may be used in other processes of the encoding device (10a).

[0135] According to an embodiment, information (e.g., maximum value, minimum value, mean, variance, basis vector, basis vector coefficient, or EVR information) used in the scaling parameter extractor (520) may be transmitted to the decoding device (10b).

[0136] According to one embodiment, a dimensionality reduction performer (530) may receive feature maps and perform feature map dimensionality reduction. The dimensionality reduction performer (530) may extract redundancy between feature maps or redundancy within feature maps as a process of reducing the dimension of the input feature maps. The dimensionality reduction performer (530) may include a feature map redundancy component extraction unit for redundancy extraction. According to an embodiment, information used in the dimensionality reduction performer (530) may be used as an input of a packing performer (560). Alternatively, an output of performing feature map dimensionality reduction may be used as an input of a feature map pruning performer (540). According to an embodiment, information (e.g., basis vector information, coefficient information, deep neural network information, or redundancy information) used in the feature map dimensionality reduction performer (530) may be transmitted to a decoding device (10b).

[0137] In one embodiment, the pruning performer (540) can perform feature map pruning using the output of the dimensionality reduction performer (530) as input. The pruning performer (540) can output a feature map in which some information of the feature maps is pruned for each frame or a series of frames based on the information received from the scaling parameter extractor (520). In one embodiment, the received information can include at least some of the mean, the maximum value, the minimum value, the median, the variance, the SSIM, the EVR, the basis vector, or the basis vector coefficient. In one embodiment, the pruning performer (540) can remove channels whose variance is less than a certain value with respect to the feature map. The output of the feature map pruning performer (540) can be used as an input of the quantization performer (550). In one embodiment, information used in the feature map pruning process (e.g., the pruned degree, the pruned index) can be transmitted to the decoding device (10b).

[0138] According to one embodiment, the quantization performer (550) may receive a feature map in a floating-point format or an integer format from the feature map truncation performer and perform quantization into an integer format. According to one embodiment, information on which quantization was performed (e.g., maximum / minimum values, median values, or clipping ranges) may be transmitted to the decoding device (10b). According to one embodiment, information on which quantization was performed may be utilized by the packing performer (560). In one embodiment, the quantization performer (550) may transmit the degree of quantization used in the process of performing quantization into an integer format to the packing performer (560) so that packing may be performed accordingly. The output on which quantization was performed may be used as an input of the feature map truncation performer (540).

[0139] According to one embodiment, a packing performer (560) may receive quantized feature maps from a quantization performer (550) and perform packing. The tensors resulting from the packing may be one or more tensors grouped into one or more groups. According to an embodiment, packing may be performed using information transmitted from a feature map dimensionality reduction performer (530). The output of the packing performer (560) may be used as an input of an internal encoding performer (570). According to an embodiment, information performed on the packing (e.g., the size of the packed image, the packing scan method, the number of tensors, the number of groups, or an ID for each group) may be transmitted to a decoding device (10b).

[0140] According to one embodiment, the internal encoding performer (570) can perform image encoding on a tensor input from the packing performer (560). The internal encoding performer (570) can perform image encoding to generate a bitstream. The output of the internal encoding performer (570) can be transmitted to the decoding device (10b) via a network or stored in a storage medium.

[0141] According to one embodiment of the present disclosure, an encoding device (10a) may perform at least one of spatial sampling, feature map dimensionality reduction, or feature map pruning on an input image including one or more feature maps, extract one or more scaling parameters related to a scale change of the one or more feature maps used in the spatial sampling process, the feature map dimensionality reduction process, or the feature map pruning process for the one or more feature maps, perform quantization of the one or more feature maps in integer form, perform packing by rearranging the one or more feature maps according to a predetermined unit, and encode the one or more feature maps, encoding information extracted in the process, and the one or more scaling parameters to generate a bitstream.

[0142] According to an embodiment, the order in which the spatial sampling operation, the feature map dimensionality reduction operation, and the feature map truncation operation are performed for the one or more feature maps is variable, and the one or more feature maps may be an input value of an operation performed first, and an output value of each operation may be an input value of a next operation.

[0143] According to an embodiment, the operation of performing spatial sampling on the one or more feature maps may perform at least one of spatial upsampling, downsampling, or cropping on the one or more feature maps.

[0144] According to an embodiment, the operation of performing dimensionality reduction on the one or more feature maps may include extracting redundant components included in the one or more feature maps, removing the redundant components, and performing mapping on feature maps from which redundant components have been removed among the one or more feature maps.

[0145] In an embodiment, the operation of performing a pruning on the one or more feature maps may be configured to remove at least a portion of the one or more feature maps.

[0146] According to an embodiment, the operation of extracting the one or more scaling parameters may include extracting one or more pieces of information about the one or more feature maps, and deriving a scaling parameter based on the one or more pieces of information, or deriving and outputting a differential value for an already derived scaling parameter.

[0147] According to an embodiment, one or more pieces of information extracted from the one or more feature maps may include at least some of a unit-wise mean, a maximum value, a minimum value, a median, a variance, a structural similarity index measure (SSIM), an explained variability ratio (EVR), a basis vector, or a basis vector coefficient.

[0148] According to an embodiment, the unit for extracting the one or more scaling parameters from the one or more feature maps includes a sequence, a frame, a frame group, a feature level, a feature, a layer or a channel, and a certain range for extracting the one or more pieces of information can be specified within each unit.

[0149] According to an embodiment, the scaling parameter may be transmitted to the decoding device (10b) in units of at least one of a task, a sequence, a frame, a frame group, a feature level, or a channel.

[0150] According to an embodiment, the encoding information may include at least some of a scanning order, a packing method, a quantization method, a quantization range, a truncation method, a truncation degree, an original data size, a dimensionality reduction method, channel grouping information, a coding method for each of the grouped channels, or a spatial sampling method, a scaling method, and a scaling parameter for the one or more feature maps.

[0151]

[0152] FIG. 6 is a block diagram including components of a spatial sampling performer according to one embodiment of the present disclosure.

[0153] A spatial sampling performer according to one embodiment (e.g., the spatial sampling performer (510) of FIG. 5) may include a spatial sampling method determination unit (511) and a spatial sampling execution unit (512). Depending on the embodiment, the order of each process may be changed or some components may be omitted.

[0154] A spatial sampling method determining unit (511) according to one embodiment may determine a spatial sampling method for an input feature map. There may be multiple spatial sampling methods, and the spatial sampling method determining unit (511) may select any one of the multiple spatial sampling methods. Depending on the embodiment, the spatial sampling method determining unit (511) may determine the sampling method in various ways. In one embodiment, the spatial sampling method determining unit (511) may determine the sampling method based on a frequency component or activity of a feature map. In another embodiment, the spatial sampling method determining unit (511) may determine the sampling method based on information of a region of interest (ROI), according to a range or ratio of an ROI region in a spatial region of the entire channel. At this time, a spatial sampling rate (M) and ROI information may be transmitted to the decoding device (10b). The sampling rate may be used as an input of the spatial sampling performing unit (512).

[0155] According to one embodiment, the spatial sampling performing unit (512) may receive a sampling method from the sampling method determining unit (511) and perform sampling. The spatial sampling performing unit (512) may perform upsampling or downsampling in various ways. In one embodiment, the spatial sampling performing unit (512) may downsample or upsample the horizontal or vertical length of one channel by M times. For example, the spatial sampling performing unit (512) may perform sampling in a manner of cropping based on ROI in one channel. The sampled feature map, which is the output of feature map sampling, may be used as an input of the feature map dimensionality reduction performing unit (530).

[0156]

[0157] FIG. 7 is a block diagram including components of a scaling parameter extractor according to one embodiment of the present disclosure.

[0158] A scaling parameter extractor (e.g., scaling parameter extractor (520) of FIG. 5) according to one embodiment may receive a feature map as input and output a scaling parameter of the input feature map. For example, the scaling parameter extractor (520) may receive an original feature map as input and output a scaling parameter of the input feature map. For example, the scaling parameter extractor (520) may output a scaling parameter of a dimensionally reduced feature map output from a dimensionality reduction performer (530). For example, the scaling parameter extractor (520) may output a scaling parameter of a truncated feature map output from a feature map truncation performer (540).

[0159] In one embodiment, the scaling parameter may be a scalar or vector value for scaling the values ​​of the restored feature map. Alternatively, the scaling parameter may be a differential value for a previously derived scaling parameter existing in the scaling parameter storage.

[0160] In one embodiment, the scaling parameter may be a statistical parameter that represents the distribution characteristics of the restored feature map (original feature map, dimensionally reduced feature map, or truncated feature map). For example, the scaling parameter may represent a set of standard deviation parameters and mean parameters that represent the distribution characteristics.

[0161] The scaling parameter extractor (520) may include a feature map information extraction unit (521), a scaling parameter derivation unit (522), and a scaling parameter storage unit (523). Depending on the embodiment, the order of each process may be changed or some components may be omitted.

[0162] According to one embodiment, the feature map information extraction unit (521) may perform a process of extracting feature map information from an input feature map. In one embodiment, the feature may represent an original feature, a feature on which dimensionality reduction has been performed, a feature on which truncation has been performed, or a quantized feature. In one embodiment, the feature map information may include at least some of a mean, a maximum value, a minimum value, a median value, a variance, SSIM, EVR, a basis vector, or a basis vector coefficient for each unit of the feature. In one embodiment, the unit from which the feature map information is extracted may be a sequence, a frame, a frame group, a feature level, a feature, a layer, or a channel, and a specific range may be designated within each unit. For example, the feature map information extraction unit (521) may extract feature map information for a special point range by utilizing ROI for a sequence, a frame, a frame group, a feature level, a feature, or a channel. The feature map information extraction unit (521) may extract feature map information for a down-sampled or up-sampled feature. In one embodiment, feature map information may include one or more pieces of information. The output of the feature map information extraction unit (521) may be transmitted to and stored in the scaling parameter storage unit (523). Alternatively, the output of the feature map information extraction unit (521) may be used as input to the scaling parameter derivation unit (522).

[0163] According to one embodiment, the scaling parameter derivation unit (522) may receive information extracted from a feature map and derive and output a scaling parameter or a differential value for an already derived scaling parameter. The scaling parameter derivation unit (522) may use the feature map information and the scaling parameter existing in the scaling parameter storage unit (523) in the scaling parameter derivation process. According to an embodiment, there may be multiple methods for deriving the scaling parameter using the input feature map information. The scaling parameter derivation unit (522) may use one or more feature map information to derive the scaling parameter. The number of scaling parameters derived using the feature map information may be one or more. For example, the number of scaling parameters derived using the feature map information may be 2.

[0164] As another example, a single induced scaling parameter may mean a single parameter set comprising multiple parameters (e.g., two). That is, the number of induced scaling parameters being one or more may mean the number of induced scaling parameter sets being one or more. For example, the number of induced scaling parameter sets may be one.

[0165] The scaling parameter derivation unit (522) can derive multiple scaling parameters using feature map information. According to an embodiment, there are multiple scaling application methods, and the scaling parameter derivation unit (522) can select any one of them to derive the scaling parameter. For example, the scaling parameter derivation unit (522) can perform scaling by multiplying the predicted and corrected scaling parameter for each unit by the dimensionally restored map for each unit. The scaling parameter derivation unit (522) can perform scaling by adding the predicted and corrected scaling parameter for each unit to the dimensionally restored map for each unit. The scaling parameter derivation unit (522) can perform scaling by adding the predicted and corrected scaling parameter for each unit to the dimensionally restored map for each unit in proportion to the size of the value of each element. According to an embodiment, the feature map scaling parameter application method can be transmitted to the decoding device (10b). The output of the scaling parameter derivation unit (522) can be used as an input of the internal encoding performer (570).

[0166] The scaling parameter storage unit (523) according to one embodiment can receive and store information extracted from a feature map and scaling parameters. In one embodiment, the scaling parameters can be stored by unit. For example, the unit can be a sequence, a frame, a frame group, a feature level, a feature, a layer, or a channel. The stored scaling parameters and feature map information can be used as inputs to the scaling parameter derivation unit (522). The scaling parameter storage unit (523) can remove or modify at least some of the stored scaling parameters. For example, the scaling parameter storage unit (523) can store the scaling parameters in chronological order and remove scaling parameters that have passed a certain amount of time. In one embodiment, the scaling parameter storage unit (523) can overwrite the scaling parameters derived by the scaling parameter derivation unit (522) with the scaling parameters already stored in the scaling parameter storage unit (523). According to an embodiment, the stored scaling parameters and feature map information may be used in other processes of the encoding device (10a) in addition to the feature map scaling parameter extractor (520). For example, the scaling parameters stored in the scaling parameter storage unit (523) may be used in a process in which dimensionality reduction is performed in a subsequent dimensionality reduction performer (530). Alternatively, the scaling parameters stored in the scaling parameter storage unit (523) may be used in a process in which feature map truncation is performed in a subsequent feature map truncation performer (540). Alternatively, the scaling parameters stored in the scaling parameter storage unit (523) may be used in a process in which quantization is performed in a subsequent quantization performer (550). The scaling parameters stored in the scaling parameter storage unit (523) may be used in a process in which packing is performed in a subsequent packing performer (560).

[0167]

[0168] FIG. 8 is a block diagram including components of a dimensionality reduction performer according to one embodiment of the present disclosure.

[0169] According to one embodiment, a dimensionality reduction performer (e.g., a dimensionality reduction performer (530) of FIG. 5) may receive a feature map as input and output a mapped feature map, which is a tensor of lower dimension than the input feature map. The dimensionality reduction performer (530) may also output information (duplicate component information) derived in the process. The dimensionality reduction performer (530) may include a feature map duplicate component extraction unit (531) and a feature map mapping performer (532). Depending on the specific example, the order of each process may be changed or some components may be omitted.

[0170] A feature map overlapping component extraction unit (531) according to one embodiment may perform a process of extracting overlapping information from input feature maps having a size of W×H×C. In one embodiment, a unit for obtaining overlapping information may mean overlapping between C W×H feature maps. Alternatively, a unit for obtaining overlapping information may mean overlapping between W×H one-dimensional vectors having a size of C. Alternatively, a unit for obtaining overlapping information may mean overlapping between one or more feature maps (three-dimensional vectors) having a size of W×H×C. According to an embodiment, there may be various methods for deriving overlapping information. A feature map overlapping component extraction unit (531) according to one embodiment may derive and extract a ratio of each component included (overlapping) in a feature map through principal component analysis of the input feature maps. A feature map overlapping component extraction unit (531) according to one embodiment may derive and extract a ratio of frequency components of each feature map through frequency analysis of the input feature maps. A feature map redundant component extraction unit (531) according to one embodiment can extract redundant components through a deep neural network composed of one or more convolutions. The derived redundant information can be transmitted to and used by a feature map group information extraction unit. / There is no description of the feature map group information extraction unit.

[0171] The output of the feature map redundant component extraction unit (531) according to one embodiment can be used as an input of the feature map mapping performing unit (532).

[0172] The feature map mapping performing unit (532) according to one embodiment may perform a process of regenerating or expressing one or more feature maps through a tensor of the same or lower dimension. In one embodiment, the number of tensors of the same or lower dimension may be 1 or more. The feature map mapping performing unit (532) may utilize the duplicate components extracted by the feature map duplicate component extraction unit (531) for feature map mapping. In one embodiment, the feature map mapping performing unit (532) may remove duplicate components and perform mapping on feature maps from which duplicate components have been removed. The feature map mapping performing unit (532) may derive a representative feature map for feature maps having a high number of duplicate components and map one or more feature maps. In one embodiment, the number of feature maps mapped to the representative feature map may be 1. In one embodiment, the representative feature map may be a mapped feature map. Alternatively, the representative feature map may be a transformed feature map. In one embodiment, a transformed feature map may mean that the value of the feature map is mapped within a certain range of values ​​based on 0. The feature map mapping performing unit (532) may signal information used or derived in the mapping process (e.g., the difference in variance between the feature map and the mapped feature map, or the difference in mean between the feature map and the mapped feature map) to the decoding device (10b). The mapped feature maps, which are the output of the feature map mapping performing unit (532), and the information used in the mapping process may be used as inputs of the quantization performing unit (550).

[0173]

[0174] FIG. 9 is a block diagram including components of a cutting performer according to one embodiment of the present disclosure.

[0175] According to one embodiment, a cutting performer (e.g., a cutting performer (540) of FIG. 5) may receive one or more mapped feature maps as input, perform a feature map cutting process, and output a cut feature map. In one embodiment, the cutting process may refer to a process of removing a portion of one or more input mapped feature maps. The cutting performer (540) may include a feature map cutting method determining unit (541) and a feature map cutting performing unit (542). Depending on the exemplary embodiment, the order of each process may be changed or some components may be omitted.

[0176] According to one embodiment, the feature map pruning method determining unit (541) may receive one or more mapped feature maps and determine a pruning process method for each unit. In one embodiment, the pruning method may be determined according to a ratio of a range of maximum and minimum values ​​of the input mapped feature maps and a range of maximum and minimum values ​​of a unit for pruning. The feature map pruning method determining unit (541) may determine a method for removing a feature map as a pruning method when a ratio of a difference between a maximum and minimum value of the input mapped feature maps and a difference between a maximum and minimum value of a unit for pruning is less than or equal to a predetermined value. According to one embodiment, the unit for pruning may be W×H or C of one or more quantized feature maps having a size of W×H×C. In another embodiment, the pruning method may be determined according to a size of a maximum and minimum value of a unit for pruning. The feature map pruning method determining unit (541) may determine a method for removing a feature map as a pruning method when the size of the maximum value is less than or equal to a predetermined value. Alternatively, the feature map pruning method determining unit (541) may determine a method of removing a feature map when the minimum value is less than a certain size as the pruning method. In another embodiment, the pruning method may be determined based on the number of feature maps mapped to the mapped feature map. According to an embodiment, a feature map or a quantized feature map to which a certain number or more feature maps are mapped may not be removed by another embodiment.

[0177] In one embodiment, the feature map pruning performer (542) may perform a pruning process according to a method determined by the feature map pruning method determination unit (541). According to an embodiment, pruning performance information (removal ratio, removed unit, etc.) may be signaled to the decoding device (10b). The pruned feature map, which is the output of the feature map pruning performer (542), may be used as an input of the packing performer (560). In FIG. 5, the quantization performer (550) is illustrated after the pruning performer (540), but the packing performer (560) may be performed after the pruning performer (540). Alternatively, the quantization performer (550) may be performed first, and then the pruning performer (540) and the packing performer (560) may be performed sequentially.

[0178]

[0179] FIG. 10 is a block diagram including components of a quantization performer according to one embodiment of the present disclosure.

[0180] A quantization performer according to one embodiment (e.g., quantization performer (550) of FIG. 5) may receive a truncated feature map as input, perform quantization, and output a quantized feature map. The quantization performer (550) may include a quantization method determination unit (551) and a quantization performing unit (552). Depending on the specific example, the order of each process may be changed or some components may be omitted.

[0181] A quantization method determination unit (551) according to one embodiment may receive truncated feature maps and determine a quantization method for each of the truncated feature maps or each truncated feature map. According to an embodiment, there are multiple quantization methods, and for example, the following quantization methods may be included.

[0182] In one embodiment, the first quantization method can divide evenly or unevenly by adjusting the maximum and minimum values ​​of the input feature map or the maximum and minimum values ​​of the clipped feature map to fit the maximum range to be expressed for each feature map or each unit.

[0183] In another embodiment, the second quantization method can divide evenly or unevenly by adjusting the maximum and minimum values ​​of each input feature map or the maximum and minimum values ​​of each clipped feature map to fit the maximum range that one or more feature maps or one or more units are intended to express. In an embodiment, the unit can be a channel, one or more feature maps, or one or more frames. In an embodiment, the adjustment can represent a process of mapping evenly or unevenly to fit the maximum range that is intended to be expressed. In an embodiment, the quantization method determination unit (551) can signal the quantization method for each quantization performing unit to the decoding device (10b) for the determined quantization method.

[0184] According to one embodiment, the quantization performing unit (552) may perform quantization on the feature map according to the method determined by the quantization method determining unit (551). According to an embodiment, when quantization is performed based on a specific value of a quantization performing unit, the quantization performing unit (552) may signal the specific value as the quantization performing unit to the decoding device (10b). The quantization performing unit (552) may determine the degree of quantization in the encoding step and signal information about the degree of quantization to the decoding device (10b). According to an embodiment, additional information about the quantization unit (e.g., truncated feature maps or each mapped feature map within the truncated feature maps) may be signaled to the decoding device (10b). The quantization performing unit (552) may perform quantization after adjusting the average of the values ​​distributed in the unit data on which quantization is performed to 0, and may signal the value used for this to the decoding device (10b). The quantized feature map, which is the output of the quantization performing unit (542), and the information used in the quantization process can be used as input to the feature map truncation performing unit (540). In Fig. 5, the quantization performing unit (550) is illustrated after the truncation performing unit (540), but the quantization performing unit (550) may be performed first and then the truncation performing unit (540).

[0185]

[0186] FIG. 11 is a block diagram including components of a packing performer according to one embodiment of the present disclosure.

[0187] A packing performer according to one embodiment (e.g., packing performer (560) of FIG. 5) may receive one or more quantized feature maps as input, perform packing, and output one or more packed feature maps. In one embodiment, packing may refer to dividing one or more input quantized feature maps into a certain unit and rearranging them into one or more two-dimensional or three-dimensional data. The packing performer (560) may include a feature map packing method determination unit (561) and a feature map packing execution unit (562). Depending on the example, the order of each process may be changed or some components may be omitted.

[0188] According to one embodiment, a feature map packing method determination unit (561) may receive a quantized feature map and determine a packing method. The feature map packing method determination unit (561) may determine a packing method according to a maximum value and a minimum value of the input quantized feature map. In one embodiment, the feature map packing method determination unit (561) may determine a different packing method according to a difference between the maximum value and the minimum value. The feature map packing method determination unit (561) may determine a different packing method according to a size of the maximum value and the minimum value. The feature map packing method determination unit (561) may determine a packing method according to mapped feature maps of the input quantized feature map. The feature map packing method determination unit (561) may determine a packing method by dividing the number of mapped feature maps by the number of representative feature maps that are a certain number or a certain ratio or more. The feature map packing method determining unit (561) can determine that representative feature maps having a certain number or a certain ratio of mapped feature maps are packed using the same packing method. The feature map packing method determining unit (561) can signal the determined packing method to the decoding device (10b).

[0189] The feature map packing performing unit (562) according to one embodiment can perform packing for the feature map packing method determined by the feature map packing method determining unit (561) and output a packed feature map. The feature map packing performing unit (562) can perform two-dimensional packing in units of W'×H' for a feature map having a size of W'×H'×C'. In one embodiment, the packing order can be various, for example, a z-scan order. In one embodiment, the number of columns / rows in units of W'×H' can be adaptively determined according to the value of C'. The feature map packing performing unit (562) performs two-dimensional packing in units of W'×H' for a feature map having a size of W'×H'×C' ((W'×X)×(H'×Y)), X×Y <C'), 이를 새로운 축 (예: 시간 축)으로 패킹을 수행할 수 있다. 피처 맵 패킹 수행부(562)는 부호화기 입력 피처 맵이 다중 레이어 피처 맵인 경우, 다중 레이어 피처 맵을 하나의 2차원 프레임으로 패킹을 수행할 수 있고, 이를 각각의 2차원 프레임으로 패킹을 수행할 수 있다. 실시예에 따르면, 패킹하는 방식은 같은 대표 피처 맵으로 맵핑 된 피처 맵들끼리 같은 순서(예: 레스터 순서)로 패킹을 수행할 수 있다. 실시예에 따르면, 피처 맵 패킹 수행부(562)의 출력은 내부 부호화 수행기(570)의 입력으로 사용될 수 있다.

[0190]

[0191] FIG. 12 is a block diagram including components of an internal encoding performer according to one embodiment of the present disclosure.

[0192] According to one embodiment, an internal encoding performer (e.g., the internal encoding performer (570) of FIG. 5) may perform encoding of feature maps and group-specific coding information to output a bitstream. According to an embodiment, muxing may be performed on bitstreams output through each encoding performer (e.g., a feature map encoding performer, a feature map information encoding performer) to generate a single bitstream. The internal encoding performer (570) may include a frame sampling performer (571) and an encoding performer (572). Depending on the example, the order of each process may be changed or some components may be omitted.

[0193] According to one embodiment, the frame sampling performing unit (571) may perform spatial downsampling on the frame during the process of performing internal encoding of the input feature map. According to one embodiment, when the frame sampling performing unit (571) performs internal encoding of one or more feature maps, it may downsample each feature map with a different sampling rate. In one embodiment, the sampling rate may be derived from a motion vector generated during the encoding process. The sampling rate may be derived through a separate optical flow algorithm. The frame sampling performing unit (571) may apply a large sampling rate when the motion vector is small or the motion obtained through the optical flow is small. The frame sampling performing unit (571) may apply a small sampling rate when the motion vector is large or the motion obtained through the optical flow is large. The frame sampling performing unit (571) may signal sampling rate information to the decoding device (10b).

[0194] The encoding unit (572) according to one embodiment may use a video encoder (e.g., AVC / H.264, HEVC / H.265, VVC / H.266, AV1, VP9). Alternatively, the encoding unit (572) may use a 2D video encoder including one or more convolutional layers. Alternatively, the encoding unit (572) may use an entropy encoding method. The encoding unit (572) may use different encoders for each piece of information. For example, the encoding unit (572) may use a video encoder (e.g., AVC / H.264, HEVC / H.265, VVC / H.266, AV1, AV2, VP8, VP9) in the process of encoding a feature map, and may use an arithmetic encoder (e.g., CABAC, deepCABAC) in the process of encoding coding information.

[0195]

[0196] FIG. 13 illustrates a block diagram of a decoding device according to one embodiment of the present disclosure.

[0197] According to one embodiment, a decoding device (10b) may receive a bitstream, perform a decoding process, and output a restored feature map. The decoding device (10b) may include an internal decoding performer (1310) for the bitstream, an unpacking performer (1320), an inverse quantization performer (1330), a feature map unsizing performer (1340), a feature map dimension restoration performer (1350), a feature map spatial unsampling performer (1360), and a feature map scaling performer (1370). Depending on the embodiment, the order of each process may be changed, or some components may be omitted. According to the embodiment, the execution order of each component of FIG. 13 may be performed in the reverse order of the execution order of the encoding device (10a) when each component of the decoding device (10b) has a process corresponding to the encoding device (10a). Alternatively, the execution order of each component of Fig. 13 may be performed regardless of the execution order of the encoding device (10a).

[0198] An internal decoding performer (1310) according to one embodiment can receive a bitstream, perform image decoding, and generate a reconstructed image. According to an embodiment, the decoding can use at least a part of a 2D video decoder (e.g., AVC / H.264, HEVC / H.265, VVC / H.266, AV1, VP9), a 2D video decoder including one or more convolution layers, or an entropy decoder. The internal decoding performer (1310) can implicitly or explicitly convert the color space of a decoded feature map resulting from the internal decoding performer to another color space, and then perform an image decoding process thereafter.

[0199] An unpacking performer (1320) according to one embodiment can receive a decoded bitstream and perform unpacking. The unpacking performer (1320) can receive a decoded feature map through internal decoding, receive a scanning order and a packing method from the encoding device (10a), and perform unpacking. The unpacking performer (1320) can receive a decoded feature map through the unpacking process and create one or more feature maps in a three-dimensional form. The unpacking performer (1320) can perform a process of merging one or more unpacked feature maps into one feature map. For example, the unpacking performer (1320) can create one unpacked feature map by concatenating two unpacked feature maps in the channel direction. According to an embodiment, the unpacked feature map can be used as an input of the dequantization performer (1330).

[0200] In one embodiment, a dequantization performer (1330) may receive an unpacked feature map as input and perform dequantization. The dequantization performer (1330) may receive a decoded image (feature map) or a decoded image on which a partial process has been performed and perform dequantization. The dequantization performer (1330) may receive information used in the quantization process from the encoding device (10a), for example, information about the quantization method and range (e.g., maximum / minimum / center / mean / variance values, clipping range), and perform dequantization using the same. In the case where quantization is performed after removing the mean value, the dequantization performer (1330) may compensate again for the mean value received from the encoding device (10a) after performing dequantization. In one embodiment, the dequantized feature map may be used as an input of the feature map de-truncation performer (1340).

[0201] According to one embodiment, the feature map uncut performer (1340) can perform feature map uncut by receiving a dequantized feature map as input. The feature map uncut performer (1340) can perform uncut on the decoded feature map by parsing information used in the uncut process (e.g., degree of uncut, size of original data) so that the decoded feature map has the same resolution and dimension as the original feature map (the same feature map as the data before feature map uncut is performed in the encoding step).

[0202] In one embodiment, information about the degree of truncation may be transmitted from the encoding device (10a) and implicitly determined based on the quantization parameters used in the encoding process. The feature map non-truncation performer (1340) may utilize information used in the dequantization process in the feature map non-truncation process. According to an embodiment, non-truncation may mean regenerating the truncation portion. The non-truncation process may be performed in various ways. The feature map non-truncation performer (1340) may perform non-truncation based on a predefined network structure. For example, the feature map non-truncation performer (1340) may change the dimension of the input dequantized feature map based on the degree of truncation and then restore the truncation information by the internal weights of the network. As another example, the feature map non-truncation performer (1340) may perform non-truncation by copying a portion of the input dequantized feature map. The feature map non-cutting performer (1340) can predict the range of values ​​of the cut region based on the degree of quantization for the non-cut region during the process of generating the cut region. According to an embodiment, the non-cut feature map can be used as input to the feature map dimension restoration performer (1350).

[0203] According to one embodiment, the feature map dimension restoration performer (1350) can receive a non-truncated feature map (W'×H'×C') as input and output a feature map of a higher dimension than the input feature map. According to an embodiment, the number of dimensions of the high-dimensional feature map can be transmitted from the encoder. Alternatively, the number of dimensions of the high-dimensional feature map can be defined in advance in the sub / decoder. The feature map dimension restoration performer (1350) can perform feature map dimension restoration by utilizing information (e.g., channel grouping information, each coding method of the grouped channels) used in the feature map dimension reduction process of the transmitted encoding device (10a). According to an embodiment, the restored dimension is the same as the feature map dimension of the input image in the encoding process, which can be signaled from the encoding device (10a) or information related thereto can be shared in advance in the sub / decoder. The feature map dimension restoration performer (1350) can perform a process of receiving an uncut feature map and reverse mapping it to the dimension of the feature map (input image) that was input to the encoding device (10a). In one embodiment, the feature map dimension restoration performer (1350) can perform dimension restoration based on the basis vectors and the coefficients for the basis vectors transmitted from the encoding device (10a) when dimension reduction through principal component analysis is performed. In one embodiment, the feature map dimension restoration performer (1350) can perform dimension restoration by generating each channel based on the representative channel and information transmitted from the encoding device (10a) (e.g., the difference in variance between the feature map and the mapped feature map, the difference in mean between the feature map and the mapped feature map) when dimension reduction through the representative channel is performed. The feature map dimension restoration performer (1350) can restore duplicate components through a deep neural network composed of one or more convolutions. According to an embodiment, the dimensionally restored feature map can be used as input to a feature map scaling performer (1370).

[0204] According to one embodiment, the feature map spatial unsampling performer (1360) can receive a dimensionally restored feature map and perform feature map spatial unsampling. In one embodiment, the unsampling can perform spatial (size of one channel) sampling, for example, upsampling, downsampling, or cropping. The feature map spatial unsampling performer (1360) can receive an uncropped feature map from the uncropping performer (1340) and perform feature map spatial unsampling. The feature map spatial unsampling performer (1360) can receive an inverse quantized feature map from the inverse quantization performer (1330) and perform feature map spatial unsampling. The feature map spatial unsampling performer (1360) can perform spatial unsampling according to an unsampling method received from the encoding device (10a). According to an embodiment, the received unsampling method may be various methods. For example, the feature map spatial unsampling performer (1360) may perform unsampling using at least one of a bilinear method, a bilateral method, a nearest neighbor method, or a network-based super resolution method. Alternatively, the feature map spatial unsampling performer (1360) may perform unsampling according to a method predefined in the encoder / decoder. The feature map spatially unsampled feature map may be used as an input to the feature map scaling performer (1370).

[0205] According to one embodiment, the feature map scaling performer (1370) can receive an unsampled feature map as input and perform feature map scaling. The feature map scaling performer (1370) can receive a dimensionally restored feature map as input and perform feature map scaling. The feature map scaling performer (1370) can receive an untruncated feature map as input and perform feature map scaling. The feature map scaling performer (1370) can receive a dequantized feature map as input and perform feature map scaling. The feature map scaling performer (1370) can perform feature map scaling by utilizing transmitted feature map scaling parameters (e.g., maximum value, minimum value, mean, variance, basis vector, basis vector coefficient, EVR information). The feature map scaled feature map is a restored feature map, which can be used as an input of a network for a subsequent task (e.g., object detection, object segmentation, object tracking).

[0206] According to one embodiment of the present disclosure, a decoding device (10b) may decode an input bitstream to generate a restored feature map, extract encoding information from the input bitstream, and, based on the encoding information, perform at least one operation of unpacking, dequantization, feature map un-truncation, feature map dimension restoration, or spatial unsampling on the restored feature map corresponding to an encoding process of the bitstream, and perform scaling on the restored feature map using a scaling parameter included in the encoding information.

[0207] According to an embodiment, the encoding information may include at least some of a scanning order, a packing method, a quantization method, a quantization range, a truncation method, a truncation degree, an original data size, a dimensionality reduction method, channel grouping information, a coding method for each of the grouped channels, or a spatial sampling method, a scaling method, and a scaling parameter for the original feature map.

[0208] According to an embodiment, the scaling parameter may include at least some of a scaling parameter value, a scaling parameter prediction value, feature map information, or a residual value for the scaling parameter prediction value.

[0209] According to an embodiment, the feature map information may include at least some of a mean, a maximum, a minimum, a median, a variance, a structural similarity index measure (SSIM), an explained variability ratio (EVR), a basis vector, or a basis vector coefficient.

[0210] According to an embodiment, the scaling parameter may be received corresponding to at least one unit of a task, a sequence, a frame, a frame group, a feature level, or a channel.

[0211] In an embodiment, the scaling parameter may be used as a prediction for another scaling parameter for the same unit.

[0212] According to an embodiment, the predicted scaling parameter may be corrected according to a residual value of the scaling parameter.

[0213] According to an embodiment, scaling can be performed on the restored feature map using a predicted or corrected scaling parameter based on a scaling method included in the encoding information.

[0214] According to an embodiment, the restored feature map may be a feature map on which at least one of unpacking, dequantization, untruncation, dimension restoration, or spatial unsampling has been performed.

[0215]

[0216] FIG. 14 is a block diagram including components of an internal decryption performer according to one embodiment of the present disclosure.

[0217] An internal decoding performer according to one embodiment (e.g., the internal encoding performer (1310) of FIG. 13) may receive a bitstream and perform decoding on coding information for each image and group. The internal decoding performer (1310) may perform demuxing on the input bitstream and input it to each decoding performer. The internal decoding performer (1310) may include a decoding performer (1311) and an upsampling performer (1312). Depending on the embodiment, the order of each process may be changed or some components may be omitted.

[0218] According to one embodiment, a decoding unit (1311) may receive a bitstream and perform feature map decoding or coding information decoding. According to an embodiment, the coding information may include information on a coding mode for each channel of the original feature map (e.g., Skip_flag). According to an embodiment, the output of the decoding unit (1311) may consist of one or more feature maps. The decoded feature map may be used as an input for the upsampling unit (1312).

[0219] According to one embodiment, an upsampling unit (1312) may receive a decoded feature map and perform spatial upsampling. The upsampling unit (1312) may perform spatial upsampling by applying a sampling rate transmitted from the encoding device (10a). The decoded feature map on which spatial upsampling has been performed may be used as an input for an unpacking unit (1320).

[0220]

[0221] FIG. 15 is a block diagram including components of a feature map scaling performer according to one embodiment of the present disclosure.

[0222] A feature map scaling performer (e.g., feature map scaling performer (1370) of FIG. 13) according to one embodiment may receive a dimensionally restored feature map and a scaling parameter from an encoding device (10a) and output a scaled feature map. According to an embodiment, the scaling parameter may include at least some of a scaling parameter value, a scaling parameter prediction value, feature map information (e.g., mean, maximum, minimum, median, variance, SSIM, EVR, basis vector, basis vector coefficient, etc.), and a residual value for the scaling parameter prediction value. The scaling parameter may be transmitted in a predefined manner and unit (e.g., task, sequence, frame, frame group, feature, channel) in a decoder / decoder. The feature map scaling performer (1370) may include a scaling parameter prediction unit (1371), a scaling parameter storage unit (1372), and a scaling performer (1373). Depending on the embodiment, the order of each process may be changed or some components may be omitted.

[0223] According to one embodiment, the scaling parameter prediction unit (1371) can receive a dimensionally restored feature map and a scaling parameter transmitted from an encoding device (10a) as inputs and output a predicted scaling parameter.

[0224] The scaling parameter prediction unit (1371) can predict scaling parameters in various ways. For example, the scaling parameter prediction unit (1371) can predict scaling parameters between tasks. The scaling parameter prediction unit (1371) can use the scaling parameter for a received task as a predicted value for the scaling parameter for another task.

[0225] For another example, the scaling parameter prediction unit (1371) can predict the scaling parameter between frames. The scaling parameter prediction unit (1371) can use the scaling parameter used in the first frame as a scaling parameter prediction value for the subsequent frames. The scaling parameter prediction unit (1371) can use the scaling parameter of an I-frame as a scaling parameter prediction value up to the next I-frame. The scaling parameter prediction unit (1371) can use the scaling parameter of the first frame within one or more frame units as a scaling parameter prediction value for one or more subsequent frame units. The scaling parameter prediction unit (1371) can use the scaling parameter values ​​of adjacent frames to predict the scaling parameter of the current frame. The scaling parameter prediction unit (1371) can derive (average / center / maximum / minimum values) the scaling parameter for the feature map of the current frame for frames that are not transmitted due to reasons such as downsampling or temporal sampling, and use the derived scaling parameter as a prediction value.

[0226] As another example, the scaling parameter prediction unit (1371) can predict the scaling parameter between feature levels (multiple feature maps from one frame). The scaling parameter prediction unit (1371) can predict the scaling parameter value based on the ratio of values ​​between each feature level. In one embodiment, the value between feature levels can mean a value related to feature map information (e.g., mean, maximum value, minimum value, median, variance, SSIM, EVR, basis vector, basis vector coefficient).

[0227] As another example, the scaling parameter prediction unit (1371) can predict scaling parameters between channels. The scaling parameter prediction unit (1371) can use the scaling parameter used in the first channel as a predicted value for the scaling parameter for other channels. The scaling parameter prediction unit (1371) can use the same scaling parameter as a predicted value for a group consisting of one or more channels. According to an embodiment, the predicted scaling parameter can be corrected through the residual of the scaling parameter transmitted from the encoder. The predicted scaling parameter can be stored in the scaling parameter storage unit (1372). The predicted scaling parameter can be used as an input for the scaling performing unit (1373).

[0228] According to one embodiment, the scaling performing unit (1373) may receive a feature map as input and perform scaling using the scaling parameters received from the scaling parameter prediction unit (1371). In one embodiment, the feature may mean one or more of a dimensionally restored feature, a non-truncated feature, a dimensionally restored feature, or a dequantized feature. The number of received scaling parameters may be one or more than one. For example, the number of received scaling parameters may be two.

[0229] As another example, a single received scaling parameter may mean a single parameter set containing multiple parameters (e.g., two). That is, the fact that the number of received scaling parameters may be one or more than one may mean that the number of received scaling parameter sets may be one or more than one. For example, the number of received scaling parameter sets may be one.

[0230]

[0231] The scaling unit (1373) can perform scaling according to a scaling method determined by the scaling application method transmitted from the encoding device (10a). For example, the scaling unit (1373) can perform scaling by multiplying the dimensionally restored map for each unit by the scaling parameter predicted and corrected for each unit according to the scaling application method. For example, the scaling unit (1373) can perform scaling by adding the scaling parameter predicted and corrected for each unit to the dimensionally restored map for each unit according to the scaling application method. For example, the scaling unit (1373) can perform scaling by adding the scaling parameter predicted and corrected for each unit to the dimensionally restored map for each unit in proportion to the size of the value of each element according to the scaling application method.

[0232] According to one embodiment, the scaling parameter storage unit (1372) can store the scaling parameter predicted by the scaling parameter prediction unit (1371), the corrected scaling parameter, and feature map information (e.g., mean, maximum, minimum, median, variance, SSIM, EVR, basis vector, basis vector coefficient). According to an embodiment, the stored information can be transferred to the scaling parameter prediction unit (1371) and used in the process of predicting the scaling parameter for another feature map. The scaling parameter storage unit (1372) can receive and store the predicted scaling parameter from the prediction unit (1371). The scaling parameter stored in the scaling parameter storage unit (1372) can be a predicted scaling parameter or a corrected scaling parameter.

[0233]

[0234] <Examples of syntax and semantics related to scaling parameters>

[0235] An encoding device (10a) (encoder / coder) can transmit information about scaling parameters to a decoding device (10b) (decoder / decoder). The following examples of syntax and semantics can be shared by the encoding device (10a) and the decoding device (10b).

[0236] (1) scale_sig_unit: This can indicate information about the unit for transmitting the scaling parameter determined by the encoder's feature map scaling parameter extractor and transmitted in sequence units. The parameters parsed by the decoder's feature map scaling performer can be applied in a manner determined by the scale_by syntax. It can be defined as 2 bits. For example, 00: frame unit, 01: frame group unit, 10: feature level unit, and 101: channel unit can indicate.

[0237] (2) scale_by: This can indicate information about the scaling application method determined by the encoder's feature map scaling parameter extractor and transmitted in fixed units. Scaling can be performed using the parsed scaling application method in the decoder's feature map scaling performer. It can be defined as 2 bits. For example, 0: Scaling is performed by adding the scaling parameter proportionally to the size of the value of each element to the dimensionally restored map for each unit, 1: Scaling is performed by multiplying the scaling parameter by the dimensionally restored map for each unit, 2: Scaling is performed by adding the scaling parameter to the dimensionally restored map for each unit.

[0238] (3) Scale_resi_flag: The scaling parameter determined by the encoder's feature map scaling parameter extractor and transmitted in a set unit can indicate information about whether it is a predicted value (scale_resi_flag is 0) or a residual value (scale_resi_flag is 1). At this time, the unit for transmitting the scaling parameter and the unit of information about whether the scaling parameter is a predicted value or a residual value may be different. The scaling parameter can be corrected based on the scaling information parsed by the decoder's feature map scaling performer.

[0239] (4) derived_scale: If the value transmitted from the encoder's feature map scaling parameter extractor is a residual value (scale_resi_flag is 1), this value can be derived from the decoder's feature map scaling performer.

[0240] (5) scale_val_minus1: Scaling parameter value transmitted per unit - 1. It can be defined as 3 bits. For example, it can be defined as in Table 1.

[0241] In case of Bit prediction value, in case of residual value, 0000.050.0050010.0750.0100100.10.0150110.1250.0201000.150.0251010.20.0301100.30.0351110.40.040

[0242] Table 2 is an example of the syntax for scale_sig_unit.

[0243] Descriptorscale_sig_unitue(2)feature_map_header_rbsp () {… scale_byue(2)scale_resi_flague(1)…}

[0244] Tables 3 through 6 provide examples of unit-specific syntax for transmitting scaling parameters. Table 3 provides an example of frame-level signaling. Table 4 provides an example of frame-group-level signaling. Table 5 provides an example of feature-level signaling. Table 6 provides an example of channel-level signaling.

[0245] Descriptorif (scale_sig_unit == 0 ) {for ( i=0; i <sps_num_frame; i++ ) {…scale=scale_val_minus1ue(v)for ( i=0; i<sps_num_level; i++ ) {…}}}

[0246] Descriptorif (scale_sig_unit == 1 ) {… for (i=0;i <sps_num_frame; i++ ) {if( POC % GOP_size){scale=scale_val_minus1ue(v)for ( i=0; i<sps_num_level; i++ ) {…}}}}

[0247] Descriptorif (scale_sig_unit== 2 ) {… for (i=0;i <sps_num_level; i++ ) {scale=scale_val_minus1ue(v)for ( j=0; j<sps_num_channels[i]; j++ ) {…}}}

[0248] Example 4) Descriptorif (scale_sig_unit== 3 ) {… for ( i=0; i <sps_num_level; i++ ) {for ( j=0; j<sps_num_channels[i]; j++ ) {scale=scale_val_minus1ue(v)}}}

[0249] Tables 7 and 8 provide examples of syntax for transmitting residual or predicted values ​​for scaling parameters. Table 7 provides an example of syntax for transmitting residual values. Table 8 provides an example of syntax for transmitting predicted values.

[0250] Example 1) Descriptorif (scale_sig_unit == 0) {for ( i=0; i <sps_num_frame; i++ ) {…scale=derived_scale + scale_val_minus1ue(v)for ( i=0; i<sps_num_level; i++ ) {…}}}

[0251] Example 3) Descriptorif (scale_sig_unit== 2 ) {… for ( i=0; i <sps_num_level; i++ ) {scale=scale_val_minus1ue(v)for ( j=0; j<sps_num_channels[i]; j++ ) {…}}}

[0252] The examples of the present disclosure shown in this specification and drawings are only specific examples to easily explain the technical contents of the present disclosure and help understand the present disclosure, and are not intended to limit the scope of the present disclosure. It will be apparent to those skilled in the art that other modifications may be implemented in addition to the examples described so far. The claims set forth in this specification may be combined in various ways. For example, the technical features of the method claims of this specification may be combined to implement a device, and the technical features of the device claims of this specification may be combined to implement a method. In addition, the technical features of the method claims of this specification and the technical features of the device claims of this specification may be combined to implement a device, and the technical features of the method claims of this specification and the technical features of the device claims of this specification may be combined to implement a method.

[0253] Although the above-described disclosure is described based on a series of steps or a flowchart, this does not limit the chronological order of the invention, and may be performed simultaneously or in a different order as needed. In addition, each component (e.g., unit, module, etc.) constituting the block diagram in the above-described disclosure may be implemented as a hardware device or software, or a plurality of components may be combined to be implemented as a single hardware device or software. The above-described disclosure may be implemented in the form of program instructions that can be executed through various computer components and recorded on a non-transitory computer-readable recording medium. The non-transitory computer-readable recording medium may include program instructions, data files, data structures, etc., singly or in combination. Examples of the non-transitory computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions such as ROMs, RAMs, and flash memories. The hardware device may be configured to operate as one or more software modules to perform processing according to the present disclosure, and vice versa. The device according to the present disclosure may have program instructions for storing or transmitting a bitstream generated by the encoding method described above.The scope of the present disclosure may include software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) that cause operations according to the methods of various embodiments to be executed on a device or a computer, and a non-transitory computer-readable medium having such software or instructions stored thereon and executable on the device or computer.

[0254] The present disclosure may be applicable to the industrial fields of image encoding / decoding methods, devices and recording media for machines.

Claims

1. In the encoding device, For an input image including one or more feature maps, at least one of spatial sampling, feature map dimensionality reduction, or feature map pruning is performed, Extracting one or more scaling parameters related to a scale change of the one or more feature maps used in the spatial sampling process for the one or more feature maps, the feature map dimensionality reduction process, or the feature map cropping process, Quantizing one or more of the above feature maps into integer form, Perform packing by reordering one or more of the above feature maps according to a certain unit, and An encoding device that generates a bitstream by encoding one or more of the above feature maps, encoding information extracted in the above process, and one or more scaling parameters.

2. In paragraph 1, The order in which the spatial sampling operation for the one or more feature maps, the feature map dimensionality reduction operation, and the feature map pruning operation are performed is variable, An encoding device, wherein the one or more feature maps are input values ​​of the first operation to be performed, and the output value of each operation becomes the input value of the next operation.

3. In paragraph 1, The operation of performing spatial sampling on one or more feature maps is: An encoding device that performs at least one of spatial upsampling, downsampling, or cropping on one or more of the above feature maps.

4. In paragraph 1, The operation of performing dimensionality reduction on one or more of the above feature maps is: An encoding device that extracts duplicate components included in one or more feature maps, removes the duplicate components, and performs mapping on feature maps from which duplicate components have been removed among the one or more feature maps.

5. In paragraph 1, The operation of performing a cut on one or more of the above feature maps is: An encoding device that removes at least a portion of one or more of the above feature maps.

6. In paragraph 1, The operation of extracting one or more of the above scaling parameters comprises: Extracting one or more pieces of information about one or more feature maps, An encoding device that derives a scaling parameter based on one or more pieces of information above, or derives and outputs a differential value for an already derived scaling parameter.

7. In paragraph 1, An encoding device, wherein one or more pieces of information extracted from the one or more feature maps include at least some of a unit-wise mean, maximum value, minimum value, median, variance, structural similarity index measure (SSIM), explained variability ratio (EVR), basis vector, or basis vector coefficient. Encoding device.

8. In paragraph 1, The unit for extracting the one or more scaling parameters from the one or more feature maps includes a sequence, a frame, a frame group, a feature level, a feature, a layer or a channel, An encoding device that specifies a certain range for extracting one or more pieces of information within each unit.

9. In paragraph 1, An encoding device wherein the scaling parameter is transmitted to a decoding device in units of at least one of a task, a sequence, a frame, a frame group, a feature level, or a channel.

10. In paragraph 1, An encoding device, wherein the encoding information includes at least some of a scanning order, a packing method, a quantization method, a quantization range, a truncation method, a truncation degree, an original data size, a dimensionality reduction method, channel grouping information, a coding method for each of the grouped channels, or a spatial sampling method, a scaling method, and a scaling parameter for the one or more feature maps.

11. In the decoding device, Decode the input bitstream to generate a restored feature map, Extract encoding information from the input bitstream, Based on the encoding information, at least one operation of unpacking, dequantization, feature map uncutting, feature map dimension restoration, or spatial unsampling is performed on the restored feature map corresponding to the encoding process of the bitstream, and A decoding device that performs scaling on a restored feature map using a scaling parameter included in the above encoding information.

12. In paragraph 11, A decoding device, wherein the encoding information includes at least some of a scanning order, a packing method, a quantization method, a quantization range, a truncation method, a truncation degree, an original data size, a dimensionality reduction method, channel grouping information, a coding method for each of the grouped channels, or a spatial sampling method, a scaling method, and a scaling parameter for the original feature map.

13. In paragraph 11, A decoding device, wherein the scaling parameter includes at least a portion of a scaling parameter value, a scaling parameter prediction value, feature map information, or a residual value for the scaling parameter prediction value.

14. In paragraph 13, A decoding device, wherein the feature map information includes at least some of a mean, a maximum value, a minimum value, a median, a variance, a structural similarity index measure (SSIM), an explained variability ratio (EVR), a basis vector, or a basis vector coefficient.

15. In paragraph 11, The above scaling parameter is received in response to at least one unit of a task, a sequence, a frame, a frame group, a feature level or a channel, a decoding device.

16. In paragraph 11, A decoding device wherein the above scaling parameter is used as a prediction for other scaling parameters for the same unit.

17. In paragraph 16, A decoding device in which the predicted scaling parameter is corrected according to the residual value of the scaling parameter.

18. In paragraph 11, A decoding device that performs scaling on the restored feature map using a predicted or corrected scaling parameter based on a scaling method included in the above encoding information.

19. In paragraph 18, A decoding device, wherein the above restored feature map is a feature map on which at least one of unpacking, dequantization, uncutting, dimension restoration, or spatial unsampling has been performed.

20. In a non-transitory computer-readable recording medium storing a bitstream generated by an encoding method, The encoding method comprises the steps of performing at least one of spatial sampling, feature map dimension reduction, or feature map pruning on an input image including one or more feature maps; A step of extracting one or more scaling parameters related to a scale change of the one or more feature maps used in a spatial sampling process for the one or more feature maps, a feature map dimensionality reduction process, or a feature map cropping process; A step of performing quantization of the one or more feature maps into integer form; A step of performing packing by reordering the one or more feature maps according to a certain unit; and A non-transitory computer-readable recording medium comprising a step of generating the bitstream by encoding the one or more feature maps, encoding information extracted in the process, and the one or more scaling parameters.

Images