Encoder and decoder using compression of region of interest in feature map

Abstract

One example of the present disclosure proposes a video coding for machines (VCM) encoding device. The encoding device includes: a feature map extraction unit for extracting one or more feature maps for an input image; a region-of-interest deriving unit for deriving one or more spatial regions of interest from the one or more feature maps; a feature map conversion unit for converting a feature map in units of coding groups including the one or more feature maps; and a feature map encoding unit for encoding the one or more feature maps or the converted feature map in units of coding groups and outputting a bitstream.

Description

Encoder and decoder using region-of-interest compression within feature maps

[0001] The present disclosure relates to an encoder and decoder using region-of-interest compression within a feature map.

[0002] With the continuous development of the information and communication industry, broadcasting services with HD (High Definition) resolution have spread worldwide.

[0003] Through this proliferation, many users have become accustomed to high-resolution and high-quality images and / or videos, and the demand for higher-resolution and high-quality images / videos, such as 4K or 8K or higher UHD (Ultra High Definition) images / videos, has increased in various fields.

[0004] The technology for coding this UHD video data was completed in 2013 through HEVC (High Efficiency Video Coding), a standard technology.

[0005] HEVC is a next-generation video compression technology with a higher compression ratio and lower complexity than the previous H.264 / AVC technology, and is a key technology for effectively compressing the massive data of HD and UHD video.

[0006] HEVC performs block-by-block encoding, like previous compression standards.

[0007] However, unlike H.264 / AVC, there is only one profile. The core encoding technologies included in HEVC's sole profile are divided into eight areas: hierarchical encoding structure technology, transform technology, quantization technology, intra-frame prediction encoding technology, inter-frame motion prediction technology, entropy encoding technology, loop filter technology, and other technologies.

[0008] Since the establishment of the HEVC video codec in 2013, the Versatile Video Coding (VVC) standard, a next-generation video codec that aims to improve performance by more than twice that of HEVC, has been developed to address the expansion of realistic video and virtual reality services utilizing 4K and 8K video images. VVC is called H.266.

[0009] H.266 (VVC) was developed with the goal of being more than twice as efficient as the previous generation codec, H.265 (HEVC). VVC was initially developed with resolutions over 4K in mind, but it was also developed for ultra-high-resolution video processing at a whopping 16K level to support 360-degree videos due to the expansion of the VR market. In addition, as the HDR market is expanding due to the development of display technology, it supports 16-bit color depth as well as 10-bit color depth to respond to this, and supports brightness expressions of 1000 nits, 4000 nits, and 10000 nits. In addition, since it is being developed with the VR market and 360-degree video market in mind, it supports partial frame rates in the range of 0 to 120 FPS.

[0010] Advances in Artificial Intelligence

[0011] Artificial intelligence (AI) is also steadily developing. AI refers to the artificial imitation of human intelligence, including the ability to recognize, classify, infer, predict, and control / decision-making.

[0012] With the advancement of artificial intelligence technology and the increase in Internet of Things (IoT) devices, machine-to-machine traffic is expected to explode, and machine-dependent image analysis is expected to become widely used.

[0013] However, as the amount of images to be analyzed by machines is expected to increase exponentially, issues with server load and power consumption are expected to arise.

[0014] Accordingly, the present disclosure aims to provide an encoder and decoder that utilize compression of a region of interest within a feature map to enable effective machine-based image analysis.

[0015] In order to achieve the above-mentioned purpose, according to one disclosure of the present specification, a VCM encoding device is provided.

[0016] An encoding device for a machine may include a feature map extraction unit that extracts one or more feature maps for an input image; a region of interest derivation unit that derives one or more spatial regions of interest from the one or more feature maps; a feature map conversion unit that converts the feature map into a coding group unit including the one or more feature maps; and a feature map encoding unit that encodes the one or more feature maps or the converted feature map into a coding group unit and outputs a bitstream.

[0017] A VCM decoding device according to one disclosure of the present specification is provided. The VCM decoding device may include a feature map decoding unit that receives a bitstream and restores one or more feature maps or feature map transform coefficients in units of coding groups; a feature map inverse transform unit that performs inverse transform by multiplying the restored feature map transform coefficients in units of coding groups by an inverse transform matrix in response to restoring the feature map transform coefficients; a feature map reconstruction unit that predicts an untransmitted feature map or a portion of a feature map based on the restored feature map to reconstruct an entire feature map; and a machine task analysis unit that performs a pre-requested machine task based on the restored entire feature map.

[0018] According to one disclosure of the present specification, a non-volatile computer-readable storage medium having instructions recorded thereon, wherein the instructions, when executed by one or more processors, cause the one or more processors to perform the steps of: extracting one or more feature maps for an input image; deriving one or more spatial regions of interest from the one or more feature maps; transforming the feature maps into coding group units including the one or more feature maps; and encoding the one or more feature maps or the transformed feature maps into coding group units to output a bitstream.

[0019] According to the present disclosure, image analysis by a machine can be effectively performed.

[0020] Figure 1 schematically illustrates an example of a video / image coding system.

[0021] Figure 2 is a drawing schematically illustrating the configuration of a video / image encoding device.

[0022] Figure 3 is a drawing schematically illustrating the configuration of a video / image decoding device.

[0023] Figures 4a to 4d are exemplary diagrams showing a VCM encoder and a VCM decoder.

[0024] Figure 5 is an exemplary diagram showing a VCM encoder that compresses a feature map and then transmits it as a bitstream, and a VCM decoder that restores the received bitstream, according to an example.

[0025] Figure 6 is an exemplary diagram showing an example of generating feature maps P2 to P5 illustrated in 5.

[0026] Figure 7 is an exemplary diagram showing in detail the MSFF execution unit illustrated in Figure 5.

[0027] FIG. 8 is a block diagram including components of a VCM encoding device according to one embodiment of the present disclosure.

[0028] FIG. 9a is an example of a single feature map extraction structure according to an embodiment of the present disclosure, and FIG. 9b is an example of a multiple feature map extraction structure according to an embodiment of the present disclosure.

[0029] FIG. 10 is an example of a spatial area of ​​interest according to an embodiment of the present disclosure.

[0030] FIG. 11 is a flowchart illustrating a process for extracting a region of interest according to an embodiment of the present disclosure.

[0031] Figure 12 shows examples of input values ​​(a) of the region of interest extraction unit and output values ​​(b) of the region of interest extraction unit.

[0032] FIG. 13 is an example of an input value (a) and an output value (b1, b2) of a coding group derivation unit (840) according to one embodiment of the present disclosure.

[0033] FIG. 14 is a flowchart including a matrix multiplication-based feature map transformation process according to one embodiment of the present disclosure.

[0034] FIG. 15 illustrates an example of a matrix multiplication-based feature map transformation according to an embodiment of the present disclosure.

[0035] FIG. 16 illustrates an example of feature map transformation based on multiple convolutional layers according to an embodiment of the present disclosure.

[0036] FIG. 17 is a block diagram including components of a VCM decoding device according to an embodiment of the present disclosure.

[0037] FIG. 18 is a flowchart including an inverse transformation process of a matrix multiplication-based feature map according to an embodiment of the present disclosure.

[0038] FIG. 19 illustrates an example of inverse transformation of a matrix multiplication-based feature map according to an embodiment of the present disclosure.

[0039] FIG. 20 illustrates an example of inverse transformation of a feature map based on multiple convolutional layers according to an embodiment of the present disclosure.

[0040] FIG. 21 is a flowchart illustrating a feature map reconstruction process according to an embodiment of the present disclosure.

[0041] FIG. 22 is an example of combining a region of interest and a region of no interest according to one embodiment of the present disclosure.

[0042] FIG. 23 illustrates an example of feature map inverse transformation based on multiple convolution layers according to an embodiment of the present disclosure.

[0043] FIG. 24 illustrates a picture_parameter_set syntax including information about a region of interest according to one embodiment of the present disclosure.

[0044] FIG. 25a illustrates a coding group header (coding_group_header) syntax according to one embodiment of the present disclosure.

[0045] FIG. 25b illustrates a decryption reference attribute (is_intra_coded) syntax included in a coding group header according to one embodiment of the present disclosure.

[0046] Specific structural or step-by-step descriptions of embodiments according to the concept of the present disclosure disclosed in this specification or application are merely illustrative for the purpose of explaining embodiments according to the concept of the present disclosure, and embodiments according to the concept of the present disclosure may be implemented in various forms, and embodiments according to the concept of the present disclosure may be implemented in various forms and should not be construed as being limited to the embodiments described in this specification or application.

[0047] Embodiments according to the concept of the present disclosure may have various modifications and take various forms. Therefore, specific embodiments are illustrated in the drawings and described in detail in this specification or application. However, this is not intended to limit embodiments according to the concept of the present disclosure to specific disclosed forms, and it should be understood that all modifications, equivalents, and alternatives included within the spirit and technical scope of the present disclosure are included.

[0048] While terms such as "first" and / or "second" may be used to describe various components, these components should not be limited by these terms. These terms are only intended to distinguish one component from another; for example, without departing from the scope of the present disclosure, a first component may be referred to as a "second component," and similarly, a second component may also be referred to as a "first component."

[0049] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. Conversely, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Other expressions that describe the relationship between components, such as "between" and "directly between" or "adjacent to" and "directly adjacent to", should be interpreted similarly.

[0050] The terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit the present disclosure. The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, it should be understood that the terms "comprises" or "has" indicate the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0051] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0052] Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0053] In describing the embodiments, description of technical contents that are well known in the technical field to which the present disclosure belongs and are not directly related to the present disclosure will be omitted.

[0054] This is to convey the gist of the present disclosure more clearly without obscuring it by omitting unnecessary explanations.

[0055] This document relates to video / image coding. For example, the method / embodiment disclosed in this document may be related to the Versatile Video Coding (VVC) standard (ITU-T Rec. H.266), the next-generation video / image coding standard after VVC, or other video coding-related standards (e.g., the High Efficiency Video Coding (HEVC) standard (ITU-T Rec. H.265), the essential video coding (EVC) standard, the AVS2 standard, etc.).

[0056] This document presents various embodiments of video / image coding, and unless otherwise stated, the embodiments may be performed in combination with each other.

[0057] In this document, "video" can refer to a series of images over time. "Picture" generally refers to a unit representing a single image from a specific time period, and "slice" / "tile" are units that constitute part of a picture in coding.

[0058] A slice / tile can contain one or more coding tree units (CTUs). A picture can consist of one or more slices / tiles. A picture can consist of one or more tile groups. A tile group can contain one or more tiles.

[0059] A pixel or pel can mean the smallest unit that constitutes a picture (or image). Additionally, the term "sample" can be used as a counterpart to a pixel. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luma component, or only the pixel / pixel value of the chroma component. Alternatively, a sample can mean a pixel value in the spatial domain, or when such a pixel value is converted to the frequency domain, it can mean a transform coefficient in the frequency domain.

[0060] A unit may represent a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to that region.

[0061] A unit can contain one luma block and two chroma (e.g., cb, cr) blocks. The term "unit" may sometimes be used interchangeably with the terms "block" or "area." In general, an MxN block can contain a set (or array) of samples (or array of samples) or transform coefficients, each consisting of M columns and N rows.

[0062] Figure 1 schematically illustrates an example of a video / image coding system.

[0063] Referring to FIG. 1, a video / image coding system may include a source device and a receiving device. The source device may transmit encoded video / image information or data to the receiving device via a digital storage medium or a network in the form of a file or streaming.

[0064] The source device may include a video source, an encoding device, and a transmission unit. The receiving device may include a reception unit, a decoding device, and a renderer.

[0065] The encoding device may be referred to as a video / image encoding device, and the decoding device may be referred to as a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display unit, and the display unit may be configured as a separate device or an external component.

[0066] A video source may obtain video / images through a process of capturing, synthesizing, or generating video / images. The video source may include a video / image capture device and / or a video / image generation device. A video / image capture device may include, for example, one or more cameras, a video / image archive containing previously captured video / images, etc. A video / image generation device may include, for example, a computer, a tablet, a smartphone, etc., and may (electronically) generate video / images. For example, a virtual video / image may be generated through a computer, etc., in which case the video / image capture process may be replaced by a process of generating related data.

[0067] An encoding device can encode input video / images. The encoding device can perform a series of procedures, such as prediction, transformation, and quantization, to improve compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0068] The transmission unit can transmit encoded video / image information or data output in bitstream form to the receiving unit of the receiving device via a digital storage medium or network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit can include an element for generating a media file using a predetermined file format and an element for transmission via a broadcasting / communication network.

[0069] The receiving unit can receive / extract the bitstream and transmit it to a decoding device.

[0070] The decoding device can decode the video / image by performing a series of procedures such as inverse quantization, inverse transformation, and prediction corresponding to the operation of the encoding device.

[0071] The renderer can render decoded video / images. The rendered video / images can be displayed through the display unit.

[0072] Figure 2 is a drawing schematically illustrating the configuration of a video / image encoding device.

[0073] The term “video encoding device” hereinafter may include a video encoding device.

[0074] Referring to FIG. 2, the encoding device (10a) may be configured to include an image partitioner (10a-10), a prediction unit (predictor) (10a-20), a residual processor (residual processor) (10a-30), an entropy encoder (entropy encoder) (10a-40), an adder (adder) (10a-50), a filter (filter) (10a-60), and a memory (10a-70). The prediction unit (10a-20) may include an inter prediction unit (10a-21) and an intra prediction unit (10a-22). The residual processing unit (10a-30) may include a transformer (10a-32), a quantizer (10a-33), a dequantizer (10a-34), and an inverse transformer (10a-35). The residual processing unit (10a-30) may further include a subtractor (10a-31). The addition unit (10a-50) may be called a reconstructor or a reconstructed block generator. The above-described image segmentation unit (10a-10), prediction unit (10a-20), residual processing unit (10a-30), entropy encoding unit (10a-40), addition unit (10a-50), and filtering unit (10a-60) may be configured by one or more hardware components (e.g., encoder chipset or processor) according to an embodiment. In addition, the memory (10a-70) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (10a-70) as an internal / external component.

[0075] The image segmentation unit (10a-10) can segment an input image (or picture, frame) input to the encoding device (10a) into one or more processing units.

[0076] For example, the processing unit may be called a coding unit (CU). In this case, the coding unit may be recursively split from a coding tree unit (CTU) or a largest coding unit (LCU) according to a Quad-tree binary-tree ternary-tree (QTBTTT) structure. For example, one coding unit may be split into multiple coding units of deeper depth based on a quad-tree structure, a binary-tree structure, and / or a ternary structure. In this case, for example, the quad-tree structure may be applied first, and the binary-tree structure and / or the ternary structure may be applied later. Alternatively, the binary-tree structure may be applied first. The coding procedure according to the present document may be performed based on the final coding unit that is no longer split. In this case, based on coding efficiency according to image characteristics, etc., the maximum coding unit can be used as the final coding unit, or, if necessary, the coding unit can be recursively divided into coding units of lower depths, and the coding unit of the optimal size can be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration described below. As another example, the processing unit may further include a prediction unit (PU) or a transformation unit (TU). In this case, the prediction unit and the transformation unit may each be divided or partitioned from the final coding unit described above. The prediction unit may be a unit of sample prediction, and the transformation unit may be a unit for deriving a transform coefficient and / or a unit for deriving a residual signal from a transform coefficient.

[0077] The term "unit" may be used interchangeably with terms such as "block" or "area" depending on the case. In general, an MxN block can represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luminance component, or only the pixel / pixel value of the chroma component. A sample can be used as a term corresponding to a pixel or pel in a picture (or image).

[0078] The subtraction unit (10a-31) can subtract the prediction signal (predicted block, prediction samples, or prediction sample array) output from the prediction unit (10a-20) from the input image signal (original block, original samples, or original sample array) to generate a residual signal (residual block, residual samples, or residual sample array), and the generated residual signal is transmitted to the conversion unit (10a-32). The prediction unit (10a-20) can perform prediction on a block to be processed (hereinafter, referred to as a current block) and generate a predicted block including prediction samples for the current block.

[0079] The prediction unit (10a-20) can determine whether intra-prediction or inter-prediction is applied to the current block or CU unit. As described later in the description of each prediction mode, the prediction unit can generate various prediction-related information, such as prediction mode information, and transmit it to the entropy encoding unit (10a-40). The prediction-related information can be encoded by the entropy encoding unit (10a-40) and output in the form of a bitstream.

[0080] The intra prediction unit (10a-22) can predict the current block by referring to samples within the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from it, depending on the prediction mode.

[0081] In intra prediction, prediction modes can include multiple non-directional modes and multiple directional modes. Non-directional modes can include, for example, DC modes and planar modes. Directional modes can include, for example, 33 directional prediction modes or 65 directional prediction modes, depending on the granularity of the prediction direction.

[0082] However, this is only an example; depending on the settings, a greater or lesser number of directional prediction modes may be used. The intra prediction unit (10a-22) may also determine the prediction mode to be applied to the current block by utilizing the prediction mode applied to the surrounding blocks.

[0083] The inter prediction unit (10a-21) can derive a predicted block for a current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on an inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring block can include a spatial neighboring block existing in the current picture and a temporal neighboring block existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The above temporal neighboring blocks may be called collocated reference blocks, collocated CUs (colCUs), etc., and a reference picture including the temporal neighboring blocks may be called a collocated picture (colPic). For example, the inter prediction unit (10a-21) may construct a motion information candidate list based on the neighboring blocks, and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction may be performed based on various prediction modes, and for example, in the case of skip mode and merge mode, the inter prediction unit (10a-21) may use the motion information of the neighboring blocks as the motion information of the current block. In the case of skip mode, unlike the merge mode, a residual signal may not be transmitted.In the motion vector prediction (MVP) mode, the motion vector of the surrounding blocks is used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0084] The prediction unit (10a-20) can generate a prediction signal based on various prediction methods described below. For example, the prediction unit can apply intra prediction or inter prediction to predict a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). In addition, the prediction unit can perform intra block copy (IBC) to predict a block. The intra block copy can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in this document.

[0085] The prediction signal generated through the inter prediction unit (10a-21) and / or the intra prediction unit (10a-22) can be used to generate a reconstructed signal or a residual signal. The transform unit (10a-32) can generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique can include a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), a Graph-Based Transform (GBT), or a Conditionally Non-linear Transform (CNT). Here, GBT refers to a transform obtained from a graph when the relationship information between pixels is expressed as a graph. CNT refers to a transform obtained based on generating a prediction signal using all previously reconstructed pixels. In addition, the transform process can be applied to a pixel block having a square equal size, or can be applied to a block of a non-square variable size.

[0086] The quantization unit (10a-33) quantizes the transform coefficients and transmits them to the entropy encoding unit (10a-40), and the entropy encoding unit (10a-40) can encode the quantized signal (information about the quantized transform coefficients) and output it as a bitstream. The information about the quantized transform coefficients can be called residual information.

[0087] The quantization unit (10a-33) can rearrange the quantized transform coefficients in the form of a block into a one-dimensional vector based on the coefficient scan order, and can also generate information about the quantized transform coefficients based on the quantized transform coefficients in the form of the one-dimensional vector. The entropy encoding unit (10a-40) can perform various encoding methods, such as, for example, exponential Golomb, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), etc.

[0088] The entropy encoding unit (10a-40) may encode, together or separately, information necessary for video / image restoration (e.g., values ​​of syntax elements, etc.) in addition to quantized transform coefficients. The encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream in the form of a network abstraction layer (NAL) unit. The video / image information may further include information regarding various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. The signaling / transmitted information and / or syntax elements described later in this document may be encoded through the above-described encoding procedure and included in the bitstream. The bitstream may be transmitted through a network or may be stored in a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The signal output from the entropy encoding unit (10a-40) may be configured as an internal / external element of the encoding device (10a) by a transmitting unit (not shown) that transmits and / or a storing unit (not shown) that stores the signal, or the transmitting unit may be included in the entropy encoding unit (10a-40).

[0089] The quantized transform coefficients output from the quantization unit (10a-33) can be used to generate a prediction signal. For example, by applying inverse quantization and inverse transformation to the quantized transform coefficients through the inverse quantization unit (10a-34) and the inverse transform unit (10a-35), a residual signal (residual block or residual samples) can be reconstructed. The addition unit (10a-50) can add the reconstructed residual signal to the prediction signal output from the prediction unit (10a-20), thereby generating a reconstructed signal (reconstructed picture, reconstructed block, reconstructed samples, or reconstructed sample array). When there is no residual for the target block to be processed, such as when skip mode is applied, the predicted block can be used as the reconstructed block. The generated restoration signal can be used for intra prediction of the next processing target block within the current picture, and can also be used for inter prediction of the next picture after filtering as described below.

[0090] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture encoding and / or restoration process.

[0091] The filtering unit (10a-60) can improve subjective / objective picture quality by applying filtering to the restoration signal. For example, the filtering unit (10a-60) can apply various filtering methods to the restoration picture to generate a modified restoration picture, and store the modified restoration picture in the memory (10a-70), specifically, in the DPB of the memory (10a-70). The various filtering methods may include, for example, deblocking filtering, sample adaptive offset (SAO), an adaptive loop filter, a bilateral filter, etc. The filtering unit (10a-60) can generate various information regarding filtering and transmit the information to the entropy encoding unit (10a-90), as described below in the description of each filtering method. The information regarding filtering may be encoded by the entropy encoding unit (10a-90) and output in the form of a bitstream.

[0092] The modified restored picture transmitted to the memory (10a-70) can be used as a reference picture in the inter prediction unit (10a-80). Through this, when inter prediction is applied, the encoding device can avoid prediction mismatch between the encoding device (10a) and the decoding device, and can also improve encoding efficiency.

[0093] The DPB of the memory (10a-70) can store the modified restored picture to be used as a reference picture in the inter prediction unit (10a-21). The memory (10a-70) can store motion information of a block from which motion information in the current picture is derived (or encoded) and / or motion information of blocks in a picture that has already been restored. The stored motion information can be transmitted to the inter prediction unit (10a-21) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (10a-70) can store restored samples of restored blocks in the current picture and transmit them to the intra prediction unit (10a-22).

[0094] Figure 3 is a drawing schematically illustrating the configuration of a video / image decoding device.

[0095] Referring to FIG. 3, the decoding device (10b) may be configured to include an entropy decoder (10b-10), a residual processor (10b-20), a predictor (10b-30), an adder (10b-40), a filter (10b-50), and a memory (10b-60). The predictor (10b-30) may include an inter-prediction unit (10b-31) and an intra-prediction unit (10b-32). The residual processor (10b-20) may include a dequantizer (10b-21) and an inverse transformer (10b-21). The entropy decoding unit (10b-10), residual processing unit (10b-20), prediction unit (10b-30), addition unit (10b-40), and filtering unit (10b-50) described above may be configured by a single hardware component (e.g., decoder chipset or processor) according to an embodiment. In addition, the memory (10b-60) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (10b-60) as an internal / external component.

[0096] When a bitstream including video / image information is input, the decoding device (10b) can restore the image corresponding to the process in which the video / image information is processed in the encoding device of FIG. 2. For example, the decoding device (10b) can derive units / blocks based on block division-related information obtained from the bitstream. The decoding device (10b) can perform decoding using a processing unit applied in the encoding device. Therefore, the processing unit of decoding may be, for example, a coding unit, and the coding unit may be divided from a coding tree unit or a maximum coding unit according to a quad tree structure, a binary tree structure, and / or a ternary tree structure. One or more transform units may be derived from the coding unit. Then, the restored image signal decoded and output by the decoding device (10b) can be reproduced through a reproduction device.

[0097] The decoding device (10b) can receive a signal output from the encoding device of FIG. 2 in the form of a bitstream, and the received signal can be decoded through the entropy decoding unit (10b-10). For example, the entropy decoding unit (10b-10) can parse the bitstream to derive information (e.g., video / image information) necessary for image restoration (or picture restoration). The video / image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information.

[0098] The decoding device can further decode the picture based on information about the parameter set and / or the general restriction information. The signaling / received information and / or syntax elements described later in this document can be decoded and obtained from the bitstream through the decoding procedure. For example, the entropy decoding unit (10b-10) can decode information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output values ​​of syntax elements required for image restoration and quantized values ​​of transform coefficients for the residual.

[0099] In more detail, the CABAC entropy decoding method receives a bin corresponding to each syntax element in a bitstream, determines a context model using information of the syntax element to be decoded and decoding information of surrounding and decoding target blocks or information of symbols / bins decoded in a previous step, and predicts the occurrence probability of the bin according to the determined context model to perform arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element. At this time, the CABAC entropy decoding method can update the context model using information of the decoded symbol / bin for the context model of the next symbol / bin after determining the context model. Among the information decoded by the entropy decoding unit (10b-10), information regarding prediction is provided to the prediction unit (10b-30), and information regarding the residual on which entropy decoding has been performed by the entropy decoding unit (10b-10), i.e., quantized transform coefficients and related parameter information, can be input to the inverse quantization unit (10b-21).

[0100] In addition, information regarding filtering among the information decoded by the entropy decoding unit (10b-10) may be provided to the filtering unit (10b-50). Meanwhile, a receiving unit (not shown) that receives a signal output from an encoding device may be further configured as an internal / external element of the decoding device (10b), or the receiving unit may be a component of the entropy decoding unit (10b-10). Meanwhile, the decoding device according to the present document may be called a video / video / picture decoding device, and the decoding device may be divided into an information decoder (video / video / picture information decoder) and a sample decoder (video / video / picture sample decoder). The above information decoder may include the entropy decoding unit (10b-10), and the sample decoder may include at least one of the inverse quantization unit (10b-21), the inverse transformation unit (10b-22), the prediction unit (10b-30), the addition unit (10b-40), the filtering unit (10b-50), and the memory (10b-60).

[0101] The inverse quantization unit (10b-21) can inverse quantize the quantized transform coefficients and output the transform coefficients. The inverse quantization unit (10b-21) can rearrange the quantized transform coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scanning order performed in the encoding device. The inverse quantization unit (10b-21) can perform inverse quantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain transform coefficients.

[0102] In the inverse transform unit (10b-22), the transform coefficients are inversely transformed to obtain a residual signal (residual block, residual sample array).

[0103] The prediction unit can perform a prediction for the current block and generate a predicted block including prediction samples for the current block.

[0104] The prediction unit can determine whether intra prediction or inter prediction is applied to the current block based on the information about the prediction output from the entropy decoding unit (10b-10), and can determine a specific intra / inter prediction mode.

[0105] The prediction unit can generate a prediction signal based on various prediction methods described below. For example, the prediction unit can apply intra prediction or inter prediction to predict a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP). In addition, the prediction unit can perform intra block copy (IBC) to predict a block. The intra block copy can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in this document.

[0106] The intra prediction unit (10b-32) can predict the current block by referencing samples within the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from it, depending on the prediction mode.

[0107] In intra prediction, prediction modes may include multiple non-directional modes and multiple directional modes. The intra prediction unit (10b-32) may determine the prediction mode to be applied to the current block by utilizing the prediction modes applied to the surrounding blocks.

[0108] The inter prediction unit (10b-31) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on an inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.).

[0109] In the case of inter prediction, the neighboring blocks may include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit (10b-31) may construct a motion information candidate list based on the neighboring blocks, and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter prediction may be performed based on various prediction modes, and the information regarding the prediction may include information indicating the mode of inter prediction for the current block.

[0110] The addition unit (10b-40) can generate a restoration signal (restored picture, restoration block, restoration sample array) by adding the acquired residual signal to the prediction signal (predicted block, prediction sample array) output from the prediction unit (10b-30). In cases where there is no residual for the block to be processed, such as when skip mode is applied, the predicted block can be used as the restoration block.

[0111] The addition unit (10b-40) may be called a restoration unit or a restoration block generation unit.

[0112] The generated restoration signal can be used for intra prediction of the next processing target block within the current picture, can be output after filtering as described below, or can be used for inter prediction of the next picture.

[0113] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture decoding process.

[0114] The filtering unit (10b-50) can improve subjective / objective image quality by applying filtering to the restoration signal. For example, the filtering unit (10b-50) can apply various filtering methods to the restoration picture to generate a modified restoration picture, and transmit the modified restoration picture to the memory (60), specifically, the DPB of the memory (10b-60). The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0115] The (corrected) reconstructed picture stored in the DPB of the memory (10b-60) can be used as a reference picture in the inter prediction unit (10b-31). The memory (10b-60) can store motion information of a block from which motion information is derived (or decoded) within the current picture and / or motion information of blocks within a picture that has already been reconstructed. The stored motion information can be transmitted to the inter prediction unit (10b-31) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (10b-60) can store reconstructed samples of reconstructed blocks within the current picture and transmit them to the intra prediction unit (10b-32).

[0116] In this specification, the embodiments described in the prediction unit (10b-30), the inverse quantization unit (10b-21), the inverse transformation unit (10b-22), and the filtering unit (10b-50) of the decoding device (10b) can be applied to the prediction unit (10a-20), the inverse quantization unit (10a-34), the inverse transformation unit (10a-35), and the filtering unit (10a-60) of the encoding device (10a) in the same manner or correspondingly.

[0117] As described above, prediction is performed to increase compression efficiency when performing video coding. Through this, a predicted block including prediction samples for a current block, which is a coding target block, can be generated. Here, the predicted block includes prediction samples in a spatial domain (or pixel domain). The predicted block is derived identically from an encoding device and a decoding device, and the encoding device can increase video coding efficiency by signaling information (residual information) about the residual between the original block and the predicted block, rather than the original sample value of the original block itself, to a decoding device. The decoding device can derive a residual block including residual samples based on the residual information, and generate a reconstructed block including reconstructed samples by combining the residual block and the predicted block, and can generate a reconstructed picture including the reconstructed blocks.

[0118] The above residual information can be generated through transformation and quantization procedures.

[0119] For example, the encoding device can derive a residual block between the original block and the predicted block, perform a transform procedure on residual samples (a residual sample array) included in the residual block to derive transform coefficients, perform a quantization procedure on the transform coefficients to derive quantized transform coefficients, and signal related residual information to a decoding device (via a bitstream). Here, the residual information can include information such as value information, position information, transform technique, transform kernel, and quantization parameter of the quantized transform coefficients. The decoding device can perform an inverse quantization / inverse transform procedure based on the residual information to derive residual samples (or residual blocks). The decoding device can generate a reconstructed picture based on the predicted block and the residual block. The encoding device can also inverse quantize / inverse transform the quantized transform coefficients to derive a residual block for reference in inter prediction of a subsequent picture, and generate a reconstructed picture based on the residual block.

[0120] <VCM(Video coding for Machines)>

[0121] With the recent advancements in various industries such as surveillance, intelligent transportation, smart cities, intelligent industry, and intelligent content, the amount of image or feature map data consumed by machines is increasing. In contrast, traditional video compression methods currently in use were developed with human vision in mind, and therefore contain unnecessary information, making them inefficient for machine tasks. For example, the resolution of images from the viewer's perspective may be higher than that of images (e.g., feature maps) from the machine's perspective. Therefore, research on video codec technologies that efficiently compress feature maps for machine tasks is needed.

[0122] The Moving Picture Experts Group (MPEG), an international standardization group for multimedia encoding, is discussing Video Coding for Machines (VCM). VCM is an image or feature map encoding technology that targets machine vision, rather than human viewer vision. In this document, feature maps can be referred to as "feature maps," and features can be referred to as "features."

[0123] Figures 4a to 4d are exemplary diagrams showing a VCM encoder and a VCM decoder.

[0124] Referring to FIG. 4a, a VCM encoder (100a) and a VCM decoder (100b) are shown.

[0125] When a VCM encoder (100a) encodes a video and / or a feature map and transmits it as a bitstream, a VCM decoder (100b) can decode and output the bitstream. At this time, the VCM decoder (100b) can output one or more videos and / or feature maps. For example, the VCM decoder (100b) can output a first feature map for machine-based analysis and a first image for user viewing. The first image can have a higher resolution than the first feature map.

[0126] Referring to FIG. 4b, a feature extractor for extracting a feature map may be connected to the front end of the VCM encoder (100a).

[0127] The VCM encoder (100a) may include a feature encoder.

[0128] The VCM decoder (100b) may include a feature decoder and a video reconstructor. The feature decoder may decode a feature map from a bitstream and output a first feature map for machine-assisted analysis. The video reconstructor may regenerate and output a first video from the bitstream for viewing by a user.

[0129] Referring to Fig. 4c, a feature extractor for extracting a feature map is connected to the front end of the VCM encoder (100). The VCM encoder (100a) may include a feature encoder.

[0130] The VCM decoder (100b) may include a feature decoder. The feature decoder may decode a feature map from a bitstream and output a first feature map for machine-based analysis. That is, the bitstream may be encoded only as a feature map, not as an image. To elaborate, the feature map may be data containing information about features for processing a specific task of a machine based on an image.

[0131] Referring to FIG. 4d, a feature extractor may be connected to the front end of the VCM encoder (100a).

[0132] The VCM encoder (100a) may include a feature converter and a video encoder. The video encoder may be the encoding device (10a) illustrated in FIG. 2.

[0133] The VCM decoder (100b) illustrated in FIG. 4d may include a video decoder and an inverse converter. The video decoder may be the decoding device (10b) illustrated in FIG. 3.

[0134] FIG. 5 is an exemplary diagram showing a VCM encoder that compresses a feature map and then transmits it as a bitstream, and a VCM decoder that restores a received bitstream, according to an example; FIG. 6 is an exemplary diagram showing an example of generating feature maps P2 to P5 shown in FIG. 5; and FIG. 7 is an exemplary diagram showing in detail the MSFF execution unit shown in FIG. 5.

[0135] As can be seen with reference to FIG. 5, the VCM encoder (100a) (or transmitter) may include a multi-scale feature fusion (MSFF) performing unit (100a-1) and a single-stream feature codec (SSFC) encoder (100a-2). The VCM decoder (100b) (or receiver) may include a SSFC decoder (100b-1) and a multi-scale feature reconstruction (MSFR) performing unit (100b-2).

[0136] The above MSFF execution unit (100a-1) aligns the feature maps P2 to P5 of the P layer and then concatenates them.

[0137] The feature maps P2 to P5 of the above P layer can be generated as shown in Fig. 6.

[0138] Referring to FIG. 6, feature maps P2 to P5 can be generated using a combination of a first artificial neural network model, e.g., ResNet, and a second artificial neural network model, e.g., Feature Pyramid Network (FPN). The first artificial neural network model may be referred to as a backbone, and the second artificial neural network model may be referred to as a neck or a head. As illustrated in FIG. 6, when an image is input, the first artificial neural network model (e.g., ResNet) may perform several stages (e.g., Stage 1 to Stage 5) in a bottom-up manner. Each stage may include, for example, convolution, batch normalization, ReLu (Rectified Linear Unit), etc. Convolution may be performed on the outputs from each stage of the first artificial neural network model (e.g., ResNet), so that feature maps C2 to C5 of the C layer may be generated. The above second artificial neural network model (e.g., FPN) can receive feature maps C2 to C5 of the C layer as input and accumulate them in a top-down manner to generate feature maps M2 to M5 of the M layer. By performing convolution on feature maps M2 to M5 of the M layer, feature maps P2 to P5 of the P layer can be generated.

[0139] Referring again to FIG. 5, the MSFF performing unit (100a-1) aligns and then concatenates the feature maps P2 to P5 of the P layer. Specifically, the MSFF performing unit (100a-1) downsamples the sizes of the feature maps P2 to P4 to match the size of the feature map P5 and then concatenates them.

[0140] Referring to FIG. 7, the MSFF execution unit (100a-1) performs pooling on each of the feature maps P2 to P5. Through the pooling, each of the feature maps P2 to P5 can be reduced to a specific size, for example, 64 x 64 or 32 x 32. Alternatively, the remaining feature maps (i.e., feature maps P2, P3, and P4) can be made to have the same size based on the feature map P5 having the smallest size among the feature maps P2 to P5. That is, pooling can be performed on each of the remaining feature maps (i.e., feature maps P2, P3, and P4) to make them the same size as the feature map P5.

[0141] Then, the MSFF execution unit (100a-1) can concatenate the feature maps P2 to P5. For example, if each of the feature maps P2 to P5 is in the form of a cube with a size of 64 x 64 and 256 channels, after concatenation, it can be in the form of a cube with a size of 64 x 64 and 1024 channels.

[0142] The above concatenated feature maps can pass through SEblock.

[0143] Referring back to FIG. 5, the SEBlock is a processing operation by SE (Squeeze-and-Excitation), which calculates channel-specific weights of a feature map and multiplies these weights by the output feature map of the residual unit. In addition, the MSFF performing unit can perform channel-wise reduction on the output of the SEBlock using convolution. The MSFF performing unit can ultimately transmit the output F to the SSFC encoder (100a-2).

[0144] When the above SSFC encoder (100a-2) receives the output F from the above MSFF execution unit (100a-1), it can reduce the number of channels and then transmit the bitstream.

[0145] For this purpose, the SSFC encoder (100a-2) may include convolution, batch normalization, and a Tanh function.

[0146] The above VCM decoder (100b) (i.e., the receiving end) may include an MSFR performing unit (100b-1) and an SSFC decoder (100b-2). When the SSFC decoder (100b-2) receives the bitstream, it may generate an output F'*. To this end, the SSFC decoder (100b-2) may include convolution, batch normalization, and PReLu, etc.

[0147] The MSFR performing unit (100b-1) of the above VCM decoder (100b) can restore the feature map P2' to the feature map P5' of the P' layer from the output F'* from the SSFC decoder (100b-2). To this end, the MSFR performing unit (100b-1) can perform upsampling / downsampling and accumulation, etc.

[0148]

[0149] FIG. 8 is a block diagram including components of a VCM encoding device according to one embodiment of the present disclosure. For convenience of explanation in the present disclosure, the encoding device may be referred to as an encoder or encoder, and the decoding device may be referred to as a decoder or decoder.

[0150] A VCM encoding device (100a) according to one embodiment of the present disclosure can extract a feature map from an input image, encode the feature map, and output a bitstream. The VCM encoding device (100a) can include a feature map extraction and selection unit (810), a region of interest derivation unit (820), a region of interest extraction unit (830), a coding group derivation unit (840), a feature map conversion unit (850), and a feature map encoding unit (860).

[0151] The feature map extraction and selection unit (810) can extract one or more feature maps from an image or video, select a feature map to be encoded from among the one or more feature maps, and further scale the selected feature map.

[0152] The region of interest derivation unit (820) can derive a region of interest in a spatial axis or channel axis from a feature map.

[0153] The region of interest extraction unit (830) can extract only the region of interest from the spatial axis or channel axis in the feature map. The region of interest extraction unit (830) can be included in the region of interest derivation unit (820). In one embodiment, the region of interest derivation unit (820) can extract the region of interest from the spatial axis or channel axis in the feature map and proceed to the next step by extracting only the region of interest.

[0154] The coding group derivation unit (840) can group feature maps to be encoded into coding groups. The feature map conversion unit (850) and the feature map encoding unit (860) can be performed independently for each coding group. The coding group is a step for feature map conversion and encoding, and can define a criterion for: i) designating all encoding targets as one group; ii) designating feature maps with full resolution from which regions of interest are not extracted and feature maps from which only regions of interest are extracted as different groups; iii) designating all feature maps with full resolution from which regions of interest are not extracted as one group; or iv) designating regions of interest having the same region of interest index as one coding group, and can classify one or more feature maps extracted from an input image into one or more groups according to the criterion.

[0155] The feature map transformation unit (850) can perform transformation on the feature map within the coding group. The transformation method may be a matrix multiplication-based method or a multi-convolution layer performing method. In one embodiment, in the case of matrix multiplication-based transformation, the feature map transformation unit (850) can derive transformation coefficients by scaling the spatial resolution of the feature map in units of coding groups, dividing the feature map into transformation units, and then performing matrix multiplication-based transformation. In another embodiment, in the case of multi-convolution layer-based transformation, the spatial resolution of the feature map can be adjusted to be the same in units of coding groups, combined along a channel axis, and then implicitly performed along the channel axis to perform multi-convolution layers. The feature map transformation unit (850) can be selectively performed at the input image or frame level.

[0156] The feature map encoding unit (860) can output a bit stream by encoding a feature map that has been converted by the feature map conversion unit (850) in units of coding groups or a feature map that has not been converted (a feature map transmitted from the region of interest derivation unit (820)). The feature map encoding unit (860) can output a bit stream by encoding a feature map or a converted feature map based on a method such as prediction, conversion, quantization, or entropy coding.

[0157]

[0158] FIG. 9a is an example of a single feature map extraction structure according to an embodiment of the present disclosure, and FIG. 9b is an example of a multiple feature map extraction structure according to an embodiment of the present disclosure.

[0159] The feature map extraction and selection unit (810) of the VCM encoding device (100a) can extract one or more feature maps from a frame input to the feature map encoder and select and scale one or more feature maps to be encoded.

[0160] In the feature map extraction step, a single feature map extraction structure may be composed of one or more convolution layers and may have a structure such as, for example, FIG. 9A. Referring to FIG. 9A, the feature map extraction and selection unit (810) may sequentially perform one or more convolution layers on an input image (one frame or multiple frames) to derive one feature map.

[0161] Alternatively, the structure for extracting multiple feature maps may be composed of one or more convolution layers, and may have a structure as in FIG. 9b, for example. Referring to FIG. 9b, the feature map extraction and selection unit (810) may derive multiple feature maps by performing a series or parallel combination of one or more convolution layers on the input image. The multiple extracted feature maps may have different spatial resolutions and different channel lengths. For example, the multiple extracted feature maps may all have the same channel length and the spatial resolution may increase linearly. In FIG. 9b, four feature maps (feature maps 0 to 3) extracted from the input image have the same channel length c, and the height h and the width w increase linearly from feature map 3 to feature map 0. Specifically, feature map 3 consists of c, h, and w, feature map 2 consists of c, 2h, and 2w, feature map 1 consists of c, 4h, and 4w, and feature map 0 consists of c, 8h, and 8w.

[0162] In the feature map selection step, the feature map extraction and selection unit (810) can select a feature map to be encoded from one or more extracted feature maps. Unselected feature maps are not encoded by the VCM encoding device (100a) and are not transmitted to the VCM decoding device (100b). However, if necessary, the VCM decoding device (100b) can generate an unrestored feature map (a feature map not selected by the VCM encoding device (100a)) based on a received and restored feature map (a feature map selected by the VCM encoding device (100a)).

[0163] The VCM encoding device (100a) can transmit information related to feature map extraction to the VCM decoding device (100b). The information related to feature map extraction includes the index of the layer from which the feature map is extracted, the total number of extracted feature maps, the number of selected feature maps, the index / width / height / channel length of each selected feature map, etc., and the information can be transmitted to the VCM decoding device (100b) at a higher level of a sequence or frame group or frame level.

[0164] In the feature map scaling step, the feature map extraction and selection unit (810) can perform spatial resolution scaling on the feature map selected in the feature map selection step. A syntax indicating whether spatial resolution scaling is performed can be transmitted at the sequence level, frame group level, or frame level. A parameter indicating a scale degree for changing the spatial resolution can be transmitted at the sequence level, frame group level, or frame level. The method for deriving and expressing the parameter indicating the scale degree can be, for example, as follows.

[0165] Method 1 for deriving parameters: The scaling value for each feature map can be derived from a specific fixed value, a user-specified value, or a specific algorithm for each sequence / frame group / frame. The specific algorithm can be, for example, an algorithm for deriving a scale parameter for downsampling all feature maps to have the same spatial resolution as the smallest feature map, or an algorithm for deriving a scaling parameter based on the size of objects present in the current frame or frame group.

[0166] The width and height of a feature map can be used by sharing a single scale parameter, or can be transmitted using separate scale parameters for the width and height. Depending on the meaning of the transmitted scale parameter, the equations for calculating the spatial resolution width and height (FM_scaled_width, FM_scaled_height) after scaling the feature map can be as shown in Equations 1 to 6. Equations 1 and 2 can be cases where the original values ​​of the scale parameters are transmitted, Equations 3 and 4 can be cases where the values ​​obtained by performing a log2 operation on the scale parameters are transmitted, and Equations 5 and 6 can be cases where the indexes of the scale parameters to be used in the scale parameter table are transmitted. The scale parameter table can be in the form of a vector in which a plurality of scale parameters are listed.

[0167]

[0168]

[0169]

[0170]

[0171]

[0172]

[0173] Method 2 for Deriving Parameters: A single scaling value commonly applied to multiple feature maps can be transmitted as a fixed value, a user-specified value, or a value derived through a specific algorithm on a per-sequence / frame-group / frame basis. The process for deriving the scaled spatial resolution width and height using the scaling parameter can be identical to Method 1.

[0174]

[0175] FIG. 10 is an example of a spatial area of ​​interest according to an embodiment of the present disclosure.

[0176] The region of interest derivation unit (820) of the VCM encoding device (100a) according to one embodiment of the present disclosure can derive one or more spatial regions of interest from a feature map.

[0177] The region of interest may be an arbitrary-shaped region composed of a continuous set of unit regions (NxM). The region of interest may be defined by location and size information of a region of interest map or polygon.

[0178] When the region of interest is defined as a region of interest map, the region of interest map can be a two-dimensional map with the spatial resolution of the feature map, as shown in (a) of Fig. 10. In the region of interest map, the values ​​of pixels included in the region of interest and those included in non-regions of interest can be different values. For example, in the region of interest map, the values ​​of pixels included in the region of interest can be set to 1, and the values ​​of pixels not included in the region of interest can be set to 0.

[0179] The region of interest can be defined by the position and size information of the polygon, as in (b) of Fig. 10. If the polygon is, for example, a rectangle, the region of interest can be defined by the upper left coordinate, width, and height of the rectangle.

[0180] The VCM encoding device (100a) can transmit information syntax regarding a region of interest to the VCM decoding device (100b). The region of interest information syntax can be included in picture_parameter_set and transmitted to the VCM decoding device (100b). The picture_parameter_set syntax can be defined as shown in FIG. 24. The region of interest information syntax can include the width (N) and height (M) of a unit region of the region of interest. If the region of interest is defined as a region of interest map, the region of interest map can be transmitted by downsampling it to the width and height of the unit region, respectively. If the region of interest is defined as a rectangle, the upper left coordinate of the rectangle, the width and height values ​​of the rectangle, and the scale parameter can be transmitted by downscaling them to the width and height of the unit region, respectively.

[0181] Regions of interest can be derived one or more times using neural network-based or non-neural network-based methods, or the user can specify regions of interest on a sequence-by-sequence or frame-by-frame basis.

[0182]

[0183] Fig. 11 is a flowchart illustrating a process for extracting a region of interest according to an embodiment of the present disclosure. Fig. 12 illustrates examples of input values ​​(a) of a region of interest extraction unit and output values ​​(b) of the region of interest extraction unit.

[0184] The region of interest extraction unit (830) of the VCM encoding device (100a) according to one embodiment of the present disclosure can extract a portion of a region from a feature map according to the region of interest derived from the region of interest derivation unit (820), and can be performed in the order of FIG. 11. Each step of FIG. 11 can be selectively performed.

[0185] In step S1101, the region of interest extraction unit (830) may receive a feature map to be encoded as input and select a feature map to be extracted. When there are multiple feature maps, the region of interest extraction unit (830) may select one or more feature maps for performing region of interest extraction. The following steps S1102 and S1103 may be performed for each feature map selected as a region of interest extraction target. The number of feature maps from which regions of interest are extracted and each index may be included in picture_parameter_set and transmitted to the VCM decoding device (100b).

[0186] In step S1102, the region of interest extraction unit (830) can scale the spatial resolution scale of the region of interest to be the same as the spatial resolution scale of the feature map from which the region of interest is to be extracted. For example, when the scale of the region of interest is the scale of the feature map with the smallest spatial resolution among the feature maps to be encoded, and the current feature map to be extracted has a width and a height that are a and b times larger in spatial resolution than the smallest feature map, the width and height of the region of interest can be scaled by multiplying them by a and b, respectively. The scaling can be performed by calculating a predetermined formula, or downsampling or upsampling based on a neural network can be performed.

[0187] In step S1103, the region of interest extraction unit (830) can extract a set of elements whose spatial axis coordinates are included in the region of interest among the elements in the feature map. FIG. 12 (a) shows the input of the region of interest extraction unit (830), and FIG. 12 (b) can show the output of the region of interest extraction unit (830). For example, when four feature maps (feature maps 0 to 3) are input, feature maps 1 to 3 may not extract regions of interest, and feature map 0 may show an example in which two regions of interest are extracted.

[0188]

[0189] Referring again to FIG. 8, the coding group derivation unit (840) can group one or more feature maps to be encoded into one or more coding groups. The coding group can be an execution unit of the feature map conversion unit (850) or feature map encoding unit (860) of FIG. 8, or a transmission unit.

[0190] The coding group derivation unit (840) can group feature maps into coding groups according to the following criteria.

[0191] Criterion 1: All feature maps to be encoded can be grouped into one coding group.

[0192] Criterion 2: Full-resolution feature maps without region of interest extraction and feature maps with only region of interest extracted can be grouped into different coding groups.

[0193] Criterion 3: Full-resolution feature maps from which regions of interest are not extracted can be grouped into one coding group.

[0194] Criterion 4: Regions of interest with the same region of interest index can be grouped into one coding group.

[0195] FIG. 13 is an example of an input value (a) and output values ​​(b1, b2) of a coding group derivation unit (840) according to an embodiment of the present disclosure. For example, the input value (a) of the coding group derivation unit (840) is a feature map to be encoded. The output value (b1) of the coding group derivation unit (840) exemplifies a result grouped according to the grouping criterion 1. The output value (b2) of the coding group derivation unit (840) exemplifies a result grouped according to the grouping criteria 2, 3, and 4.

[0196] Coding group information can be included in the header (coding_group_header) of each coding group and transmitted to the VCM decoding device (100b). The coding_group_header syntax can be defined as shown in Fig. 25a. Coding group information can include a coding group index, the number of feature maps included in the coding group, the index of each feature map, and an ROI index if the feature map is ROI extracted.

[0197]

[0198] Referring back to FIG. 8, the feature map transformation unit (850) can perform transformation on feature maps within a coding group in units of coding groups. The type of transformation may be, for example, matrix multiplication-based transformation, multi-convolution layer-based transformation, etc. FIG. 15 illustrates an example of matrix multiplication-based feature map transformation according to an embodiment of the present disclosure, and FIG. 16 illustrates an example of multi-convolution layer-based feature map transformation according to an embodiment of the present disclosure. The reference attribute or type of transformation may be included in the coding_group_header and transmitted to the VCM decoding device (100b). The reference attribute of the transformation may mean whether only information within the feature map is used when performing transformation on an arbitrary feature map or whether information from another feature map is also used. The type of transformation may mean the type of algorithm that performs the transformation.

[0199] FIG. 14 is a flowchart including a matrix multiplication-based feature map transformation process according to an embodiment of the present disclosure. When the type of transformation is matrix multiplication-based transformation, the feature map transformation unit (850) may perform the process of FIG. 14, and each step of FIG. 14 may be selectively performed. FIG. 15 is an example of a matrix multiplication-based feature map transformation according to an embodiment of the present disclosure, in which (a) represents an input feature map, (b) represents a scale-transformed feature map, and (c) represents each example of a transformation unit.

[0200] In step S1401, the feature map conversion unit (850) can perform spatial resolution scaling on each feature map within the coding group. The feature map conversion unit (850) can specify the degree of spatial resolution scaling for each feature map, and the scaling parameter can be included in the coding_group_header and transmitted to the VCM decoding device (100b). Alternatively, the feature map conversion unit (850) can fixedly use a method of downscaling each feature map to the spatial resolution of the feature map with the smallest spatial resolution within the conversion group. FIG. 15 (a) represents a coding group including a plurality of feature maps (feature maps 1 to 3) as an input value of the feature map conversion unit (850). FIG. 15 (b) can represent an example in which downsampling is performed on each feature map (feature map 1, feature map 2) to the spatial resolution of feature map 3 with the smallest spatial resolution.

[0201] In step S1402, the feature map transformation unit (850) can divide each feature map into transformation units (transformation units) and list them. The transformation units can be data divided into sizes of w_tu x h_tu x c_tu that do not overlap the feature map. Each of w_tu, h_tu, and c_tu can have a value that is a divisor of the minimum width, minimum height, and minimum channel length among the scaled feature maps. FIG. 15 (c) is an example in which, when the scaled feature maps have the same width, height, and channel length, they are divided into transformation units having sizes of feature map width / 2, feature map height / 2, and 1, respectively. The w_tu, h_tu, and c_tu of the transformation units can represent the case where they are feature map width / 2, feature map height / 2, and 1, respectively. In this case, since the width and height of each feature map are reduced by half, channel length * 4 transformation units per feature map can be derived.

[0202] In step S1403, the feature map transformation unit (850) can perform matrix multiplication-based transformation on each derived transformation unit. For each transformation unit, a multiplication operation can be performed with matrix_num matrices to derive matrix_num transformation coefficients.

[0203]

[0204] Fig. 16 illustrates an example of feature map transformation based on multiple convolution layers according to an embodiment of the present disclosure. When the type of transformation is multiple convolution layer-based transformation, the feature map transformation unit (850) may perform multiple convolution layers that adjust the spatial resolution of one or more feature maps in a coding group to be the same, combine them into a channel axis, and then encapsulate them into a channel axis. Referring to Fig. 16, the feature map transformation unit (850) may receive a coding group including feature maps 1 to 3, and adjust the spatial resolution to be the same based on feature map 3 having the smallest spatial resolution. The feature map transformation unit (850) may downscale the spatial resolution of feature maps 1 and 2 to be the same as the spatial resolution of feature map 3. The feature map conversion unit (850) can combine feature maps 1 to 3 with the same spatial resolution along a channel axis, and then perform a multi-convolution layer that encapsulates the feature maps along the channel axis, thereby outputting a feature map in which feature maps 1 to 3 are combined and converted.

[0205]

[0206] Referring again to FIG. 8, the feature map encoding unit (860) of the VCM encoding device (100a) can output a bitstream by encoding a transformed feature map output from the feature map conversion unit (850) or a feature map on which transformation has not been performed (a feature map output from the region of interest derivation unit (820)) in groups.

[0207] The feature map encoding unit (860) determines an encoding method to be used for feature map encoding among various encoding methods, and a syntax indicating the determined encoding method can be included in the coding_group_header and transmitted to the VCM decoding device (100b).

[0208] The encoding method that can be used in the feature map encoding unit (860) may depend on the encoding reference attribute. The encoding reference attribute may indicate whether only information within the feature map is used when encoding an arbitrary feature map, or whether information from other feature maps is also used. The encoding reference attribute syntax may be included in the coding_group_header and transmitted to the VCM decoding device (100b), or may not be transmitted and the same value as the transform reference attribute may be used. When the encoding reference attribute information exists, the feature map encoding unit (860) may transmit an index of an encoding algorithm to be used among encoding algorithms that can be performed on the encoding reference attribute of the current frame. For example, when the encoding reference attribute is intra, the encoding method may be an entropy coding-based method. For example, when the encoding reference attribute is inter, the encoding method may be a video-based non-deep learning method, a video-based deep learning method, etc. If the encoded reference attribute information is not transmitted, the same value as the value of the converted reference attribute information may be used.

[0209] 1) When the encoding method is a video-based non-deep learning method, the encoding process may be composed of stages such as prediction, transformation, quantization, and entropy coding, and each stage may be omitted. For example, the prediction, transformation, and quantization stages may be omitted and only the entropy coding stage may be performed. The feature map encoding unit (860) may reconstruct the feature map or the transformed feature map within the coding group into a one-dimensional array, a two-dimensional frame, or a three-dimensional matrix before performing the feature map encoding, and then perform the encoding. When the feature map or the transformed feature map is reconstructed into a two-dimensional frame and video-based compression consisting of the prediction, transformation, quantization, and entropy coding stages is performed, the feature map encoding unit (860) may designate a local quantization parameter within the frame based on the region of interest received from the region of interest derivation unit (820). The feature map encoding unit (860) may designate different quantization parameters for the region of interest and the non-region of non-interest. For example, the feature map encoding unit (860) can designate a low quantization parameter for a region of interest and a high quantization parameter for a region of no interest.

[0210] 2) When the encoding method is a video-based deep learning method, the feature map encoding unit (860) can perform encoding using a neural network structure composed of multiple convolution layers.

[0211]

[0212] FIG. 17 is a block diagram including components of a VCM decoding device according to an embodiment of the present disclosure.

[0213] A VCM decoding device (100b) according to an embodiment of the present disclosure can receive a bitstream in which a feature map is compressed, restore the feature map, perform machine task analysis, and output the analysis result. The VCM decoding device (100b) can include a feature map decoding unit (1710), a feature map inverse transformation unit (1720), a feature map reconstruction unit (1730), and a machine task analysis unit (1740). Each component can be selectively performed at the sequence or frame level.

[0214] The feature map decoding unit (1710) can restore a feature map or feature map transformation coefficient in units of coding groups from the received bitstream.

[0215] The feature map inverse transformation unit (1720) can perform inverse transformation on the feature map transformation coefficients restored in units of coding groups.

[0216] The feature map reconstruction unit (1730) can restore the number and size of feature maps initially extracted by the feature map extraction and selection unit (810) of the VCM encoding device (100a) by generating a feature map or a portion of a feature map that has not been transmitted or restored based on the restored feature map.

[0217] The machine task analysis unit (1740) can perform machine tasks such as object detection, region segmentation, object tracking, action recognition, and situation interpretation using the restored feature map. The structure of the machine task analysis unit (1740) can be composed of multiple convolution layers and can include multiple convolution layers in front of the feature map extraction and selection unit (810) of the VCM encoding device (100a) that extracts the feature map. The machine task analysis unit (1740) can induce which layer among the multiple convolution layers of the machine task analysis unit (1740) will input the restored feature map to perform machine analysis based on the feature map extraction layer position (FM_extract_layer_idx) transmitted from the picture level parameter set (picture_parameter_set).

[0218] The feature map decoding unit (1710) can restore a feature map or feature map transformation coefficient from the received bitstream in units of coding groups. If a decoding reference attribute exists, the value transmitted in the coding group header (coding_group_header) can be parsed (is_intra_coded) or derived to the same value as the transformation reference attribute value. The is_intra_coded syntax can be defined as in Fig. 25b. Depending on the parsed or derived decoding reference attribute value, an index (intra_coding_method_idx, inter_coding_method_idx) indicating a decoding method to be used among the possible decoding methods can be parsed from the coding group header (coding_group_header). If the decoding reference attribute does not exist, the index (coding_method_idx) indicating the decoding method can be parsed independently.

[0219] The feature map decoding unit (1710) may be performed using a neural network-based or non-neural network-based method. If performed using a non-neural network, the decoding process may consist of steps such as prediction, transformation, quantization, and entropy coding, and each step may be omitted.

[0220] The restored or transformed feature map in the feature map decoding unit (1710) may be in a reconstructed form, and data restored to the original form of the feature map may be reversely reconstructed. The reconstructed form may be, for example, in the form of a one-dimensional array, a two-dimensional frame, or a three-dimensional matrix.

[0221] The feature map inverse transformation unit (1720) can perform inverse transformation on feature map transformation coefficients within a coding group in units of coding groups. The type of inverse transformation may be, for example, an inverse transformation based on matrix multiplication, an inverse transformation based on performing multiple convolution layers, etc. The feature map inverse transformation unit (1720) can perform an inverse transformation corresponding to an inverse transformation method index (transform_method_idx) included in a coding group header (coding_group_header). The feature map inverse transformation unit (1720) can first parse a transformation reference attribute (is_intra_transformed) in the coding group header (coding_group_header), and when it is a parsed reference attribute, can parse an inverse transformation type index (intra_transform_method_idx, inter_transform_method_idx) to be used among possible inverse transformation methods. The feature map inverse transformation unit (1720) can parse the intra_transform_method_idx syntax as an inverse transformation type index when the transformation reference attribute (is_intra_transformed) is true, and can parse the inter_transform_method_idx syntax as an inverse transformation type index when the transformation reference attribute (is_intra_transformed) is false. The inverse transformation can be performed using a predefined inverse transformation method mapped to the value of intra_transform_method_idx or inter_transform_method_idx. The feature map inverse transformation unit (1720) can only parse the inverse transformation type index (transform_method_idx) when the transformation reference attribute (is_intra_transformed) syntax does not exist.

[0222] If the inverse transformation type of the feature map inverse transformation unit (1720) is a matrix multiplication-based method, it can be performed by the process of FIG. 18.

[0223] FIG. 18 is a flowchart including a process of inverse transformation of a feature map based on matrix multiplication according to an embodiment of the present disclosure. FIG. 19 illustrates an example of inverse transformation of a feature map based on matrix multiplication according to an embodiment of the present disclosure.

[0224] In step S1801, the feature map inverse transform unit (1720) can perform inverse transformation by multiplying the feature map transformation coefficients of each inverse transform unit by an inverse transform matrix. An inverse transform matrix that matches the width and height (TU_width, TU_height) of the inverse transform unit transmitted in the coding group header (coding_group_header) can be selected and inverse transformation can be performed to derive feature values ​​of the size of the inverse transform unit. (a) of Fig. 19 can represent the result data in which inverse transformation is performed for each inverse transform unit for an arbitrary coding group.

[0225] In step S1802, the feature map inverse transformation unit (1720) can combine the inversely transformed data into the original feature map form. Figure 19 (b) can show the result of rearranging the inversely transformed data to the inverse transformation unit position in the original feature map.

[0226] In step S1803, the feature map inverse transformation unit (1720) can perform inverse scaling on the inversely transformed feature map. The inverse scaling parameter (scale_param) of each feature map can be transmitted by being included in the coding group header (coding_group_header) or can be derived in a fixed manner. The fixed manner can calculate the inverse scaling parameter based on, for example, the size of the inversely transformed feature map and the original size of the feature map included in pps. The inverse scaling can be performed based on a fixed formula or can be performed using a neural network-based method. (c) of Fig. 19 can represent an inversely scaled feature map. (a), (b), and (c) of Fig. 19 respectively correspond to (c), (b), and (a) of Fig. 15.

[0227]

[0228] FIG. 20 illustrates an example of inverse transformation of a feature map based on multiple convolutional layers according to an embodiment of the present disclosure.

[0229] The feature map inverse transformation unit (1720) can input transformed data into a neural network composed of a combination of convolution layers and output an inversely transformed feature map when the type of inverse transformation is a multi-convolution layer-based inverse transformation. The structure and input / output data of the neural network performing the inverse transformation may be as shown in Fig. 20. Each step of Fig. 20 corresponds to the reverse order of each step of Fig. 16.

[0230]

[0231] Referring back to FIG. 17, the feature map reconstruction unit (1730) can predict a feature map or a portion of a feature map that has not been transmitted and restored based on the feature map restored by the feature map decoding unit (1710) or the feature map inverse transformation unit (1720), and restore the number and size of feature maps initially extracted by the feature map extraction and selection unit (810) of the VCM encoding device (100a). Among the feature maps initially extracted by the feature map extraction and selection unit (810), the feature map restored by the VCM decoding device (100b) and the region information within the feature map can be derived by combining information such as the number of feature maps, index, width, height, and channel length transmitted at the sequence or frame group or frame level. The feature map reconstruction unit (1730) can reconstruct a feature map according to each step of FIG. 21.

[0232] FIG. 21 is a flowchart showing a feature map reconstruction process according to an embodiment of the present disclosure. The VCM encoding device (100a) can extract a region of interest for a feature map to be encoded, and since a region not included in the region of interest in a feature map from which a region of interest has been extracted is not encoded, the feature map can be restored before extracting the region of interest by predicting a part (a region not included in the region of interest) that was not transmitted from the VCM encoding device (100a) based on the restored part in the VCM decoding device (100b). An example of extracting a region of interest in the VCM encoding device (100a) is as shown in FIGS. 12a and 12b, and when a feature map is reconstructed according to FIG. 21, FIG. 22 can show that FIG. 12a is restored.

[0233] In step S2101, the feature map reconstruction unit (1730) can place the region of interest in the original position in the feature map if the region of interest is restored in any feature map, and predict the feature values ​​of the non-interest region that has not been restored, and combine the region of interest with the predicted non-interest region. Fig. 22 is an example of combining the region of interest and the non-interest region according to one embodiment of the present disclosure. Referring to Fig. 22, the feature map reconstruction unit (1730) can predict the feature values ​​of the non-restored non-interest region of feature map 0 based on the restored feature maps 1 to 3, and combine the predicted non-interest region and the restored region of interest to generate the restored feature map 0.

[0234] The feature map reconstruction unit (1730) can predict the spatial resolution of a non-interest region of a feature map in which only the region of interest is restored using one or more feature maps in which the entire spatial resolution is restored. A neural network-based method can be used as a non-interest region prediction method. The entire original resolution of the feature map can be predicted, or prediction can be performed by excluding features included in the region of interest from the entire resolution. Alternatively, the feature values ​​of the non-interest region can be predicted based on the boundary feature values ​​(boundary values) of the restored region of interest. Alternatively, the feature values ​​of the non-interest region can be designated as specific, predetermined values.

[0235] The feature map reconstruction unit (1730) can restore a full-resolution feature map by combining the restored region of interest and the predicted non-region of interest. If feature values ​​within the region of interest are also predicted in the non-region of interest prediction step of FIG. 22, the predicted values ​​can be replaced with the restored region of interest values.

[0236] In step S2102, the feature map reconstruction unit (1730) can descale the spatial resolution of the restored feature map. Whether to perform spatial resolution descale on the restored feature map of the current frame can be derived based on a syntax indicating whether to perform spatial resolution descale on a sequence, a frame group, or a frame unit. The order of transmission levels can be descending in the order of sequence->frame group->frame, and if the performance of spatial resolution descale transmitted at any transmission level means not to perform descale, the syntax indicating whether to perform spatial resolution descale on all lower levels can be omitted and spatial resolution descale on the feature map of the current frame can be omitted.

[0237] When the feature map reconstruction unit (1730) performs spatial resolution inverse scaling on the current frame, it can parse the scale parameter for each feature map or parse the scale parameter common to the feature map. The scale parameter can be transmitted in the form of an original parameter value, a log2-operated value, an index value, etc. When the scale parameter is an original value, the relationship between the original resolution and the scaled resolution can be as in mathematical equations 7 and 8, when the scale parameter is a log2-operated value, the relationship between the original resolution and the scaled resolution can be as in mathematical equations 9 and 10, and when the scale parameter is in the form of an index value of a scale table, the relationship between the original resolution and the scaled resolution can be as in mathematical equations 11 and 12.

[0238]

[0239]

[0240]

[0241]

[0242]

[0243]

[0244] The feature map reconstruction unit (1730) can perform reverse scaling using a deep learning-based or non-deep learning-based spatial resolution conversion algorithm so that the restored feature map has the original spatial resolution.

[0245] In step S2103, the feature map reconstruction unit (1730) can predict a feature map that is not selected by the feature map extraction and selection unit (810) of the VCM encoding device (100a) to generate a feature map and thus is not encoded or decoded, using the decoded feature map. The feature map can be generated using a method based on multiple convolution layers. FIG. 23 shows an example of feature map inverse transformation based on multiple convolution layers according to an embodiment of the present disclosure. For example, when there are four feature maps (feature maps 1, 2, 3, and 4) initially extracted from the VCM encoding device (100a) and only feature map 3 and feature map 2 are encoded and decoded, the feature values ​​of the unrestored feature map 1 and feature map 0 can be predicted using the restored feature map 2 and feature map 3.

[0246]

[0247] Figures 24, 25a, and 25b illustrate syntaxes according to an embodiment of the present disclosure. Figure 24 illustrates a picture_parameter_set syntax that includes information about a region of interest. Figure 25a illustrates a coding group header (coding_group_header) syntax. Figure 25b illustrates a decoding reference attribute (is_intra_coded) syntax included in a coding group header.

[0248]

[0249] The examples of the present disclosure presented in this specification and drawings are intended solely to facilitate the technical content of the present disclosure and to aid understanding thereof, and are not intended to limit the scope of the present disclosure. It will be apparent to those skilled in the art that other variations are possible in addition to the examples described above.

[0250] The claims set forth in this specification may be combined in various ways. For example, the technical features of the method claims of this specification may be combined and implemented as a device, and the technical features of the device claims of this specification may be combined and implemented as a method. Furthermore, the technical features of the method claims and the technical features of the device claims of this specification may be combined and implemented as a device, and the technical features of the method claims and the technical features of the device claims of this specification may be combined and implemented as a method.

Claims

1. In a VCM (video coding for machines) encoding device, A feature map extraction unit that extracts one or more feature maps for an input image; An area of ​​interest derivation unit for deriving one or more spatial areas of interest from the one or more feature maps; A feature map conversion unit that converts a feature map into a coding group unit including one or more feature maps; and A VCM encoding device comprising a feature map encoding unit that encodes one or more feature maps or the converted feature maps in coding group units and outputs a bitstream.

2. In paragraph 1, A VCM encoding device wherein the above feature map extraction unit changes the spatial resolution of one or more extracted feature maps using a scale parameter.

3. In accordance with paragraph 2, A VCM encoding device wherein the scale parameter is specified for each of the one or more feature maps, and is specified in units of a sequence, a group of frames, or a frame.

4. In paragraph 1, A VCM encoding device, wherein said one or more spatial regions of interest are composed of a contiguous set of unit regions and are defined by position and size information of a region of interest map or polygon.

5. In paragraph 4, Information about one or more of the above spatial regions of interest is included in a set of picture parameters and transmitted to a VCM decoding device, A VCM encoding device, wherein information about one or more spatial regions of interest includes the number of feature maps from which regions of interest are extracted and their respective indices.

6. In paragraph 1, Including further coding group derivation, The above coding group derivation unit groups the one or more feature maps into one or more coding groups according to a predetermined grouping criterion, A VCM encoding device, wherein the above-described predetermined grouping criteria are defined based on at least one of whether the encoding target is selected, whether the region of interest is extracted, and whether the region of interest includes the same region of interest.

7. In paragraph 1, The above feature map conversion unit is a VCM encoding device that converts a feature map based on matrix multiplication or multi-convolution layer execution.

8. In paragraph 7, A VCM encoding device in which the above feature map transformation unit scales the spatial resolution of a feature map included in a first coding group, divides the feature map into transformation units, and performs matrix multiplication-based transformation on the divided feature maps to derive transformation coefficients.

9. In paragraph 7, A VCM encoding device in which the above feature map conversion unit adjusts the spatial resolution of one or more feature maps in a coding group to be the same, combines the one or more feature maps along a channel axis, and performs a multi-convolution layer that encapsulates one combined feature map along the channel axis.

10. In paragraph 1, Information about the feature map transformation is included in the coding group header and transmitted to the VCM decoding device. A VCM encoding device, wherein the information regarding the feature map transformation includes at least one of a reference attribute indicating whether only internal information of the feature map is used when performing transformation on an arbitrary feature map or information of another feature map is referenced, and a transformation type indicating the type of algorithm performing the transformation.

11. In paragraph 1, Information about the encoding method is included in the coding group header and transmitted to the VCM decoding device. A VCM encoding device, wherein the information regarding the encoding method includes at least one of an encoding reference attribute indicating whether only internal information of a feature map is used when encoding an arbitrary feature map or information of another feature map is referenced, and a type of encoding algorithm.

12. In paragraph 1, The above feature map encoding unit performs encoding according to a video-based non-deep learning method or a video-based deep learning method, The above video-based non-deep learning method performs encoding according to the steps of prediction, transformation, quantization, and entropy coding. The above video-based deep learning method is a VCM encoding device that performs encoding using a neural network structure composed of multiple convolutional layers.

13. In paragraph 12, The above video-based non-deep learning method is a VCM encoding device that, when encoding is performed by reconstructing one or more feature maps or transformed feature maps into two-dimensional frames, specifies different quantization parameters for regions of interest and regions of non-interest.

14. In a VCM (video coding for machines) decoding device, A feature map decoding unit that receives a bitstream and restores one or more feature maps or feature map transform coefficients in units of coding groups; A feature map inverse transformation unit that performs inverse transformation by multiplying the restored feature map transformation coefficients by an inverse transformation matrix in response to restoring the above feature map transformation coefficients in units of coding groups; A feature map reconstruction unit that reconstructs the entire feature map by predicting a feature map or a portion of a feature map that was not transmitted based on the restored feature map; and A VCM decoding device comprising a machine task analysis unit that performs a pre-requested machine task based on the above-described restored entire feature map.

15. In paragraph 14, The above pre-requested machine task corresponds to one of object detection, region segmentation, object tracking, action recognition, and situation interpretation, VCM decoding device.

16. In paragraph 14, A VCM decoding device in which the above feature map decoding unit performs decoding of the bitstream using a decoding method according to a decoding reference attribute included in a coding group header for the bitstream.

17. In paragraph 14, A VCM decoding device in which the above feature map inverse transform unit performs inverse transform on the feature map or feature map transform coefficients using an inverse transform method according to a transform reference attribute included in a coding group header for the bitstream.

18. In paragraph 17, The above feature map inverse transform unit inversely transforms the feature map or feature map transformation coefficients according to a matrix multiplication-based inverse transform method or a multi-convolution layer-based inverse transform method, The feature map inverse transform unit according to the matrix multiplication-based inverse transform method performs inverse transform by multiplying the feature map transform coefficient of each inverse transform unit by an inverse transform matrix, rearranges the position of the inversely transformed feature map based on feature map information included in a coding group header for the bitstream, and inversely scales the feature map using an inverse scaling parameter included in the coding group header. A VCM decoding device, wherein the feature map inverse transformation unit according to the above multi-convolution layer-based inverse transformation method performs inverse transformation on the restored feature map using a neural network composed of a combination of convolution layers.

19. In paragraph 14, The above feature map reconstruction unit is a VCM decoding device that places a restored region of interest in an original position of a feature map based on feature map information for the bitstream transmitted at a sequence, frame group, or frame level, predicts feature values ​​for non-interest regions that have not been restored using one or more feature maps in which the full spatial resolution has been restored, and restores a feature map of full resolution by combining the restored region of interest and the predicted non-interest regions.

20. A non-volatile computer-readable storage medium that records commands, The above instructions, when executed by one or more processors, cause the one or more processors to: A step of extracting one or more feature maps for an input image; A step of deriving one or more spatial regions of interest from the one or more feature maps; A step of converting a feature map into a coding group unit including one or more feature maps; and A non-transitory computer-readable storage medium that performs a step of encoding the above one or more feature maps or the above-mentioned transformed feature maps in coding group units and outputting a bitstream.

Images