Neural network-based method and apparatus for compressing multi-layer feature map

Abstract

A method and an apparatus for encoding a multi-layer feature map, and a recording medium, of the present disclosure may comprise the steps of: extracting, from an input image, a multi-layer feature map including a plurality of feature maps in a hierarchical form; outputting a single-layered fusion latent representation by sequentially encoding the multi-layer feature map through consecutive encoding blocks; and encoding the single-layered fusion latent representation into a bit stream.

Description

Neural network-based multi-layer feature map compression method and device

[0001] The present disclosure relates to a technology for reducing the compression bit amount of a feature map while minimizing performance degradation of a machine task performed using a decoded feature map by providing a method for encoding a feature map extracted through an artificial neural network.

[0002] As machine learning becomes more widely used across a variety of devices, including mobile devices as well as large servers, the means for extracting feature maps and the means for performing tasks are increasingly located on different devices rather than within the same device.

[0003] When the feature map extraction means and the task execution means are separated, the extracted feature map must be transferred to the task execution means, but since the amount of data in the feature map is very large, a feature map encoding method is needed to drastically reduce the amount of data in the feature map while minimizing the degradation of task execution performance.

[0004] The conventional technology performs encoding by independently encoding multi-layer feature maps or by creating a single fused feature map. In the former case, there is a problem of increased complexity due to encoding multiple times in parallel, and in the latter case, there is a problem of increased complexity due to the fusion process and latent expression extraction process being performed serially, and there is a drawback of not being able to fully utilize the sequential properties between layers.

[0005] The multilayer feature map encoding method, device, and recording medium of the present disclosure may include a step of extracting a multilayer feature map including a plurality of feature maps in a layered form from an input image, a step of sequentially encoding the multilayer feature map through consecutive encoding blocks to output a fused latent representation of a single layer, and a step of encoding the fused latent representation of the single layer into a bitstream.

[0006] In the multi-layer feature map encoding method, device, and recording medium of the present disclosure, the consecutive encoding blocks may include a first encoding block having as input a feature map having the highest resolution of the multi-layer feature map, a second encoding block having as input a first intermediate latent representation which is an output of the first encoding block, a third encoding block having as input a first fused latent representation which is an output of the second encoding block, and a fourth encoding block having as input a second fused latent representation which is an output of the third encoding block and outputting a fused latent representation of the single layer.

[0007] In the multi-layer feature map encoding method, device, and recording medium of the present disclosure, the inputs of the second encoding block, the third encoding block, and the fourth encoding block may include feature maps of the multi-layer feature map having the same resolution as the input intermediate latent representation or fused latent representation.

[0008] In the multi-layer feature map encoding method, device, and recording medium of the present disclosure, the first encoding block, the second encoding block, and the third encoding block may include a convolutional neural network layer using a kernel of size nxn and a nonlinear layer of LeakyReLU with the same structure.

[0009] In the multi-layer feature map encoding method, device and recording medium of the present disclosure, the fourth encoding block may not include the nonlinear layer of the LeakyReLU.

[0010] In the multilayer feature map encoding method, device, and recording medium of the present disclosure, the resolution of the output of the first encoding block, the second encoding block, the third encoding block, and the fourth encoding block may be lower than the resolution of the input.

[0011] In the multi-layer feature map encoding method, device, and recording medium of the present disclosure, the consecutive encoding blocks may include a first encoding block having as input a feature map having the highest resolution of the multi-layer feature map, a second encoding block having as input a first intermediate latent representation which is an output of the first encoding block, a third encoding block having as input a feature map of the multi-layer feature map having a lower resolution than the first fused latent representation which is an output of the second encoding block, and a fourth encoding block having as input the second intermediate latent representation which is an output of the third encoding block and the first fused latent representation and outputting the fused latent representation of the single layer.

[0012] In the multi-layer feature map encoding method, device, and recording medium of the present disclosure, the input of the second encoding block and the fourth encoding block may include a feature map of the multi-layer feature map having the same resolution as the input intermediate latent representation or fused latent representation.

[0013] In the multi-layer feature map encoding method, device, and recording medium of the present disclosure, the first encoding block, the second encoding block, and the third encoding block may include a convolutional neural network layer using a kernel of size nxn and a nonlinear layer of LeakyReLU with the same structure.

[0014] In the multi-layer feature map encoding method, device and recording medium of the present disclosure, the fourth encoding block may not include the nonlinear layer of the LeakyReLU.

[0015] In the multilayer feature map encoding method, device, and recording medium of the present disclosure, the resolution of the output of the first encoding block, the second encoding block, and the fourth encoding block may be lower than the resolution of the input, and the resolution of the output of the third encoding block may be higher than the resolution of the input.

[0016] The multi-layer feature map decoding method, device, and recording medium of the present disclosure may include the steps of decoding a single-layer fused latent representation for a multi-layer feature map including a plurality of feature maps in a hierarchical form from a bitstream; and the steps of sequentially decoding the multi-layer feature map through consecutive decoding blocks from the single-layer fused latent representation.

[0017] In the multi-layer feature map decoding method, device, and recording medium of the present disclosure, the consecutive decoding blocks may include a first decoding block having as input the fused latent representation of the single layer, a second decoding block having as input the first feature map of the multi-layer feature map having the lowest resolution, which is an output of the first decoding block, a third decoding block having as input the second feature map, which is an output of the second decoding block having a higher resolution than the first feature map, and a fourth decoding block having as input the third feature map, which is an output of the third decoding block having a higher resolution than the second feature map, and outputting the fourth feature map of the multi-layer feature map having the highest resolution.

[0018] In the multi-layer feature map decoding method, device, and recording medium of the present disclosure, the first decoding block, the second decoding block, and the third decoding block may include a convolutional neural network layer using a kernel of size nxn and a nonlinear layer of LeakyReLU with the same structure.

[0019] In the multi-layer feature map decoding method, device and recording medium of the present disclosure, the fourth decoding block may not include the nonlinear layer of the LeakyReLU.

[0020] In the multi-layer feature map decoding method, device, and recording medium of the present disclosure, the resolution of the output of the first decoding block, the second decoding block, the third decoding block, and the fourth decoding block may be higher than the resolution of the input.

[0021] In the multi-layer feature map decoding method, device, and recording medium of the present disclosure, the consecutive decoding blocks may include a first decoding block that takes as input the fused latent representation of the single layer, a second decoding block that takes as input the first feature map which is the output of the first decoding block and outputs the second feature map of the multi-layer feature map having the lowest resolution, a third decoding block that takes as input the first feature map and outputs the third feature map having a higher resolution than the first feature map, and a fourth decoding block that takes as input the third feature map and outputs the fourth feature map of the multi-layer feature map having the highest resolution.

[0022] In the multi-layer feature map decoding method, device, and recording medium of the present disclosure, the second decoding block, the third decoding block, and the fourth decoding block may include a convolutional neural network layer using a kernel of size nxn and a nonlinear layer of LeakyReLU with the same structure.

[0023] In the multi-layer feature map decoding method, device, and recording medium of the present disclosure, the resolution of the output of the first decoding block, the third decoding block, and the fourth decoding block may be higher than the resolution of the input, and the resolution of the output of the second decoding block may be lower than the resolution of the input.

[0024] The present disclosure can achieve high encoding performance with a simple neural network structure while effectively reducing redundancy between layers by sequentially fusing feature maps of each layer of a multi-layer feature map.

[0025] The present disclosure can achieve significantly better encoding efficiency compared to feature map compression using existing image compression codecs, and has the advantage of being a feature map compression technology composed only of a neural network structure without the need to use an artificial neural network and an image compression codec simultaneously.

[0026] Additionally, high encoding efficiency can be achieved even when used with video compression codecs.

[0027] Figure 1 shows an example of the results of machine work for object detection and classification using Fast R-CNN, one type of artificial neural network.

[0028] Figure 2 illustrates an example in which a multi-layer feature map is extracted and machine work is performed.

[0029] Figure 3 illustrates the structure of Mask R-CNN, an artificial neural network model frequently used for object region segmentation.

[0030] Figure 4 shows a multi-layer feature map P extracted through FPN of Mask R-CNN. K , which shows an example.

[0031] Figure 5 illustrates an embodiment of a multi-layer feature map.

[0032] Figure 6 illustrates an example in which a feature map extraction means exists on a mobile device, but a specific task execution means, such as object segmentation, disparity map estimation, or image restoration, exists on a cloud server.

[0033] Figure 7 illustrates the process of encoding a neural network-based image.

[0034] Figure 8 illustrates an example of the encoding neural network of Figure 7.

[0035] Figure 9 illustrates an example of the decryption neural network of Figure 7.

[0036] Figure 10 illustrates a process of performing neural network-based multi-layer feature map compression (or decoding / decoding).

[0037] Figure 11 illustrates a multilayer feature map sequential fusion encoding neural network, which is an example of the encoding neural network of Figure 10.

[0038] Figure 12 illustrates a multilayer feature map sequential fusion decoding neural network, which is an example of the decoding neural network of Figure 10.

[0039] Figure 13 illustrates a multi-layer feature map encoding method.

[0040] Figure 14 illustrates a multi-layer feature map decoding method.

[0041] FIG. 15 illustrates a first embodiment at a high level of a multi-layer feature map encoding and decoding system using components of the present disclosure.

[0042] FIG. 16 illustrates a second embodiment at a high level of a multi-layer feature map encoding and decoding system using components of the present disclosure.

[0043] Figure 17 illustrates a detailed implementation of the neural network block used in this embodiment.

[0044] Figure 18 illustrates a first embodiment of a multilayer feature map sequential fusion encoding neural network, which is a multilayer feature map sequential fusion encoding means.

[0045] Figure 19 illustrates a second embodiment of a multilayer feature map sequential fusion encoding neural network, which is a multilayer feature map sequential fusion encoding means.

[0046] Figure 20 illustrates a third embodiment of a multilayer feature map sequential fusion encoding neural network, which is a multilayer feature map sequential fusion encoding means.

[0047] Figure 21 illustrates a fourth embodiment of a multilayer feature map sequential fusion encoding neural network, which is a multilayer feature map sequential fusion encoding means.

[0048] Figure 22 illustrates a first embodiment of a multilayer feature map sequential fusion decoding neural network, which is a multilayer feature map sequential fusion decoding means.

[0049] Figure 23 illustrates a second embodiment of a multi-layer feature map sequential fusion decoding neural network, which is a multi-layer feature map sequential fusion decoding means.

[0050] Figure 24 illustrates a third embodiment of a multi-layer feature map sequential fusion decoding neural network, which is a multi-layer feature map sequential fusion decoding means.

[0051] FIG. 25 illustrates an example of multilayer feature map general encoding that can be used with the multilayer sequential fusion decoding of the present disclosure.

[0052] Figure 26 illustrates a fourth embodiment of a multi-layer feature map sequential fusion decoding neural network, which is a multi-layer feature map sequential fusion decoding means.

[0053] Figure 27 illustrates a fifth embodiment of a multi-layer feature map sequential fusion decoding neural network, which is a multi-layer feature map sequential fusion decoding means.

[0054] Figure 28 illustrates an example of neural network-based fused latent representation entropy encoding.

[0055] Figure 29 illustrates an example of sequentially estimating a probability distribution by unequally dividing the latent representation for an image in the channel direction.

[0056] Figure 30 illustrates an example of performing probability distribution estimation 10 times sequentially after dividing the latent expression into 10 groups.

[0057] FIG. 31 illustrates an example of performing feature map compression (or sub-decoding) based on a multi-layer feature map sequential fusion sub-decoder / decoder of the present disclosure and a sub-decoder / decoder of an existing video compression codec, and an example of using an artificial neural network for feature map compression (or sub-decoding).

[0058] Figure 32 illustrates details of the NN video encoder and NN video decoder of Figure 31.

[0059] Figure 33 illustrates the process of performing fused latent representation encoding and decoding using the multi-layer feature map sequential fusion unit / decoder of the present disclosure and an existing image compression codec.

[0060] Figure 34 illustrates the configuration of a fused latent representation feature map.

[0061] Figure 35 illustrates frame packing through spatial, temporal, and spatiotemporal arrangement for inputting feature maps into an existing video compression codec.

[0062] Figure 36 illustrates an example of a block diagram for applying a multi-layer feature map decoder / decoder using a multi-layer feature map that has been decoded temporally previously.

[0063] FIG. 37 illustrates an example of a neural network that applies a multilayer feature map sequential fusion encoding means to an image by utilizing the multilayer temporal context of FIG. 18, which is an example of a multilayer feature map sequential fusion encoding neural network.

[0064] Figure 38 illustrates an example of a neural network applied to a neural network-based fused latent representation entropy decoder / decoder utilizing multi-layer temporal context.

[0065] Figure 39 illustrates an example of a multilayer temporal context sequential fusion neural network, which is a multilayer temporal context sequential fusion means.

[0066] Figure 40 illustrates an example of a neural network that applies a multi-layer feature map sequential fusion decoding method to a video by utilizing a multi-layer temporal context.

[0067] Figure 41 illustrates an example of a neural network that applies a multi-layer feature map sequential fusion decoding method to an image by utilizing a multi-layer temporal context.

[0068] Figure 42 illustrates an example of a neural network that extracts the temporal context of each previously decoded layer feature map without using motion information.

[0069] Figure 43 illustrates an example of a neural network that extracts the temporal context of each previously decoded layer feature map using motion information.

[0070] Figure 44 illustrates a detailed implementation of the neural network block and function block diagram of the present disclosure.

[0071] Figure 45 illustrates an example of a motion information extraction neural network, which is a means for extracting motion information.

[0072] Figure 46 illustrates an example of a motion vector encoding neural network, which is a motion vector encoding means.

[0073] Figure 47 illustrates an example of a motion vector decoding neural network, which is a motion vector decoding means.

[0074] Figure 48 illustrates an example of a motion compensation neural network, which is a motion compensation means.

[0075] Figure 49 illustrates an example of a neural network that extracts the temporal context of each previously decoded measurement feature map using a deformable convolutional neural network layer.

[0076] Figure 50 illustrates an example of a deformable convolutional neural network.

[0077] Figure 51 illustrates an example of an offset extraction means neural network for extracting offsets used in a deformable convolutional neural network.

[0078] Figure 52 illustrates an example of an offset encoding neural network, which is an offset encoding means.

[0079] Figure 53 illustrates an example of an offset decoding neural network, which is an offset decoding means.

[0080] Figure 54 illustrates an example of a motion compensation neural network, which is a motion compensation means using a deformable convolutional neural network layer.

[0081] The present disclosure is susceptible to various modifications and embodiments. Therefore, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present disclosure to specific embodiments, but rather to include all modifications, equivalents, and substitutes falling within the spirit and scope of the present disclosure. In the drawings, similar reference numerals designate the same or similar functions throughout. The shapes and sizes of elements in the drawings may be exaggerated for clarity. The detailed description of the exemplary embodiments described below refers to the accompanying drawings, which illustrate specific embodiments by way of example. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that the various embodiments, while different from each other, are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present disclosure. Furthermore, it should be understood that the positions or arrangements of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiment. Accordingly, the detailed description set forth below is not intended to be taken in a limiting sense, and the scope of the exemplary embodiments, if properly described, is defined only by the appended claims, along with the full scope equivalents to which such claims are entitled.

[0082] While terms such as "first" and "second" may be used herein to describe various components, these components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present disclosure, a first component may be referred to as a "second component," and similarly, a second component may also be referred to as a "first component." The term "and / or" includes a combination of multiple related items described herein or any of multiple related items described herein.

[0083] When a component of the present disclosure is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components present in between. Conversely, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components present in between.

[0084] The components shown in the embodiments of the present disclosure are independently depicted to represent different characteristic functions, and do not imply that each component is composed of separate hardware or a single software component. That is, each component is listed and included as a separate component for convenience of explanation, and at least two components among each component may be combined to form a single component, or a single component may be divided into multiple components to perform a function, and such integrated and separate embodiments of each component are also included in the scope of the present disclosure as long as they do not deviate from the essence of the present disclosure.

[0085] The terminology used in this disclosure is only used to describe specific embodiments and is not intended to limit the present disclosure. The singular expression includes plural expressions unless the context clearly indicates otherwise. In this disclosure, it should be understood that terms such as "comprise" or "have" are intended to specify the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof. In other words, the description of a specific configuration in this disclosure as "comprising" does not exclude configurations other than the specified configuration, and means that additional configurations may be included in the scope of the implementation or technical idea of ​​the present disclosure.

[0086] Some components of the present disclosure may not be essential components that perform essential functions of the present disclosure, but may be optional components merely for performance enhancement. The present disclosure may be implemented by including only components essential to implementing the essence of the present disclosure, excluding components used solely for performance enhancement. A structure that includes only essential components, excluding optional components used solely for performance enhancement, is also within the scope of the present disclosure.

[0087] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In describing the embodiments of this specification, if a detailed description of a related known configuration or function is judged to obscure the gist of this specification, the detailed description will be omitted. The same reference numerals will be used for identical components in the drawings, and duplicate descriptions of the same components will be omitted.

[0088] There are increasing cases of utilizing artificial neural networks (ANNs) in various machine vision tasks such as object classification, object recognition, object detection, object segmentation, and object tracking, as well as various image processing tasks such as super-resolution and frame interpolation.

[0089] Figure 1 shows an example of the results of machine work for object detection and classification using Fast R-CNN, one type of artificial neural network.

[0090] Referring to Figure 1, the recognized objects in Figure 1 can be classified into automobiles, ships, people, traffic signals, etc., and each region can be configured as a rectangle, and probability information (or accuracy) regarding whether the region matches the classified object can be included. Here, the rectangle is an example of the region shape, and the region shape can include circles, polygons, etc. In addition, through region segmentation within the region, the range of the region and the object can be matched as much as possible.

[0091] An artificial neural network model that performs a machine task can typically be comprised of a feature map extraction means that extracts features from input data or an input image, and a task execution means that actually performs a specific machine task based on the extracted features. In this case, when the input is in the form of an image, the "extracted features" may typically be referred to as a feature map. While the present disclosure describes the invention using the term "feature map," the present disclosure can equally be applied to features that are not in the form of a map.

[0092] Figure 2 illustrates an example in which a multi-layer feature map is extracted and machine work is performed.

[0093] The present disclosure may be applicable to a pyramid-structured feature map or a multi-layer feature map. A multi-layer feature map may have a pyramid structure in which feature maps with different resolutions are formed in multiple layers. For example, as a feature map belongs to a higher layer (or as the layer index increases), the resolution of the feature map may decrease, and as a feature map belongs to a lower layer (or as the layer index decreases), the resolution of the feature map may increase. In another example, as a feature map belongs to a higher layer (or as the layer index increases), the resolution of the feature map may increase, and as a feature map belongs to a lower layer (or as the layer index decreases), the resolution of the feature map may decrease. Furthermore, as illustrated in FIG. 34, which will be described later, feature maps within the same layer may have the same resolution.

[0094] Layer information (e.g., layer index, layer resolution, etc.) of each layer in a multi-layer feature map can be included in the feature map information and transmitted when performing machine tasks.

[0095] The present disclosure may relate to a compression (or encoding / decoding) method, device, and recording medium for such a multi-layer feature map.

[0096] Figure 3 illustrates the structure of Mask R-CNN, an artificial neural network model frequently used for object region segmentation.

[0097] In the Mask R-CNN structure of Fig. 3, a feature pyramid network (FPN) can be used as a means of extracting a multi-layer feature map, and a region proposal network (RPN) and region of interest heads (ROI heads) can be used as a means of performing machine tasks.

[0098] A feature pyramid network (FPN) is an example of extracting a multi-layer feature map, and a C-layer feature map and a P-layer feature map can be extracted by the FPN. Here, both the C-layer feature map and the P-layer feature map can be multi-layer feature maps of the present disclosure.

[0099] Hereinafter, in the present disclosure, the invention will be described using the P-layer feature map of FIG. 3. However, this is merely for convenience of explanation, and the description of the present disclosure can be equally applied to the C-layer feature map of FIG. 3 or other types of multi-layer feature maps.

[0100] When the input is an image, the shape of the feature map can be expressed as a two-dimensional array in width x height, and since the feature map of one layer is usually composed of multiple channels as shown in Figure 34, which will be explained later, the feature map of each layer can be expressed as a three-dimensional array with a size equal to width_size x height_size x number_of_channels.

[0101] That is, the feature map of layer k is F k If you do F k is a three-dimensional array F consisting of extracted feature values. k It can be expressed as [x][y][c], where x and y represent the horizontal and vertical positions of the feature values, respectively, and c can represent the channel index. For example, a multi-layer feature map C extracted from FPN k or multilayer feature map P k Multilayer feature map F of the disclosure k It can be said that the three-dimensional array F k It can be expressed as [x][y][c].

[0102] Figure 4 shows a multi-layer feature map P extracted through FPN of Mask R-CNN. k , which shows an example.

[0103] Referring to Fig. 4, the P-layer feature map P is a multi-layer feature map extracted through the FPN of the Mask R-CNN of Fig. 3. kHere we can see an example of the first channel, where the layer index k can be 2, 3, 4, 5, or 6.

[0104] In other words, in FPN, the feature map of each layer can be composed of 256 channels, and Fig. 4 may only show the feature map corresponding to the first channel among the feature maps of each layer as an example. For reference, in FPN, as the layer becomes deeper (as the layer index increases), the width and height of the feature map can become smaller and smaller than the size (resolution) of the input image.

[0105] Figure 5 illustrates an embodiment of a multi-layer feature map.

[0106] Referring to Fig. 5, even in a model for machine tasks such as YOLO v3, a multi-layer feature map (Output 1, Output 2, Output 3) consisting of three layers can be extracted in a similar manner to the P-layer feature map as in Fig. 5 and used to perform machine tasks.

[0107] Specifically, the above multi-layer feature map may be a case where a pyramid structure similar to FPN is used together with darknet 53 as a means of extracting multi-layer feature maps within YOLO v3.

[0108] As machine learning becomes more widely used across a variety of devices, including mobile devices as well as large servers, the means for extracting feature maps and the means for performing tasks are increasingly located on different devices rather than within the same device.

[0109] Figure 6 illustrates an example in which a feature map extraction means exists on a mobile device, but a specific task execution means, such as object segmentation, disparity map estimation, or image restoration, exists on a cloud server.

[0110] A feature map extracted from a mobile device is transmitted to a server, and a task is performed on the server based on the transmitted feature map, and the result of the task can be transmitted back to the mobile device.

[0111] In cases where the feature map extraction means and the task execution means are separated, as in this example, the extracted feature map must be transferred to the task execution means, and a feature map encoding method may be required to minimize the amount of data of the feature map to be transferred or stored while minimizing the degradation of task execution performance.

[0112] As another example, even if the feature map extraction means and the task execution means exist in one device, there may be a case where the extracted feature map is stored through a storage device and then used by the task execution means later. In this case as well, the feature map compression (or encoding / decoding) method as described above may be required.

[0113] In the present disclosure, the term “image” may refer to various types of images, such as natural images acquired through a camera, computer graphics, holographic images, feature map images extracted through a neural network, and ultrasound images.

[0114] Figure 7 illustrates the process of neural network-based image compression.

[0115] The core components of a neural network-based image compression (or decoding / decoding) method may include an encoding neural network, a decoding neural network, quantization, a latent representation probability model, and entropy encoding and decoding, and the process of performing image compression (or decoding / decoding) is described as follows 1) to 6) together with FIG. 7.

[0116] 1) In the encoder, the image to be encoded (i.e., the input image) x can be converted into a latent representation y through an encoding neural network.

[0117] Latent representation can mean one of a latent vector, a latent representation, and a latent feature map.

[0118] 2) Each component y of latent expression yi can be quantized.

[0119] 3) The quantized latent representation ^y is a learnable latent representation probability model P (^y) It can be transmitted to the decoder in the form of a bitstream through entropy encoding based on (^y).

[0120] 4) The bitstream transmitted from the encoder can be restored to ^y through entropy decoding based on the same latent representation probability model as in the encoder.

[0121] 5) The decoding neural network can restore an image by inputting ^y. In this disclosure, the restored image can be expressed as ^x.

[0122] 6) The neural network parameters of the core components can be learned through the loss function and backpropagation algorithm.

[0123] In Fig. 7, a block with a dotted border may represent a learnable parameter or a neural network.

[0124] In the present disclosure, the term 'encoding block' refers to a neural network block commonly used in an encoder of a neural network-based image compression (or encoding / decoding) method, and may have the following characteristics.

[0125] The encoding block can take as input an input image or an output feature map of another encoding block and output a feature map calculated by learned weights.

[0126] The encoding block may include a neural network structure such as a convolution layer, a transposed convolution layer, a fully connected layer, a vision transformer, etc.

[0127] Additionally, the encoding block may include nonlinear functions such as ReLU (Rectified Linear Unit), Tanh (Tangent hyperbolic), and Sigmoid function.

[0128] Additionally, the encoding block may include element-wise arithmetic operations such as residual (skip) connection.

[0129] The encoding block can perform dimensionality reduction or dimensionality expansion on the input by adjusting the stride of the neural network structure (convolution layer, fully connected layer) used or the number of nodes in the hidden layer, and in particular, can increase or decrease the spatial resolution (width, height) of the input.

[0130] In the following description, the encoding block (s↑) may mean an encoding block that increases the horizontal and vertical sizes of the input by s times, and the encoding block (s↓) may mean an encoding block that reduces the horizontal and vertical sizes of the input by s times.

[0131] In the present disclosure, a decoding block means a neural network block commonly used in a decoder of a neural network-based image compression method, and may have the following characteristics.

[0132] The decoding block can output a feature map calculated by learned weights using a hidden vector for the input image or an output feature map of another decoding block as input.

[0133] The decryption block may include neural network structures such as a convolution layer, a transposed convolution layer, a fully connected layer, and a vision transformer.

[0134] Additionally, the decryption block may include non-linear functions such as ReLU (Rectified Linear Unit), Tanh (Tangent hyperbolic), and Sigmoid function.

[0135] Additionally, the decryption block may include element-wise arithmetic operations such as residual (skip) connection.

[0136] By adjusting the stride of the neural network structure (Convolution layer, Fully connected layer) used by the decryption block or the number of nodes in the hidden layer, dimensionality reduction or dimension expansion can be performed on the input, and in particular, the spatial resolution (width, height) of the input can be increased or decreased.

[0137] In the following explanation, a decryption block (s↑) may mean a decryption block that doubles the horizontal and vertical sizes of the input, and a decryption block (s↓) may mean a decryption block that doubles the horizontal and vertical sizes of the input.

[0138] (1) Encoding neural network and decoding neural network

[0139] In a neural network-based image compression method, an input image x is converted into a latent representation y through a series of encoding blocks, and the composition of these encoding blocks can be referred to as an encoding neural network.

[0140] In a neural network-based image compression method, a (quantized) latent representation ^y for an image is converted into a restored image ^x through a series of decoding blocks, and the composition of these decoding blocks can be referred to as a decoding neural network.

[0141] Figure 8 illustrates an example of the encoding neural network of Figure 7.

[0142] An encoding neural network can take as input an image x to be encoded and output a latent representation y of the input image through N consecutive encoding blocks. In this regard, Fig. 8 can represent an embodiment in the case where N = 4. Here, N is a natural number greater than or equal to 1 and may include 2, 3, 4, 5, 6, 7, 8, etc. In addition, N may be a predefined value or a value determined by information signaled from a bitstream.

[0143] Figure 9 illustrates an example of the decryption neural network of Figure 7.

[0144] The decoding neural network can take a quantized latent representation ^y as input and output a restored image ^x through N consecutive decoding blocks. In this regard, Fig. 9 can show an embodiment when N=4. Here, N is a natural number greater than or equal to 1 and may include 2, 3, 4, 5, 6, 7, 8, etc. In addition, N may be a predefined value or a value determined by information signaled from a bitstream.

[0145] For example, the latent representation of an input image may be a feature map format with the same number of channels as or greater than the input image and a smaller spatial resolution (height and width). Alternatively, the latent representation of an input image may be a feature map format with the same number of channels as or greater than the input image and a larger spatial resolution (height and width).

[0146] In the present disclosure, the intermediate latent representation means the output of the encoding block used in the encoding process of the neural network-based image encoding method, as shown in FIG. 8. For example, in an encoding neural network using four encoding blocks, the output of the first encoding block that takes the input image x as input can be called the first intermediate latent representation l1, and the previous intermediate latent representation l k-1The output of the kth encoding block with input (k=2, 3, 4) is the kth intermediate latent representation l k (k=2, 3, 4). In this case, l4 can be equal to the latent representation y. (l4=y).

[0147] (2) Latent representation quantization

[0148] Quantization is the process of mapping values ​​within a specific range to specific values, and can be used to limit the number of values ​​that can be expressed.

[0149] Quantization can be broadly divided into uniform quantization and non-uniform quantization. Uniform quantization refers to the case where the range to which each value is mapped is the same, while non-uniform quantization refers to the case where the range to which each value is mapped is different.

[0150] In neural network-based image compression methods, the values ​​of latent representation components are quantized, and since the encoding neural network guarantees nonlinear transformation, uniform quantization can be mainly used.

[0151] In particular, quantization in neural network-based image compression can be replaced with a process of adding uniform noise between -0.5 and 0.5 (Equation 1) instead of quantization at training time so that the backpropagation algorithm can be used, and a rounding function (Equation 2) can be used at inference time.

[0152] Formula 1:

[0153] Formula 2:

[0154] Here, ~y is the latent representation with uniform noise added, ^y is the quantized latent representation, u(a,b) is the uniform distribution between a and b, and Round(·) can be a typical rounding operation.

[0155] (3) Latent expression probability model P (^y) (^y)

[0156] In the neural network-based image compression method, the probability distribution of ^y is estimated to entropy encode the quantized latent representation ^y, and the neural network-based probability model for estimating this probability distribution is called the latent representation probability model P. (^y) It is called (^y).

[0157] Probabilistic model of latent expression P (^y) (^y) is all components of ^y ^y i Assuming that are independent random variables, it can be calculated as in Equation 3.

[0158] At this time, P (^y) (^y) is P (y) Since it can be calculated by convolution between (y) and the uniform distribution u(-1 / 2,1 / 2), the probability model of the latent expression can be expressed as in Equation 4.

[0159]

[0160]

[0161] Here, * can mean convolution, and u(a,b) can mean a uniform distribution with interval (a, b).

[0162] Component-wise probability model P of unquantized latent representation (yi) (y i ) can be predicted through a learnable neural network, and in this disclosure, this process can be called latent expression probability distribution estimation.

[0163] When performing latent expression probability distribution estimation, P yi Gaussian distribution Assuming that the parameters of the distribution can be predicted through a learnable neural network (which is referred to as a latent expression probability distribution estimation means in the core configuration of the invention). At this time, higher prediction performance can be achieved by using y or ^y as an additional input to the neural network.

[0164] (4) Entropy encoding and decoding of latent representation

[0165] Entropy coding is a coding method that uses the probability distribution of a random variable when the probability distribution is known, and can use arithmetic coding or range coding.

[0166] Neural network-based image compression methods can perform entropy encoding and entropy decoding on the quantized latent representation ^y in the encoder and decoder, respectively.

[0167] At this time, entropy encoding and decoding are performed using the estimated latent representation probability distribution P (^y) It can be performed based on (^y).

[0168] Neural network-based image compression is performed through neural networks composed of learnable parameters. Learning of these neural networks can be performed through a backpropagation algorithm that updates the parameters and weights of the neural network in a direction that minimizes a specific loss function calculated from training data.

[0169] In the case of an image compression model, learning can be performed to simultaneously minimize the distortion between the input image and the restored image and the bit rate of the bitstream transmitted from the encoder to the decoder.

[0170] At this time, the distortion loss function L DAs in Equation 6, the mean square error (MSE) or MS-SSIM (Multi-Scale Structural Similarity Index Measure) between the input image x and the restored image ^x can be used.

[0171] E in Equation 6 x~px (·) can mean calculating the expected value (average) for the input x of all learning data.

[0172] Bitrate loss function L R It can be calculated using Equation 7, and can be said to be the cross-entropy between the actual latent expression distribution and the latent expression distribution estimated through the latent expression probability model, and in this sense, it can be called an entropy loss function.

[0173] Final rate-distortion loss function L for learning RD can be expressed as Equation 8, and the distortion loss function L D and the entropy loss function L R A constant λ is used to determine the ratio between the two, and the desired image quality level and bit rate for the output of the trained model can be determined based on λ. (Generally, the larger λ, the higher the image quality.)

[0174] x is the input image data sample to be used for learning, p x may be the distribution of the training data.

[0175]

[0176] Since a multi-layer feature map is composed of multiple feature maps with different spatial resolutions, a different encoding method may be required than when encoding a natural image, and in the present disclosure, the input image is a multi-layer feature map {F k} k=1, …, LA neural network-based image compression method for the case where (L is the number of layers) is described, especially as a neural network-based multi-layer feature map compression method. As an example, a multi-layer feature map {F k} k=1, …, L The smaller k can mean a feature map of a layer with a larger spatial resolution. As another example, a multi-layer feature map {F k} k=1, …, L A smaller k may mean a feature map of a layer with a smaller spatial resolution.

[0177] Figure 10 illustrates a process of performing neural network-based multi-layer feature map compression (or decoding / decoding).

[0178] The core components of the neural network-based multi-layer feature map compression (or sub- / decoding) method are also an encoding neural network, a decoding neural network, quantization, a latent representation probability model, and entropy encoding and decoding, and the process of performing neural network-based multi-layer feature map compression (or sub- / decoding) is explained through Figs. 10 and 1) to 6).

[0179] 1) In the encoder, the multi-layer feature map {F to be encoded k} k=1, …, L can be converted into a latent representation y through a coding neural network.

[0180] 2) Each component y of latent expression y i can be quantized.

[0181] 3) The quantized latent representation ^y is a learnable latent representation probability model P (^y) (^y) It can be transmitted to the decoder in the form of a bitstream through entropy encoding.

[0182] 4) The bitstream transmitted from the encoder can be restored to ^y through entropy decoding based on the same latent representation probability model as in the encoder.

[0183] 5) The decoding neural network can restore a multi-layer feature map by taking ^y as input. In this disclosure, the restored multi-layer feature map is {^F k} k=1, …, L It can be expressed as follows.

[0184] 6) The parameters of the core components can be learned through the loss function and backpropagation algorithm.

[0185] In Fig. 10, blocks with dotted borders may represent learnable parameters or neural networks.

[0186] Neural network-based multi-layer feature map compression is performed through neural networks composed of learnable parameters. These neural networks are called neural network-based multi-layer feature map compression models, and the learning of these neural networks can be performed through a backpropagation algorithm that updates the parameters and weights of the neural network in a direction that minimizes a specific loss function calculated from training data.

[0187] When training a neural network-based multi-layer feature map compression model, either rate-distortion optimization or rate-performance optimization can be used.

[0188] The rate-distortion optimization method is a learning method that simultaneously minimizes the distortion between the input multi-layer feature map and the restored multi-layer feature map and the bit rate of the bitstream transmitted from the encoder to the decoder.

[0189] The rate-performance optimization method is a learning method that simultaneously minimizes the performance of machine tasks performed through restored multi-layer feature maps and the bit rate of the bitstream transmitted from the encoder to the decoder.

[0190] The distortion loss function used in the neural network-based multi-layer feature map compression model is as shown in Equation 9, where the input multi-layer feature map {F k} k=1, …, L and restored multilayer feature map {^F k} k=1, …, L The mean square error (MSE) or multi-scale structural similarity index measure (MS-SSIM) is calculated by weighting w for each layer. k It can be used by weighting the sum.

[0191] The bit rate can be approximated by the cross-entropy between the probability distribution of the latent expression estimated by the latent expression probability model and the actual latent expression, as in Equation 10.

[0192] Machine work loss function L P It refers to the performance of machine tasks performed from compressed and restored multi-layer feature maps, and can be calculated through a comparison between the correct label and the inference result of the machine task.

[0193] At this time, classification loss function, bounding box loss function, mask loss function, etc. can be used depending on the type of machine task.

[0194] Loss function L for rate-distortion optimization RD can be expressed as Equation 11, and the distortion loss function L D and cross entropy-based loss function L R A constant λ is used to determine the ratio between the two, and depending on λ, the desired restoration level and bit rate for the output of the trained model can be determined. (Generally, the larger λ, the higher the restoration level.)

[0195] Loss function L for rate-performance optimization RPcan be expressed as in Equation 12, and the performance loss function L P and cross entropy-based loss function L R A constant λ is used to determine the ratio between the two, and the desired restoration level and bit rate for the output of the trained model can be determined based on λ. (Generally, the larger λ, the higher the performance of the machine task.)

[0196]

[0197] For multi-layer feature map encoding, L multi-layer feature maps can be encoded independently or combined into a single fused feature map. However, the former method increases complexity due to L parallel encodings, while the latter method increases complexity due to the serial execution of the fusion and latent representation extraction processes, and fails to fully utilize the sequential properties between layers.

[0198] The present disclosure relates to a neural network-based multi-layer feature map compression method including a multi-layer feature map sequential fusion encoding or multi-layer feature map sequential fusion decoding process. Through the sequential fusion encoding or decoding of the present disclosure, the multi-layer feature map fusion process and the latent representation extraction process are sequentially processed as a single process, thereby reducing complexity and improving compression performance by utilizing the sequential properties of the multi-layer feature map.

[0199] Figure 11 illustrates a multilayer feature map sequential fusion encoding neural network, which is an example of the encoding neural network of Figure 10.

[0200] Referring to Fig. 11, a multilayer feature map sequential fusion encoding neural network for performing multilayer feature map sequential fusion encoding is a multilayer feature map {F k} k=1, …, L To effectively encode f, we use a fused latent representation through consecutive encoding blocks. kIt can be an encoding neural network that sequentially generates and finally outputs a single-layer fused latent representation y for the multi-layer feature map.

[0201] At this time, sequential encoding may mean that the layer index k of the fused latent representation increases or decreases by one each time each encoding block is passed.

[0202] In particular, the structure of FIG. 10 may be an example in which the layer index k of the fused latent representation or intermediate latent representation increases by one.

[0203] Fusion latent expression f k In the coding neural network for multilayer feature map encoding, the intermediate latent representation l, which is the output of the previous coding block, k-1 (or fusion latent expression f k-1 ) as well as the encoded target feature map F with the same spatial resolution. k It can mean the output of the encoding block that takes as an additional input. In the above process, only the case where the intermediate latent representation and the target feature map have the same resolution was explained, but the above process can be applied even when the resolution is different by simply adding a process to convert the resolution to the same. For example, the intermediate latent representation l k-1 and the encoded target feature map F k If the resolutions are different, change the resolution of one of the two to obtain the intermediate latent expression l k-1 and the encoded target feature map F k The above process can be performed by making the resolution the same.

[0204] At this time, l k-1 (or f k-1 ) and F k Concatenation between channels can be used to use them as inputs to a single encoding block.

[0205] Such fused latent representations may contain fused information about different layers of a multilayer feature map.

[0206] The special feature of multi-layer feature map sequential fusion encoding is that since it encodes the feature maps of all layers as a single fused latent representation, it can improve encoding performance by minimizing the redundancy between layers of multi-layer feature maps, and since it fuses the information of the feature maps of all layers via the fused latent representation despite encoding the feature maps of all layers, it can achieve high encoding performance without a large increase in complexity compared to the existing neural network-based natural image compression method.

[0207] Multilayer feature map {F) input to the encoding neural network of the present disclosure k} k=1, …, L may be the result value after performing additional preprocessing steps. In this case, the preprocessing may include a resolution-based filtering process, a resolution-transforming process, etc.

[0208] Figure 12 illustrates a multilayer feature map sequential fusion decoding neural network, which is an example of the decoding neural network of Figure 10.

[0209] A multi-layer feature map sequential fusion decoding neural network for performing multi-layer feature map sequential fusion decoding is a neural network that performs sequential fusion decoding of multi-layer feature maps by decoding multi-layer feature maps {^F through consecutive decoding blocks from a single layer of fused latent representation y for multi-layer feature maps as shown in Fig. 12 in order to effectively decode multi-layer feature maps. k} k=1, …, L It can be a decryption neural network that sequentially decrypts.

[0210] At this time, sequential decoding may mean that the layer index k of the multi-layer feature map restored increases or decreases by one each time each decoding block is passed.

[0211] In particular, the structure of FIG. 11 may be an embodiment of a case where the layer index k of the multi-layer feature map being restored is reduced by one.

[0212] A single fused latent representation for multi-layer feature maps can be quantized for entropy encoding and decoding (i.e., ^y).

[0213] The multilayer feature map sequential fusion decoding neural network takes y as input and computes the feature map ^F of the layer with the lowest resolution. L From the layer with the highest resolution, the feature map ^F1 can be sequentially restored.

[0214] At this time, the output of the kth decryption block is the target of decryption, ^F L-K+1 , and may be characterized by being the input to the next decryption block at the same time.

[0215] Decrypted feature map ^F through decryption block k is the final restored feature map {^F k} k=1, …, L This can be done, but may require additional post-processing. In this case, post-processing may include resolution-dependent filtering, resolution transformation, etc.

[0216] The special feature of multi-layer feature map sequential fusion decoding is that since the feature maps of all layers are decoded from a single fused latent representation, the redundancy between layers of the multi-layer feature maps can be minimized, thereby improving the encoding performance. In addition, despite decoding the feature maps of all layers, it can achieve high encoding performance without a large increase in complexity compared to the decoding neural network of the existing neural network-based natural image compression method due to the simple structure that uses the output of the previous decoding block as the input of the next decoding block.

[0217] Figure 13 illustrates a multi-layer feature map encoding method.

[0218] The multi-layer feature map encoding method and device of FIG. 13 may include a multi-layer feature map extraction means [E1] for extracting a multi-layer feature map from an input image, a multi-layer feature map sequential fusion encoding means [E2] for sequentially converting the extracted multi-layer feature map into a fused latent representation and finally outputting a single-layer fused latent representation, and a fused latent representation encoding means [E3] for converting the fused latent representation into a bitstream.

[0219] Here, the extracted multi-layer feature map may have been normalized using a conventional method such as Min-Max normalization or Z-score normalization [E2-1].

[0220] The fused latent representation encoding means may be one of fused latent representation encoding using an image compression codec [E3-1] and fused latent representation entropy encoding based on a neural network [E3-2].

[0221] Fused latent representation encoding using a video compression codec [E3-1] may include a fused latent representation transformation means [E3-1-1] for performing processes such as quantization, padding or resizing, and rearrangement on the fused latent representation to create a format suitable for encoding, and an encoding means [E3-1-2] of a video compression codec for encoding the transformed fused latent representation in the form of a bitstream.

[0222] The neural network-based fused latent representation entropy encoding [E3-2] may include a fused latent representation quantization means [E3-2-1] for quantizing a fused latent representation, a fused latent representation probability distribution estimation means [E3-2-2] for estimating a probability distribution of the quantized fused latent representation, and a fused latent representation entropy encoding means [E3-2-3] for entropy encoding the quantized fused latent representation using the probability distribution of the estimated fused latent representation.

[0223] As a method for applying the multi-layer feature map sequential fusion encoding means of [E2] of Fig. 13 to a video, a temporal context can be utilized through a method [E2-2] that utilizes a multi-layer feature map that was decoded temporally earlier than the multi-layer feature map currently being encoded.

[0224] The method [E2-2] of utilizing a temporally previously decoded multi-layer feature map may be one of a method [E2-2-1] of extracting a multi-layer temporal context based on a temporally previously decoded multi-layer feature map and a method [E2-2-2] of directly utilizing a temporally previously decoded multi-layer feature map.

[0225] The method for extracting multi-layer temporal context [E2-2-1] may be one of a multi-layer temporal context extraction means using motion information [E2-2-1-1], a multi-layer temporal context extraction means not using motion information [E2-2-1-2], and a multi-layer temporal context extraction means using a deformable synthetic convolutional neural network [E2-2-1-3].

[0226] The multi-layer temporal context extraction means using motion information [E2-2-1-1] may include a motion vector extraction means [E2-2-1-1-1] for extracting motion information between a previously decoded multi-layer feature map and a current encoding target multi-layer feature map, a motion vector encoding means [E2-2-1-1-2] for outputting a latent representation of the extracted motion vector, a motion latent representation encoding means [E2-2-1-1-3] for converting the motion latent representation into a bitstream, a motion latent representation decoding means [E2-2-1-1-4] for decoding the motion latent representation from the bitstream, a motion vector decoding means [E2-2-1-1-5] for decoding the motion vector from the motion latent representation, and a motion compensation means [E2-2-1-1-6] for predicting a multi-layer feature map of a current frame based on the decoded motion vector and the previously decoded multi-layer feature map.

[0227] The multi-layer temporal context extraction means [E2-2-1-3] using a deformable convolutional neural network may include an offset extraction means [E2-2-1-3-1] for extracting an offset of a deformable convolutional neural network using a previously decoded multi-layer feature map and a currently encoded multi-layer feature map, an offset coding means [E2-2-1-3-2] for outputting a latent representation of the extracted offset, an offset latent representation encoding means [E2-2-1-3-3] for converting the offset latent representation into a bitstream, an offset latent representation decoding means [E2-2-1-3-4] for decoding the offset latent representation from the bitstream, an offset decoding means [E2-2-1-3-5] for decoding an offset vector from the offset latent representation, and a deformable convolutional neural network [E2-2-1-3-6] for predicting a multi-layer feature map of a current frame based on the decoded offset and the previously decoded multi-layer.

[0228] Neural network-based fused latent representation entropy encoding [E3-2] is a method for applying a neural network-based fused latent representation entropy encoding method to a video, and can utilize temporal context through a method [E2-2] that utilizes a multi-layer feature map that was decoded temporally before the multi-layer feature map currently being encoded.

[0229] Figure 14 illustrates a multi-layer feature map decoding method.

[0230] A method and device for decoding a multi-layer feature map from an input bitstream may include a fused latent representation decoding means [D1] for decoding a latent representation from an input bitstream, and a multi-layer feature map sequential fused decoding means [D2] for sequentially decoding a multi-layer feature map using the fused latent representation as input.

[0231] The fused latent representation decoding means may be at least one of fused latent representation decoding using an image compression codec [D1-1] and fused latent representation entropy decoding based on a neural network [D1-2].

[0232] The fused latent representation decoding using a video compression codec may include a decoding means [D1-1-1] of the video compression codec for decoding a fused latent representation from an input bitstream, and a fused latent representation inverse transform means [D1-1-2] for performing an inverse process of a quantization, padding or resizing, or rearrangement process on the fused latent representation.

[0233] The neural network-based fused latent representation entropy encoding may include a fused latent representation probability distribution estimation means [D1-2-1] for estimating a probability distribution of a quantized fused latent representation, and a fused latent representation entropy decoding means [D1-2-2] for entropy-decoding a quantized fused latent representation from an input bitstream using the estimated probability distribution of the fused latent representation.

[0234] A method for applying a neural network-based fused latent representation entropy decoding method [D1-2] to a video can utilize temporal context through a method [D2-1] that utilizes a multi-layer feature map that was decoded temporally earlier than the multi-layer feature map currently being decoded.

[0235] The method [D2-1] for utilizing a previously decoded multi-layer feature map may be one of a method [D2-1-1] for extracting a multi-layer temporal context based on a previously decoded multi-layer feature map, and a method [D2-1-2] for directly utilizing a previously decoded multi-layer feature map.

[0236] The multi-layer temporal context extraction means [D2-1-1] may be one of a multi-layer temporal context extraction means using motion information [D2-1-1-1], a multi-layer temporal context extraction means not using motion information [D2-1-1-2], and a multi-layer temporal context extraction means using a deformable synthetic convolutional neural network [D2-1-1-3].

[0237] A method for performing temporal context extraction using motion information may include a motion latent representation decoding means [D2-1-1-1-1] for decoding a motion latent representation from a bitstream, a motion vector decoding means [D2-1-1-1-2] for decoding a motion vector from the motion latent representation, and a motion compensation means [D2-1-1-1-3] for predicting a multi-layer feature map of a current frame based on the motion vector and a previously decoded multi-layer feature map.

[0238] A method for performing temporal context extraction using a deformable convolutional neural network may include an offset latent representation decoding means [D2-1-1-3-1] for decoding an offset latent representation from a bitstream, an offset decoding means [D2-1-1-3-2] for decoding an offset vector from the offset latent representation, and a deformable convolutional neural network [D2-1-1-3-3] for predicting a multi-layer feature map of a current frame based on the decoded offset and a previously decoded multi-layer feature map.

[0239] FIG. 15 illustrates a first embodiment at a high level of a multi-layer feature map encoding and decoding system using components of the present disclosure.

[0240] In the first embodiment, fused latent representation encoding [E3-1] and decoding [D1-1] using an image compression codec can be used as a fused latent representation encoding means [E3] and a decoding means [D1]. The description of each detailed component will be provided in more detail in the following embodiments.

[0241] FIG. 16 illustrates a second embodiment at a high level of a multi-layer feature map encoding and decoding system using components of the present disclosure.

[0242] In the second embodiment, neural network-based fused latent representation entropy encoding [E3-2] and decoding [D1-2] can be used as the fused latent representation encoding means [E3] and the decoding means [D1]. The description of each detailed component will be provided in more detail in the following embodiments.

[0243] When performing fusion latent representation entropy encoding and decoding based on a neural network, a multi-layer feature map that has been decoded previously in time can be utilized. The method ([E2-2], [D2-1]) of utilizing a multi-layer feature map that has been decoded previously in time can be used as a method for applying a multi-layer feature map sequential fusion encoding means [E2] and decoding means [D2] and a neural network-based fusion latent representation entropy encoding means [E3-2] and decoding means [D1-2] to a video.

[0244] The encoding and decoding quality or bit rate of the neural network model is determined using the parameter λ as in Equation 11 or Equation 12 above. If λ is lower than a specific threshold, a multi-layer feature map decoded previously in time is utilized, and if λ is higher than the threshold, a switching method that does not utilize a multi-layer feature map decoded previously in time can be used. At this time, the specific threshold can be a fixed value determined in advance, or it can be adaptively determined and used depending on the type of task, the input feature map extraction means, or the input image or feature map.

[0245] Figure 17 illustrates a detailed implementation of the neural network block used in this embodiment.

[0246] The neural network block of the present disclosure can be expressed as an encoding block in an encoding neural network and as a decoding block in a decoding neural network.

[0247] In this embodiment, a multi-layer feature map {F k} k=1, …, L This particular P layer feature map {P k} k=2, …, 5An embodiment of the present disclosure will be described using an example. In addition, a detailed implementation of the neural network block used in this embodiment can be described with reference to FIG. 17.

[0248] In the case of Mask R-CNN, the P6 layer in the P layer may not be compressed. This is because it is highly correlated with the P5 layer, so there is no difference in machine task performance even if P5 downsampling is used instead of P6.

[0249] In Fig. 17, the n Х n Conv block may mean a convolutional neural network layer that uses a kernel of size n Х n.

[0250] In Fig. 17, the n Х n Subpixel block may mean a sub-pixel convolution layer using a kernel of size n Х n proposed in Wenzhe Shi et. al., "Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network"; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 1874-1883.

[0251] In Fig. 17, LeakyReLU may refer to a LeakyReLU nonlinear layer commonly used in neural networks for image processing.

[0252] In Fig. 17, GDN and IGDN refer to nonlinear layers proposed in J. Balle, et. al., "Density modeling of ´ images using a generalized normalization transformation," arXiv preprint arXiv:1511.06281, 2015., which can be commonly used in neural network-based image compression methods.

[0253] ^P in Fig. 17 H , ^P L refers to the feature maps of two adjacent layers among the layers of a multi-layer feature map, and in this case, ^P H is a feature map with relatively higher spatial resolution, ^P L may mean a feature map with relatively lower spatial resolution.

[0254] In Fig. 17, + indicates element-wise addition between two feature maps, and Concat may indicate concatenation between channels.

[0255] Figure 18 illustrates a first embodiment of a multilayer feature map sequential fusion encoding neural network, which is a multilayer feature map sequential fusion encoding means.

[0256] The first embodiment of Fig. 18 may be an example of a case where the layer index of the fused latent representation or the intermediate latent representation sequentially increases when sequentially converting a multi-layer feature map into a fused latent representation (the number of layers N = 4).

[0257] In the first embodiment of Fig. 18, the P layer feature map {P k} k=2, …, 5 By sequentially encoding multi-layer feature maps into fused latent representations using as input, the final fused latent representation y can be output.

[0258] The first embodiment of Fig. 18 may be an embodiment of step [E2] of Fig. 13.

[0259] Encoding block 1, which takes P2 as input, consists of Resblock and Resblock 2↓, and can output l1, which is an intermediate latent representation.

[0260] Encoding block 2, which takes P3 and l1 as input, is also composed of Resblock and Resblock 2↓, and can output f2, which is a fused latent representation.

[0261] Encoding block 3, which takes P4 and f2 as input, is also composed of Resblock and Resblock 2↓, and can output f3, which is a fused latent representation.

[0262] Finally, the encoding block 4, which takes P5 and f3 as input, consists of a 3Х3 convolutional neural network layer (2↓) and an attention module, and can output the final fused latent representation y (=f4).

[0263] Attention Module may refer to the attention module proposed in Z. Cheng et. al., "Learned image compression with discretized Gaussian mixture likelihoods and attention modules," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 7939-7948.

[0264] At this time, in addition to the encoding block in this embodiment, conventional neural network-based encoding blocks for image processing can be combined and used.

[0265] Figure 19 illustrates a second embodiment of a multilayer feature map sequential fusion encoding neural network, which is a multilayer feature map sequential fusion encoding means.

[0266] The second embodiment of Fig. 19 may be an example of a case where the layer index of the fused latent representation or the intermediate latent representation sequentially increases when sequentially converting a multi-layer feature map into a fused latent representation (number of layers N = 4).

[0267] Referring to FIG. 19, it can be seen that an additional embodiment of a multi-layer feature map sequential fusion encoding means having a structure similar to the previous embodiments is illustrated, and a P layer feature map {P k} k=2, …, 5By sequentially encoding multi-layer feature maps into fused latent representations using as input, the final fused latent representation y can be output.

[0268] Encoding block 1, which takes P2 as input, consists of one 5 Х 5 convolutional neural network layer (2↓) and three Resblocks, which can output the intermediate latent representation l1.

[0269] The encoding block 2, which takes P3 and l1 as input, also consists of one 5 Х 5 convolutional neural network layer (2↓, three Resblocks, and one Attention Module), and can output the fused latent representation f2.

[0270] The encoding block 3, which takes P4 and f2 as input, also consists of one 5 Х 5 convolutional neural network layer (2↓) and three Resblocks, and can output the fused latent representation f3.

[0271] Finally, the encoding block 4, which takes P5 and f3 as input, consists of a 5 Х 5 convolutional neural network layer 2↓ and an attention module, and can output the final fused latent representation y (= f4).

[0272] Figure 20 illustrates a third embodiment of a multilayer feature map sequential fusion encoding neural network, which is a multilayer feature map sequential fusion encoding means.

[0273] The third embodiment of Fig. 20 may be an example of a case where the layer index of the fused latent representation or the intermediate latent representation sequentially increases when sequentially converting a multi-layer feature map into a fused latent representation (number of layers N = 3).

[0274] Referring to Fig. 20, an additional embodiment of a multi-layer feature map sequential fusion encoding means having a structure similar to the previous embodiments can be confirmed. As an example, a case where Darknet 53 is used as a feature extractor, as in the YOLO v3 model, can be taken. That is, the encoding target feature map {F k}k=1, …, 3 This may be an example of a case where the number of layers is 3.

[0275] In this case, a multi-layer feature map sequential fusion encoding means can be configured with three encoding blocks as shown in Fig. 20.

[0276] Encoding block 1, which takes F1 as input, consists of one 5 Х 5 convolutional neural network layer (2↓), three Resblocks, and one Attention Module, and can output the intermediate latent representation l1.

[0277] Encoding block 2, which takes F2 and l1 as input, consists of one 5 Х 5 convolutional neural network layer (2↓) and three Resblocks, and can output the fused latent representation f2.

[0278] Encoding block 3, which takes F3 and f2 as input, consists of 5 Х 5 convolutional neural network layers 2↓ and an Attention Module, and can output the final fused latent representation y (= f3).

[0279] Figure 21 illustrates a fourth embodiment of a multilayer feature map sequential fusion encoding neural network, which is a multilayer feature map sequential fusion encoding means.

[0280] The third embodiment of Fig. 21 may be an example of a case where a structure in which the layer index of the fused latent representation or intermediate latent representation sequentially increases and a structure in which the layer index of the fused latent representation sequentially decreases are merged when sequentially converting a multi-layer feature map into a fused latent representation.

[0281] In the fourth embodiment of Fig. 21, it can be shown that when converting a multi-layer feature map sequentially into a fused latent representation, the spatial resolution of the final fused latent representation can be varied by varying the encoding order of the coding blocks and the spatial resolution of the intermediate latent representation and the fused latent representation.

[0282] The fourth embodiment of Fig. 21 may be an embodiment of step [E2] of Fig. 13.

[0283] Encoding block 1, which takes P2 as input, consists of Relock and Resblock 2↓, and can output l1, which is an intermediate latent representation.

[0284] Encoding block 2, which takes P3 and l1 as input, is also composed of Resblock and Resblock 2↓, and can output f2, which is a fused latent representation.

[0285] Encoding block 3, which takes P4 as input, consists of Resblock and Resblock 2↑, and can output l3, which is an intermediate latent representation.

[0286] Encoding block 4, which takes P4, f2, and l3 as inputs, consists of a 3Х3 convolutional neural network layer (2↓) and an Attention Module, and can output the final fused latent representation y (=f4).

[0287] The final fused latent representation y (=f4) in this additional embodiment may have a higher spatial resolution than in the first embodiment.

[0288] This feature has the additional advantage of optimizing encoding performance by adjusting the spatial resolution of the final fused latent representation depending on the characteristics of the multi-layer feature map to be encoded.

[0289] Figure 22 illustrates a first embodiment of a multi-layer feature map sequential fusion decoding neural network, which is a multi-layer feature map sequential fusion decoding means.

[0290] The first embodiment of FIG. 22 may be an embodiment in which the layer index of the restored multi-layer feature map sequentially decreases when sequentially decoding the multi-layer feature map.

[0291] In the first embodiment of Fig. 22, the P-layer feature map can be sequentially decoded using the fused latent representation ^y as input. As a result, the restored P-layer feature map {^P k} k=2, …, 5 can be printed.

[0292] The first embodiment of Fig. 22 may be an embodiment of step [D2] of Fig. 14.

[0293] Decoding block 1, which takes as input the quantized fused latent representation ^y, is composed of Resblock and Resblock 2↑, and can output the feature map ^P5, which has the smallest resolution among the feature maps of the P layer feature map.

[0294] Decryption block 2, which takes ^P5 as input, also consists of Resblock and Resblock 2↑, and can output the P layer feature map ^P4.

[0295] Decryption block 3, which takes ^P4 as input, also consists of Resblock and Resblock 2↑, and can output the P layer feature map ^P3.

[0296] Finally, the decryption block 4 that takes ^P3 as input consists of a 3Х3 convolutional neural network layer (2↑) and an attention module, and can output a feature map ^P2 with the largest resolution among the feature maps of the P layer feature map.

[0297] Attention Module may refer to the attention module proposed in Z. Cheng et. al., "Learned image compression with discretized Gaussian mixture likelihoods and attention modules," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 7939-7948.

[0298] At this time, in addition to the decoding block in this embodiment, conventional neural network-based decoding blocks for image processing can be combined and used.

[0299] Figure 23 illustrates a second embodiment of a multi-layer feature map sequential fusion decoding neural network, which is a multi-layer feature map sequential fusion decoding means.

[0300] The second embodiment of Fig. 23 may be an embodiment in which a structure in which the layer index of a multi-layer feature map that is restored when sequentially decoding a multi-layer feature map sequentially increases and a structure in which the layer index of the multi-layer feature map sequentially decreases are combined.

[0301] The second embodiment of Fig. 23 shows that when sequentially restoring a multi-layer feature map, encoding performance can be optimized to suit the characteristics of the multi-layer feature map to be encoded by configuring the decoding order of decoding blocks in various ways.

[0302] The second embodiment of Fig. 23 may be an embodiment of step [D2] of Fig. 14.

[0303] Decoding block 1, which takes as input the quantized fused latent representation ^y, consists of 3Х3 convolutional neural network layers (2↑) and an Attention Module, and can output a P-layer feature map ^P4.

[0304] Decryption block 2, which takes ^P4 as input, consists of Resblock and Resblock 2↓, and can output a P layer feature map ^P5.

[0305] Decryption block 3, which takes ^P4 as input, consists of Resblock and Resblock 2↑, and can output a P layer feature map ^P3.

[0306] The decryption block 4 that takes ^P3 as input also consists of Resblock and Resblock 2↑, and can output the P layer feature map ^P2.

[0307] Figure 24 illustrates a third embodiment of a multi-layer feature map sequential fusion decoding neural network, which is a multi-layer feature map sequential fusion decoding means.

[0308] The third embodiment of Fig. 24 may be an embodiment in which the layer index of the restored multi-layer feature map sequentially increases when sequentially decoding the multi-layer feature map.

[0309] The third embodiment of Fig. 24 may be an embodiment of step [D2] of Fig. 14.

[0310] The third embodiment of FIG. 24 can show that when sequentially decoding a multi-layer feature map, the inputs of decoding block 2, decoding block 3, and decoding block 4 can have additional inputs in addition to the outputs of the previous decoding blocks.

[0311] Decoding block 1, which takes as input the quantized fused latent representation ^y, is composed of Resblock, Resblock 2↑, Resblock, Resblock 2↑, Attention Module, Resblock, Resblock 2↑, Resblock, Subpixel2↑ blocks, and can output a P-layer feature map ^P2.

[0312] Decryption block 2, which takes as input ^P2, the output of decryption block 1, is composed of a Feature Mixing block and can output a P layer feature map ^P3.

[0313] Decoding block 2 can additionally input the feature map extracted from the fused latent representation ^y through a series of neural network blocks consisting of Resblock, Resblock 2↑, Resblock, Resblock 2↑, Attention Module, Resblock, and Resblock 2↑.

[0314] Decryption block 3, which takes as input ^P3, the output of decryption block 2, is composed of a Feature Mixing block and can output a P layer feature map ^P4.

[0315] Decryption block 3 can additionally input feature maps extracted from the fused latent representation ^y through a series of neural network blocks consisting of Resblock, Resblock 2↑, Resblock, and Resblock 2↑.

[0316] Decryption block 4, which takes as input ^P4, the output of decryption block 3, is composed of a Feature Mixing block and can output a P layer feature map ^P5.

[0317] Decryption block 4 can additionally input feature maps extracted from the fused latent representation ^y through a series of neural network blocks consisting of Resblock and Resblock 2↑.

[0318] FIG. 25 illustrates an example of multilayer feature map general encoding that can be used with the multilayer sequential fusion decoding of the present disclosure.

[0319] The multi-layer feature map sequential fusion encoding and decoding of the present disclosure may not necessarily be used together. For example, the sequential fusion decoding of the present disclosure may be used together with the multi-layer feature map general encoding as illustrated in FIG. 25. In another example, the sequential fusion encoding of the present disclosure may be used together with the multi-layer feature map general decoding.

[0320] Even if a multi-layer feature map sequential fusion encoding method is used to encode a multi-layer feature map, it is not necessary to restore the multi-layer feature map through multi-layer feature map sequential fusion decoding, and a multi-layer feature map general decoding method can be used to restore the multi-layer feature map.

[0321] Likewise, even if another combination of encoding blocks is used for encoding a multi-layer feature map, as in Fig. 25 (general encoding method for a multi-layer feature map), if the final output can be expressed in the form of a single-layer feature map, the multi-layer feature map can be restored through sequential fusion decoding of the multi-layer feature map.

[0322] Figure 26 illustrates a fourth embodiment of a multi-layer feature map sequential fusion decoding neural network, which is a multi-layer feature map sequential fusion decoding means.

[0323] The fourth embodiment of Fig. 26 may be an embodiment in which the layer index of the restored multi-layer feature map sequentially increases when sequentially decoding the multi-layer feature map.

[0324] The fourth embodiment of Fig. 26 may be an embodiment of step [D2] of Fig. 14.

[0325] The fourth embodiment of Fig. 26 can show that when sequentially decoding a multi-layer feature map, the inputs of decoding block 2, decoding block 3, and decoding block 4 can have additional inputs in addition to the outputs of the previous decoding blocks.

[0326] Decoding block 1, which takes as input the quantized fused latent representation ^y, is composed of a combination of blocks of 5x5 Tconv 2↑, Resblock, Attention Module, and 1x1 Conv, and can output a P-layer feature map ^P2.

[0327] Decryption block 2, which takes as input ^P2, the output of decryption block 1, is composed of a Feature Mixing block and can output a P layer feature map ^P3.

[0328] Decoding block 2 can additionally input the feature map extracted from the fused latent representation ^y through a series of neural network blocks consisting of a combination of blocks of 5x5 Tconv 2↑, Resblock, Attention Module, and 1x1 Conv.

[0329] Decryption block 3, which takes as input ^P3, the output of decryption block 2, is composed of a Feature Mixing block and can output a P layer feature map ^P4.

[0330] Decoding block 3 can additionally input the feature map extracted through a series of neural network blocks consisting of a combination of blocks of 5x5 Tconv 2↑, Resblock, and 1x1 Conv from the fused latent representation ^y.

[0331] Decryption block 4, which takes as input ^P4, the output of decryption block 3, is composed of a Feature Mixing block and can output a P layer feature map ^P5.

[0332] Decoding block 4 can additionally input the feature map extracted through a series of neural network blocks consisting of combinations of blocks of 5x5 Tconv 2↑, Resblock, and 1x1 Conv from the fused latent representation ^y.

[0333] The TConv block can mean a typical transposed convolution layer.

[0334] Figure 27 illustrates a fifth embodiment of a multi-layer feature map sequential fusion decoding neural network, which is a multi-layer feature map sequential fusion decoding means.

[0335] The fifth embodiment of Fig. 27 may be an embodiment in which the layer index of the restored multi-layer feature map sequentially increases when sequentially decoding the multi-layer feature map.

[0336] The fifth embodiment of Fig. 27 may be an embodiment of step [D2] of Fig. 14.

[0337] The fifth embodiment of Fig. 27 can show that when sequentially decoding a multi-layer feature map, decoding block 2 and decoding block 3 can have additional inputs in addition to the output of the previous decoding block. (Number of layers N = 3)

[0338] Decoding block 1, which takes as input the quantized fused latent representation ^y, is composed of a combination of blocks of 5x5 Tconv 2↑, Resblock, Attention Module, and 1x1 Conv, and can output a multi-layer feature map ^F1.

[0339] Decryption block 2, which takes as input ^F1, the output of decryption block 1, is composed of a Feature Mixing block and can output a multi-layer feature map ^F2.

[0340] Decoding block 2 can additionally input the feature map extracted through a series of neural network blocks consisting of a combination of blocks of 5x5 Tconv 2↑, Resblock, and 1x1 Conv from the fused latent representation ^y.

[0341] Decryption block 3, which takes as input ^F2, the output of decryption block 2, is composed of a Feature Mixing block and can output a multi-layer feature map ^F3.

[0342] Decoding block 3 can additionally input the feature map extracted through a series of neural network blocks consisting of a combination of blocks of 5x5 Tconv 2↑, Resblock, and 1x1 Conv from the fused latent representation ^y.

[0343] Figure 28 illustrates an example of neural network-based fused latent representation entropy encoding.

[0344] Specific examples of [E3-2] of FIG. 13 and [D1-2] of FIG. 14 are explained through FIG. 28.

[0345] [E3-2-1] In relation to this, the Q block of Fig. 28 is an example of a fused latent representation quantization means, and all components y of the fused latent representation y i It performs uniform quantization on (at this time, the usual rounding operation can be used), and can output the quantized fused latent representation ^y.

[0346] In neural network-based image compression, quantization can be replaced with a process of adding uniform noise between -0.5 and 0.5 instead of quantization at training time to enable the use of the backpropagation algorithm, and a rounding function can be used at inference time.

[0347] In addition, in order to reduce the mismatch between the learning process and the inference process, a rounding function such as is also used in the learning process. In this case, an identity function can be used when calculating the gradient for the rounding function.

[0348] [E3-2-2], [D1-2-1] In relation to the entropy model of Fig. 28, an example of a means of estimating the probability distribution of the fused latent representation can be constructed based on the entropy model proposed in David Minnen et. al., "Joint autoregressive and hierarchical priors for learned image compression," Advances in neural information processing systems, vol. 31, 2018.

[0349] By first encoding the hyperprior ^z, which is additional information (side information) from the fused latent representation ^y, and transmitting it to the decoder, a more accurate probability distribution estimation for the fused latent representation can be made possible.

[0350] Also, the components of the already decoded fused latent representation ^y <i The component of the fused latent representation to be encoded currently based on ^y i Additional information (context) can be extracted and used for encoding.

[0351] In addition, the entropy model used in the neural network-based technologies below can be used as a means of estimating the probability distribution of the fused latent representation.

[0352] [E3-2-3], [D1-2-2] In relation to FIG. 28, AE and AD are examples of a fused latent representation entropy encoding means and a fused latent representation entropy decoding means, respectively, and represent arithmetic encoding and arithmetic decoding. In addition to arithmetic encoding, range coding can also be used.

[0353] Figure 29 illustrates an example of sequentially estimating a probability distribution by unequally dividing the latent representation for an image in the channel direction.

[0354] Figure 30 illustrates an example of performing probability distribution estimation 10 times sequentially after dividing the latent expression into 10 groups.

[0355] [E3-2-2], [D1-2-1] The Space-Channel ConTeXt (SCCTX) entropy model can be used as a means of estimating the probability distribution of the fused latent representation.

[0356] SCCTX divides the latent representation for the image into five groups (^y) unequally in the channel direction, as shown in Fig. 30(a). (1) , ... ,^y (5) ), and then each latent expression group ^y (k) When sequentially estimating the probability distribution for the already decoded latent representation group ^y (<k) This may include ways to utilize additional information. This additional information may be referred to as channel context.

[0357] Additionally, each latent expression group ^y as in Fig. 29 (k) is spatially divided into two subgroups ^y1 in a checkerboard pattern. (k), ^y2 (k) By dividing the data into subgroups, we can estimate the probability distribution of the remaining subgroup using one already encoded subgroup as additional information. This additional information can be called spatial context.

[0358] Finally, SCCTX can maximize the entropy encoding efficiency of the latent representation by dividing the latent representation into 10 groups and then sequentially performing a total of 10 probability distribution estimations (Fig. 30(b)).

[0359] For the fusion latent expression of the present disclosure, this SCCTX can also be used as a means for estimating the probability distribution of the fusion latent expression.

[0360] FIG. 31 illustrates an example of performing feature map compression (or sub-decoding) based on a multi-layer feature map sequential fusion sub-decoder / decoder of the present disclosure and a sub-decoder / decoder of an existing video compression codec, and an example of using an artificial neural network for feature map compression (or sub-decoding).

[0361] In Figure 31, two classification examples of feature coding for machines can be seen.

[0362] Figure 32 illustrates details of the NN video encoder and NN video decoder of Figure 31.

[0363] Figure 31 (a) shows an example of performing feature map encoding based on an encoder and decoder of an existing video compression codec such as HEVC or VVC, and (b) shows an example of performing feature map encoding based on an encoder and decoder of an artificial neural network codec for feature map encoding.

[0364] In feature map encoding for machine learning, the feature map may typically be a multi-layer feature map. While the present disclosure describes a multi-layer feature map encoding / decoding method, the multi-layer feature map encoding / decoding method also includes a single layer, so the present disclosure may not be limited to multi-layer feature maps. The following description will focus solely on multi-layer feature map encoding.

[0365] Fig. 31 may be related to step [E3-1] of Fig. 13 and step [D1-1] of Fig. 14.

[0366] As an embodiment of (a) of FIG. 31, the multi-layer feature map sequential fusion encoding and decoding of the present disclosure can be combined with an image compression codec. In FIG. 31 (a), the Format Conversion part uses a multi-layer feature map sequential fusion encoding neural network. In terms of the feature map encoding method of FIG. 10, in the embodiment of FIG. 10, the encoder of the image codec may be used instead of the quantization part and entropy encoding. This can generate a bitstream by encoding the fused latent representation, which is the output of the fusion encoding, with the encoder of the image codec, as shown in FIG. 33. In the decoding process, the decoder of the image codec may be used instead of the entropy decoding method and the dequantization process.

[0367] Figure 33 illustrates the process of performing fused latent representation encoding and decoding using the multi-layer feature map sequential fusion unit / decoder of the present disclosure and an existing image compression codec.

[0368] In Figure 33, it can be seen that a fused latent representation transformation process has been added. This process takes the form of a feature map, as shown in Figure 34, which is the output of a multi-layer feature map sequential fusion encoding neural network, and this feature map cannot be directly used as input to a video compression codec encoder. Typically, feature maps can be represented as floating-point values. Since video compression codecs require integer value representation, this transformation may be necessary. In addition, since video compression codecs receive input in the form of a single frame or a continuous frame (sequence), this transformation may also be necessary. The fused latent representation transformation can play a role in performing these two processes.

[0369] The decryption process is the reverse of the encoding process, so its explanation is omitted.

[0370] In Fig. 33, since the fused latent representation is generally configured in the form of a feature map, the fused latent representation transformation and inverse transformation process is described below using the feature map transformation and inverse transformation process as an example.

[0371] Figure 34 illustrates the configuration of a fused latent representation feature map.

[0372] The feature map transformation and inverse transformation process can be composed of steps 1) to 3) below.

[0373] 1) Means for arranging and rearranging feature map channels

[0374] 2) Quantization and dequantization means of feature values

[0375] 3) Feature map conversion information

[0376] Below, we will review the specific details.

[0377] 1) Means for arranging and rearranging feature map channels

[0378] As in the feature map of Figure 34, the feature map can usually be composed of multiple channels.

[0379] In this figure, the number of channels is N C (or c), the horizontal size of the channel is W C (or x), the vertical size of the channel is H C (or y) can be expressed as .

[0380] To encode multiple channels, these channels may need to be converted into frames (or pictures), which are the units input to the encoder of the video codec.

[0381] In the feature map arrangement method, one of the spatial arrangement method, temporal arrangement method, and spatio-temporal arrangement method can be used when converting channels into frames.

[0382] The feature map frame thus transformed can be compressed by an encoder of a video compression codec and restored by a decoder of the video compression codec. The restored feature map frame can be restored to its original channel configuration form by a feature map rearrangement means.

[0383] Figure 35 illustrates frame packing through spatial, temporal, and spatiotemporal arrangement for inputting feature maps into an existing video compression codec.

[0384] Figures 35 (a), (b), and (c) can represent spatial arrangement, temporal arrangement, and spatiotemporal arrangement methods, respectively.

[0385] Spatial arrangement refers to arranging feature map channels m horizontally and n vertically in a tile-like shape to form one feature map frame, as shown in Figure (a), where m and n are the product of the total number of channels to be arranged, N. c It is set to be the same as . The spatially arranged feature map frame can be encoded using the intra-frame prediction method in the encoding means.

[0386] Temporal arrangement refers to arranging each feature map channel temporally so that it becomes one frame, as shown in Figure (b), and the total number of frames for one feature map is the total number of channels N that compose the feature map. c It can be set to be the same as . The temporally arranged feature map frames can be encoded using the inter-frame prediction method in the encoder of the video compression codec. In the present disclosure, the term inter-frame prediction, which is widely used in video encoding standards, is used to help understanding, but it may be more appropriate to call it inter-channel prediction to distinguish it from the existing inter-frame prediction.

[0387] The spatiotemporal arrangement is a temporal arrangement of spatially arranged frames as shown in Figure (c), and the total number of channels to be arranged is N, which is the product of the total number of frames multiplied by mxn, the number of channels that constitute one frame. c It can be set to be the same as .

[0388] 2) Quantization and dequantization of feature values

[0389] Quantization of feature values ​​can be quantized into n-bit integers using uniform quantization or non-uniform quantization methods.

[0390] Equations 13 and 14 are the feature values ​​F, respectively. Original n-bit uniform quantization process and restored feature values ​​after quantization ~ F Converted can represent the uniform dequantization process.

[0391] In the quantization process, rounding may be applied to reduce errors in the integerization process, as in Equation 13, and a process of clipping values ​​that exceed the range of an n-bit integer may be included.

[0392] For quantization and dequantization, the number of bits n of the quantized result and the range of feature values ​​are required. max A value must be set. The n value can be set to a value that can be input to a typical video encoder, such as 8 or 10, or it can be set to a smaller value, such as 4 or 6, to improve compression performance. Range max The value can be obtained by subtracting the maximum value from the minimum value of the feature values ​​to be encoded, or a pre-determined value can be used.

[0393] The above n value and Range max The value is a parameter related to feature value quantization, and the same value used in the encoding process must be used in the decoding process as well. Therefore, it can be preset in the multi-layer feature map encoder or included in the bitstream and passed to the multi-layer feature map decoder.

[0394] Quantization of feature values ​​can be performed before or after the feature map array.

[0395] Dequantization of feature values ​​can be performed before or after rearranging the feature maps. However, if feature value quantization was performed before rearranging the feature maps, dequantization should be performed after rearranging. If feature value quantization was performed after rearranging the feature maps, dequantization should be performed before rearranging to reduce computational errors.

[0396]

[0397]

[0398] 3) Feature map conversion information

[0399] Number of bits n used for feature value quantization and feature value range Range max may need to be passed to the dequantization means.

[0400] In addition, the feature map rearrangement means must know parameters such as how the feature map channels are arranged among spatial, temporal, or spatiotemporal arrangements, and if spatial or spatiotemporal arrangements are used, the number of horizontal channels or vertical channels constituting one frame, etc., so that rearrangement can be performed according to the original channel configuration.

[0401] Therefore, it may be desirable to predetermine such feature map transformation information or to encode it through a feature map encoding means, decode it through a feature map decoding means, and then transmit it to a feature map inverse transformation means.

[0402] As a video compression codec, an encoder and decoder of an existing video compression standard such as HEVC (High Efficiency Video Coding) or VVC (Versatile Video Coding) can be used, and in addition, an encoder or decoder of a codec for still image compression can be used.

[0403] Figure 36 illustrates an example of a block diagram for applying a multi-layer feature map decoder / decoder using a multi-layer feature map that has been decoded temporally previously.

[0404] When applying a multi-layer feature map sequential fusion encoding / decoding means and an entropy encoding / decoding means to a video, encoding performance can be improved by utilizing a multi-layer feature map that was decoded temporally earlier than the multi-layer feature map currently being encoded.

[0405] In the present disclosure, the multilayer feature map to be encoded is {P i} i=2, …, 5 It is indicated as {P i} i=2, …, 5 The multilayer feature map that was previously encoded and decoded is {^P i,ref} i=2, …, 5 can be expressed as

[0406] Decrypted multilayer feature map {^P i,ref}i=2, …, 5 As indicated by the blue arrows in the figure, it can be utilized as an additional input of the multi-layer feature map sequential fusion encoder, the neural network-based fused latent representation entropy decoder, and the multi-layer feature map sequential fusion decoder of the present disclosure.

[0407] At this time, {^P is applied to all three components: multi-layer feature map sequential fusion encoder, neural network-based fusion latent representation entropy decoder / decoder, and multi-layer feature map sequential fusion decoder. i,ref} i=2, …, 5 It can be used as an additional input, or it can be used as an additional input only for some components.

[0408] Also, the decrypted multilayer feature map {^P i,ref} i=2, …, 5 {^P, a multi-layer temporal context, is extracted through a multi-layer temporal context extraction means using as input i context} i=2, …, 5 After printing this {^P i context} i=2, …, 5 As indicated by the red arrow in the figure, it can also be used as an additional input of the multi-layer feature map sequential fusion encoder, the neural network-based fused latent representation entropy decoder, and the multi-layer feature map sequential fusion decoder of the present disclosure.

[0409] At this time, {^P is applied to all three components: multi-layer feature map sequential fusion encoder, neural network-based fusion latent representation entropy decoder / decoder, and multi-layer feature map sequential fusion decoder. i context} i=2, …, 5 It can be used as an additional input, or it can be used as an additional input only for some components.

[0410] FIG. 37 illustrates an example of a neural network that applies a multilayer feature map sequential fusion encoding means to an image by utilizing the multilayer temporal context of FIG. 18, which is an example of a multilayer feature map sequential fusion encoding neural network.

[0411] When the multilayer feature map sequential fusion encoding means of the present disclosure is applied to a video, a multilayer temporal context extracted from a multilayer feature map decoded temporally earlier than the multilayer feature map currently being encoded can be utilized.

[0412] Current encoding target multilayer feature map {P i} i=2, …, 5 And multi-layered temporal context {^P i context} i=2, …, 5 By sequentially encoding multi-layer feature maps into fused latent representations using as input, the final fused latent representation y can be output.

[0413] Encoding block 1 is the temporal context of P2 and P2 layer ^P2 context The combined results between channels can be input.

[0414] Encoding block 2 is the temporal context of P3 and P3 layer ^P3 context , the combined result between the channels of l1 can be input.

[0415] Encoding block 3 is the temporal context of P4 and P4 layer ^P4 context , the combined result between the channels of f2 can be input.

[0416] Finally, encoding block 4 is the temporal context of P5 and P5 layer ^P5 context , the combined result between the channels of f3 can be input.

[0417] As described in Fig. 36, the multi-layer temporal context {^P of Fig. 37 i context} i=2, …, 5 is a decrypted multilayer feature map {^P i,ref} i=2, …, 5 can be used as a substitute for .

[0418] Figure 38 illustrates an example of a neural network applied to a neural network-based fused latent representation entropy decoder / decoder utilizing multi-layer temporal context.

[0419] When the neural network-based fused latent representation entropy encoding / decoding means of the present disclosure is applied to a video, additional multi-layer temporal context can be utilized.

[0420] Specifically, FIG. 38 may be an example of a neural network that applies a neural network-based fused latent representation entropy decoder / decoder to FIG. 28, which is an example of a neural network-based fused latent representation entropy decoder / decoder, by utilizing a multi-layer temporal context.

[0421] The multi-layer temporal context sequential fusion means of Fig. 38 is a multi-layer temporal context {^P i context} i=2, …, 5 By inputting , we can output the fused temporal latent expression t, which is information for estimating a more accurate latent expression probability distribution. In addition, t is f of Fig. 28. ep can be used as additional input.

[0422] As explained in Figure 36 above, the multi-layer temporal context of Figure 38, {^P i context} i=2, …, 5 is a decrypted multilayer feature map {^P i,ref} i=2, …, 5 can be used as a substitute for .

[0423] Figure 39 illustrates an example of a multilayer temporal context sequential fusion neural network, which is a multilayer temporal context sequential fusion means.

[0424] Referring to Figure 39, multi-layer temporal context {^P i context} i=2, …, 5 By sequentially fusing the multi-layer temporal context into a fused temporal latent representation through fusion blocks using fusion blocks as input, the fused temporal latent representation t can be finally output. At this time, the fusion block can be an encoding block or a decoding block described above as a neural network block.

[0425] {^P2, the temporal context of the P2 layer context} as input can be composed of 5 Х 5 2↓ convolutional neural network layers and GDN, and can output the intermediate latent representation l1. {^P3, the temporal context of the P3 layer context} and l1 as input, the fusion block 2 can also be composed of 5 Х 5 2↓ convolutional neural network layers and GDN, and can output the fused latent representation f2. The temporal context of the P4 layer, {^P4 context} and f2 as inputs, the fusion block 3 can also be composed of 5 Х 5 2↓ convolutional neural network layers and GDN, and can output the fused latent representation f3. Finally, the temporal context of the P5 layer, {^P5 ontext} and f3 inputs can be composed of 5 Х 5 2↓ convolutional neural network layers, and can output the final fused temporal latent representation t(=f4).

[0426] At this time, in addition to the fusion block in this embodiment, conventional neural network-based fusion blocks for image processing can be combined and used. As described above in Fig. 36, the multi-layer temporal context {^P of Fig. 39 i context} i=2, …, 5 is a decrypted multilayer feature map {^P i,ref} i=2, …, 5 can be used as a substitute for .

[0427] Figure 40 illustrates an example of a neural network that applies a multi-layer feature map sequential fusion decoding method to a video by utilizing a multi-layer temporal context.

[0428] When applying a multi-layer feature map sequential fusion decoding method to a video, a multi-layer temporal context can be utilized. Specifically, FIG. 40 shows an example of a neural network that applies a multi-layer feature map sequential fusion decoding method to a video by utilizing the multi-layer temporal context in FIG. 22, which is an example of a multi-layer feature map sequential fusion decoding neural network, and the decoded fused latent representation ^y multi-layer temporal context {^P i context} i=2, …, 5 The P-layer feature maps can be sequentially decoded using the input. As a result, the restored P-layer feature map {^P i} i=2, …, 5 can be output. At this time, additional decoding blocks (decoding blocks 5 to 8 of Fig. 40) can be used to utilize multi-layer temporal context.

[0429] The intermediate latent representation l1 and the temporal context of the P5 layer {^P5 context} as input can be composed of Resblock and can output P-layer feature map ^P5. The fused latent representation f2 and the temporal context of P4 layer {^P4 context} as input can be composed of Resblock and can output P-layer feature map ^P4. The fused latent representation f3 and the temporal context of P3 layer {^P3 context} as input can be composed of Resblock and can output P-layer feature map ^P3. Finally, f4, which is a fused latent representation, and {^P2, which is the temporal context of P2 layer. context} as input can be composed of Resblock and can output P-layer feature map ^P2.

[0430] At this time, in addition to the decoding block in this embodiment, conventional neural network-based decoding blocks for image processing can be combined and used. As described above in Fig. 36, the multi-layer temporal context {^P of Fig. 40 i context} i=2, …, 5 is a decrypted multilayer feature map {^P i,ref} i=2, …, 5 can be used as a substitute for .

[0431] Figure 41 illustrates an example of a neural network that applies a multi-layer feature map sequential fusion decoding method to an image by utilizing a multi-layer temporal context.

[0432] Specifically, FIG. 41(a) shows another example of a neural network that applies a multi-layer feature map sequential fusion decoding means to an image by utilizing a multi-layer temporal context in FIG. 24, which is another example of a multi-layer feature map sequential fusion decoding neural network, and can sequentially decode P-layer feature maps through a combination of other decoding blocks by using a fused latent representation ^y and a multi-layer temporal context as inputs.

[0433] The intermediate latent representation l1 and the temporal context of the P2 layer ^P2 context Feature Mixing block 1, which takes as input, can be configured as in Fig. 41(b-1) and can output ^P2. The intermediate latent representation l2 and ^P2 and the temporal context of the P3 layer, ^P3 context Feature Mixing block 2 with input can be configured as in Fig. 41(b-2) and can output ^P3. The intermediate latent representation l3 and ^P3 and the temporal context of the P4 layer ^P4 context Feature Mixing block 3, which takes as input, can be configured as in Fig. 41(b-2) and can output ^P4. Finally, the intermediate latent representation l4 and ^P4 and the temporal context of the P5 layer, ^P5 contextFeature Mixing block4 that takes as input can be configured as in Fig. 41(b-2) and can output ^P5.

[0434] As previously described in Figure 36, the multi-layer temporal context {^P of Figure 41 i context} i=2, …, 5 is a decrypted multilayer feature map {^P i,ref} i=2, …, 5 can be used as a substitute for .

[0435] Figure 42 illustrates an example of a neural network that extracts the temporal context of each previously decoded layer feature map without using motion information.

[0436] When extracting multi-layer temporal context, motion information may or may not be used, and a deformable convolutional neural network may be used.

[0437] Referring to Figure 42, the previously decrypted multi-layer feature map {^P i,ref} i=2, …, 5 {^P, which is the temporal context of each layer feature map, is input as i context} i=2, …, 5 can be output. At this time, in addition to the temporal context extraction block in this embodiment, conventional temporal context extraction blocks for image processing can be combined and used.

[0438] Figure 43 illustrates an example of a neural network that extracts the temporal context of each previously decoded layer feature map using motion information.

[0439] Referring to Figure 43, the previously decrypted multi-layer feature map {^P i,ref} i=2, …, 5 and the multilayer feature map of the current encoding target {P i} i=2, …, 5 Temporal context of multilayer feature map using {^P as input i context} i=2, …, 5 can be extracted.

[0440] A motion information extraction means that takes as input a previously decoded multi-layer feature map and a multi-layer feature map of a current encoding target can extract a motion vector based on the two inputs. A motion vector encoding means that takes as input a motion vector can output a motion latent representation, which is a latent representation of the motion vector. A motion latent representation encoding means that takes as input a motion latent representation can output a bitstream. A motion compensation means that takes as input a previously decoded multi-layer feature map and a motion vector can output a multi-layer temporal context.

[0441] Figure 44 illustrates a detailed implementation of the neural network block and function block diagram of the present disclosure.

[0442] Figure 44(a) shows an example of a motion residual flow block, which is a component of a motion information extraction means to be described later, and includes a motion vector and a multi-layer feature map {P of the predicted P layer current encoding target. i} i=2, …, 5 The motion residual flow can be extracted using the input. In Fig. 44(a), the 7Х7 Conv block may refer to a convolutional neural network layer using a kernel of size 7Х7. In Fig. 44(a), ReLU may refer to a ReLU nonlinear layer commonly used in neural networks for image processing. At this time, in addition to the motion residual flow generation block diagram in this embodiment, conventional neural network-based motion residual flow generation block diagrams for image processing may be combined and used.

[0443] Figure 44(b) shows an example of a motion compensation block diagram, which shows a motion vector and a previously decoded multi-layer feature map {^P i,ref} i=2, …, 5 The predicted P layer can be extracted by inputting . In Fig. 44(b), bilinear warping is performed on the previously decoded multi-layer feature map {^P i,ref} i=2, …, 5It may be a function that performs coordinate transformation based on a motion vector. At this time, in addition to the motion compensation block diagram of the present embodiment, a typical motion compensation function for image processing may be combined and used.

[0444] Fig. 44(c) shows an example of a motion vector scaling block diagram, which can extract a scaled motion vector by inputting a motion vector. In Fig. 44(c), bilinear interpolation can be a function that scales the motion vector. In this case, in addition to the motion vector scaling block diagram of the present embodiment, a typical scaling function for image processing can be combined and used.

[0445] Figure 45 illustrates an example of a motion information extraction neural network, which is a means for extracting motion information.

[0446] Referring to Figure 45, the previously decrypted multilayer feature map {^P i,ref} i=2, …, 5 and the multilayer feature map of the current encoding target {P i} i=2, …, 5 Motion information can be extracted by inputting .

[0447] The P5 layer of the current encoding target and the previously decrypted P5 layer ^P 5,ref The motion residual flow block 1, which inputs a feature map filled with 0, can output v1, which is an intermediate motion residual flow of the P5 layer size, and then output V1 by doubling the width and height. Previously decoded P4 layer ^P 4,refThe motion compensation block 1, which takes V1 and V2 as inputs, can output the predicted P layer m1 of the size of the P4 layer through bilinear warping. The motion residual flow block 2, which takes the P4 layer P4 of the current encoding target and V1 and m1 as inputs, can generate v2, which is an intermediate motion residual flow of the size of the P4 layer, and then output V2 by doubling the width and height after adding components with V1. Previously decoded P3 layer ^P 3,ref The motion compensation block 2, which takes V2 and m2 as inputs, can output the predicted P layer m2 of the size of the P3 layer through bilinear warping. The motion residual flow block 3, which takes the P3 layer p3 of the current encoding target and V2 and m2 as inputs, can generate v3, which is an intermediate motion residual flow of the size of the P3 layer, and then double the width and height after adding components with V2 to output V3. Previously decoded P2 layer ^P 2,ref Motion compensation block 3, which takes V3 as input, can output a predicted P layer m3 of P2 layer size through bilinear warping. Motion residual flow block 4, which takes P2 layer p2 of the current encoding target, V3 and m3 as input, can generate v4, which is an intermediate motion residual flow of P2 layer size, and then output the final motion vector V after component-to-component addition with V3.

[0448] Figure 46 illustrates an example of a motion vector encoding neural network, which is a motion vector encoding means.

[0449] Referring to Figure 46, the final motion latent representation y is obtained by encoding the motion vector into a motion latent representation by taking the motion vector V as input. MV can output. The motion vector encoding means that takes motion vector V as input can be composed of four 5 Х 5 2↓ convolutional neural network layers and three GDN layers, and the final motion latent representation y MVcan be printed.

[0450] Figure 47 illustrates an example of a motion vector decoding neural network, which is a motion vector decoding means.

[0451] Referring to Figure 47, the motion potential expression ^y MV By decoding the motion vector using ^V as input, the final motion vector ^V can be output. The motion latent representation ^y MV The motion vector encoding means that takes as input is composed of four 5 Х 5 2↓ convolutional neural network layers and three GDN layers, and can output the final motion vector ^V.

[0452] Figure 48 illustrates an example of a motion compensation neural network, which is a motion compensation means.

[0453] Referring to Figure 48, the motion vector ^V and the previously decoded multilayer feature map {^P i,ref} i=2, …, 5 The final multi-layer temporal information {^P) is obtained through motion compensation using input i context} i=2, …, 5 can be printed.

[0454] Motion vector ^V and previously decoded P2 layer ^P 2,ref Motion compensation block 1, which takes as input the P2 layer of the current frame predicted through bilinear warping, Predicted can output the predicted current frame's P2 layer P2 Predicted The temporal context extraction block 1 with as input can be composed of Resblock and 3 Х 3 convolutional neural network layers, and the final P2 layer's temporal ^P2 context can be printed.

[0455] Motion vector scaling block 1, which takes motion vector ^V as input, can output motion vector ^V3 of P3 layer size by reducing the motion vector by the size of P3 layer through bilinear interpolation. Motion vector ^V3 of P3 layer size and previously decoded P3 layer ^P 3,ref Motion compensation block 2, which takes as input the P3 layer of the current frame predicted through bilinear warping, Predicted can output the predicted current frame's P3 layer P3 Predicted The temporal context extraction block 2 that takes as input can be composed of Resblock and 3 Х 3 convolutional neural network layers, and the temporal information of the final P3 layer ^P3 context can be printed.

[0456] At this time, in addition to the temporal context extraction block in this embodiment, conventional neural network-based temporal context extraction blocks for image processing can be used in combination.

[0457] Figure 49 illustrates an example of a neural network that extracts the temporal context of each previously decoded measurement feature map using a deformable convolutional neural network layer.

[0458] Referring to Figure 49, the previously decrypted multilayer feature map {^P i,ref} i=2, …, 5 and the multilayer feature map of the current encoding target {P i} i=2, …, 5 Temporal context of multilayer feature map using {^P as input i context} i=2, …, 5 can be extracted.

[0459] An offset extraction means that takes as input a previously decoded multi-layer feature map and a multi-layer feature map of a current encoding target can extract an offset to be used in a deformable convolutional neural network based on the two inputs. An offset encoding means that takes as input an offset can output an offset latent representation, which is a latent representation of the offset. An offset latent representation encoding means that takes as input an offset latent representation can output a bitstream. A deformable convolutional neural network that takes as input a previously decoded multi-layer feature map and an offset can output a multi-layer temporal context.

[0460] Figure 50 illustrates an example of a deformable convolutional neural network.

[0461] Referring to Figure 50, after extracting an offset through an offset extraction block that takes a single-layer feature map as input, a single-layer feature map can be output through a convolutional neural network block with a transformation based on the offset and the input single-layer feature map as input.

[0462] The offset extraction block and the deformable convolutional neural network block can use other typical convolutional neural network layers for image processing in addition to the blocks in this embodiment.

[0463] Figure 51 illustrates an example of an offset extraction means neural network for extracting offsets used in a deformable convolutional neural network.

[0464] Referring to Figure 51, the previously decrypted multi-layer feature map {^P i,ref} i=2, …, 5 and the multilayer feature map of the current encoding target {P i} i=2, …, 5 Multilayer offset {V with input i} i=2, …, 5 can be extracted.

[0465] Multilayer feature map of the current encoding target {P i} i=2, …, 5 and previously decrypted multilayer feature map {^P i,ref}i=2, …, 5 The offset extraction block that takes as input can output a multi-layer offset of the same size and channel as the multi-layer feature map of the current encoding target. At this time, the offset extraction block can be composed of two 3 Х 3 convolutional neural networks, but the offset extraction block can be used in combination with other typical convolutional neural network layers for image processing.

[0466] Figure 52 illustrates an example of an offset encoding neural network, which is an offset encoding means.

[0467] Specifically, FIG. 52 can represent an example of a neural network that applies a multi-layer feature map sequential fusion encoding means to offset encoding by utilizing FIG. 18, which is an example of a multi-layer feature map sequential fusion encoding neural network.

[0468] Multilayer offset {V i} i=2, …, 5 By sequentially encoding the multi-layer offsets into a fused latent representation using V2 as input, the final fused latent representation y can be output. Encoding block 1 can take V2 as input. Encoding block 2 can take the combined result between the channels of V3 and l1 as input. Encoding block 3 can take the combined result between the channels of V4 and l2 as input. Finally, encoding block 4 can take the combined result between the channels of V5 and l3 as input. At this time, the encoding block can be used in combination with other typical convolutional neural network layers for image processing.

[0469] Figure 53 illustrates an example of an offset decoding neural network, which is an offset decoding means.

[0470] Specifically, FIG. 53 can represent an example of a neural network that applies a multi-layer feature map sequential fusion decoding means to offset decoding by utilizing FIG. 22, which is an example of a multi-layer feature map sequential fusion decoding neural network.

[0471] The decoded fused latent representation ^y is input and the multi-layer offsets are sequentially decoded. The resulting restored multi-layer offset {^V i} i=2, …, 5 can output. Decoding block 1 with intermediate latent representation l1 as input can output ^V5. Decoding block 2 with intermediate latent representation l2 as input can output ^V4. Decoding block 3 with intermediate latent representation l3 as input can output ^V3. Finally, decoding block 4 with intermediate latent representation l4 as input can output ^V2. At this time, the decoding block can be used in combination with other typical convolutional neural network layers for image processing.

[0472] Figure 54 illustrates an example of a motion compensation neural network, which is a motion compensation means using a deformable convolutional neural network layer.

[0473] Referring to Figure 54, the multi-layer offset {^V i} i=2, …, 5 and the previously decrypted multilayer feature map {^P i,ref} i=2, …, 5 The final multilayer temporal information {^P) is obtained through a deformable convolutional neural network using as input i context} i=2, …, 5 can be printed.

[0474] Multi-layer offset{^V i} i=2, …, 5 and previously decrypted multilayer feature map {^P i,ref} i=2, …, 5 Block 1 of the deformable convolutional neural network that takes as input {f i} i=2, …, 5 can output {f i} i=2, …, 5 and previously decrypted multilayer feature map {^P i,ref} i=2, …, 5Motion compensation block 2, which takes as input , can be composed of 3 Х 3 convolutional neural network layers, and multilayer feature map {P of the predicted current frame i predicted} i=2, …, 5 can output the predicted multilayer feature map {P i predicted} i=2, …, 5 The temporal context extraction block that takes as input can be composed of Resblock and 3 Х 3 convolutional neural network layers, and the final multi-layer temporal context {^P i context} i=2, …, 5 can be output. At this time, in addition to the neural network block in this embodiment, conventional neural network-based blocks for image processing can be combined and used.

[0475] The step of extracting a multilayer feature map including a plurality of feature maps in a hierarchical form from an input image of the present disclosure may be performed in a multilayer feature map extraction unit. The step of sequentially encoding the multilayer feature map through consecutive encoding blocks to output a fused latent representation of a single layer may be performed in a fused latent representation output unit. The step of encoding the fused latent representation of the single layer into a bitstream may be performed in a bitstream encoding unit. In addition, the multilayer feature map extraction unit, the fused latent representation output unit, and the bitstream encoding unit may be included in a multilayer feature map encoding device. The specific contents of each unit are specifically described in the means or methods above, and are therefore omitted below.

[0476] The step of decoding a single-layer fused latent representation for a multi-layer feature map including a plurality of feature maps in a hierarchical form from a bitstream of the present disclosure may be performed by a bitstream decoding unit. The step of sequentially decoding the multi-layer feature map through consecutive decoding blocks from the single-layer fused latent representation may be performed by a multi-layer feature map decoding unit. The bitstream decoding unit and the multi-layer feature map decoding unit may be included in a multi-layer feature map decoding device. The specific contents of each unit are specifically described in the means or methods above and are therefore omitted below.

[0477] While the exemplary methods of this disclosure are presented as a series of operations for clarity of description, this is not intended to limit the order in which the steps are performed, and individual steps may be performed simultaneously or in different orders, if desired. To implement a method according to this disclosure, additional steps may be included in addition to the steps illustrated, some steps may be excluded and the remaining steps included, or some steps may be excluded and additional steps included.

[0478] The various embodiments of the present disclosure are not intended to list all possible combinations but rather to illustrate representative aspects of the present disclosure, and the matters described in the various embodiments may be applied independently or in combinations of two or more.

[0479] Additionally, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. In the case of hardware implementation, the embodiments may be implemented by one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), general processors, controllers, microcontrollers, microprocessors, etc.

[0480] The scope of the present disclosure includes software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) that cause operations according to the methods of various embodiments to be executed on a device or a computer, and a non-transitory computer-readable medium having such software or instructions stored thereon and executable on the device or computer.

[0481] The present disclosure can be used in the field of a method and device for encoding / decoding a feature map extracted through an artificial neural network.

Claims

1. A step of extracting a multi-layer feature map including multiple feature maps in a layered form from an input image; A step of sequentially encoding the above multi-layer feature map through consecutive encoding blocks to output a fused latent representation of a single layer; and A multi-layer feature map encoding method, characterized in that it includes a step of encoding the fused latent representation of the single layer into a bitstream.

2. In paragraph 1, A multi-layer feature map encoding method, characterized in that the above consecutive encoding blocks include a first encoding block having as input a feature map having the highest resolution of the multi-layer feature map, a second encoding block having as input a first intermediate latent representation which is an output of the first encoding block, a third encoding block having as input a first fused latent representation which is an output of the second encoding block, and a fourth encoding block having as input a second fused latent representation which is an output of the third encoding block and outputting a fused latent representation of the single layer.

3. In paragraph 2, A multilayer feature map encoding method, characterized in that the inputs of the second encoding block, the third encoding block, and the fourth encoding block include feature maps of the multilayer feature map having the same resolution as the input intermediate latent representation or fused latent representation.

4. In paragraph 2, A multi-layer feature map encoding method, characterized in that the first encoding block, the second encoding block, and the third encoding block include a convolutional neural network layer using a kernel of size nxn and a nonlinear layer of LeakyReLU with the same structure.

5. In paragraph 4, A multi-layer feature map encoding method, characterized in that the fourth encoding block does not include a non-linear layer of the LeakyReLU.

6. In paragraph 2, A multilayer feature map encoding method, characterized in that the resolution of the output of the first encoding block, the second encoding block, the third encoding block, and the fourth encoding block is lower than the resolution of the input.

7. In paragraph 1, A multilayer feature map encoding method, characterized in that the above consecutive encoding blocks include a first encoding block having as input a feature map having the highest resolution of the multilayer feature map, a second encoding block having as input a first intermediate latent representation which is an output of the first encoding block, a third encoding block having as input a feature map of the multilayer feature map having a lower resolution than the first fused latent representation which is an output of the second encoding block, and a fourth encoding block having as input the second intermediate latent representation which is an output of the third encoding block and the first fused latent representation and outputting a fused latent representation of the single layer.

8. In paragraph 7, A multi-layer feature map encoding method, characterized in that the input of the second encoding block and the fourth encoding block includes a feature map of the multi-layer feature map having the same resolution as the input intermediate latent representation or fused latent representation.

9. In paragraph 7, A multi-layer feature map encoding method, characterized in that the first encoding block, the second encoding block, and the third encoding block include a convolutional neural network layer using a kernel of size nxn and a nonlinear layer of LeakyReLU with the same structure.

10. In paragraph 9, A multi-layer feature map encoding method, characterized in that the fourth encoding block does not include a non-linear layer of the LeakyReLU.

11. In paragraph 7, The resolution of the output of the first encoding block, the second encoding block, and the fourth encoding block is lower than the resolution of the input, A multi-layer feature map encoding method, characterized in that the resolution of the output of the third encoding block is higher than the resolution of the input.

12. A step of decoding a single-layer fused latent representation for a multi-layer feature map including multiple feature maps in a hierarchical form from a bitstream; and A multi-layer feature map decoding method, characterized in that it includes a step of sequentially decoding the multi-layer feature map through consecutive decoding blocks from the fused latent representation of the single layer.

13. In paragraph 12, A multi-layer feature map decoding method, characterized in that the above consecutive decoding blocks include a first decoding block which inputs the fused latent representation of the single layer, a second decoding block which inputs the first feature map of the multi-layer feature map having the lowest resolution, which is an output of the first decoding block, a third decoding block which inputs the second feature map, which is an output of the second decoding block having a higher resolution than the first feature map, and a fourth decoding block which inputs the third feature map, which is an output of the third decoding block having a higher resolution than the second feature map, and outputs the fourth feature map of the multi-layer feature map having the highest resolution.

14. In paragraph 13, A multi-layer feature map decoding method, characterized in that the first decoding block, the second decoding block, and the third decoding block include a convolutional neural network layer using a kernel of size nxn and a nonlinear layer of LeakyReLU with the same structure.

15. In paragraph 14, A multi-layer feature map decoding method, characterized in that the fourth decoding block does not include a non-linear layer of the LeakyReLU.

16. In paragraph 13, A multi-layer feature map decoding method, characterized in that the resolution of the output of the first decoding block, the second decoding block, the third decoding block, and the fourth decoding block is higher than the resolution of the input.

17. In paragraph 12, A multi-layer feature map decoding method, characterized in that the above-mentioned consecutive decoding blocks include a first decoding block which takes as input the fused latent representation of the single layer, a second decoding block which takes as input the first feature map which is the output of the first decoding block and outputs the second feature map of the multi-layer feature map having the lowest resolution, a third decoding block which takes as input the first feature map and outputs the third feature map having a higher resolution than the first feature map, and a fourth decoding block which takes as input the third feature map and outputs the fourth feature map of the multi-layer feature map having the highest resolution.

18. In paragraph 17, A multi-layer feature map decoding method, characterized in that the second decoding block, the third decoding block, and the fourth decoding block include a convolutional neural network layer using a kernel of size nxn and a nonlinear layer of LeakyReLU with the same structure.

19. In paragraph 17, The resolution of the output of the first decoding block, the third decoding block, and the fourth decoding block is higher than the resolution of the input, A multi-layer feature map decoding method, characterized in that the resolution of the output of the second decoding block is lower than the resolution of the input.

20. A computer-readable recording medium storing a bitstream generated by a multi-layer feature map encoding method, The above multi-layer feature map encoding method comprises the steps of: extracting a multi-layer feature map including a plurality of feature maps in a layered form from an input image; A step of sequentially encoding the above multi-layer feature map through consecutive encoding blocks to output a fused latent representation of a single layer; and A computer-readable recording medium, characterized in that it comprises a step of encoding the fused latent representation of the single layer into a bitstream.

Images