Multi-layer feature map encoding / decoding method and device based on feature map adaptation and sequential restoration, and recording medium

Abstract

A multi-layer feature map encoding method and device of the present disclosure comprises the steps of: acquiring a reduced feature map by reducing a multi-layer feature map extracted from an input image; and encoding the reduced feature map into a bitstream, wherein the reduction of the multi-layer feature map can be performed using a multi-layer feature map reduction means including a plurality of multi-layer feature map reduction blocks. In addition, the multi-layer feature map decoding method and device of the disclosure may comprise the steps of: acquiring a reduced feature map by decoding a bitstream; and restoring a multi-layer feature map by using the reduced feature map as an input of a multi-layer feature map restoration means including a plurality of multi-layer feature map restoration blocks.

Description

Multilayer feature map encoding / decoding method, device, and recording medium based on feature map adaptation and sequential restoration

[0001] The present disclosure relates to the technical field of a method, device, and recording medium for encoding / decoding a feature map extracted through an artificial neural network.

[0002] As machine learning becomes more widely used across a variety of devices, including mobile devices as well as large servers, the means for extracting feature maps and the means for performing tasks are increasingly located on different devices rather than within the same device.

[0003] When the feature map extraction means and the task execution means are separated, the extracted feature map must be transferred to the task execution means, but since the amount of data in the feature map is very large, a feature map encoding method is needed to drastically reduce the amount of data in the feature map while minimizing the degradation of task execution performance.

[0004] The purpose of this disclosure is to provide a method for preventing information flow from being restricted in a sequential multi-layer feature map restoration process by obtaining a feature map of each layer to be restored through a feature map adaptation block.

[0005] In addition, the present disclosure aims to provide a method for maximizing feature map encoding efficiency while minimizing computational complexity and structural redundancy by utilizing a serial-parallel structure of a feature map restoration means.

[0006] The multi-layer feature map decoding method and device of the present disclosure may include a step of decoding a bitstream to obtain a reduced feature map, and a step of restoring a multi-layer feature map by using the reduced feature map as an input to a multi-layer feature map restoration means including a plurality of multi-layer feature map restoration blocks.

[0007] In the multi-layer feature map decoding method and device of the present disclosure, restoration of the multi-layer feature map can be performed by a sequential multi-layer feature map restoration method that restores the multi-layer feature map using a feature map adaptation block.

[0008] In the multi-layer feature map decoding method and device of the present disclosure, the feature map adaptation block includes a first feature map adaptation block and a second feature map adaptation block that reconstructs a second feature map having a smaller resolution than a first feature map reconstructed from the first feature map adaptation block, and the number of input channels of the second feature map adaptation block may be greater than or equal to the number of input channels of the first feature map adaptation block.

[0009] In the multi-layer feature map decoding method and device of the present disclosure, restoration of the multi-layer feature map can be performed by a serial-parallel multi-layer feature map restoration method that restores the multi-layer feature map by configuring the plurality of multi-layer feature map restoration blocks in a serial-parallel mixed form.

[0010] In the multi-layer feature map decoding method and device of the present disclosure, the resolution of feature maps output from the plurality of multi-layer feature map restoration blocks can sequentially increase or decrease depending on the order in which the plurality of multi-layer feature map restoration blocks are sequentially performed.

[0011] In the multi-layer feature map decoding method and device of the present disclosure, all or part of the feature maps output from the plurality of multi-layer feature map restoration blocks can be restored as a hierarchical feature map of the multi-layer feature map through the feature map adaptation block.

[0012] In the multi-layer feature map decoding method and device of the present disclosure, feature maps output from the remaining multi-layer feature map restoration blocks, excluding the last multi-layer feature map restoration block according to the order in which the plurality of multi-layer feature map restoration blocks are sequentially performed among the plurality of multi-layer feature map restoration blocks, can be restored as hierarchical feature maps of the multi-layer feature map through the feature map adaptation block.

[0013] In the multi-layer feature map decoding method and device of the present disclosure, the feature map adaptation block can transform the input feature map through a neural network without changing the spatial resolution of the input and output feature maps.

[0014] The multilayer feature map encoding method and device of the present disclosure include a step of obtaining a reduced feature map by reducing a multilayer feature map extracted from an input image, and a step of encoding the reduced feature map into a bitstream, wherein the reduction of the multilayer feature map can be performed using a multilayer feature map reduction means including a plurality of multilayer feature map reduction blocks.

[0015] In the multi-layer feature map encoding method and device of the present disclosure, the resolution of feature maps output from the plurality of multi-layer feature map reduction blocks can sequentially increase or decrease depending on the order in which the plurality of multi-layer feature map reduction blocks are sequentially performed.

[0016] In the multi-layer feature map encoding method and device of the present disclosure, the number of channels of a feature map output from a first layer of a last multi-layer feature map reduction block according to the order in which the plurality of multi-layer feature map reduction blocks are sequentially performed among the plurality of multi-layer feature map reduction blocks may be smaller than the number of channels of a final output feature map of the last multi-layer feature map reduction block.

[0017] The present disclosure prevents information flow restrictions during sequential multi-layer feature map restoration, thereby maximizing encoding / decoding efficiency with minimal complexity. Furthermore, by utilizing a serial-parallel structure, the present disclosure minimizes the computational complexity and structural redundancy of the multi-layer feature map restoration method without significantly compromising encoding efficiency.

[0018] Figure 1 shows an example of the results of machine work for object detection and classification using Fast R-CNN, one type of artificial neural network.

[0019] Figure 2 illustrates an example in which a multi-layer feature map is extracted and machine work is performed.

[0020] Figure 3 illustrates the structure of Mask R-CNN, an artificial neural network model frequently used for object region segmentation.

[0021] Figure 4 shows a multi-layer feature map P extracted through FPN of Mask R-CNN. k , which shows an example.

[0022] Figure 5 illustrates an embodiment of a multi-layer feature map.

[0023] Figure 6 illustrates an example in which a feature map extraction means exists on a mobile device, but a specific task execution means, such as object segmentation, disparity map estimation, or image restoration, exists on a cloud server.

[0024] Figure 7 illustrates an example of a process of encoding and decoding a multi-layer feature map.

[0025] Fig. 8 illustrates an example of a multi-layer feature map reduction means. Fig. 9 illustrates an example of a specific configuration of the multi-layer feature map reduction blocks used in Fig. 8.

[0026] Figures 10 and 11 illustrate examples of the detailed structure of the neural network block used in Figure 9.

[0027] Figure 12 illustrates a first embodiment of a multi-layer feature map restoration means.

[0028] Figure 13 illustrates an example of the configuration of the multi-layer feature map restoration block and neural network blocks for additional information used in Figure 12.

[0029] Figure 14 illustrates a second embodiment of a multi-layer feature map restoration means.

[0030] Figure 15 illustrates a third embodiment of a multi-layer feature map restoration means.

[0031] Figure 16 illustrates an example of a parallel multi-layer feature map restoration means.

[0032] Figure 17 illustrates an example of a resolution top-down sequential multi-layer feature map restoration means.

[0033] Figure 18 illustrates an example of a resolution-upward sequential multi-layer feature map restoration means.

[0034] Figure 19 illustrates an example of a parallel multi-layer feature map restoration means and the amount of information.

[0035] Figure 20 illustrates an example of a resolution-upward sequential multi-layer feature map restoration means and the amount of information.

[0036] Figure 21 illustrates an example of a resolution top-down sequential multi-layer feature map restoration means and the amount of information.

[0037] Figure 22 illustrates an example of a resolution-upward sequential multi-layer feature map restoration means based on feature map adaptation and the amount of information.

[0038] Figure 23 illustrates an example of a resolution top-down sequential multi-layer feature map restoration means based on feature map adaptation and the amount of information.

[0039] Figure 24 illustrates an example of a resolution-upward sequential multi-layer feature map restoration means based on feature map adaptation omitting the last feature map adaptation block and the amount of information.

[0040] Figure 25 illustrates an example of a resolution top-down sequential multi-layer feature map restoration means based on feature map adaptation omitting the last feature map adaptation block and the amount of information.

[0041] Figure 26 illustrates an example of a multi-layer feature map reduction means based on latent feature map adaptation.

[0042] Figure 27 illustrates an example of a multi-layer feature map reduction means based on latent feature map adaptation omitting the first latent feature map adaptation block.

[0043] FIG. 28 illustrates an embodiment of a drawing in which a multi-layer feature map restoration block 0 is added to a parallel multi-layer feature map restoration means and the amount of information.

[0044] FIG. 29 illustrates an embodiment of a drawing in which a multi-layer feature map restoration block 0 is added to a parallel multi-layer feature map restoration means and the amount of information.

[0045] Figure 30 illustrates an example of resolution-upward inter-layer compensation.

[0046] Figure 31 illustrates an example of resolution top-down inter-layer compensation.

[0047] Figure 32 illustrates an example in which a feature map adaptation block and an inter-layer compensation block are used together.

[0048] Figure 33 illustrates an embodiment of a multi-layer feature map encoding method and device.

[0049] Figure 34 illustrates an embodiment of a multi-layer feature map decoding method and device.

[0050] Figure 35 illustrates an example of a neural network block.

[0051] Figure 36 is a drawing for explaining an example of SubpelPixel.

[0052] Figure 37 shows an example of using DepthConv3x3(P, P) and CONV1x1(P, P / 3) together.

[0053] Figures 38, 39 and 40 illustrate examples of spatial mixing and channel mixing.

[0054] Figure 41 shows an example of the definition of WSiLU used in Figures 38 and 39.

[0055] Figure 42 illustrates an embodiment of a multi-layer feature map reduction means.

[0056] Figure 43 illustrates an embodiment of a multi-layer feature map reduction means.

[0057] Figure 44 illustrates an example of extracting a reduced feature map so that the resolution is the same as that of the feature map with the smallest resolution among the input feature maps.

[0058] Figure 45 illustrates an example of extracting a reduced feature map so that the resolution is the same as that of the second smallest feature map.

[0059] Figure 46 illustrates an embodiment of the last block of the multi-layer feature map reduction means.

[0060] Figure 47 illustrates an embodiment of the last block of the multi-layer feature map reduction means.

[0061] Figure 48 illustrates an embodiment of the last block of the multi-layer feature map reduction means.

[0062] Figures 49 and 50 illustrate examples of implementations of a multi-layer feature map reduction means that apply weight reduction to the last block of the multi-layer feature map reduction means.

[0063] Figure 51 illustrates an embodiment of a multi-layer feature map reduction means based on latent feature map adaptation.

[0064] Figure 52 illustrates an embodiment of a multi-layer feature map reduction means based on latent feature map adaptation.

[0065] Figure 53 illustrates an embodiment of a multi-layer feature map reduction means based on latent feature map adaptation where the first latent feature map adaptation block is omitted.

[0066] Figure 54 illustrates an embodiment of a multi-layer feature map reduction means based on latent feature map adaptation where the first latent feature map adaptation block is omitted.

[0067] FIG. 55 illustrates an embodiment of a resolution top-down sequential multi-layer feature map restoration means based on feature map adaptation.

[0068] FIG. 56 illustrates an embodiment of a resolution top-down sequential multi-layer feature map restoration means based on feature map adaptation.

[0069] FIG. 57 illustrates an embodiment of a resolution top-down sequential multi-layer feature map restoration means based on feature map adaptation omitting the last feature map adaptation block.

[0070] Figure 58 illustrates an example of restoring a multi-layer feature map from a reduced feature map having the same resolution as the feature map with the smallest resolution among the input feature maps.

[0071] Figure 59 illustrates an example of restoring a multi-layer feature map from a downscaled feature map whose resolution is the same as that of the second smallest feature map.

[0072] FIG. 60 illustrates an embodiment of a resolution-upward sequential multi-layer feature map restoration means based on feature map adaptation.

[0073] FIG. 61 illustrates an embodiment of a resolution-upward sequential multi-layer feature map restoration means based on feature map adaptation.

[0074] FIG. 62 illustrates an embodiment of a resolution-upward sequential multi-layer feature map restoration means based on feature map adaptation omitting the last feature map adaptation block.

[0075] Figure 63 illustrates an embodiment of a resolution top-down sequential multi-layer feature map restoration means based on feature map adaptation using an asymmetric number of channels considering information flow and spatial resolution.

[0076] Figure 64 illustrates an embodiment of a resolution top-down sequential multi-layer feature map restoration means based on feature map adaptation using an asymmetric number of channels considering information flow and spatial resolution.

[0077] Figure 65 illustrates an embodiment of a resolution-upward sequential multi-layer feature map restoration means based on feature map adaptation using an asymmetric number of channels considering information flow and spatial resolution.

[0078] Figure 66 illustrates an embodiment of a sequential to parallel multi-layer feature map restoration means.

[0079] Figure 67 illustrates an embodiment of a composite use of a feature map adaptation block and an inter-layer compensation block.

[0080] Figure 68 illustrates an example of a multi-layer feature map restoration means when the number of layers is three.

[0081] Figures 69 and 70 illustrate examples of a multi-layer feature map reduction means and a multi-layer feature map restoration means when the number of layers is two, respectively.

[0082] Figure 71 illustrates an example of a multi-layer feature map reduction means when the number of layers is 1.

[0083] Figures 72 and 73 illustrate examples of a multi-layer feature map restoration means when the number of layers is 1.

[0084] Figure 74 illustrates an example of reduced feature map entropy encoding.

[0085] Figure 75 illustrates an example of spatially dividing a latent expression group into a checkerboard pattern.

[0086] Figure 76 illustrates an example of additionally utilizing information of already decoded latent expression groups when sequentially estimating probability distributions for latent expression groups.

[0087] Figure 77 illustrates an example of a feature map encoding system.

[0088] Figure 78 illustrates an example of a process for performing reduced feature map encoding and decoding using an image compression codec.

[0089] Figure 79 illustrates an example of the configuration of a feature map.

[0090] Figure 80 shows an example of a spatially arranged feature map.

[0091] Figure 81 shows an example of a temporally arranged feature map.

[0092] Figure 82 shows an example of a spatiotemporally arranged feature map.

[0093] The present disclosure is susceptible to various modifications and embodiments. Therefore, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present disclosure to specific embodiments, but rather to include all modifications, equivalents, and substitutes falling within the spirit and scope of the present disclosure. In the drawings, similar reference numerals designate the same or similar functions throughout. The shapes and sizes of elements in the drawings may be exaggerated for clarity. The detailed description of the exemplary embodiments described below refers to the accompanying drawings, which illustrate specific embodiments by way of example. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that the various embodiments, while different from each other, are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present disclosure. Furthermore, it should be understood that the positions or arrangements of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiment. Accordingly, the detailed description set forth below is not intended to be taken in a limiting sense, and the scope of the exemplary embodiments, if properly described, is defined only by the appended claims, along with the full scope equivalents to which such claims are entitled.

[0094] While terms such as "first" and "second" may be used herein to describe various components, these components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present disclosure, a first component may be referred to as a "second component," and similarly, a second component may also be referred to as a "first component." The term "and / or" includes a combination of multiple related items described herein or any of multiple related items described herein.

[0095] When a component of the present disclosure is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components present in between. Conversely, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components present in between.

[0096] The components shown in the embodiments of the present disclosure are independently depicted to represent different characteristic functions, and do not imply that each component is composed of separate hardware or a single software component. That is, each component is listed and included as a separate component for convenience of explanation, and at least two components among each component may be combined to form a single component, or a single component may be divided into multiple components to perform a function, and such integrated and separate embodiments of each component are also included in the scope of the present disclosure as long as they do not deviate from the essence of the present disclosure.

[0097] The terminology used in this disclosure is only used to describe specific embodiments and is not intended to limit the present disclosure. The singular expression includes plural expressions unless the context clearly indicates otherwise. In this disclosure, it should be understood that terms such as "comprise" or "have" are intended to specify the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof. In other words, the description of a specific configuration in this disclosure as "comprising" does not exclude configurations other than the specified configuration, and means that additional configurations may be included in the scope of the implementation or technical idea of ​​the present disclosure.

[0098] Some components of the present disclosure may not be essential components that perform essential functions of the present disclosure, but may be optional components merely for performance enhancement. The present disclosure may be implemented by including only components essential to implementing the essence of the present disclosure, excluding components used solely for performance enhancement. A structure that includes only essential components, excluding optional components used solely for performance enhancement, is also within the scope of the present disclosure.

[0099] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In describing the embodiments of this specification, if a detailed description of a related known configuration or function is judged to obscure the gist of this specification, the detailed description will be omitted. The same reference numerals will be used for identical components in the drawings, and duplicate descriptions of the same components will be omitted.

[0100] There are increasing cases of utilizing artificial neural networks (ANNs) in various machine vision tasks such as object classification, object recognition, object detection, object segmentation, and object tracking, as well as various image processing tasks such as super-resolution and frame interpolation.

[0101] Figure 1 shows an example of the results of machine work for object detection and classification using Fast R-CNN, one type of artificial neural network.

[0102] Referring to Figure 1, the recognized objects in Figure 1 can be classified into automobiles, ships, people, traffic signals, etc., and each region can be configured as a rectangle, and probability information (or accuracy) regarding whether the region matches the classified object can be included. Here, the rectangle is an example of the region shape, and the region shape can include circles, polygons, etc. In addition, through region segmentation within the region, the range of the region and the object can be matched as much as possible.

[0103] An artificial neural network model that performs a machine task can typically be comprised of a feature map extraction means that extracts features from input data or an input image, and a task execution means that actually performs a specific machine task based on the extracted features. In this case, when the input is in the form of an image, the "extracted features" may typically be referred to as a feature map. While the present disclosure describes the invention using the term "feature map," the present disclosure can equally be applied to features that are not in the form of a map.

[0104] Figure 2 illustrates an example in which a multi-layer feature map is extracted and machine work is performed.

[0105] The present disclosure may be applicable to a pyramid-structured feature map or a multi-layer feature map. A multi-layer feature map may have a pyramid structure in which feature maps with different resolutions are formed in multiple layers. For example, as a feature map belongs to a higher layer (or as the layer index increases), the resolution of the feature map may decrease, and as a feature map belongs to a lower layer (or as the layer index decreases), the resolution of the feature map may increase. In another example, as a feature map belongs to a higher layer (or as the layer index increases), the resolution of the feature map may increase, and as a feature map belongs to a lower layer (or as the layer index decreases), the resolution of the feature map may decrease. Furthermore, feature maps within the same layer may have the same resolution.

[0106] Layer information (e.g., layer index, layer resolution, etc.) of each layer in a multi-layer feature map can be included in the feature map information and transmitted when performing machine tasks.

[0107] The present disclosure may relate to a compression (or encoding / decoding) method, device, and recording medium for such a multi-layer feature map.

[0108] Figure 3 illustrates the structure of Mask R-CNN, an artificial neural network model frequently used for object region segmentation.

[0109] In the Mask R-CNN structure of Fig. 3, a feature pyramid network (FPN) can be used as a means of extracting a multi-layer feature map, and a region proposal network (RPN) and region of interest heads (ROI heads) can be used as a means of performing machine tasks.

[0110] A feature pyramid network (FPN) is an example of extracting a multi-layer feature map, and a C-layer feature map and a P-layer feature map can be extracted by the FPN. Here, both the C-layer feature map and the P-layer feature map can be multi-layer feature maps of the present disclosure.

[0111] Hereinafter, in the present disclosure, the invention will be described using the P-layer feature map of FIG. 3. However, this is merely for convenience of explanation, and the description of the present disclosure can be equally applied to the C-layer feature map of FIG. 3 or other types of multi-layer feature maps.

[0112] Figure 4 shows a multi-layer feature map P extracted through FPN of Mask R-CNN. k , which shows an example.

[0113] If the input is an image, the shape of the feature map can be expressed as a two-dimensional array in width x height, and since the feature map of one layer is usually composed of multiple channels, the feature map of each layer can be expressed as a three-dimensional array with a size equal to width_size x height_size x number_of_channels.

[0114] That is, the feature map of layer k is F k If you do F k is a three-dimensional array F consisting of extracted feature values. k It can be expressed as [x][y][c], where x and y represent the horizontal and vertical positions of the feature values, respectively, and c can represent the channel index. For example, a multi-layer feature map C extracted from FPN k or multilayer feature map P k Multilayer feature map F of the disclosure k It can be said that the three-dimensional array F k It can be expressed as [x][y][c].

[0115] Referring to Fig. 4, the P-layer feature map P is a multi-layer feature map extracted through the FPN of the Mask R-CNN of Fig. 3. kAn example of the first channel can be seen in . In FPN, the feature map of each layer consists of 256 channels, and Fig. 4 may only show the feature map corresponding to the first channel among the feature maps of each layer as an example. Note that in FPN, as the layer becomes deeper, the width and height of the feature map may become smaller and smaller than the size of the input image.

[0116] But in general, the layer index k and the feature map F k The relationship between the resolutions is not fixed, and depending on the type of multi-layer feature map, as the index k of the layer increases, the width and height of the feature map may become smaller or larger than the size of the input image.

[0117] Figure 5 illustrates an embodiment of a multi-layer feature map.

[0118] Referring to Fig. 5, even in a model for machine tasks such as YOLO v3, a multi-layer feature map (Output 1, Output 2, Output 3) consisting of three layers can be extracted in a similar manner to the P-layer feature map as in Fig. 5 and used to perform machine tasks.

[0119] Specifically, the above multi-layer feature map may be a case where a pyramid structure similar to FPN is used together with darknet 53 as a means of extracting multi-layer feature maps within YOLO v3.

[0120] As machine learning becomes more widely used in a variety of devices, including mobile devices as well as large servers, the means for extracting feature maps and the means for performing machine learning are increasingly located not within the same device but within different devices.

[0121] Figure 6 illustrates an example in which a feature map extraction means exists on a mobile device, but a specific task execution means, such as object segmentation, disparity map estimation, or image restoration, exists on a cloud server.

[0122] A feature map extracted from a mobile device is transmitted to a server, and a task is performed on the server based on the transmitted feature map, and the result of the task can be transmitted back to the mobile device.

[0123] In cases where the feature map extraction means and the task execution means are separated, as in this example, the extracted feature map must be transferred to the task execution means, and a feature map encoding method may be required to minimize the amount of data of the feature map to be transferred or stored while minimizing the degradation of task execution performance.

[0124] As another example, even if the feature map extraction means and the task execution means exist in one device, there may be a case where the extracted feature map is stored through a storage device and then used by the task execution means later. In this case as well, the feature map compression (or encoding / decoding) method as described above may be required.

[0125] In the present disclosure, the term “image” may refer to various types of images, such as natural images acquired through a camera, computer graphics, holographic images, feature map images extracted through a neural network, and ultrasound images.

[0126] Figure 7 illustrates an example of a process of encoding and decoding a multi-layer feature map.

[0127] A multi-layer feature map is composed of multiple layers of feature maps with different spatial resolutions, and each layer's feature map has multiple channels, so it can have a much higher dimensionality than a typical image / video. Therefore, a conventional multi-layer feature map encoding device may include 1) a multi-layer feature map reduction means, 2) a reduced feature map conversion means, and 3) a reduced feature map encoding means.

[0128] The process of performing multi-layer feature map encoding can be explained as follows through steps 1) to 3) of Figure 7.

[0129] 1) Multi-layer feature map reduction means can be used to drastically reduce the dimensionality of multi-layer feature maps. This multi-layer feature map reduction means multi-layer feature map {F k} (k=1,…,L) With input as input, one or more of the detailed processes of the multi-layer feature map reduction means below can be performed, and then a reduced feature map y can be output. The detailed processes of the multi-layer feature map reduction means below can be performed independently, but two or more detailed processes can be combined and performed as a single process.

[0130] The detailed process of the multi-layer feature map reduction means may include i) reducing the temporal or spatial resolution of each layer feature map of the multi-layer feature map, ii) reducing the number of layers through fusion between layer feature maps of the multi-layer feature map, and iii) expressing each layer feature map of the multi-layer feature map or the fused feature map between layers by compressing it into a smaller number of channels.

[0131] 2) The reduced feature map conversion means takes the reduced feature map y as input and performs one or more of the detailed processes of the reduced feature map conversion means below, and then converts the converted feature map y. c can be output. This conversion process can be used to convert the format of the reduced feature map so that it can be used as input to the reduced feature map encoding means, or to improve the encoding efficiency of the reduced feature map encoding and decoding means. However, the reduced feature map conversion means can be omitted depending on the type of the reduced feature map encoding means.

[0132] The detailed process of the reduced feature map transformation means may include i) rearranging the dimensions of the feature map and ii) performing quantization on the component values ​​of the feature map.

[0133] 3) The reduced feature map encoding means is the transformed feature map y c It can take as input and output an encoded bitstream.

[0134] The reduced feature map encoding means can be performed through an encoder of a standard video codec such as HEVC (High Efficiency Video Coding) or VVC (Versatile Video Coding). Alternatively, the reduced feature map encoding means can be performed through an entropy encoding method based on an artificial neural network.

[0135] A multi-layer feature map decoding device for restoring a multi-layer feature map from a bitstream output through the multi-layer feature map encoding method described above may include 4) a reduced feature map decoding means, 5) a reduced feature map inverse transformation means, and 6) a multi-layer feature map restoration means.

[0136] The process of performing multi-layer feature map decoding can be explained as follows through steps 4) to 6) in Figure 7. (The order of the encoding process may be reversed.)

[0137] 4) The reduced feature map decoding means takes the encoded bitstream as input and converts it into a transformed feature map ^y. c can be restored. (Note that the restored data, i.e. the intermediate and final outputs from the decryptor, are marked with the ^( ) (hat) symbol.)

[0138] The reduced feature map decoding means can use a decoder for a standard video codec such as HEVC (High Efficiency Video Coding) or VVC (Versatile Video Coding). Alternatively, the reduced feature map decoding means can use an entropy decoding method based on an artificial neural network.

[0139] 5) The reduced feature map inverse transformation means is the reverse process of the reduced feature map transformation means of the multi-layer feature map encoding method, and the transformed feature map ^y cAfter performing one or more of the detailed processes of the reduced feature map inverse transformation means below with input as input, the reduced feature map ^y can be restored. However, the reduced feature map inverse transformation means may be omitted depending on the type of the reduced feature map decoding means. For example, if the type of the reduced feature map decoding means uses an entropy decoding method based on an artificial neural network, the reduced feature map inverse transformation means may be omitted. In other words, the reduced feature map inverse transformation means may be performed only when the type of the reduced feature map decoding means uses an entropy decoding method based on a standard video codec.

[0140] The detailed process of the reduced feature map inverse transformation means may include i) rearranging the dimensions of the feature map and ii) performing inverse quantization on the component values ​​of the feature map.

[0141] 6) The multi-layer feature map restoration means can be used to restore the original multi-layer feature map from the information contained in the reduced feature map. This multi-layer feature map restoration means takes the reduced feature map ^y as input, performs one or more of the detailed processes of the multi-layer feature map restoration means below, and then restores the multi-layer feature map {^F k} (k=1,…,L) can be restored. The detailed process of the multi-layer feature map restoration method below can be performed independently, but two or more processes can be combined and performed as one process.

[0142] The detailed process of the multi-layer feature map restoration means may include i) a process of restoring each layer of the original multi-layer feature map from the reduced feature map, ii) a process of restoring the temporal or spatial resolution of the reduced feature map or the restored hierarchical feature map, and iii) a process of restoring the number of channels of the reduced feature map or the restored hierarchical feature map.

[0143] In the present disclosure, a multi-layer feature map reduction block may mean a unit of a structure composed of a neural network block commonly used in an artificial neural network-based image processing technique or a classical filtering technique, dimension reduction and expansion technique for image processing.

[0144] A multi-layer feature map reduction block can have as input one or more of the layer feature maps that constitute a multi-layer feature map and the output of another multi-layer feature map reduction block.

[0145] A multi-layer feature map reduction block may include neural network structures such as a convolutional neural network layer, a transposed convolutional neural network layer, a fully connected layer, and a vision transformer.

[0146] Additionally, the multi-layer feature map reduction block may include non-linear functions such as ReLU (Rectified Linear Unit), Tanh (Tangent hyperbolic), and Sigmoid function.

[0147] Additionally, the multi-layer feature map reduction block may include element-wise arithmetic operations such as residual (skip) connection.

[0148] Additionally, the multi-layer feature map reduction block may include traditional downsampling techniques such as bilinear interpolation and bicubic interpolation.

[0149] Additionally, the multi-layer feature map reduction block may include traditional upsampling techniques such as bilinear extrapolation and bicubic extrapolation.

[0150] In addition, the multi-layer feature map reduction block can perform dimensionality reduction or dimensionality expansion on the input by adjusting the stride of the neural network structure (convolution layer, fully connected layer) used or the number of nodes in the hidden layer, and in particular, can increase or decrease the spatial resolution (width, height) of the input.

[0151] In the following description, a multi-layer feature map reduction block (s↑) may refer to a multi-layer feature map reduction block that increases the horizontal and vertical sizes of the input by s times, and a multi-layer feature map reduction block (s↓) may refer to a multi-layer feature map reduction block that reduces the horizontal and vertical sizes of the input by s times.

[0152] In the present disclosure, a multi-layer feature map restoration block may mean a unit of a structure composed of a neural network block commonly used in an artificial neural network-based image processing technique or a classical filtering technique, dimension reduction and expansion technique for image processing.

[0153] The multi-layer feature map restoration block can take as input one or more of a reduced feature map generated by fusing and transforming a multi-layer feature map and the output of another multi-layer feature map restoration block.

[0154] The multi-layer feature map restoration block may include neural network structures such as a convolutional neural network layer, a transposed convolutional neural network layer, a fully connected layer, and a vision transformer.

[0155] Additionally, the multi-layer feature map restoration block may include non-linear functions such as ReLU (Rectified Linear Unit), Tanh (Tangent hyperbolic), and Sigmoid function.

[0156] Additionally, the multi-layer feature map restoration block may include element-wise arithmetic operations such as residual (skip) connection.

[0157] Additionally, the multi-layer feature map restoration block may include traditional downsampling techniques such as bilinear interpolation and bicubic interpolation.

[0158] Additionally, the multi-layer feature map restoration block may include traditional upsampling techniques such as bilinear extrapolation and bicubic extrapolation.

[0159] In addition, the multi-layer feature map restoration block can perform dimensionality reduction or dimensionality expansion on the input by adjusting the stride of the neural network structure (convolution layer, fully connected layer) used or the number of nodes in the hidden layer, and in particular, can increase or decrease the spatial resolution (width, height) of the input.

[0160] In the following description, a multi-layer feature map restoration block (s↑) may refer to a multi-layer feature map restoration block that increases the horizontal and vertical sizes of the input by s times, and a multi-layer feature map restoration block (s↓) may refer to a multi-layer feature map restoration block that reduces the horizontal and vertical sizes of the input by s times.

[0161] Figure 8 illustrates an example of a multi-layer feature map reduction means.

[0162] For convenience of explanation, this example assumes that a P-layer feature map among multi-layer feature maps is encoded.

[0163] Figure 8 can represent the structure of FENet (Fusion and Encoding Network), which is a multi-layer feature map reduction means.

[0164] FENet consists of a series of multi-layer feature map reduction blocks (multi-layer feature map reduction blocks 1, 2, 3, 4), and can sequentially fuse and transform multi-layer feature maps to output a reduced feature map y.

[0165] Multi-layer feature map reduction block 1 can take the feature map p2 of the lowest layer as input and output a latent feature map y1 with a spatial resolution reduced by ½ times in width and ½ times in height compared to the input, and with the number of channels P.

[0166] Multi-layer feature map reduction block 2 can take as input the latent feature map y1 and the layer feature map p3 with the same spatial resolution as the latent feature map, and output the latent feature map y2 with a spatial resolution reduced by ½ times in width and ½ times in height compared to the input, and with the number of channels P.

[0167] Multi-layer feature map reduction block 3 can take as input a latent feature map y2 and a layer feature map p4 with the same spatial resolution as the latent feature map, and output a latent feature map y3 with a spatial resolution reduced by ½ times in width and ½ times in height compared to the input, and with the number of channels P.

[0168] Multi-layer feature map reduction block 4 can take as input the latent feature map y3 and the layer feature map p5 with the same spatial resolution as it, and output the latent feature map y4 with a spatial resolution reduced by ½ times in width and ½ times in height compared to the input, and with the number of channels P.

[0169] The fourth intermediate latent feature map y4 can be viewed as a reduced feature map y, and can additionally undergo channel-specific scaling by the gain unit.

[0170] Figure 9 illustrates an example of a specific configuration of the multi-layer feature map reduction blocks used in Figure 8.

[0171] Figures 10 and 11 illustrate examples of the detailed structure of the neural network block used in Figure 9.

[0172] Figure 12 illustrates a first embodiment of a multi-layer feature map restoration means.

[0173] The multi-layer feature map restoration means of this example may be a parallel multi-layer feature map restoration means that independently restores each layer feature map using a multi-layer feature map restoration block from the same input ^y.

[0174] For convenience of explanation, in this example, it is assumed that the reduced feature map ^y output after feature reduction is performed on the P-layer feature map is used as input to decode the P-layer feature map among the multi-layer feature maps. However, the reduced feature map ^y may be compressed and restored by the reduced feature map encoder and decoder.

[0175] Figure 12 can represent the structure of DRNet (Decoding and Reconstruction Network), which is a multi-layer feature map restoration means.

[0176] DRNet consists of a series of multi-layer feature map restoration blocks 1, 2, 3, 4 and inter-layer compensation blocks 1, 2, 3, which restore each layer feature map of the multi-layer feature map to form a multi-layer feature map {^P k} (k=2,…,5) can be printed.

[0177] The reduced feature map ^y can be additionally scaled per channel by the Inverse Gain unit.

[0178] Multi-layer feature map restoration block 1, multi-layer feature map restoration block 2, multi-layer feature map restoration block 3, and multi-layer feature map restoration block 4 can restore each layer feature map ^p2, ^p3, ^p4, and ^p5 by taking the reduced feature map ^y as input, respectively.

[0179] Figure 13 illustrates an example of the configuration of the multi-layer feature map restoration block and neural network blocks for additional information used in Figure 12.

[0180] TConv5x5(P,P) ↑2 can represent a transposed convolutional neural network with a normal 5x5 sized kernel and a stride of 2.

[0181] Figure 14 illustrates a second embodiment of a multi-layer feature map restoration means.

[0182] The multi-layer feature map restoration means of this example may be a resolution top-down sequential multi-layer feature map restoration means that first restores the layer feature map with the largest spatial resolution through a multi-layer feature map restoration block, and then uses the restored layer feature map as an input to the next multi-layer feature map restoration block to restore the feature map of the next layer with a smaller spatial resolution.

[0183] The multi-layer feature map restoration means in this example consists of a series of multi-layer feature map restoration blocks 1, 2, 3, 4, which take the reduced feature map ^y as input and P feature map {^P k} (k=2,…,5) Prints out.

[0184] Multi-layer feature map restoration block 1 can restore the feature map ^P2 of the lowest layer by taking the reduced feature map ^y as input.

[0185] Multi-layer feature map restoration block 2 can restore the feature map ^P3 of the next layer by using the layer feature map ^P2, which is the output of multi-layer feature map restoration block 1, as input.

[0186] Multi-layer feature map restoration block 3 can restore the feature map ^P4 of the next layer by using the layer feature map ^P3, which is the output of multi-layer feature map restoration block 2, as input.

[0187] Multi-layer feature map restoration block 4 can restore the feature map ^P5 of the next layer by using the layer feature map ^P4, which is the output of multi-layer feature map restoration block 3, as input.

[0188] A description of the components of each multi-layer feature map restoration block may be as follows.

[0189] UpBlock ↑2 can be composed of a combination of a pre-convolution layer with a 3x3 or 5x5 kernel with a stride of 2, a Batch Normalization layer, which is mainly used in artificial neural network-based image processing, and a P-ReLU activation function.

[0190] Reconst. Block ↓2 can be a convolution layer with 3x3 or 5x5 kernels with stride 2.

[0191] A Conv block can be a convolutional layer with 3x3 or 5x5 kernels with stride 1.

[0192] Figure 15 illustrates a third embodiment of a multi-layer feature map restoration means.

[0193] The multi-layer feature map restoration means of this example may be a parallel multi-layer feature map restoration means that independently restores each layer feature map using a multi-layer feature map restoration block from the same input ^y.

[0194] The multi-layer feature map restoration means in this example consists of a series of multi-layer feature map restoration blocks 1, 2, 3, 4 and inter-layer compensation blocks 1, 2, 3, and takes a reduced feature map ^y as input and generates P feature maps {^p k} (k=2,…,5) can be printed.

[0195] Multi-layer feature map restoration block 1, multi-layer feature map restoration block 2, multi-layer feature map restoration block 3, and multi-layer feature map restoration block 4 can restore each layer feature map ^p2, ^p3, ^p4, and ^p5 by taking the reduced feature map ^y as input, respectively.

[0196] The components of each multi-layer feature map restoration block and inter-layer compensation block may be described as follows.

[0197] A Conv block can be a convolutional layer with 3x3 or 5x5 kernels with stride 1.

[0198] Upsample ↑2 blocks can be nearest neighbor interpolation, bilinear interpolation, or bicubic interpolation.

[0199] Figure 16 illustrates an example of a parallel multi-layer feature map restoration means.

[0200] The parallel multi-layer feature map restoration means is characterized by restoring each layer feature map in parallel regardless of the spatial resolution of the layer feature map, as shown in Fig. 16, and can additionally be characterized by having the same input of all multi-layer feature map restoration blocks as the reduced feature map ^y.

[0201] Figure 17 illustrates an example of a resolution top-down sequential multi-layer feature map restoration means.

[0202] The resolution top-down sequential multi-layer feature map restoration means is characterized by sequentially restoring hierarchical feature maps having increasingly smaller spatial resolutions starting from a hierarchical feature map having the largest spatial resolution, as shown in FIG. 17, and at this time, the reduced feature map ^y is used as an input of the first multi-layer feature map restoration block, and the hierarchical feature map having a larger spatial resolution restored in the previous order is used as an input of the multi-layer feature map restoration block in the subsequent order.

[0203] Figure 18 illustrates an example of a resolution-upward sequential multi-layer feature map restoration means.

[0204] The resolution bottom-up sequential multi-layer feature map restoration means is characterized by sequentially restoring hierarchical feature maps having increasingly larger spatial resolutions starting from a hierarchical feature map having the smallest spatial resolution, as shown in FIG. 18, and at this time, the reduced feature map ^y is used as an input of the first multi-layer feature map restoration block, and the hierarchical feature map having a smaller spatial resolution restored in the previous order is used as an input of the multi-layer feature map restoration block in the subsequent order.

[0205] For reference, hierarchical feature maps ^F of Figs. 16 to 29 k It can be assumed that the spatial resolution increases as the layer index k increases.

[0206] Neural network-based multi-layer feature map encoding is performed through neural networks composed of learnable parameters. These neural networks are called neural network-based multi-layer feature map encoding models, and the learning of these neural networks can be performed through a backpropagation algorithm that updates the parameters and weights of the neural network in a direction that minimizes a specific loss function calculated from training data.

[0207] When training a neural network-based multi-layer feature map encoding model, either the rate-distortion optimization method or the rate-performance optimization method can be used.

[0208] The rate-distortion optimization method is a learning method that simultaneously minimizes the distortion between the input multi-layer feature map and the restored multi-layer feature map and the bit rate of the bitstream transmitted from the encoder to the decoder.

[0209] The rate-performance optimization method is a learning method that simultaneously minimizes the performance of machine tasks performed through restored multi-layer feature maps and the bit rate of the bitstream transmitted from the encoder to the decoder.

[0210] The distortion loss function used in the neural network-based multi-layer feature map encoding model is as shown in Equation 1, where the input multi-layer feature map {F k} (k=1,…,L) and restored multilayer feature map {^F k} (k=1,…,L) The mean square error (MSE) or multi-scale structural similarity index measure (MS-SSIM) is calculated by weighting w for each layer. k It can be used by weighting the sum.

[0211] The bit rate can be approximated by the cross-entropy between the probability distribution of the latent expression estimated by the latent expression probability model and the actual latent expression, as in Equation 2.

[0212] Machine work loss function L P It refers to the performance of machine tasks performed from compressed and restored multi-layer feature maps, and can be calculated through a comparison between the correct label and the inference result of the machine task.

[0213] At this time, classification loss function, bounding box loss function, mask loss function, etc. can be used depending on the type of machine task.

[0214] Loss function L for rate-distortion optimization RD can be expressed as Equation 3, and the distortion loss function L D and cross entropy-based loss function L RA constant λ is used to determine the ratio between the two, and depending on λ, the desired restoration level and bit rate for the output of the trained model can be determined. (Generally, the larger λ, the higher the restoration level.)

[0215] Loss function L for rate-performance optimization RP can be expressed as in Equation 4, and the performance loss function L P and cross entropy-based loss function L R A constant λ is used to determine the ratio between the two, and the desired restoration level and bit rate for the output of the trained model can be determined based on λ. (Generally, the larger λ, the higher the performance of the machine task.)

[0216] Formula 1

[0217]

[0218] Formula 2

[0219]

[0220] Formula 3

[0221]

[0222] Formula 4

[0223]

[0224] Figure 19 illustrates an example of a parallel multi-layer feature map restoration means and the amount of information.

[0225] In the case of the parallel multi-layer feature map restoration means, as shown in Fig. 19, since the input of all multi-layer feature map restoration blocks is the reduced feature map ^y, there is an advantage in that restoration performance is improved by utilizing all information of the reduced feature map ^y as the input of each multi-layer feature map restoration block without information loss, but the computational complexity may increase because the structural redundancy increases.

[0226] Figure 20 illustrates an example of a resolution-upward sequential multi-layer feature map restoration means and the amount of information.

[0227] Figure 21 illustrates an example of a resolution top-down sequential multi-layer feature map restoration means and the amount of information.

[0228] In the case of the resolution bottom-up sequential multilayer feature map restoration means and the resolution top-down sequential multilayer feature map restoration means, the layer feature map ^F restored in the previous order as in FIG. 20 and FIG. 21 (k-1) (or ^F (k+1) ) hierarchical feature map ^F k By using the multilayer feature map restoration block as an input for restoring the feature map ^y, there may be an advantage in minimizing the neural network structure and computational complexity compared to the method of independently restoring each layer feature map from the reduced feature map ^y. However, from the perspective of information flow, the layer feature map ^F restored in the previous order (k-1) (or ^F (k+1) ) contains less information than the reduced feature map ^y, so the amount of information flowing to the multi-layer feature map restoration block used in the subsequent sequence may decrease, resulting in a decrease in restoration performance.

[0229] The thickness of the arrows in Figures 19 to 21 can indicate the amount of information.

[0230] Feature map adaptation is a method to not limit the information flow in the bottom-up sequential multi-layer feature map restoration means and the top-down sequential multi-layer feature map restoration means, and in the process of restoring each layer feature map, the output of the multi-layer feature map restoration block is restored as is. k A latent feature map that contains all the information needed to restore the feature maps later, rather than using ^~F (1,…,k) (or ^~F (k,…,L) ) and converts it through additional image processing techniques or neural network blocks to finally restore the hierarchical feature map ^F k It may be a way to output .

[0231] The combination of additional image processing techniques or neural network blocks used at this time is called a feature map adaptation block.

[0232] The feature map adaptation block of the present disclosure is characterized by transforming the input feature map to be adapted similarly to the feature map to be restored without changing the spatial resolution of the input and output feature maps.

[0233] Figure 22 illustrates an example of a resolution-upward sequential multi-layer feature map restoration means based on feature map adaptation and the amount of information.

[0234] Referring to FIG. 22, a resolution-up sequential multi-layer feature map restoration means using one or more feature map adaptation blocks can be referred to as a resolution-up sequential multi-layer feature map restoration means based on feature map adaptation.

[0235] Figure 23 illustrates an example of a resolution top-down sequential multi-layer feature map restoration means based on feature map adaptation and the amount of information.

[0236] Referring to FIG. 23, a resolution top-down sequential multi-layer feature map restoration means using one or more feature map adaptation blocks can be referred to as a resolution top-down sequential multi-layer feature map restoration means based on feature map adaptation.

[0237] Figure 24 illustrates an example of a resolution-upward sequential multi-layer feature map restoration means based on feature map adaptation omitting the last feature map adaptation block and the amount of information.

[0238] Figure 25 illustrates an example of a resolution top-down sequential multi-layer feature map restoration means based on feature map adaptation omitting the last feature map adaptation block and the amount of information.

[0239] Feature map adaptation is a method to not limit the information flow in the bottom-up sequential multi-layer feature map restoration method and the top-down sequential multi-layer feature map restoration method, and the intermediate latent feature map ^~F (1,…,k) (or ^~F (k,…,L) ) and feature map adaptation block ^Fk This is a method for differentiating information. However, since the output of the multi-layer feature map restoration block 4, ~^F4 (or ~^F1), does not pass through an additional multi-layer feature map restoration block thereafter, there is no need to differentiate information from the final output feature map, ~^F4 (or ~^F1). In other words, the feature map adaptation block corresponding to the multi-layer feature map restoration block 4 can be omitted or have a lightweight structure.

[0240] Figure 26 illustrates an example of a multi-layer feature map reduction means based on latent feature map adaptation.

[0241] The feature map adaptation described above can be applied in a similar manner to multi-layer feature map reduction methods.

[0242] Feature map adaptation applied to multi-layer feature map reduction means can be called latent feature map adaptation means.

[0243] Latent feature map adaptation may be a method for transforming the input feature map of each layer in a multi-layer feature map reduction means into a latent space common to the latent feature maps combined with it across channels.

[0244] Latent feature map y in the channel-to-channel combination process of each layer (k-1) The input feature map that is combined with p (k+1) Instead of doing it as y, we transform it through a latent feature map adaptation means. (k-1) ~p which has a common potential space with (k+1) It can be characterized by being.

[0245] At this time, the latent feature map adaptation means can be configured through a combination of conventional image processing techniques or neural network blocks.

[0246] In the examples of multi-layer feature map restoration means shown in Fig. 22, Fig. 23, Fig. 24 and Fig. 25, a common latent feature map (ex.~^F 1234) and the information of the feature map of the layer to be currently restored are separated from each other, and a feature map adaptation block is used to transform the latent feature map to be restored to be similarly adapted to the target feature map without changing the spatial resolution. Similarly, a latent feature map adaptation block can be used in a multi-layer feature map reduction means such as that of FIG. 26 to transform the feature map of the layer to be currently encoded to be similarly adapted to the latent feature map.

[0247] Specifically, in the multi-layer feature map reduction means shown in Fig. 8, the feature map of the kth layer, p k Through this multi-layer feature map reduction method, y (k-1) is converted to , and then y (k-1) Wow p (k+1) These channels can be combined. However, y (k-1) is a latent feature map and p (k+1) Since it is a feature map that is not transformed, it is not a feature map that exists in the same space. Therefore, as in Fig. 26, p (k+1) y goes through the latent feature map adaptation block (k-1) ~p( which has a potential space like k+1) After being converted to y (k-1) It can be characterized by being connected to a channel.

[0248] A multi-layer feature map reduction means using one or more latent feature map adaptation blocks is called a multi-layer feature map reduction means based on latent feature map adaptation and can be represented as shown in Fig. 26.

[0249] Figure 27 illustrates an example of a multi-layer feature map reduction means based on latent feature map adaptation omitting the first latent feature map adaptation block.

[0250] Feature map adaptation that omits the last feature map adaptation may explain that the latent feature map adaptation block can be omitted if information differentiation is not required between the output feature map and the latent feature map. Similarly, in latent feature map adaptation that omits the first latent feature map adaptation, the latent feature map adaptation block can be omitted if there is no cross-channel coupling between the input feature map and the latent feature map.

[0251] Specifically, for the input feature map of the first layer, the latent feature map adaptation block can be omitted because it is not combined with the latent feature map and entered into the multi-layer feature map reduction block.

[0252] A multi-layer feature map reduction means based on latent feature map adaptation omitting the first latent feature map adaptation block can be represented as shown in Fig. 27.

[0253] A resolution bottom-up sequential multi-layer feature map restoration method based on feature map adaptation and a resolution top-down sequential multi-layer feature map restoration method based on feature map adaptation, in which the latent feature map is the output of the multi-layer feature map restoration block ~^F (1,…,k) (or ~^F (k,…,L) ) may be a method for making the multi-layer feature map restoration block lightweight while minimizing the restriction of information flow by making the number of channels different.

[0254] In a resolution bottom-up sequential multi-layer feature map restoration method based on feature map adaptation as in Fig. 22, latent feature map ~^F 1234 , ~^F 234 , ~^F 34 , ~^F4, by making the number of channels of the latent feature map with a small spatial resolution larger than or equal to the number of channels of the latent feature map with a large spatial resolution smaller than or equal to the number of channels of the latent feature map with a large spatial resolution, the outputs of each multi-layer feature map reconstruction block can be maintained to have similar dimensions and information content. That is, ~^F 1234 , ~^F 234 , ~^F 34, ~^F4, if the number of channels is c1, c2, c3, c4, then c1≥c2≥c3≥c4 can be satisfied.

[0255] Similarly, in the resolution top-down sequential multi-layer feature map restoration method based on feature map adaptation such as Fig. 23, the latent feature map ~^F 1234 , ~^F 123 , ~^F 12 , ~^F1, by making the number of channels of the latent feature map with a small spatial resolution larger than or equal to the number of channels of the latent feature map with a large spatial resolution smaller than or equal to the number of channels of the latent feature map with a large spatial resolution, the outputs of each multi-layer feature map restoration block can be maintained to have similar dimensions and information content. That is, ~^F 1234 , ~^F 123 , ~^F 12 , ~^F1, if the number of channels is c1, c2, c3, c4, then c1≤c2≤c3≤c4 is satisfied.

[0256] In particular, since the output latent feature map of the sequentially preceding multi-layer feature map restoration block is used as the input of the subsequent multi-layer feature map restoration block in the resolution-bottom sequential sequential multi-layer feature map restoration means, it is more desirable to use this asymmetrical channel count usage method in the case of the resolution-bottom sequential multi-layer feature map restoration means, since the output latent feature map of the sequentially preceding multi-layer feature map restoration block is used as the input of the subsequent multi-layer feature map restoration block, the later latent feature map having a larger number of channels in the sequentially preceding latent feature map has a greater effect in minimizing the restriction of information flow.

[0257] As a method for reducing the weight of a parallel multi-layer feature map restoration means, an additional multi-layer feature map restoration block that takes a reduced feature map ^y as input before a parallel multi-layer feature map restoration block may be used to minimize the complexity of the subsequent parallel multi-layer feature map restoration block.

[0258] FIG. 28 illustrates an embodiment of a drawing in which a multi-layer feature map restoration block 0 is added to a parallel multi-layer feature map restoration means and the amount of information.

[0259] For example, if the spatial resolutions of the hierarchical feature maps ^F1, ^F2, ^F3, and ^F4 are 2, 4, 8, and 16 times greater than the spatial resolution of the reduced feature map ^y, respectively, then the multi-layer feature map restoration blocks 1, 2, 3, and 4 must include the neural network or image processing technique for doubling the spatial resolution 1, 2, 3, and 4, respectively. At this time, if the multi-layer feature map restoration block 0, which doubles the spatial resolution of the input, is added as shown in FIG. 28, the number of neural networks or image processing techniques for doubling the spatial resolution can be reduced to 0, 1, 2, and 3 in the subsequent multi-layer feature map restoration blocks 1, 2, 3, and 4, respectively, thereby reducing structural redundancy and reducing complexity without information loss.

[0260] FIG. 29 illustrates an embodiment of a drawing in which a multi-layer feature map restoration block 0 is added to a parallel multi-layer feature map restoration means and the amount of information.

[0261] The spatial resolution increased in these additional multi-layer feature map restoration blocks is not specified as being doubled, and if multi-layer feature map restoration block 0 that increases the spatial resolution by four times as shown in FIG. 29 is added, then in multi-layer feature map restoration block 1, the neural network or image processing technique for reducing the spatial resolution by two times may be included once, and in multi-layer feature map restoration blocks 2, 3, and 4, the neural network or image processing technique for increasing the spatial resolution by two times may be included less often, 0, 1, and 2 times, respectively.

[0262] Inter-layer compensation refers to the layer feature map to be restored in the multi-layer feature map restoration method ^F k Feature maps of adjacent layers in the restoration process ^F (k-1) (or ^F (k+1)) may mean utilizing additional information. This inter-layer compensation may be configured using conventional image processing techniques or artificial neural network layers, and in the present disclosure, this may be referred to as an inter-layer compensation block.

[0263] Figure 30 illustrates an example of resolution-upward inter-layer compensation.

[0264] The method of utilizing a feature map with a small spatial resolution to restore a feature map with a large spatial resolution, as shown in Fig. 30, can be called resolution-up interlayer compensation.

[0265] Figure 31 illustrates an example of resolution top-down inter-layer compensation.

[0266] As shown in Fig. 31, a method of utilizing a feature map with a large spatial resolution to restore a feature map with a small spatial resolution can be classified into resolution top-down interlayer compensation.

[0267] Considering the structural processing delay, it may be more efficient to align the direction of inter-layer compensation with the direction of multi-layer feature map restoration.

[0268] For example, for a resolution-upward sequential multi-layer feature map restoration method, using resolution-upward inter-layer compensation may be more efficient from a structural processing delay perspective.

[0269] Figure 32 illustrates an example in which a feature map adaptation block and an inter-layer compensation block are used together.

[0270] As shown in Figure 32, the feature map adaptation block and the inter-layer compensation block can be used together, and the inter-layer compensation block can be positioned sequentially before or after the feature map adaptation block. In this case, positioning the feature map adaptation block sequentially before the inter-layer compensation block can be more efficient from the perspectives of information flow and complexity.

[0271] Figure 33 illustrates an embodiment of a multi-layer feature map encoding method and device.

[0272] In one embodiment, a multi-layer feature map encoding method and device may be characterized by including, when encoding a multi-layer feature map extracted from an input image, [E2] a multi-layer feature map reduction means for reducing the dimension of the extracted multi-layer feature map and outputting a reduced feature map; and [E3] a reduced feature map encoding means for encoding the reduced feature map as an input and outputting a bitstream.

[0273] The above-mentioned reduced feature map encoding means may be characterized by being one of [E3-1] reduced feature map encoding using an image compression codec and [E3-2] reduced feature map encoding using entropy encoding based on an artificial neural network.

[0274] The reduced feature map encoding using the above image compression codec can be performed by including: [E3-1-1] a feature map conversion means for performing processes such as quantization, padding or resizing, and rearrangement on the reduced feature map to make it into a format suitable for encoding; and [E3-1-2] an encoder of the image compression codec for encoding the converted feature map in the form of a bitstream.

[0275] The reduced feature map encoding using the artificial neural network-based entropy encoding may be performed by including: [E3-2-1] a feature map quantization means for quantizing the reduced feature map; [E3-2-2] a feature map probability distribution estimation means for estimating a probability distribution of the quantized feature map; and [E3-2-3] a feature map entropy encoding means for entropy encoding the quantized feature map using the estimated feature map probability distribution.

[0276] Figure 34 illustrates an embodiment of a multi-layer feature map decoding method and device.

[0277] In one embodiment, a multi-layer feature map decoding method and device may include, in decoding a multi-layer feature map from an input bitstream, [D1] a reduced feature map decoding means for decoding a reduced feature map from an input bitstream; and [D2] a multi-layer feature map restoration means for restoring a multi-layer feature map using the decoded reduced feature map as an input.

[0278] The above-mentioned reduced feature map decoding means may be characterized by being one of [D1-1] reduced feature map decoding using an image compression codec and [D1-2] reduced feature map decoding using entropy decoding based on an artificial neural network.

[0279] In performing the reduced feature map decoding using the above image compression codec, the reduced feature map decoding using the image compression codec may include [D1-1-1] a decoder of the image compression codec for decoding a feature map converted from an input bitstream, and [D1-1-2] a feature map inverse transformation means for performing the reverse process of the quantization, padding, or size adjustment, and rearrangement process on the converted feature map to output the reduced feature map.

[0280] In performing the reduced feature map decoding using the artificial neural network-based entropy decoding, the reduced feature map decoding using the artificial neural network-based entropy decoding may include [D1-2-1] a feature map probability distribution estimation means for estimating the probability distribution of a quantized feature map, and [D1-2-2] a feature map entropy decoding means for entropy-decoding the quantized feature map using the estimated feature map probability distribution.

[0281] In this embodiment, a multi-layer feature map {F k} (k=1,…,L) This particular P-layer feature map {p k} (k=2,…,5)Although the present disclosure is described by taking as an example the case where Darknet 53 is used as a feature extractor, such as in the YOLO v3 model, the present disclosure is not limited to such multi-layer feature maps and can be similarly applied to conventional multi-layer feature maps.

[0282] Figure 35 illustrates an example of a neural network block.

[0283] The FeatureMixing(P, P) block of the present disclosure can represent a neural network block as in FIG. 35.

[0284] x in Fig. 35 H can mean the feature map with a larger spatial resolution among the two inputs of FeatureMixing.

[0285] x in Fig. 35 L can mean the feature map with a smaller spatial resolution among the two inputs of FeatureMixing.

[0286] The ResBlock(P, P) block can represent a neural network block as shown in Fig. 10.

[0287] The AttnBlock(P, P) block can represent a neural network block as in Fig. 11.

[0288] A ConvNxN(A, B) block can mean a convolutional neural network layer that uses kernels of size NХN and has input channels A and output channels B.

[0289] When it is denoted as ConvNxN(A, B) ↓, it indicates a decrease in spatial resolution of the output feature map compared to the input feature map, and s can represent the stride of the convolutional neural network layer. If there is no notation, it can indicate the case where s=1.

[0290] The TconvNxN(A, B) block can mean a transposed convolutional neural network layer that uses a kernel of size NХN and has input channels A and output channels B.

[0291] When TconvNxN(A, B) ↑s is expressed as an increase in the spatial resolution of the output feature map compared to the input feature map, s can represent the stride of the transposed convolutional neural network layer. If there is no indication, it can represent the case where s=1.

[0292] ReLU, LeakyReLU, and Sigmoid can refer to nonlinear layers commonly used in neural networks for image processing.

[0293] (+) indicates element-wise addition between two feature maps, and Concat indicates concatenation between channels. ⓧ indicates element-wise multiplication between two feature maps.

[0294] Figure 36 is a drawing for explaining an example of SubpelPixel.

[0295] Figure 37 shows an example of using DepthConv3x3(P, P) and CONV1x1(P, P / 3) together.

[0296] Figures 38, 39 and 40 illustrate examples of spatial mixing and channel mixing.

[0297] Embodiments of the multi-layer feature map reduction means and multi-layer feature map restoration means of the present disclosure may be examples using layers such as ConvNxN, TconvNxN, etc., and neural network blocks such as ResBlock, AttnBlock, etc. In this case, some layers or neural network blocks may be modified and used to reduce complexity. These examples are described below.

[0298] Example 1) In the case of ConvNxN and TconvNxN, the complexity of the corresponding layer can be adjusted by changing the size of N.

[0299] For example, if you use layers such as Conv3x3 or Conv2x2 instead of Conv5x5, you can reduce the complexity.

[0300] For example, if you use layers such as Tconv3x3 or Tconv2x2 instead of Tconv5x5, you can reduce the complexity.

[0301] Additionally, for TconvNxN, it can be performed using a method such as pixel shuffle shown in Fig. 36.

[0302] Additionally, in the case of ConvNxN, it can be replaced with DepthConvNxN, etc. as shown in Fig. 37.

[0303] Example 2) Resblock, AttnBlock, etc. used in the present disclosure can be replaced with low complexity blocks. In this case, low complexity blocks may often perform spatial mixing and channel mixing separately.

[0304] The pre-convolution operation (CONV-1) can be replaced with subpelPixel. SubpelPixel is a technique that increases the number of channels of the input tensor x through the CONV operation and then rearranges them (pixelSuffle) in units of pixels to change them to high resolution. In the case of subpelPixel(M, P, 2↑), it can input a tensor with M channels and output a P-channel tensor with twice the resolution in width and height.

[0305] Referring to FIG. 36 for an example of SubpelPixel, FIG. 36 can represent an example in which the number of input channels M is 1 and the number of output channels P is 1.

[0306] When the input tensor with resolution f1×f2 is x, x increases the number of channels through the CONV5x5(1, n_1) operation (n 1>1 ).

[0307] After that CONV3x3(n1, n2)... CONV3x3(n(l-2) , n (l-1) ) operations can be added. (n i and l can be set arbitrarily)

[0308] The tensor immediately preceding the high-resolution output is r 2 (n l ) can have a number of channels.

[0309] r 2 The feature map with the number of channels can be rearranged pixel by pixel to produce a final output with a resolution increased by r times in width and height.

[0310] DepthConv3x3(P, P) is mainly CONV1x1(P, C out ) can be used with.

[0311] Figure 37 can show an example of using DepthConv3x3(P, P) and CONV1x1(P, P / 3) together.

[0312] A typical CONV3x3(P, P) can process all channels of an input tensor simultaneously (i.e., it can include operations between channels). It can take as input x with P channels and output x' with P channels.

[0313] However, DepthConv3x3(P,P) can process each input channel independently and combine them later.

[0314] In Fig. 37, when P is 3, x with 3 channels is input, and CONV3x3(1, 1) operation is performed on each of the 3 channels to output 3 tensors with 1 channel, and then the final ^x' with 3 channels can be output through channel combination.

[0315] DepthConv3x3(P,P) has a disadvantage in that it cannot utilize information between channels because there is no cross-channel operation compared to CONV3x3(P, P). Therefore, in the general case, CONV1x1(P, C) is followed by DepthConv3x3(P, P).out ) can be used to perform cross-channel operations.

[0316] In Fig. 37, C out This could be the case for 1.

[0317] Figures 38, 39 and 40 illustrate examples of spatial mixing and channel mixing.

[0318] The structure of channel mixing can have a structure that performs attention with low complexity, including gating, which divides the path into two paths with the same number of channels, as shown in Fig. 40, and performs an element-wise product operation of the result of one path with the result of the other path.

[0319] The channel split used in FIG. 38, FIG. 39, and FIG. 40 may divide the number of channels of the input tensor into two or three groups.

[0320] For example, when dividing the number of channels into two groups, if the input tensor x has the shape [B, C1, H, W] (B: batch size, C: number of channels, HxW: resolution), then when going through channel split, x is divided into x1 which is [B, C1, H, W] and x2 which is [B, C2, H, W]. Here, C1+C2=C.

[0321] Figure 41 shows an example of the definition of WSiLU used in Figures 38 and 39.

[0322] WSiLU (Weighted SiLU) can be a structure that multiplies the learnable parameter w to the SiLU activation function.

[0323] Weighted SiLU (WSiLU) can also be used as an activation function that reduces the number of channels in an input tensor. Below is an example of a channel-separated activation function.

[0324] Here, x with C channels can be divided into x1 and x2 through channel separation. At this time, the number of channels of x1 and x2 can be the same as C / 2.

[0325] This may enable more sophisticated information selection than a simple activation function, x=x1·WSiLU(x2).

[0326] This allows to halve the number of channels in a conventional input.

[0327] Figure 42 illustrates an embodiment of a multi-layer feature map reduction means.

[0328] Figure 42 shows an embodiment of a multi-layer feature map reduction means, and P layer feature map {p k} (k=2,…,5) It can output a reduced feature map y by taking as input.

[0329] Multi-layer feature map reduction block 1 can take as input the layer feature map p2 with the largest spatial resolution and output a latent feature map ~p2 with both width and height reduced to ½ size.

[0330] Multi-layer feature map reduction block 2 takes as input the layer feature map p3 with the same spatial resolution as ~p2 and connects them across channels to produce a latent feature map ~p that is reduced to ½ in both width and height. 23 can be printed.

[0331] Multilayer feature map reduction block 3 is ~p 23 With the layer feature map p4 with the same spatial resolution as it, the latent feature map ~p is input with both width and height reduced to ½ size. 234 can be printed.

[0332] Multilayer feature map reduction block 4 is ~p 234 A reduced feature map y=~p with both width and height reduced to ½ size by inputting a layer feature map p5 with the same spatial resolution as it and connecting the channels. 2346 can be printed.

[0333] At this time, P and M, the number of channels constituting each neural network layer, can be set to 192, 320, etc., respectively, and F, the number of channels of the P layer feature map, can be 256.

[0334] Figure 43 illustrates an embodiment of a multi-layer feature map reduction means.

[0335] Figure 43 shows an additional embodiment of a multi-layer feature map reduction means, and can be used as an example when Darknet 53 is used as a feature extractor as in the YOLO v3 model. That is, the feature map to be encoded {F k} (k=1,…,3) This may be an example of a case where the number of layers is 3.

[0336] Multi-layer feature map reduction block 1 can take as input the layer feature map F1 with the largest spatial resolution and output the latent feature map ~F1 with both width and height reduced to ½ size.

[0337] Multi-layer feature map reduction block 2 takes as input the layer feature map F2 with the same spatial resolution as ~F1 and concatenates them across channels to produce a latent feature map ~F, which is reduced in both width and height to ½ size. 12 can be printed.

[0338] Multilayer feature map reduction block 4 is ~F 12 A reduced feature map y=~F with both width and height reduced to ½ size by inputting a layer feature map F3 with the same spatial resolution as it and connecting the channels. 123 can be printed.

[0339] At this time, P and M, which are the number of channels constituting each neural network layer, can be set to 192, 320, etc., respectively, and F can be set to 128 or 256, etc., depending on the channel configuration for each layer of the feature map extracted from Darknet 53.

[0340] Figure 44 illustrates an example of extracting a reduced feature map so that the resolution is the same as that of the feature map with the smallest resolution among the input feature maps.

[0341] Figure 45 illustrates an example of extracting a reduced feature map so that the resolution is the same as that of the second smallest feature map.

[0342] The above embodiments may be examples in which the resolution of the reduced feature map is reduced by half in both width and height compared to the resolution of the feature map with the smallest resolution among the input feature maps. However, the multi-layer feature map reduction means of the present disclosure can extract reduced feature maps with various resolutions, such as the resolutions of feature maps in other layers. For example, the reduced feature map can be extracted so that the resolution is the same as that of the feature map with the smallest resolution among the input feature maps, or the reduced feature map can be extracted so that the resolution is the same as that of the feature map with the second smallest resolution.

[0343] Figure 44 shows the P layer feature map {p k} (k=2,…,5) By taking as input, a reduced feature map y having the same resolution as p5, which has the smallest resolution among the input feature maps, can be output. This embodiment shows only some of the order of various multi-layer feature map reduction blocks and the configuration method of neural network blocks, and the resolution of the reduced feature map can be made to p5 level using various methods.

[0344] Multi-layer feature map reduction block 1 can take as input the layer feature map p2 with the largest spatial resolution and output a latent feature map ~p2 with both width and height reduced to ½ size.

[0345] Multi-layer feature map reduction block 2 takes as input the layer feature map p3 with the same spatial resolution as ~p2 and connects them across channels to produce a latent feature map ~p that is reduced to ½ in both width and height. 23 can be printed.

[0346] Multilayer feature map reduction block 3 is ~p 23With the layer feature map p4 with the same spatial resolution as it, the latent feature map ~p is input with both width and height reduced to ½ size. 234 can be printed.

[0347] Multilayer feature map reduction block 4 is ~p 234 With the same spatial resolution as that of the layer feature map p5, the reduced feature map y=~p is input as the channel-to-channel concatenation. 2345 can be output. Here, the spatial resolution of the reduced feature map y can be equal to p5.

[0348] At this time, P and M, the number of channels constituting each neural network layer, can be set to 192, 320, etc., respectively, and F, the number of channels of the P layer feature map, can be 256.

[0349] Figure 45 illustrates an example of the order of multi-layer feature map reduction blocks and the method of configuring neural network blocks.

[0350] Figure 45 shows the P layer feature map {p k} (k=2,…,5) With as input, a reduced feature map y having the same resolution as p4, which has the second smallest resolution among the input feature maps, can be output. This embodiment shows only some of the order of various multi-layer feature map reduction blocks and the configuration method of neural network blocks, and the resolution of the reduced feature map can be made to p4 level using various methods.

[0351] Multi-layer feature map reduction block 1 can take as input the layer feature map p2 with the largest spatial resolution and output a latent feature map ~p2 with both width and height reduced to ½ size.

[0352] Multi-layer feature map reduction block 2 takes as input the layer feature map p3 with the same spatial resolution as ~p2 and connects them across channels to produce a latent feature map ~p that is reduced to ½ in both width and height. 23 can be printed.

[0353] Multi-layer feature map reduction block 3 can take as input the layer feature map p5 with the smallest spatial resolution and output a latent feature map ~p5 whose size is doubled in both width and height.

[0354] Multilayer feature map reduction block 4 has the same spatial resolution ~p 23 , p4 and ~p5 are connected between channels as input, and the reduced feature map y=~p 2345 can be output. Here, the spatial resolution of the reduced feature map y can be equal to p4.

[0355] At this time, P and M, the number of channels constituting each neural network layer, can be set to 192, 320, etc., respectively, and F, the number of channels of the P layer feature map, can be 256.

[0356] In the present disclosure, a lightweight structure can be proposed to reduce the complexity of the last block of a multi-layer feature map reduction means and increase performance.

[0357] The last block of the multi-layer feature map reduction means may refer to multi-layer feature map reduction block 4 in FIG. 42 and multi-layer feature map reduction block 3 in FIG. 43. In other words, it may refer to a multi-layer feature map reduction block that outputs a reduced feature map y.

[0358] The number of channels in the input feature map of the last block is c in , and the number of channels of the output feature map is c out When said, c of the last block shown in Fig. 42 and Fig. 43 in is P+F or P+4F, and c out is M. M is ultimately the number of channels of the reduced feature map y, and can be designed to have a sufficiently large number of channels. For example, M can be set to a number greater than P or F.

[0359] The first layer (Conv5x5) of the last block shown in Figures 42 and 43 is c inInput a feature map with the number of channels c out It can output a feature map with the number of channels c. However, as described above, out =Since M is usually set to a large number, the complexity of the first layer can increase significantly.

[0360] Figure 46 illustrates an embodiment of the last block of the multi-layer feature map reduction means.

[0361] In this disclosure, the first layer of the last block is c as shown in Fig. 46. in We input a feature map with the number of channels Q( <M)의 채널 수를 갖는 특징맵을 출력할 수 있다. 첫 번째 레이어 이후의 레이어는 Q의 채널 수를 갖는 특징맵을 입력 받아 M의 채널 수를 갖는 특징맵을 출력할 수 있다. c in Input a feature map with the number of channels c out Instead of outputting a feature map with the number of channels c in We input a feature map with the number of channels Q( <M)의 채널 수를 갖는 특징맵을 출력하고, 이후 레이어는 Q의 채널 수를 갖는 특징맵을 입력 받아 M의 채널 수를 갖는 특징맵을 출력하므로, 첫 번째 레이어의 복잡도를 감소시킬 수 있으며, Q를 충분히 작은 값으로 설정하면 이후 레이어의 복잡도 또한 낮기 때문에 전체적으로 마지막 블록의 복잡도를 감소시킬 수 있다.

[0362] Figure 47 illustrates an embodiment of the last block of the multi-layer feature map reduction means.

[0363] Figure 48 illustrates an embodiment of the last block of the multi-layer feature map reduction means.

[0364] Here, the layers after the first layer of the last block are described in detail. As shown in Fig. 46, a lightweight structure such as conv1x1 can be used to input a feature map with Q channels and output a feature map with M channels. In addition, as shown in Fig. 47, after the first layer that outputs a feature map with Q channels, layers that input a feature map with Q channels and output a feature map with Q channels can be added to ensure sufficient spatial transformation, and then a layer that converts from Q to the final output channel size M is passed. As shown in Fig. 48, after adding a layer that inputs a feature map with Q channels and outputs a feature map with M channels, layers that input a feature map with M channels and output a feature map with M channels can be added thereafter. In both cases of FIG. 47 and FIG. 48, additional performance improvements can be achieved through sufficient spatial transformation, and in particular, when using blocks such as those in FIG. 47, additional complexity reductions can be achieved.

[0365] Figure 51 illustrates an embodiment of a multi-layer feature map reduction means based on latent feature map adaptation.

[0366] Additionally, Fig. 51 may illustrate the additional use of a latent feature map adaptation block in a multi-layer feature map reduction means.

[0367] Latent feature map adaptation block 1 can take hierarchical feature map p2 as input and output latent feature map ~p2.

[0368] Latent feature map adaptation block 2 can take hierarchical feature map p3 as input and output latent feature map ~p3.

[0369] Latent feature map adaptation block 3 can take hierarchical feature map p4 as input and output latent feature map ~p4.

[0370] Latent feature map adaptation block 4 can take hierarchical feature map p5 as input and output latent feature map ~p5.

[0371] Multi-layer feature map reduction block 1 can take a latent feature map ~p2 as input and output a latent feature map y1 that is reduced to ½ in both width and height.

[0372] Multi-layer feature map reduction block 2 can take as input the latent feature map y1 and the latent feature map ~p3 with the same resolution as it, and output the latent feature map y2 with both the width and height reduced to ½.

[0373] Multi-layer feature map reduction block 3 can take as input the latent feature map y2 and the latent feature map ~p4 with the same resolution as it, which is a combination of channels, and output the latent feature map y3 with both the width and height reduced to ½.

[0374] Multi-layer feature map reduction block 4 can take as input the latent feature map y3 and the latent feature map ~p5 with the same resolution as it, which is a combination of channels, and output the latent feature map y4, which is reduced in both width and height to ½.

[0375] Figure 52 illustrates an embodiment of a multi-layer feature map reduction means based on latent feature map adaptation.

[0376] Additionally, Fig. 52 may be an example in which a latent feature map adaptation block with lower complexity is applied compared to Fig. 51.

[0377] Latent feature map adaptation block 1 can take hierarchical feature map p2 as input and output latent feature map ~p2.

[0378] Latent feature map adaptation block 2 can take hierarchical feature map p3 as input and output latent feature map ~p3.

[0379] Latent feature map adaptation block 3 can take hierarchical feature map p4 as input and output latent feature map ~p4.

[0380] Latent feature map adaptation block 4 can take hierarchical feature map p5 as input and output latent feature map ~p5.

[0381] Multi-layer feature map reduction block 1 can take a latent feature map ~p2 as input and output a latent feature map y1 that is reduced to ½ in both width and height.

[0382] Multi-layer feature map reduction block 2 can take as input the latent feature map y1 and the latent feature map ~p3 with the same resolution as it, and output the latent feature map y2 with both the width and height reduced to ½.

[0383] Multi-layer feature map reduction block 3 can take as input the latent feature map y2 and the latent feature map ~p4 with the same resolution as it, which is a combination of channels, and output the latent feature map y3 with both the width and height reduced to ½.

[0384] Multi-layer feature map reduction block 4 can take as input the latent feature map y3 and the latent feature map ~p5 with the same resolution as it, which is a combination of channels, and output the latent feature map y4, which is reduced in both width and height to ½.

[0385] Figure 53 illustrates an embodiment of a multi-layer feature map reduction means based on latent feature map adaptation where the first latent feature map adaptation block is omitted.

[0386] Additionally, FIG. 53 can show that a latent feature map adaptation block is additionally used for input layer maps excluding the first input layer map in a multi-layer feature map reduction means.

[0387] Latent feature map adaptation block 2 can take hierarchical feature map p3 as input and output latent feature map ~p3.

[0388] Latent feature map adaptation block 3 can take hierarchical feature map p4 as input and output latent feature map ~p4.

[0389] Latent feature map adaptation block 4 can take hierarchical feature map p5 as input and output latent feature map ~p5.

[0390] Multi-layer feature map reduction block 1 can take a layered feature map p2 as input and output a latent feature map y1 that is reduced to ½ in both width and height.

[0391] Multi-layer feature map reduction block 2 can take as input the latent feature map y1 and the latent feature map ~p3 with the same resolution as it, and output the latent feature map y2 with both the width and height reduced to ½.

[0392] Multi-layer feature map reduction block 3 can take as input the latent feature map y2 and the latent feature map ~p4 with the same resolution as it, which is a combination of channels, and output the latent feature map y3 with both the width and height reduced to ½.

[0393] Multi-layer feature map reduction block 4 can take as input the latent feature map y3 and the latent feature map ~p5 with the same resolution as it, which is a combination of channels, and output the latent feature map y4, which is reduced in both width and height to ½.

[0394] Figure 54 illustrates an embodiment of a multi-layer feature map reduction means based on latent feature map adaptation where the first latent feature map adaptation block is omitted.

[0395] Additionally, Fig. 54 can represent an example in which a latent feature map adaptation block having lower complexity than Fig. 53 is applied.

[0396] Latent feature map adaptation block 2 can take hierarchical feature map p3 as input and output latent feature map ~p3.

[0397] Latent feature map adaptation block 3 can take hierarchical feature map p4 as input and output latent feature map ~p4.

[0398] Latent feature map adaptation block 4 can take hierarchical feature map p5 as input and output latent feature maps ~p6.

[0399] Multi-layer feature map reduction block 1 can take a layered feature map p2 as input and output a latent feature map y1 that is reduced to ½ in both width and height.

[0400] Multi-layer feature map reduction block 2 can take as input the latent feature map y1 and the latent feature map ~p3 with the same resolution as it, and output the latent feature map y2 with both the width and height reduced to ½.

[0401] Multi-layer feature map reduction block 3 can take as input the latent feature map y2 and the latent feature map ~p4 with the same resolution as it, which is a combination of channels, and output the latent feature map y3 with both the width and height reduced to ½.

[0402] Multi-layer feature map reduction block 4 can take as input the latent feature map y3 and the latent feature map ~p5 with the same resolution as it, which is a combination of channels, and output the latent feature map y4, which is reduced in both width and height to ½.

[0403] Figures 49 and 50 illustrate examples of implementations of a multi-layer feature map reduction means that apply weight reduction to the last block of the multi-layer feature map reduction means.

[0404] The case of Fig. 49 is an example of an implementation using four multi-layer feature map reduction blocks, and the case of Fig. 50 may be an example of an implementation using three multi-layer feature map reduction blocks.

[0405] The last block of Fig. 49 and Fig. 50 is c, as in Fig. 47. inIt is possible to input a feature map with a number of channels of M and output a feature map with a number of channels of Q that is smaller than M. In addition, layers that input a feature map with a number of channels of Q and output a feature map with a number of channels of Q can be added to ensure sufficient spatial transformation. Afterwards, a layer that transforms from Q to the final output channel size M can be passed. Since sufficient spatial transformation is performed, additional performance improvement is expected, and in cases where the complexity of the added layers is not large compared to the complexity reduction of the first layer (conv5x5), a complexity reduction can also be expected.

[0406] At this time, it may be more efficient to make the number of intermediate channels Q used in the last multi-layer feature map reduction block the same as the number of output channels P of the previous multi-layer feature map reduction block.

[0407] FIG. 55 illustrates an embodiment of a resolution top-down sequential multi-layer feature map restoration means based on feature map adaptation.

[0408] Referring to Figure 55, we can see an example of additionally using a feature map adaptation block in a top-down sequential multi-layer feature map restoration process that sequentially restores layer feature maps with increasingly smaller resolutions starting from a layer feature map with the largest spatial resolution.

[0409] Multilayer feature map restoration block 1 takes the reduced feature map ^y as input and creates a latent feature map ~^p that is sized 16 times in both width and height. 2345 can be printed.

[0410] Multilayer feature map restoration block 2 is a latent feature map ~^p 2345 The latent feature map with both width and height reduced to ½ size using ~^p 345 can be printed.

[0411] Multilayer feature map restoration block 3 is a latent feature map ~^p 345The latent feature map with both width and height reduced to ½ size using ~^p 45 can be printed.

[0412] Multilayer feature map restoration block 4 is the latent feature map ~^p 45 By inputting , we can output a latent feature map ~^p5 with both width and height reduced to ½ size.

[0413] Feature map adaptation block 1 is a latent feature map ~^p 2345 The final restored layer feature map ^p2 can be output by taking as input.

[0414] Feature map adaptation block 2 is a latent feature map ~^p 345 The final restored layer feature map ^p3 can be output by taking as input.

[0415] Feature map adaptation block 3 is a latent feature map ~^p 45 The final restored layer feature map ^p4 can be output by taking as input.

[0416] Feature map adaptation block 4 can take the latent feature map ~^p5 as input and output the final restored layer feature map ^p5.

[0417] At this time, P and M, the number of channels constituting each neural network layer, can be set to 192, 320, etc., respectively, and F, the number of channels of the P layer feature map, can be 256.

[0418] The above embodiments have described examples in which the resolution of the reduced feature map is reduced by half in both width and height compared to the resolution of the feature map with the smallest resolution among the restored feature maps. However, the multi-layer feature map restoration means of the present disclosure can restore a multi-layer feature map from reduced feature maps of various resolutions, such as the resolutions of feature maps in other layers. For example, a multi-layer feature map can be restored from a reduced feature map having the same resolution as the feature map with the smallest resolution among the restored feature maps, or a multi-layer feature map can be restored from a reduced feature map having the same resolution as the feature map with the second-smallest resolution.

[0419] Figure 58 illustrates an example of restoring a multi-layer feature map from a reduced feature map having the same resolution as the feature map with the smallest resolution among the input feature maps.

[0420] Figure 59 illustrates an example of restoring a multi-layer feature map from a downscaled feature map whose resolution is the same as that of the second smallest feature map.

[0421] Fig. 58 may represent another specific embodiment for Fig. 25. In Fig. 58, unlike Fig. 55, the size of the spatial resolution of ^y may be equal to ^p5.

[0422] Multilayer feature map restoration block 1 takes the reduced feature map ^y as input and creates a latent feature map ~^p that is 8 times larger in both width and height. 2345 can be printed.

[0423] Multilayer feature map restoration block 2 is a latent feature map ~^p 2345 The latent feature map with both width and height reduced to ½ size using ~^p 345 can be printed.

[0424] Multilayer feature map restoration block 3 is a latent feature map ~^p 345 The latent feature map with both width and height reduced to ½ size using ~^p 45 can be printed.

[0425] Multilayer feature map restoration block 4 is the latent feature map ~^p 45 By inputting , we can output a latent feature map ~^p5 with both width and height reduced to ½ size.

[0426] Feature map adaptation block 1 is a latent feature map ~^p 2345 The final restored layer feature map ^p2 can be output by taking as input.

[0427] Feature map adaptation block 2 is a latent feature map ~^p 345 The final restored layer feature map ^p3 can be output by taking as input.

[0428] Feature map adaptation block 3 is a latent feature map ~^p 45 The final restored layer feature map ^p4 can be output by taking as input.

[0429] Since the latent feature map ~^p5 does not pass through an additional multi-layer feature map restoration block, there is no need to restrict the information flow. Therefore, the feature map adaptation block corresponding to multi-layer feature map restoration block 4 can be omitted.

[0430] At this time, P and M, the number of channels constituting each neural network layer, can be set to 192, 320, etc., respectively, and F, the number of channels of the P layer feature map, can be 256.

[0431] Fig. 59 illustrates another specific embodiment for Fig. 25.

[0432] In Fig. 59, unlike Fig. 55, the size of the spatial resolution of ^y can be the same as ^p4.

[0433] Multilayer feature map restoration block 1 takes the reduced feature map ^y as input and expands the latent feature map ~^p to 4 times the size in both width and height. 2345 can be printed.

[0434] Multilayer feature map restoration block 2 is a latent feature map ~^p 2345The latent feature map with both width and height reduced to ½ size using ~^p 345 can be printed.

[0435] Multilayer feature map restoration block 3 is a latent feature map ~^p 345 The latent feature map with both width and height reduced to ½ size using ~^p 45 can be printed.

[0436] Multilayer feature map restoration block 4 is the latent feature map ~^p 45 By inputting , we can output a latent feature map ~^p5 with both width and height reduced to ½ size.

[0437] Feature map adaptation block 1 is a latent feature map ~^p 2345 The final restored layer feature map ^p2 can be output by taking as input.

[0438] Feature map adaptation block 2 is a latent feature map ~^p 345 The final restored layer feature map ^p3 can be output by taking as input.

[0439] Feature map adaptation block 3 is a latent feature map ~^p 45 The final restored layer feature map ^p4 can be output by taking as input.

[0440] Since the latent feature map ~^p5 does not pass through an additional multi-layer feature map restoration block, there is no need to restrict the information flow. Therefore, the feature map adaptation block corresponding to multi-layer feature map restoration block 4 can be omitted.

[0441] At this time, P and M, the number of channels constituting each neural network layer, can be set to 192, 320, etc., respectively, and F, the number of channels of the P layer feature map, can be 256.

[0442] FIG. 56 illustrates an embodiment of a resolution top-down sequential multi-layer feature map restoration means based on feature map adaptation.

[0443] Fig. 56 may represent an embodiment using a feature map adaptation block having more layers compared to Fig. 55.

[0444] Multilayer feature map restoration block 1 takes the reduced feature map ^y as input and creates a latent feature map ~^p that is sized 16 times in both width and height. 2345 can be printed.

[0445] Multilayer feature map restoration block 2 is a latent feature map ~^p 2345 The latent feature map with both width and height reduced to ½ size using ~^p 345 can be printed.

[0446] Multilayer feature map restoration block 3 is a latent feature map ~^p 345 The latent feature map with both width and height reduced to ½ size using ~^p 45 can be printed.

[0447] Multilayer feature map restoration block 4 is the latent feature map ~^p 45 By inputting , we can output a latent feature map ~^p5 with both width and height reduced to ½ size.

[0448] Feature map adaptation block 1 is a latent feature map ~^p 2345 The final restored layer feature map ^p2 can be output by taking as input.

[0449] Feature map adaptation block 2 is a latent feature map ~^p 345 The final restored layer feature map ^p3 can be output by taking as input.

[0450] Feature map adaptation block 3 is a latent feature map ~^p 45 The final restored layer feature map ^p4 can be output by taking as input.

[0451] Feature map adaptation block 4 can take the latent feature map ~^p5 as input and output the final restored layer feature map ^p5.

[0452] At this time, P and M, the number of channels constituting each neural network layer, can be set to 192, 320, etc., respectively, and F, the number of channels of the P layer feature map, can be 256.

[0453] FIG. 57 illustrates an embodiment of a resolution top-down sequential multi-layer feature map restoration means based on feature map adaptation omitting the last feature map adaptation block.

[0454] Fig. 57 can represent a specific embodiment for Fig. 25. In Fig. 57, the last feature map adaptation block used in Fig. 56 is omitted, and instead, a conv layer is added to match the number of output channels to the last multi-layer feature map restoration block.

[0455] Multilayer feature map restoration block 1 takes the reduced feature map ^y as input and creates a latent feature map ~^p that is sized 16 times in both width and height. 2345 can be printed.

[0456] Multilayer feature map restoration block 2 is a latent feature map ~^p 2345 The latent feature map with both width and height reduced to ½ size using ~^p 345 can be printed.

[0457] Multilayer feature map restoration block 3 is a latent feature map ~^p 345 The latent feature map with both width and height reduced to ½ size using ~^p 45 can be printed.

[0458] Multilayer feature map restoration block 4 is the latent feature map ~^p 45 By inputting , we can output a latent feature map ~^p5 with both width and height reduced to ½ size.

[0459] Feature map adaptation block 1 is a latent feature map ~^p 2345 The final restored layer feature map ^p2 can be output by taking as input.

[0460] Feature map adaptation block 2 is a latent feature map ~^p345 The final restored layer feature map ^p3 can be output by taking as input.

[0461] Feature map adaptation block 3 is a latent feature map ~^p 45 The final restored layer feature map ^p4 can be output by taking as input.

[0462] Since the latent feature map ~^p5 does not pass through an additional multi-layer feature map restoration block, there is no need to restrict the information flow. Therefore, the feature map adaptation block corresponding to multi-layer feature map restoration block 4 can be omitted.

[0463] At this time, P and M, the number of channels constituting each neural network layer, can be set to 192, 320, etc., respectively, and F, the number of channels of the P layer feature map, can be 256.

[0464] FIG. 60 illustrates an embodiment of a resolution-upward sequential multi-layer feature map restoration means based on feature map adaptation.

[0465] FIG. 60 illustrates an embodiment of a resolution-upward sequential multi-layer feature map restoration means based on feature map adaptation, showing the additional use of a feature map adaptation block in a bottom-up sequential multi-layer feature map restoration process that sequentially restores layer feature maps having increasingly larger resolutions starting from a layer feature map having the smallest spatial resolution.

[0466] Multilayer feature map restoration block 1 takes the reduced feature map ^y as input and creates a latent feature map ~^p that is doubled in both width and height. 2345 can be printed.

[0467] Multilayer feature map restoration block 2 is a latent feature map ~^p 2345 The latent feature map is doubled in both width and height by using ~^p 234 can be printed.

[0468] Multilayer feature map restoration block 3 is a latent feature map ~^p 234The latent feature map is doubled in both width and height by using ~^p 23 can be printed.

[0469] Multilayer feature map restoration block 4 is the latent feature map ~^p 23 By inputting , we can output a latent feature map ~^p2 that is double in both width and height.

[0470] Feature map adaptation block 1 is a latent feature map ~^p 2345 The final restored layer feature map ^p5 can be output by taking as input.

[0471] Feature map adaptation block 2 is a latent feature map ~^p 234 The final restored layer feature map ^p4 can be output by taking as input.

[0472] Feature map adaptation block 3 is a latent feature map ~^p 23 The final restored layer feature map ^p3 can be output by taking as input.

[0473] Feature map adaptation block 4 can take a latent feature map ~^p2 as input and output the final restored layer feature map ^p2.

[0474] At this time, P and M, the number of channels constituting each neural network layer, can be set to 192, 320, etc., respectively, and F, the number of channels of the P layer feature map, can be 256.

[0475] FIG. 61 illustrates an embodiment of a resolution-upward sequential multi-layer feature map restoration means based on feature map adaptation.

[0476] Fig. 61 can represent an embodiment using a feature map adaptation block having more layers compared to Fig. 60.

[0477] Multilayer feature map restoration block 1 takes the reduced feature map ^y as input and creates a latent feature map ~^p that is doubled in both width and height. 2345 can be printed.

[0478] Multilayer feature map restoration block 2 is a latent feature map ~^p 2345 The latent feature map is doubled in both width and height by using ~^p 234 can be printed.

[0479] Multilayer feature map restoration block 3 is a latent feature map ~^p 234 The latent feature map is doubled in both width and height by using ~^p 23 can be printed.

[0480] Multilayer feature map restoration block 4 is the latent feature map ~^p 23 By inputting , we can output a latent feature map ~^p2 that is double in both width and height.

[0481] Feature map adaptation block 1 is a latent feature map ~^p 2345 The final restored layer feature map ^p5 can be output by taking as input.

[0482] Feature map adaptation block 2 is a latent feature map ~^p 234 The final restored layer feature map ^p4 can be output by taking as input.

[0483] Feature map adaptation block 3 is a latent feature map ~^p 23 The final restored layer feature map ^p3 can be output by taking as input.

[0484] Feature map adaptation block 4 can take a latent feature map ~^p2 as input and output the final restored layer feature map ^p2.

[0485] At this time, P and M, the number of channels constituting each neural network layer, can be set to 192, 320, etc., respectively, and F, the number of channels of the P layer feature map, can be 256.

[0486] FIG. 62 illustrates an embodiment of a resolution-upward sequential multi-layer feature map restoration means based on feature map adaptation omitting the last feature map adaptation block.

[0487] Fig. 62 may represent a specific embodiment for Fig. 24. In Fig. 62, the last feature map adaptation block used in Fig. 61 is omitted, and instead, a conv layer is added to match the number of output channels to the last multi-layer feature map restoration block.

[0488] Multilayer feature map restoration block 1 takes the reduced feature map ^y as input and creates a latent feature map ~^p that is doubled in both width and height. 2345 can be printed.

[0489] Multilayer feature map restoration block 2 is a latent feature map ~^p 2345 The latent feature map is doubled in both width and height by using ~^p 234 can be printed.

[0490] Multilayer feature map restoration block 3 is a latent feature map ~^p 234 The latent feature map is doubled in both width and height by using ~^p 23 can be printed.

[0491] Multilayer feature map restoration block 4 is the latent feature map ~^p 23 By inputting , we can output a latent feature map ~^p2 that is double in both width and height.

[0492] Feature map adaptation block 1 is a latent feature map ~^p 2345 The final restored layer feature map ^p5 can be output by taking as input.

[0493] Feature map adaptation block 2 is a latent feature map ~^p 234 The final restored layer feature map ^p4 can be output by taking as input.

[0494] Feature map adaptation block 3 is a latent feature map ~^p 23 The final restored layer feature map ^p3 can be output by taking as input.

[0495] Since the latent feature map ~^p2 does not pass through an additional multi-layer feature map restoration block, there is no need to restrict the information flow. Therefore, the feature map adaptation block corresponding to multi-layer feature map restoration block 4 can be omitted.

[0496] At this time, P and M, the number of channels constituting each neural network layer, can be set to 192, 320, etc., respectively, and F, the number of channels of the P layer feature map, can be 256.

[0497] Figure 63 illustrates an embodiment of a resolution top-down sequential multi-layer feature map restoration means based on feature map adaptation using an asymmetric number of channels considering information flow and spatial resolution.

[0498] Figure 63 shows that an asymmetric number of channels can be used considering information flow and spatial resolution in a resolution top-down sequential multi-layer feature map restoration means based on feature map adaptation.

[0499] Multilayer feature map restoration block 1 takes the reduced feature map ^y as input and creates a latent feature map ~^p that is sized 16 times in both width and height. 2345 can be printed.

[0500] Multilayer feature map restoration block 2 is a latent feature map ~^p 2345 The latent feature map with both width and height reduced to ½ size using ~^p 345 can be printed.

[0501] Multilayer feature map restoration block 3 is a latent feature map ~^p 345 The latent feature map with both width and height reduced to ½ size using ~^p 45 can be printed.

[0502] Multilayer feature map restoration block 4 is the latent feature map ~^p 45 By inputting , we can output a latent feature map ~^p5 with both width and height reduced to ½ size.

[0503] Feature map adaptation block 1 is a latent feature map ~^p2345 The final restored layer feature map ^p2 can be output by taking as input.

[0504] Feature map adaptation block 2 is a latent feature map ~^p 345 The final restored layer feature map ^p3 can be output by taking as input.

[0505] Feature map adaptation block 3 is a latent feature map ~^p 45 The final restored layer feature map ^p4 can be output by taking as input.

[0506] Feature map adaptation block 4 can take the latent feature map ~^p5 as input and output the final restored layer feature map ^p5.

[0507] At this time, the number of channels constituting each neural network layer can be set asymmetrically, such as M=320, c1=64, c2=128, c3=192, c4=256, and F, the number of channels of the P layer feature map, can be 256.

[0508] Figure 64 illustrates an embodiment of a resolution top-down sequential multi-layer feature map restoration means based on feature map adaptation using an asymmetric number of channels considering information flow and spatial resolution.

[0509] Figure 64 demonstrates that a top-down sequential multi-layer feature map restoration method based on feature map adaptation can utilize an asymmetric number of channels that take into account information flow and spatial resolution. In particular, it can be shown that the number of channels is used asymmetrically within multi-layer feature map restoration block 1, taking spatial resolution into account as well.

[0510] Multilayer feature map restoration block 1 takes the reduced feature map ^y as input and creates a latent feature map ~^p that is sized 16 times in both width and height. 2345 can be printed.

[0511] Multilayer feature map restoration block 2 is a latent feature map ~^p 2345The latent feature map with both width and height reduced to ½ size using ~^p 345 can be printed.

[0512] Multilayer feature map restoration block 3 is a latent feature map ~^p 345 The latent feature map with both width and height reduced to ½ size using ~^p 45 can be printed.

[0513] Multilayer feature map restoration block 4 is the latent feature map ~^p 45 By inputting , we can output a latent feature map ~^p5 with both width and height reduced to ½ size.

[0514] Feature map adaptation block 1 is a latent feature map ~^p 2345 The final restored layer feature map ^p2 can be output by taking as input.

[0515] Feature map adaptation block 2 is a latent feature map ~^p 345 The final restored layer feature map ^p3 can be output by taking as input.

[0516] Feature map adaptation block 3 is a latent feature map ~^p 45 The final restored layer feature map ^p4 can be output by taking as input.

[0517] Feature map adaptation block 4 can take the latent feature map ~^p5 as input and output the final restored layer feature map ^p5.

[0518] At this time, the number of channels constituting each neural network layer can be set asymmetrically, such as M=320, c1=64, c2=128, c3=192, c4=256, and F, the number of channels of the P layer feature map, can be 256.

[0519] Figure 65 illustrates an embodiment of a resolution-upward sequential multi-layer feature map restoration means based on feature map adaptation using an asymmetric number of channels considering information flow and spatial resolution.

[0520] Figure 65 shows that an asymmetric number of channels can be used considering information flow and spatial resolution in a resolution-up sequential multi-layer feature map restoration means based on feature map adaptation.

[0521] Multilayer feature map restoration block 1 takes the reduced feature map ^y as input and creates a latent feature map ~^p that is doubled in both width and height. 2345 can be printed.

[0522] Multilayer feature map restoration block 2 is a latent feature map ~^p 2345 The latent feature map is doubled in both width and height by using ~^p 234 can be printed.

[0523] Multilayer feature map restoration block 3 is a latent feature map ~^p 234 The latent feature map is doubled in both width and height by using ~^p 23 can be printed.

[0524] Multilayer feature map restoration block 4 is the latent feature map ~^p 23 By inputting , we can output a latent feature map ~^p2 that is double in both width and height.

[0525] Feature map adaptation block 1 is a latent feature map ~^p 2345 The final restored layer feature map ^p5 can be output by taking as input.

[0526] Feature map adaptation block 2 is a latent feature map ~^p 234 The final restored layer feature map ^p4 can be output by taking as input.

[0527] Feature map adaptation block 3 is a latent feature map ~^p 23 The final restored layer feature map ^p3 can be output by taking as input.

[0528] Feature map adaptation block 4 can take a latent feature map ~^p2 as input and output the final restored layer feature map ^p2.

[0529] At this time, the number of channels constituting each neural network layer can be set asymmetrically, such as M=320, c1=256, c2=192, c3=128, c4=64, and F, the number of channels of the P layer feature map, can be 256.

[0530] Figure 66 illustrates an embodiment of a sequential to parallel multi-layer feature map restoration means.

[0531] Figure 66 shows a serial-parallel multi-layer feature map restoration means, and shows that the complexity of the parallel-configured multi-layer feature map restoration block can be reduced overall by adding the multi-layer feature map restoration block 0 before the parallel-configured multi-layer feature map restoration blocks 1, 2, 3, and 4.

[0532] Considering that the spatial resolutions of the P-layer feature maps ^p2, ^p3, ^p4, and ^p5 are 16 times, 8 times, 4 times, and 2 times greater than the spatial resolution of the reduced feature map ^y, by adding a multi-layer feature map restoration block 0 that increases the spatial resolution by 2 times by taking the reduced feature map ^y as input, it can be shown that the spatial resolutions only need to be increased by 8 times, 4 times, 2 times, and 1 times in the multi-layer feature map restoration blocks 1, 2, 3, and 4 configured in parallel.

[0533] Multilayer feature map restoration block 0 takes the reduced feature map ^y as input and creates a latent feature map ~^p 2345 can be printed.

[0534] Multilayer feature map restoration block 1 is a latent feature map ~^p 2345 The final restored layer feature map ^p2 can be output by taking as input.

[0535] Multilayer feature map restoration block 2 is a latent feature map ~^p 2345 The final restored layer feature map ^p3 can be output by taking as input.

[0536] Multilayer feature map restoration block 3 is the latent feature map ~^p 2345The final restored layer feature map ^p4 can be output by taking as input.

[0537] Multilayer feature map restoration block 4 is a latent feature map ~^p 2345 The final restored layer feature map ^p5 can be output by taking as input.

[0538] Figure 67 illustrates an embodiment of a composite use of a feature map adaptation block and an inter-layer compensation block.

[0539] Figure 67 is an example of a case where feature map adaptation and inter-layer compensation are used in combination in a resolution-upward sequential multi-layer feature map restoration means based on feature map adaptation.

[0540] Multilayer feature map restoration block 1 takes the reduced feature map ^y as input and creates a latent feature map ~^p that is doubled in both width and height. 2345 can be printed.

[0541] Multilayer feature map restoration block 2 is a latent feature map ~^p 2345 The latent feature map is doubled in both width and height by using ~^p 234 can be printed.

[0542] Multilayer feature map restoration block 3 is a latent feature map ~^p 234 The latent feature map is doubled in both width and height by using ~^p 23 can be printed.

[0543] Multilayer feature map restoration block 4 is the latent feature map ~^p 23 By inputting , we can output a latent feature map ~^p2 that is double in both width and height.

[0544] Feature map adaptation block 1 is a latent feature map ~^p 2345 The final restored layer feature map ^p5 can be output by taking as input.

[0545] Feature map adaptation block 2 is a latent feature map ~^p 234The final restored layer feature map ^p4 can be output by taking as input.

[0546] Feature map adaptation block 3 is a latent feature map ~^p 23 The final restored layer feature map ^p3 can be output by taking as input.

[0547] Feature map adaptation block 4 can take a latent feature map ~^p2 as input and output the final restored layer feature map ^p2.

[0548] The inter-layer compensation block 1 can take the output of the feature map adaptation block 2 and ^p5 as inputs and output the final restored layer feature map ^p4.

[0549] The inter-layer compensation block 2 can take the output of the feature map adaptation block 3 and ^p4 as inputs and output the final restored layer feature map ^p3.

[0550] Inter-layer compensation block 3 can take the output of feature map adaptation block 4 and ^p3 as input and output the final restored layer feature map ^p2.

[0551] At this time, P and M, the number of channels constituting each neural network layer, can be set to 192, 320, etc., respectively, and F, the number of channels of the P layer feature map, can be 256.

[0552] At this time, the Upsample ↑2 block used in inter-layer compensation blocks 1, 2, and 3 may be nearest neighbor interpolation.

[0553] In the case of inter-layer compensation, a method such as a component-to-component operation of two feature maps or a channel-to-channel concatenation can be used to combine the output of the feature map adaptation block and the layer feature map to be used for compensation. In this case, since the channel-to-channel concatenation may increase the complexity of the inter-layer compensation block, it may be preferable to use a component-to-component operation of two feature maps, as in the embodiment of Fig. 67. The embodiment of Fig. 67 shows an example using a component-to-component sum as the component-to-component operation. Another method may be a method that uses a component-to-component product or a method that simultaneously uses a component-to-component sum and a product.

[0554] This disclosure primarily describes an embodiment in which the number of layers in a multi-layer feature map is four. However, this disclosure can operate for various numbers of layers.

[0555] Based on the examples of the multi-layer feature map reduction means of FIG. 49 and the multi-layer feature map restoration means of FIG. 57 described above, an embodiment when the number of layers changes is described below.

[0556] In the embodiment described below, the feature of the present disclosure, in which the feature maps output from the multi-layer feature map restoration blocks are restored as hierarchical feature maps of the multi-layer feature map through the feature map adaptation block, can still be effectively applied to the remaining blocks except for the last multi-layer feature map restoration block among the multi-layer feature map restoration blocks.

[0557] Figure 68 illustrates an example of a multi-layer feature map restoration means when the number of layers is three.

[0558] Figures 50 and 68 can show examples of a multi-layer feature map reduction means and a multi-layer feature map restoration means when the number of layers is three, respectively.

[0559] Figures 69 and 70 illustrate examples of a multi-layer feature map reduction means and a multi-layer feature map restoration means when the number of layers is two, respectively.

[0560] Figure 71 illustrates an example of a multi-layer feature map reduction means when the number of layers is 1.

[0561] Figures 72 and 73 illustrate examples of a multi-layer feature map restoration means when the number of layers is 1.

[0562] Here, Fig. 72 is an example in which a feature map adaptation block is used, and Fig. 73 may be an example in which the feature map adaptation block is omitted.

[0563] Figure 74 illustrates an example of reduced feature map entropy encoding.

[0564] Figure 74 is an example of reduced feature map entropy encoding, and explains specific examples of [E3-2] and [D1-2].

[0565] [E3-2-1] The Q block of Fig. 74 is an example of a reduced feature map quantization means, which quantizes all components y of the reduced feature map y. i It performs uniform quantization on (at this time, the usual rounding operation can be used), and can output the quantized reduced feature map ^y.

[0566] In neural network-based image compression, quantization can be replaced with a process of adding uniform noise between -0.5 and 0.5 instead of quantization at training time to enable the use of the backpropagation algorithm, and a rounding function can be used at inference time.

[0567] In addition, in order to reduce the mismatch between the learning process and the inference process, a rounding function such as is also used in the learning process. In this case, an identity function can be used when calculating the gradient for the rounding function.

[0568] [E3-2-2], [D1-2-1] The entropy model of Fig. 74 may be an example of a reduced feature map probability distribution estimation means.

[0569] For example, it can be characterized by first encoding hyperprior ^z, which is additional information (side information) from the reduced feature map ^y, and transmitting it to the decoder, thereby enabling more accurate probability distribution estimation for the reduced feature map.

[0570] Also, as an example, the components of the already decoded reduced feature map ^y <j ^y components of the reduced feature map to be encoded based on i Additional information (context) can be extracted and used as a feature for encoding.

[0571] In addition, the entropy model used in various neural network-based technologies below can be used as a means of estimating the probability distribution of reduced feature maps.

[0572] [E3-2-3], [D1-2-2] AE and AD of Fig. 74 are examples of a reduced feature map entropy encoding means and a reduced feature map entropy decoding means, respectively, and represent arithmetic encoding and arithmetic decoding. In addition to arithmetic encoding, range coding can also be used.

[0573] Figure 75 illustrates an example of spatially dividing a latent expression group into a checkerboard pattern.

[0574] Figure 76 illustrates an example of additionally utilizing information of already decoded latent expression groups when sequentially estimating probability distributions for latent expression groups.

[0575] SCCTX divides the latent representation for the image into five groups (^y) unequally in the channel direction, as shown in Figure 5 of Fig. 76. (1) , … , ^y (5) ), and then each latent expression group ^y (k)When sequentially estimating the probability distribution for the already decoded latent representation group ^y (<k) It includes a method of utilizing additional information. This additional information can be called channel context.

[0576] Additionally, each latent expression group ^y as in Fig. 75 (k) is spatially divided into two subgroups ^y1 in a checkerboard pattern. (1) , ^y1 (2) By dividing the data into subgroups, we can estimate the probability distribution of the remaining subgroup using one already encoded subgroup as additional information. This additional information can be called spatial context.

[0577] Finally, SCCTX can maximize the entropy encoding efficiency of the latent representation by dividing the latent representation into 10 groups and then sequentially performing a total of 10 probability distribution estimations (Figure 6 of FIG. 76).

[0578] For the reduced feature map of the present disclosure, this SCCTX can also be used as a means of estimating the reduced feature map probability distribution.

[0579] Figure 77 illustrates an example of a feature map encoding system.

[0580] The reduced feature map encoding and decoding using the image compression codec of the present disclosure can be configured similarly to the feature map encoding system of FIG. 77(a) below.

[0581] This feature map encoding system can be largely composed of a feature map extraction means (Feature Extraction), a feature map conversion means (Format Conversion), a feature map encoder / decoder (Feature Encoder / Decoder), and a feature map inverse conversion means (Feature Inverse-Conversion).

[0582] First, the feature map extraction means (Feature Extraction) can extract a feature map from an input image.

[0583] The extracted feature map can be converted into a format suitable for encoding through a feature map conversion means (Format Conversion) including processes such as quantization, padding or resizing, and rearrangement.

[0584] The transformed feature map is compressed through a feature map encoder and can be restored through a feature map decoder.

[0585] The restored feature map is then reconverted to a format similar to the originally extracted feature map through feature inverse conversion. This transformed feature map can then be input into a task execution mechanism for use in various machine tasks.

[0586] Figure 77 (a) shows an example of performing feature map encoding based on an encoder and decoder of an existing video compression codec such as HEVC or VVC, and (b) shows an example of using an artificial neural network for feature map encoding.

[0587] In order to use a feature map encoding system in various situations, it may be advantageous for the system to support various types of feature map extraction methods and feature map transformation methods. To this end, in the present disclosure, information regarding parameters or feature map extraction methods used in the feature map extraction process (feature map information) is encoded, if necessary, through a feature map encoder so that it can be finally transmitted to a task execution means, and information regarding parameters or feature map transformation methods used in the feature map transformation process (feature map transformation information) is also encoded, if necessary, so that it can be finally transmitted to a feature map inverse transformation means.

[0588] A feature of the reduced feature map encoding and decoding using the image compression codec of the present disclosure may be that the feature map encoding system is applied to the reduced feature map instead of the extracted feature map.

[0589] Figure 78 illustrates an example of a process for performing reduced feature map encoding and decoding using an image compression codec.

[0590] Figure 78 shows a process of performing reduced feature map encoding and decoding using a video compression codec, and can be configured with a feature map transformation and inverse transformation, and an encoder and decoder of a video compression codec.

[0591] The feature map transformation and inverse transformation process may include 1) arrangement and rearrangement means of feature map channels, 2) quantization and inverse quantization means of feature values, and 3) feature map transformation means.

[0592] Figure 79 illustrates an example of the configuration of a feature map.

[0593] Referring to Figure 79, the feature map can be composed of multiple channels. In addition, the number of channels is N C , the horizontal size of the channel is W C , the vertical size of the channel is H C can be expressed as

[0594] In order to encode multiple channels, these channels must be converted into frames (or pictures), which are the input units for encoding.

[0595] In the feature map arrangement means, one of the spatial arrangement method, the temporal arrangement method, and the spatio-temporal arrangement method can be used when converting channels into frames.

[0596] The feature map frame thus transformed can be compressed by an encoder of a video compression codec and restored by a decoder of the video compression codec. The restored feature map frame can be reconstructed into its original channel configuration form by a feature map rearrangement means.

[0597] Figure 80 shows an example of a spatially arranged feature map.

[0598] Figure 81 shows an example of a temporally arranged feature map.

[0599] Figure 82 shows an example of a spatiotemporally arranged feature map.

[0600] Spatial arrangement refers to arranging feature map channels m horizontally and n vertically in a tile-like shape to form one feature map frame, as shown in Fig. 80, and m and n can be set so that their product is equal to the total number of channels to be arranged, NC. The spatially arranged feature map frame can be encoded using an intra-frame prediction method in an encoding means.

[0601] Temporal arrangement refers to temporally arranging each feature map channel to become one frame, as shown in FIG. 81, and the total number of frames for one feature map can be set to be equal to the total number of channels NC constituting the feature map. The temporally arranged feature map frames can be encoded using the inter-frame prediction method in the encoder of the video compression codec. In the present disclosure, the term inter-frame prediction, which is widely used in video encoding standards, is used to help understanding, but it may be more appropriate to call it inter-channel prediction to distinguish it from the existing inter-frame prediction.

[0602] The spatiotemporal arrangement is a temporal arrangement of spatially arranged frames as in Fig. 82, and can be set so that the product of the total number of frames multiplied by mxn, the number of channels constituting one frame, is equal to the total number of channels to be arranged, NC.

[0603] The quantization of the feature map latent representation can be quantized into n-bit (n=bitdepth) integers using either uniform quantization or non-uniform quantization.

[0604] Equations 5 and 6 are the feature values ​​x respectively. p n-bit uniform quantization process and restored feature values ​​after quantization ^x q can represent the uniform dequantization process.

[0605] In the quantization process, rounding, up and down may be applied to reduce errors in the integerization process, and a process of clipping values ​​that exceed the range of an n-bit integer may be included.

[0606] For quantization and dequantization, you may need to set the number of bits n of the quantized result and the min and max values ​​of the feature values. The n value can be set to a value that can be input to a typical image encoder, such as 8 or 10, or it can be set to a smaller value, such as 4 or 6, to improve compression performance. The min and max values ​​can be obtained from the maximum and minimum values ​​of the feature values ​​to be encoded, or can use preset values. The min and max values ​​can be used for each machine vision task, each dataset, each image sequence, or each frame.

[0607] The above n value and min, max values ​​are parameters related to feature value quantization, and the values ​​used in the encoding process must be used in the decoding process as well. Therefore, it may be more efficient to pre-determine them in the feature map encoder or to include them in the bitstream and transmit them to the feature map decoder.

[0608] Number of bits n used for feature value quantization and feature value range Range max may need to be passed to the dequantization means.

[0609] In addition, the feature map rearrangement means must know parameters such as how the feature map channels are arranged among spatial, temporal, or spatiotemporal arrangements, and if spatial or spatiotemporal arrangements are used, the number of horizontal channels or vertical channels constituting one frame, etc., so that rearrangement can be performed according to the original channel configuration.

[0610] Therefore, such feature map transformation information can be determined in advance or encoded through a feature map encoding means, decoded by a feature map decoding means, and then transmitted to a feature map inverse transformation means.

[0611] Additionally, Equation 7 can be used instead of Equation 6, in which case only the above n value can be preset in the feature map encoder or included in the bitstream and transmitted to the feature map decoder.

[0612] Quantization of the latent representation can be performed before or after the latent representation array. Dequantization of the latent representation can also be performed before or after the latent representation rearrangement. However, if latent representation quantization is performed before the latent representation array, dequantization is performed after the rearrangement, and if latent representation quantization is performed after the latent representation array, dequantization is performed before the rearrangement to reduce computational errors.

[0613] Formula 5

[0614]

[0615]

[0616] Formula 6

[0617]

[0618]

[0619] Formula 7

[0620]

[0621]

[0622] In the encoder and decoder of the video compression codec [E3-1-2], [D1-1-1], the encoder and decoder of the existing video compression standard such as HEVC (High Efficiency Video Coding) or VVC (Versatile Video Coding) can be used as the video compression codec, and in addition, the encoder or decoder of the codec for still image compression can be used.

[0623]

[0624] While the exemplary methods of this disclosure are presented as a series of operations for clarity of description, this is not intended to limit the order in which the steps are performed, and individual steps may be performed simultaneously or in different orders, if desired. To implement a method according to this disclosure, additional steps may be included in addition to the steps illustrated, some steps may be excluded and the remaining steps included, or some steps may be excluded and additional steps included.

[0625] The various embodiments of the present disclosure are not intended to list all possible combinations but rather to illustrate representative aspects of the present disclosure, and the matters described in the various embodiments may be applied independently or in combinations of two or more.

[0626] Additionally, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. In the case of hardware implementation, the embodiments may be implemented by one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), general processors, controllers, microcontrollers, microprocessors, etc.

[0627] The scope of the present disclosure includes software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) that cause operations according to the methods of various embodiments to be executed on a device or a computer, and a non-transitory computer-readable medium having such software or instructions stored thereon and executable on the device or computer.

[0628] The present disclosure may be applicable to an image-related industry that utilizes a method, device, and recording medium for encoding / decoding a feature map extracted through an artificial neural network.

Claims

1. A step of decoding a bitstream to obtain a reduced feature map; and A multi-layer feature map decoding method, characterized in that it comprises a step of restoring a multi-layer feature map by using the reduced feature map as an input to a multi-layer feature map restoration means including a plurality of multi-layer feature map restoration blocks.

2. In paragraph 1, A multi-layer feature map decoding method, characterized in that the restoration of the multi-layer feature map is performed by a sequential multi-layer feature map restoration method that restores the multi-layer feature map using a feature map adaptation block.

3. In paragraph 2, The above feature map adaptation block includes a first feature map adaptation block and a second feature map adaptation block that restores a second feature map having a smaller resolution than the first feature map restored from the first feature map adaptation block. A multi-layer feature map decoding method, characterized in that the number of input channels of the second feature map adaptation block is greater than or equal to the number of input channels of the first feature map adaptation block.

4. In paragraph 1, A multi-layer feature map decoding method, characterized in that the restoration of the multi-layer feature map is performed by a serial-parallel multi-layer feature map restoration method that restores the multi-layer feature map by configuring the plurality of multi-layer feature map restoration blocks in a serial-parallel mixed form.

5. In paragraph 1, A multi-layer feature map decoding method, characterized in that the resolution of feature maps output from the plurality of multi-layer feature map restoration blocks sequentially increases or decreases according to the order in which the plurality of multi-layer feature map restoration blocks are sequentially performed.

6. In paragraph 2, A multi-layer feature map decoding method, characterized in that all or part of the feature maps output from the plurality of multi-layer feature map restoration blocks are restored as hierarchical feature maps of the multi-layer feature map through the feature map adaptation block.

7. In paragraph 2, A multi-layer feature map decoding method, characterized in that the feature maps output from the remaining multi-layer feature map restoration blocks, excluding the last multi-layer feature map restoration block according to the order in which the plurality of multi-layer feature map restoration blocks are sequentially performed among the plurality of multi-layer feature map restoration blocks, are restored as hierarchical feature maps of the multi-layer feature map through the feature map adaptation block.

8. In paragraph 1, The above feature map adaptation block is, A multi-layer feature map decoding method characterized by transforming an input feature map through a neural network without changing the spatial resolution of the input and output feature maps.

9. A step of obtaining a reduced feature map by reducing the multi-layer feature map extracted from the input image; and Including a step of encoding the above reduced feature map into a bitstream, A multi-layer feature map encoding method, characterized in that the reduction of the above multi-layer feature map is performed using a multi-layer feature map reduction means including a plurality of multi-layer feature map reduction blocks.

10. In paragraph 9, A multi-layer feature map encoding method, characterized in that the resolution of feature maps output from the plurality of multi-layer feature map reduction blocks sequentially increases or decreases according to the order in which the plurality of multi-layer feature map reduction blocks are sequentially performed.

11. In paragraph 9, A multi-layer feature map encoding method, characterized in that the number of channels of a feature map output from a first layer of a last multi-layer feature map reduction block according to the order in which the plurality of multi-layer feature map reduction blocks are sequentially performed among the plurality of multi-layer feature map reduction blocks is smaller than the number of channels of a final output feature map of the last multi-layer feature map reduction block.

12. A computer-readable recording medium storing a bitstream generated by a multi-layer feature map encoding method, The above multi-layer feature map encoding method is, A step of obtaining a reduced feature map by reducing a multi-layer feature map extracted from an input image; and Including a step of encoding the above reduced feature map into a bitstream, A computer-readable recording medium, characterized in that the reduction of the multi-layer feature map is performed using a multi-layer feature map reduction means including a plurality of multi-layer feature map reduction blocks.

Images