Feature map encoding device, feature map encoding method, feature map decoding device, and feature map decoding method

Abstract

To provide a technology for encoding and decoding feature maps.SOLUTION: A feature map encoding device includes a packing unit that generates a feature frame by combining multiple feature maps into one frame, a packing adjustment unit that adds padding of a predetermined size to the feature frame, and a feature map internal encoding unit that internally encodes information about the padding used by the packing adjustment unit and the feature frame with the padding added.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] Coding and decoding of feature maps in neural networks. [Background technology]

[0002] A well-known neural network technology used for image recognition, such as detecting objects at various scales in an image, dividing regions into individual objects, or tracking objects, is the FPN (Feature Pyramid Network) described in Non-Patent Document 1. In FPN, multiple feature maps at various scales are generated from the image to be processed, and various image recognitions are performed using the feature maps.

[0003] FPNs used for image recognition generate multiple feature maps from images, and their structure utilizes a CNN (Convolutional Neural Network). CNNs can be divided into a feature extraction section (backbone) that reads an image and generates a feature map using convolution and pooling, and a classification section (head) that is composed of hierarchical fully connected layers and generates output suitable for tasks such as object detection, instance segmentation, and object tracking. FPNs utilize the CNN backbone.

[0004] The feature extraction unit of the FPN is typically configured with a hierarchical structure in which a convolution process 301, an activation process 302, and a pooling process 303 shown in FIG. 3 are configured as one basic unit, and this basic unit is repeated.

[0005] Figure 4 shows the structure of the FPN. The FPN consists of a bottom-up processor 322 that uses a CNN backbone to generate a multi-scale feature map consisting of multiple hierarchical layers, and a top-down processor 324 that aggregates features from deep-layer feature maps to shallow-layer feature maps using the inverse structure of the CNN backbone. The bottom-up processor 322 reduces the resolution of the feature map by half by repeating the basic units of convolution 301, activation 302, and pooling 303 shown in Figure 3, generating a pyramid of feature maps with multiple layers. Meanwhile, the top-down processor 324 adds feature maps with resolutions corresponding to those of the bottom-up processor 322, increasing the resolution of the feature map to the same resolution as the input image, generating a pyramid of feature maps. In other words, the FPN generates multiple feature maps for each layer from the image 326 to be subjected to feature extraction processing.

[0006] The convolution processing unit 301 performs convolution processing on the data to be processed (image or feature map) using multiple types of predetermined filters (kernels). In the convolution processing in the convolution processing unit 301, predetermined filtering is performed on all of the data to be processed while sliding at predetermined intervals. The sliding interval is called the stride. The convolution processing unit 301 may determine the stride based on the number of data to be processed. For example, the convolution processing unit 301 may determine the stride to be 1 if the number of data to be processed is smaller than a predetermined value, and may determine the stride to be 2 if the number is equal to or greater than the predetermined value. Multiple feature maps are generated by preparing multiple predetermined filters at each layer and generating one feature map for each filter. The unit of the feature map is the channel. If the number (types) of predetermined filters is N (N types), N feature maps (N channels) are generated.

[0007] The activation processing unit 302 performs activation processing to nonlinearly convert the feature map output from the convolution processing unit 301. Here, the function used in the activation processing is called an activation function. The activation processing unit 302 uses a ReLU (Rectified Linear Unit) function, a sigmoid function, or the like as the activation function.

[0008] The pooling processing unit 303 is a process for downsampling the feature map output from the activation processing unit 302 by replacing local values ​​of the feature map with representative values.

[0009] On the other hand, when performing classification using a neural network, it has the ability to perform tasks using feature maps of multiple channels at each layer.

[0010] In image classification, the feature maps of multiple channels at each layer are convolved for each predetermined size based on the scale of the feature map, and the probability of the object class is calculated for each pixel. [Prior art documents] [Non-patent literature]

[0011] [Non-Patent Document 1] Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature Pyramid Networks for Object Detection. In CVPR, 2017. Summary of the Invention [Problem to be solved by the invention]

[0012] The amount of information in a feature map is enormous, making it unsuitable for transmission and storage. In view of the above problems, the present invention has an object to provide a technology for encoding and decoding feature maps. [Means for solving the problem]

[0013] In order to solve the above problem, a feature map coding device of one embodiment of the present invention includes: a packing unit that generates a feature frame by combining multiple feature maps into one frame; a packing adjustment unit that adds padding of a predetermined size to the feature frame; and a feature map internal coding unit that internally codes information about the padding used by the packing adjustment unit and the feature frame with the padding added.

[0014] Furthermore, a feature map decoding device according to another embodiment of the present invention includes a feature frame in which multiple feature maps are combined into one frame and padding is added, a feature map internal decoding unit that internally decodes information about the padding, an unpacking adjustment unit that deletes the padding from the feature frame based on the information about the padding, and an unpacking unit that extracts multiple feature maps from the feature frame from which the padding has been deleted. [Effects of the Invention]

[0015] According to the present invention, feature maps can be efficiently coded and decoded with a small amount of processing. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a block diagram illustrating the configuration of a feature map encoding device 100. [Figure 2] FIG. 2 is a block diagram illustrating the configuration of a feature map decoding device 200. [Figure 3] FIG. 1 is a block diagram for explaining the basic unit of processing in each layer of the FPN. [Figure 4] FIG. 1 is a block diagram illustrating the structure of an FPN. [Figure 5] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map reduction unit 102. [Figure 6] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map restoration unit 203. [Figure 7] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map conversion unit 103. [Figure 8] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map inverse conversion unit 202. [Figure 9] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map internal coding unit 104. [Figure 10] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map internal decoding unit 201. [Figure 11] FIG. 10 is a diagram for explaining the number of channels of feature maps x1, x2, and x3, the width of the feature maps, and the height of the feature maps. [Figure 12] FIG. 10 is a diagram illustrating a state in which feature maps of multiple channels are packed into one frame. [Figure 13] FIG. 10 is a diagram illustrating flipping when packing feature maps of multiple channels into one frame. [Figure 14] FIG. 10 is a block diagram for explaining another example of the feature map conversion unit 103 and another example of the feature map inverse conversion unit 202. [Figure 15] FIG. 10 is a diagram for explaining packing feature frames supplied to the packing adjuster 710 and the operation of the packing adjuster 710. [Figure 16] 10 is a flowchart illustrating the operation of the packing adjustment unit 710. [Figure 17] 10 is a diagram illustrating an example of the syntax of information about padding used by the packing adjustment unit 710. [Figure 18] 10 is a flowchart for explaining the operation of the unpacking adjuster 810. [Figure 19] FIG. 10 is a diagram illustrating another example of the packing adjustment unit 710. [Figure 20] 10 is a diagram illustrating an example of the syntax of information about padding used by the packing adjustment unit 710. DETAILED DESCRIPTION OF THE INVENTION

[0017] The technologies and technical terms used in this embodiment will be defined below.

[0018] <Features and feature maps> In a convolutional neural network (CNN), a filter is used to scan the image (input layer data) from which features are extracted, and the results of the convolution operation between the data and the filter coefficients are used as features or a feature map.

[0019] <Packing> Frame packing refers to combining two or more frames (pictures) into a single frame (picture) by arranging them in a tiled pattern. In this application, packing refers to combining feature maps from multiple channels into a single frame. Figure 12 shows an example of frame packing.

[0020] <Data Type> A data type that represents an integer value is called an integer type, and a data type that represents a decimal value is called a decimal type.

[0021] (First embodiment) A feature map encoding device 100 and a feature map decoding device 200 according to a first embodiment of the present invention will be described.

[0022] 1 is a block diagram of a feature map encoding device 100 according to a first embodiment. The feature map encoding device 100 according to the embodiment includes a feature map reduction unit 102, a feature map conversion unit 103, and a feature map internal encoding unit 104. The feature map encoding device 100 encodes a feature map generated by a neural network feature extraction unit 101 to generate and output a bitstream.

[0023] The neural network feature extraction unit 101 reads an image from which features are to be extracted, generates a feature map by FPN convolution processing, activation processing, and pooling processing, and supplies the generated feature map to the feature map reduction unit 102. In this embodiment, a three-layer multi-scale feature map of x1, x2, and x3 is generated.

[0024] The feature map reduction unit 102 converts the three-layer multi-scale feature maps x1, x2, and x3 acquired from the neural network feature extraction unit 101 into a single-layer single-scale feature map xf, and supplies it to the feature map conversion unit 103. The feature map reduction unit 102 will be described in detail with reference to FIG. 5.

[0025] The feature map conversion unit 103 performs packing and quantization on the decimal-type single-scale feature map xf supplied from the feature map reduction unit 102, converting it into an integer-type packed feature frame, and supplies it to the feature map internal coding unit 104.

[0026] The feature map conversion unit 103 will be described in detail with reference to FIG.

[0027] The feature map intra-encoding unit 104 encodes the integer-type packing feature frames supplied from the feature map conversion unit 103 in accordance with an image encoding standard such as VVC, HEVC, or AV1 to generate and output a bitstream. The output bitstream is supplied to the feature map decoding device 200 or the like via a network or the like.

[0028] The feature map internal coding unit 104 will be described in detail with reference to FIG.

[0029] 2 is a block diagram showing the configuration of a feature map decoding device 200 according to an embodiment of the present invention, which corresponds to the feature map encoding device 100 in FIG. 1. The feature map decoding device 200 according to this embodiment includes a feature map internal decoding unit 201, a feature map inverse conversion unit 202, and a feature map restoration unit 203. The feature map decoding device 200 receives as input a bit stream encoded by the feature map encoding device 100, decodes the bit stream, and generates three-layer multi-scale feature maps x1up, x2up, and x3up, which it supplies to a neural network classification unit 204.

[0030] The feature map internal decoding unit 201 decodes the bitstream coded by the feature map internal coding unit 104 in accordance with an image coding standard such as VVC, HEVC, or AV1, generates an integer-type packing feature map, and supplies it to the feature map inverse conversion unit 202.

[0031] The feature internal decoding unit 201 will be described in detail with reference to FIG.

[0032] The feature map inverse conversion unit 202 performs inverse quantization and unpacking on the integer-type packed feature map supplied from the feature internal decoding unit 201, converting it into a decimal-type single-scale feature map xr, and supplies it to the feature map restoration unit 203.

[0033] The feature map inverse conversion unit 202 will be described in detail with reference to FIG.

[0034] The feature map restoration unit 203 converts the single-scale feature map xr supplied from the feature map inverse conversion unit 202 into three-layer multi-scale feature maps x1up, x2up, and x3up, and supplies them to the neural network identification unit 204 as the output of the feature map decoding device 200.

[0035] The feature map restoration unit 203 will be described in detail with reference to FIG.

[0036] The neural network classification unit 204 performs classification processing such as classification of objects, places, landscapes, people, and living things in the classification target image based on the three-layer multi-scale feature maps x1up, x2up, and x3up supplied by the feature map restoration unit 203.

[0037] <Feature map reduction and restoration> The feature map reduction unit 102 has a function of converting the multi-layered multi-scale feature map acquired from the neural network feature extraction unit 101 into a single-layered single-scale feature map.

[0038] The feature map reduction unit 102 will be described in detail using Figure 5. The feature map reduction unit 102 is composed of a first feature map reduction unit 501, a first channel combination unit 502, a second feature map reduction unit 503, a second channel combination unit 504, a third feature map reduction unit 505, a first padding unit 506, a second padding unit 507, and a third padding unit 508. The feature map reduction unit 102 in Figure 5 is an example of a configuration that converts a three-layer multi-scale feature map into a single-scale feature map.

[0039] The feature map reduction unit 102 receives three-layer multi-scale feature maps, namely, a first feature map x1, a second feature map x2, and a third feature map x3, as input, converts them into a single-layer single-scale feature map xf, and supplies the result to the feature map conversion unit 103. Here, n denotes an index indicating a layer, Cn denotes the number of channels in the nth layer, Wn denotes the width of the feature map, and Hn denotes the height of the feature map. In this embodiment, the values ​​of Cn, Wn, and Hn for each layer are as shown in FIG. 11, where H and W denote the width and height, respectively, of the image from which features are extracted.

[0040] The first padding unit 506 has a function of padding the first feature map x1 to generate a first padded feature map x1pad. The first padding unit 506 determines the padding size so that the width and height of x1pad are multiples of 8. The number of channels of x1pad is the same as that of x1, which is 256.

[0041] The second padding unit 507 performs padding on the second feature map x2 by folding to generate a second padded feature map x2pad. The second padding unit 507 determines the padding size so that the width and height of x2pad are multiples of 4. The number of channels of x2pad is 256, the same as that of x2.

[0042] The third padding unit 508 performs padding on the third feature map x3 by folding to generate a third padded feature map x3pad. The third padding unit 508 determines the padding size so that the width and height of x3pad are multiples of 2. The number of channels in x3pad is 256, the same as that of x3.

[0043] In the first padding unit 506, the second padding unit 507, and the third padding unit 508, the left and right padding sizes are approximately the same, and the upper and lower padding sizes are approximately the same. That is, the feature maps x1, x2, and x3 are positioned approximately at the centers of x1pad, x2pad, and x3pad, respectively.

[0044] The first feature map reduction unit 501 performs convolution in the spatial and channel directions on the first padded feature map x1pad obtained from the first padding unit 506 to generate a first intermediate feature map y1. The number of channels of y1 is 192, its width is Wx1pad / 2, and its height is Hx1pad / 2, where Wx1pad and Hx1pad are the width and height of the first padded feature map x1pad, respectively.

[0045] The first channel combining unit 502 has a function of combining the first intermediate feature map y1 obtained from the first feature map reduction unit 501 and the second padded feature map x2pad obtained from the second padding unit 507 in the channel direction to generate an intermediate feature map y1Cx2pad. Since y1 has 192 channels and x2pad has 256 channels, the intermediate feature map y1Cx2pad has 448 channels (192 + 256).

[0046] The second feature map reduction unit 503 generates a second intermediate feature map y2 by performing convolution in the spatial and channel directions on the intermediate feature map y1Cx2pad obtained from the first channel combination unit 502. The number of channels of y2 is 192, its width is Wy1Cx2pad / 2, and its height is Hy1Cx2pad / 2, where Wy1Cx2pad and Hy1Cx2pad are the width and height of the intermediate feature map y1Cx2pad, respectively.

[0047] The second channel combining unit 504 has a function of combining the second intermediate feature map y2 obtained from the second feature map reduction unit 503 and the third padded feature map x3pad obtained from the third padding unit 508 in the channel direction to generate an intermediate feature map y2Cx3pad. Since the number of channels in the intermediate feature map y2 is 192 and the number of channels in x3pad is 256, the number of channels in y1Cx2pad is 448 (192 + 256).

[0048] The third feature map reduction unit 505 generates a third intermediate feature map y3 by performing convolution in the spatial and channel directions on the intermediate feature map y2Cx3pad obtained from the second channel combination unit 504. The number of channels of y3 is 192, its width is Wy2Cx3pad / 2, and its height is Hy2Cx3pad / 2, where Wy2Cx3pad and Hy2Cx3pad are the width and height of the intermediate feature map y2Cx3pad, respectively.

[0049] The feature map reduction unit 102 outputs the third intermediate feature map y3 as a single-scale feature map xf and supplies it to the feature map conversion unit 103.

[0050] The feature map restoration unit 203 has a function of converting the single-scale feature map xr acquired from the feature map inverse conversion unit 202 into three-layer multi-scale feature maps x1up, x2up, and x3up.

[0051] The feature map restoration unit 203 will be described in detail using Fig. 6. The feature map restoration unit 203 is composed of an 8x enlargement unit 601, a 4x enlargement unit 602, a 2x enlargement unit 603, a first feature map blending unit 604, a second feature map blending unit 605, a first de-padding unit 606, a second de-padding unit 607, and a third de-padding unit 608.

[0052] The 8x enlargement unit 601 performs transposition convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr obtained from the feature map inverse transformation unit 202 to enlarge the feature map and reduce the channels, thereby generating an intermediate feature map z1. The number of channels in z1 is 196. If the width and height of the single-scale feature map xr are xrwidth and xrheight, respectively, the width and height of z1 are xrwidth × 8 and xrheight × 8, respectively. Here, rwidth × 8 and xrheight × 8 are the same as the width and height, respectively, of the first padded feature map x1pad, which is the output of the first padding unit 506 of the feature extraction unit 102.

[0053] The 4x enlargement unit 602 performs transposition convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr obtained from the feature map inverse transformation unit 202 to enlarge the features and reduce the channels, generating an intermediate feature map z2. The number of channels in z2 is 196. The width and height of z2 are xrwidth×4 and xrheight×4, respectively. Here, rwidth×4 and xrheight×4 are the same as the width and height, respectively, of the second padded feature map x2pad, which is the output of the second padding unit 507 of the feature extraction unit 102.

[0054] The 2x enlargement unit 603 performs transposition convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr obtained from the feature map inverse transformation unit 202 to enlarge the feature map and reduce the channels, thereby generating an intermediate feature map z3. The number of channels in z3 is 196. The width and height of z3 are xrwidth×2 and xrheight×2, respectively. Here, rwidth×2 and xrheight×2 are the same as the width and height, respectively, of the third padded feature map x3pad, which is the output of the third padding unit 508 in the feature extraction unit 102.

[0055] The first feature map mixing unit 604 has a function of generating an intermediate feature map z2up by improving the quality of the intermediate feature map z2 obtained from the 4x enlargement unit 602, using the intermediate feature map z1 obtained from the 8x enlargement unit 601.

[0056] The second feature map mixing unit 605 has a function of generating an intermediate feature map z3up that improves the quality of the intermediate feature map z3 obtained from the 2x enlargement unit 603, by using the intermediate feature map z2up obtained from the first feature map mixing unit 604.

[0057] The padding removal unit 606 removes padding from the intermediate feature map z1 acquired from the 8x enlargement unit 601 to generate a first output feature map x1up. The width and height of x1up are the same as the width and height of the first feature map x1 input to the feature extraction unit 102.

[0058] The padding removal unit 607 has a function of removing padding from the intermediate feature map z2up acquired from the first feature map blending unit 604 to generate a second output feature map x2up. The width and height of x2up are the same as the width and height of the second feature map x2 input to the feature extraction unit 102.

[0059] The padding removal unit 608 has a function of removing padding from the intermediate feature map z3up acquired from the second feature map blending unit 605 to generate a third output feature map x3up. The width and height of x3up are the same as the width and height of the third feature map x3 input to the feature extraction unit 102.

[0060] In the first padding removal unit 606, the second padding removal unit 607, and the third padding removal unit 608, the left-side padding size and the right-side padding size are set to be approximately the same, and the upper-side padding size and the lower-side padding size are set to be approximately the same, similar to the first padding unit 506, the second padding unit 507, and the third padding unit 508 of the feature extraction unit 102. In other words, the padding on the top, bottom, left, and right sides is removed assuming that each output feature map x1up, x2up, and x3up is positioned approximately at the center of each intermediate feature map z1, z2up, and z3up, respectively.

[0061] <Feature map transformation and feature map inverse transformation> The feature map conversion unit 103 has a function of performing packing and quantization on the multi-channel integer-type single-scale feature map xf supplied from the feature map reduction unit 102, and converting it into an integer-type packed feature frame to be supplied to the feature map internal coding unit 104.

[0062] The feature map conversion unit 103 on the encoding side will be described in detail with reference to Fig. 7. The feature map conversion unit 103 is made up of a packing unit 701 and a feature map quantization unit 702.

[0063] The packing unit 701 has a function of generating a packed feature frame by combining input feature maps of multiple channels into one frame. Fig. 12 is a diagram explaining the state in which feature maps of multiple channels are packed into one frame. The feature maps of each channel are sequentially arranged in one frame in raster scan order, from left to right and top to bottom.

[0064] Furthermore, the packing unit 701 performs flipping based on the position in the frame where the channel feature map is placed. Whether or not to perform flipping can be selected, and the selection may be transmitted from the encoding side to the decoding side via a bitstream.

[0065] Flipping is performed when packing feature maps of multiple channels into one frame, as shown in FIG. 13. In flipping, the positions of the elements (pixels) of each channel's feature map are flipped horizontally (left and right), vertically (up and down), or horizontally and vertically (up and down, left and right) when packing the feature maps of each channel. The feature maps of four channels, A (upper left), B (upper right), C (lower left), and D (lower right) in FIG. 13, are grouped together as one set. No flipping is performed at position A in FIG. 13. At position B, the feature map is flipped horizontally (left and right). At position C, the feature map is flipped vertically (up and down). At position D, the feature map is flipped horizontally and vertically (up and down, left and right). When the distribution of elements in the feature maps of each channel is similar, flipping based on the position within the frame where the channels are placed reduces the boundaries between the feature maps of each channel, improving coding efficiency.

[0066] The feature map quantization unit 702 has a function of converting elements of a decimal-type packing feature frame (feature map of all channels) into N-bit integer type elements (N=an integer from about 8 to 16) within a predetermined range, and outputting an integer-type packing feature frame. In this embodiment, the elements are converted into 10-bit integer type elements ranging from 0 to 1023. The feature map quantization unit 702 detects the minimum and maximum values ​​of the elements of the decimal-type packing feature frame, and transmits the detected minimum and maximum values ​​of the elements of the packing feature frame (feature map of all channels) to the decoding side as metadata. To convert from a decimal-type packing feature frame (feature map) to an integer-type packing feature frame (feature map), a linear conversion is performed in which the minimum value of the decimal type is equivalent to the minimum value of the integer type, and the maximum value of the decimal type is equivalent to the maximum value of the integer type. For example, when the integer type range is expressed in 10 bits, the minimum value of the elements of the integer-type packing feature frame (feature map) is 0 and the maximum value is 1023 (2 10 -1). Values ​​between the minimum and maximum values ​​are linearly quantized.

[0067] Next, the feature map inverse transform unit 202 has a function of performing inverse quantization and unpacking on the integer-type packing feature frame decoded by VVC, HEVC, AV1, etc., supplied from the feature map internal decoding unit 201, and performing inverse transform into a decimal-type single-scale feature map xr to be supplied to the feature map restoration unit 203.

[0068] The feature map inverse transformation unit 202 on the decoding side will be described in detail with reference to Fig. 8. The feature map inverse transformation unit 202 performs the inverse process of the feature map transformation unit 103, and is made up of a feature map inverse quantization unit 801 and an unpacking unit 802.

[0069] The feature map inverse quantization unit 801 performs the inverse process of the feature map quantization unit 702 on the encoding side, and has a function of converting elements of integer-type packing feature frames from integer type to decimal type. The feature map inverse quantization unit 801 converts the integer-type packing feature frame decoded by the feature map internal decoding unit 201 into a decimal-type packing feature frame using the minimum and maximum decimal values ​​transmitted as metadata. A linear conversion is performed to convert the integer minimum value to the decimal minimum value and the integer maximum value to the decimal maximum value. Linear inverse quantization is performed on values ​​between the minimum and maximum values.

[0070] The unpacking unit 802 extracts a feature map for each channel from the packing feature frames arranged in one frame in raster scan order and supplies it to the feature map restoration unit 203 as a single-scale feature map xr.

[0071] <Feature map inner encoding and decoding> The feature map internal coding unit 104 will be described in detail using Figure 9. The feature map internal coding unit 104 is made up of a switch 901, a VVC encoding unit 902, an HEVC encoding unit 903, and an AV1 encoding unit 904. The switch 901 selects the encoding standard for internally coding the feature map converted by the feature map conversion unit 103. The VVC encoding unit 902 encodes the feature map in the VVC standard and outputs a bitstream that complies with the VVC standard. The HEVC encoding unit 903 encodes the feature map in the HEVC standard and outputs a bitstream that complies with the HEVC standard. The AV1 encoding unit 904 encodes the feature map in the AV1 standard and outputs a bitstream that complies with the AV1 standard.

[0072] In the VVC, HEVC, and AV1 standards, images are divided into blocks of a specified size and then encoded.

[0073] It is also possible to implement only one of VVC, HEVC, and AV1, and it is also possible to use image coding methods other than VVC, HEVC, and AV1.

[0074] Next, the feature map internal decoding unit 201 will be described in detail with reference to Fig. 10. The feature map internal decoding unit 201 is made up of a switch 1001, a VVC decoding unit 1002, an HEVC decoding unit 1003, and an AV1 decoding unit 1004. The switch 1001 selects the encoding standard for internal decoding based on information for selecting internal decoding in the input bitstream. The VVC decoding unit 1002 decodes the feature map in accordance with the VVC standard. The HEVC decoding unit 1003 decodes the feature map in accordance with the HEVC standard. The AV1 encoding unit 1004 decodes the feature map in accordance with the AV1 standard.

[0075] In the VVC, HEVC, and AV1 standards, decoding is performed for each predetermined block size.

[0076] It is also possible to implement only one of VVC, HEVC, and AV1, and it is also possible to use image coding methods other than VVC, HEVC, and AV1.

[0077] <About packing aligned with encoding boundaries> 14(a) is a block diagram illustrating another example of the feature map conversion unit 103. The feature map conversion unit 103 is configured with a packing unit 701, a packing adjustment unit 710, and a feature map quantization unit 702.

[0078] The packing unit 701 combines input feature maps of multiple channels into one frame and generates the result as a packing feature frame. The packing adjustment unit 710 performs predetermined adjustments on the packing feature frame supplied from the packing unit 701. The feature map quantization unit 702 converts the adjusted decimal-type packing feature frame supplied from the packing adjustment unit 710 into an integer-type packing feature frame.

[0079] The packing adjustment unit 710 performs predetermined adjustments on the packing feature frame as follows. The packing adjustment unit 710 acquires, as the block size, the size of a coding block used as a coding unit by an image coding standard in the feature map intra coding unit 104 at a subsequent stage. That is, the packing adjustment unit 710 acquires both the width and height of the block size. The packing adjustment unit 710 may acquire either the width or the height of the block size, or may acquire the block size assuming that the width and height of the block size are the same. The packing adjustment unit 710 may acquire one or more block sizes.

[0080] The packing adjustment unit 710 may store a predetermined block size in advance. The packing adjustment unit 710 may provide the pre-stored predetermined block size to the feature map intra coding unit 104. The feature map intra coding unit 104 may select a coding block size based on the predetermined block size provided from the packing adjustment unit 710, and encode the packing feature frame.

[0081] FIG. 15(a) is a diagram illustrating a packing feature frame supplied to the packing adjustment unit 710. In FIG. 15(a), the packing feature frame is indicated as 1500, and the feature map of the first channel is indicated as 1502. The feature map 1502 of the first channel corresponds to CH1 in FIG. 12. The horizontal size of the feature map 1502 of the first channel is indicated as Wf, and the vertical size of the feature map 1502 of a predetermined channel is indicated as Hf. Here, in the packing feature frame 1500, the feature maps of all channels may have the same size, or the feature maps of all channels may have different sizes.

[0082] FIG. 15(b) is a diagram illustrating the operation of the packing adjustment unit 710. FIG. 16 is a flowchart illustrating the operation of the packing adjustment unit 710. The packing adjustment unit 710 selects a feature map of a predetermined channel from the packing feature frame supplied from the packing unit 701 as a predetermined feature map 1502 (S1602). Here, the block size is indicated as 1504. The packing adjustment unit 710 calculates, as an adjusted block size, a value that is equal to or greater than the size of the predetermined feature map, among values ​​that are integer multiples of the block size (S1604). In other words, the packing adjustment unit 710 calculates, as an adjusted block width Wa, a value that is equal to or greater than the horizontal size Wf of the predetermined feature map, among values ​​that are integer multiples of the width of the block size. Similarly, the packing adjustment unit 710 calculates, as an adjusted block height Ha, a value that is equal to or greater than the vertical size Hf of the predetermined feature map, among values ​​that are integer multiples of the height of the block size.

[0083] The packing adjustment unit 710 calculates the difference between the size of the predetermined feature map and the adjusted block size as an adjusted padding size (S1606). That is, the packing adjustment unit 710 calculates the difference between the horizontal size Wf of the predetermined feature map and the adjusted block width Wa as an adjusted padding width Wp. Similarly, the packing adjustment unit 710 calculates the difference between the vertical size Hf of the predetermined feature map and the adjusted block height Ha as an adjusted padding height Hp.

[0084] The packing adjustment unit 710 may set a predetermined number of paddings when the adjusted padding size is 0. That is, the packing adjustment unit 710 may set a predetermined number of paddings when at least one of the adjusted padding width Wp or the adjusted padding height Hp is 0. The packing adjustment unit 710 may calculate, as the adjusted block size, a value that exceeds a predetermined feature map size, from among values ​​that are integer multiples of the block size.

[0085] The packing adjustment unit 710 adds padding to the predetermined feature map in the number of pieces equal to the adjusted padding size (S1608). That is, the packing adjustment unit 710 adds padding to the right side of the predetermined feature map in the number of pieces equal to the adjusted padding width Wp. Similarly, the packing adjustment unit 710 adds padding to the bottom side of the predetermined feature map in the number of pieces equal to the adjusted padding height Hp. For example, when the packing adjustment unit 710 adds padding to the predetermined feature map 1502, an area with a width of Wp on the right side of the predetermined feature map 1502 and an area with a height of Hp on the bottom side are filled with predetermined values, as shown in FIG. 15(b). If the adjusted padding size is 0, the packing adjustment unit 710 may skip S1608 and proceed to the process of S1610 without adding padding.

[0086] The packing adjustment unit 710 may set the padding value to 0, or may set the padding value to a predetermined value. The packing adjustment unit 710 may set the padding value to a value included in a predetermined feature map, or may set the padding value to a statistical value of values ​​included in the predetermined feature map, or may set the padding value to a statistical value of values ​​included in the feature maps of multiple channels. The packing adjustment unit 710 may set the padding value to a value included in a predetermined feature map adjacent to the padding. When multiple padding pieces are to be added, the packing adjustment unit 710 may set the padding values ​​to the same value or to different values. The packing adjustment unit 710 may set the padding value and set the padding type based on the value.

[0087] The packing adjustment unit 710 determines whether there is a feature map of a channel that has not yet been selected as a predetermined feature map (S1610). If there is a feature map of a channel that has not yet been selected as a predetermined feature map (S1610: Yes), the packing adjustment unit 710 selects that feature map as a predetermined feature map (S1602). Then, the packing adjustment unit 710 repeats the processing from S1602. If there is no feature map of a channel that has not yet been selected as a predetermined feature map (S1610: No), the packing adjustment unit 710 ends the processing.

[0088] The packing adjustment unit 710 adds padding of the adjusted padding size to the feature maps of all channels included in the packing feature frame supplied from the packing unit 701, and generates an adjusted packing feature frame 1510. In other words, the packing adjustment unit 710 generates a packing feature frame adjusted to fit the boundaries of coding blocks. The packing adjustment unit 710 may add padding to the feature maps of some of the channels of the packing feature frame.

[0089] The packing adjustment unit 710 adds padding to the feature map to align the position of the feature map with the boundary of the coding block. This prevents two adjacent feature maps from being mixed in a given coding block in the feature map internal coding unit 104. Because the feature map internal coding unit 104 does not mix two adjacent feature maps in a coding block, prediction efficiency is improved compared to when two adjacent feature maps are mixed. Therefore, the feature map coding device of this embodiment has the effect of improving coding efficiency.

[0090] FIG. 17 is an example of a syntax 1700 of information regarding padding used by the packing adjustment unit 710. In the syntax 1700 of FIG. 17, channel_size represents the number of channels of the feature map, padding_type represents the type of padding, block_size_x represents the width of the block size, block_size_y represents the height of the block size, adjust_block_size_x represents the width of the adjustment block size, adjust_block_size_y represents the height of the adjustment block size, adjust_padding_size_x represents the width of the adjustment padding size, and adjust_padding_size_y represents the height of the adjustment padding size. In FIG. 17, the description "for(i=0; i<M; i++)" indicates that with the initial value i = 0, while the condition i < maximum value M is satisfied, the syntax enclosed in parentheses is processed and 1 is added to i. Also, in FIG. 17, the curly brackets indicate a predetermined unit for processing. That is, the packing adjustment unit 710 sets the syntax of the number of channels of the feature map, the type of padding for each channel, the block size for each channel, the adjustment block size for each channel, and the adjustment padding size for each channel. The packing adjustment unit 710 may set a plurality of channels together as a channel group. The packing adjustment unit 710 may set the syntax 1700 for each channel group. If the information regarding the padding of a plurality of channels is the same, the packing adjustment unit 710 may set the syntax of the information regarding the padding of those plurality of channels together.

[0091] The packing adjustment unit 710 may supply at least a part included in the syntax 1700 to the feature map internal encoding unit 104. The feature map internal encoding unit 104 may select the size of the encoding block based on at least a part included in the syntax 1700 supplied from the packing adjustment unit 710 and encode the packing feature frame. The feature map internal encoding unit 104 may encode at least a part included in the syntax 1700 supplied from the packing adjustment unit 710 and generate a bit stream.

[0092] The processing by the packing adjustment unit 710 may be performed together with the processing by the packing unit 701, may be performed before the processing by the packing unit 701, or may be performed after the processing by the feature map quantization unit 702.

[0093] 14(b) is a block diagram illustrating another example of the feature map inverse transformation unit 202. The feature map inverse transformation unit 202 performs the inverse process of the feature map transformation unit 103, and is configured by a feature map inverse quantization unit 801, an unpacking adjustment unit 810, and an unpacking unit 802.

[0094] The feature map inverse quantization unit 801 converts the integer-type packing feature frame decoded by the feature map internal decoding unit 201 into a decimal-type packing feature frame by using the decimal-type maximum and minimum values ​​transmitted as metadata. The unpacking adjustment unit 810 performs predetermined adjustment on the packing feature frame supplied from the feature map internal decoding unit 201. The unpacking unit 802 extracts a feature map of each channel from the packing feature frame supplied from the unpacking adjustment unit 810 in raster scan order.

[0095] The unpacking adjuster 810 performs predetermined adjustments on the packing feature frame as follows. The unpacking adjuster 810 receives at least a portion of the syntax 1700 decoded by the feature map internal decoder 201 at the previous stage and supplied to the unpacking adjuster 810. The feature map internal decoder 201 supplies at least a portion of the syntax 1700 to the unpacking adjuster 810. The feature map internal decoder 201 may decode at least a portion of the syntax 1700 from the bitstream. The feature map internal decoder 201 may select a coding block size based on at least a portion of the syntax 1700 and decode the packing feature frame. Now, it is assumed that the unpacking adjuster 810 receives a predetermined feature map size and adjusted padding size from the feature map internal decoder 201. That is, the unpacking adjuster 810 receives the horizontal size Wf of the feature map, the vertical size Hf of the predetermined feature map, the adjusted padding width Wp, and the adjusted padding height Hp from the feature map internal decoder 201.

[0096] The unpacking adjuster 810 may store the predetermined block size stored by the packing adjuster 710 in advance.

[0097] FIG. 15(b) is a diagram illustrating the operation of the unpacking adjustment unit 810. FIG. 18 is a flowchart illustrating the operation of the unpacking adjustment unit 810. The unpacking adjustment unit 810 selects a feature map of a predetermined channel from the packing feature frame supplied from the feature map internal decoding unit 201 as the predetermined feature map 1502 (S1802). The unpacking adjustment unit 810 adds the size of the predetermined feature map and the adjusted padding size to calculate an adjusted block size (S1804). That is, the unpacking adjustment unit 810 adds the horizontal size of the predetermined feature map and the adjusted padding width to calculate an adjusted block width Wa. Similarly, the unpacking adjustment unit 810 adds the vertical size of the predetermined feature map and the adjusted padding height to calculate an adjusted block height Ha.

[0098] The unpacking adjustment unit 810 deletes an area of ​​the padding size from an area of ​​the adjusted block size that includes a predetermined feature map (S1806). That is, the unpacking adjustment unit 810 deletes an area of ​​the padding width Wp from the right side of an area of ​​the adjusted block width Wa that includes the predetermined feature map. Furthermore, the unpacking adjustment unit 810 deletes an area of ​​the padding height Hp from the bottom of an area of ​​the adjusted block height Ha that includes the predetermined feature map.

[0099] The unpacking adjuster 810 determines whether there is a feature map of a channel that has not yet been selected as a predetermined feature map (S1808). If there is a feature map of a channel that has not yet been selected as a predetermined feature map (S1808: Yes), the unpacking adjuster 810 selects that feature map as a predetermined feature map (S1802). The unpacking adjuster 810 then repeats the process from S1802. If there is no feature map of a channel that has not yet been selected as a predetermined feature map (S1808: No), the unpacking adjuster 810 ends the process.

[0100] The unpacking adjuster 810 removes padding of the adjusted padding size from the feature maps of all channels included in the packing feature frame supplied from the feature map internal decoder 201 , and generates a packing feature frame 1500 .

[0101] The processing by the unpacking adjustment unit 810 may be performed together with the processing by the unpacking unit 802, or may be performed before the feature map inverse quantization unit 801, or may be performed after the unpacking unit 802.

[0102] <About packing with offset> FIG. 19 is a diagram illustrating another example of the packing adjustment unit 710. The packing adjustment unit 710 selects a feature map of a predetermined channel from the packing feature frame supplied from the packing unit 701 as a predetermined feature map 1902 (S1602). Here, the block size is denoted as 1504. The packing adjustment unit 710 calculates, as an adjusted block size, a value that exceeds the size of the predetermined feature map among values ​​that are integer multiples of the block size (S1604). That is, the packing adjustment unit 710 calculates, as an adjusted block width Wa, a value that exceeds the horizontal size Wf of the predetermined feature map among values ​​that are integer multiples of the width of the block size. Similarly, the packing adjustment unit 710 calculates, as an adjusted block height Ha, a value that exceeds the vertical size Hf of the predetermined feature map among values ​​that are integer multiples of the height of the block size.

[0103] The packing adjustment unit 710 calculates the difference between the size of the predetermined feature map and the adjusted block size as an adjusted padding size (S1606). That is, the packing adjustment unit 710 calculates the difference between the horizontal size Wf of the predetermined feature map and the adjusted block width Wa as an adjusted padding width Wp. Similarly, the packing adjustment unit 710 calculates the difference between the vertical size Hf of the predetermined feature map and the adjusted block height Ha as an adjusted padding height Hp.

[0104] The packing adjustment unit 710 moves the predetermined feature map by a predetermined amount within the adjusted padding size, and then adds padding (S1608). Here, the predetermined amount of movement is referred to as an offset. That is, the packing adjustment unit 710 selects an offset by which to move the predetermined feature map in the horizontal direction as a horizontal offset offset_x, and an offset by which to move the predetermined feature map in the vertical direction as a vertical offset offset_y. The packing adjustment unit 710 moves the predetermined feature map by the offset (offset_x, offset_y) to obtain a moved predetermined feature map 1904.

[0105] The packing adjustment unit 710 may select an offset by comparing a predetermined feature map with a feature map adjacent to the predetermined feature map. Two adjacent feature maps may be similar, and one feature map may be similar to a shifted version of the other feature map. In this case, shifting the feature maps to align their positions is expected to improve the coding efficiency of the subsequent feature map intra-coding unit 104. Therefore, the packing adjustment unit 710 may estimate the amount of motion for the two adjacent feature maps using a well-known motion estimation technique and select an offset based on the amount of motion. The packing adjustment unit 710 may divide one of the two feature maps into blocks of a predetermined size. Then, the packing adjustment unit 710 may select a portion of the other feature map from a predetermined range for each predetermined block in the other feature map, calculate the difference in absolute value between the selected range and the predetermined block as an evaluation value, and select the position at which the evaluation value is minimum as the offset. The packing adjustment unit 710 may calculate the degree of similarity for at least some of the feature maps adjacent to a given feature map, and select an offset based on the adjacent feature map with the greatest degree of similarity.

[0106] In addition, the packing adjustment unit 710 adds padding to the predetermined feature map based on the offset and the adjusted padding size. That is, the packing adjustment unit 710 adds padding to the left of the predetermined feature map in the number of horizontal offsets offset_x. The packing adjustment unit 710 also adds padding to the top of the predetermined feature map in the number of vertical offsets offset_y. The packing adjustment unit 710 also adds padding to the right of the predetermined feature map in the number of paddings equal to the difference between the adjusted padding width Wp and the horizontal offset offset_x. The packing adjustment unit 710 also adds padding to the bottom of the predetermined feature map in the number of paddings equal to the difference between the adjusted padding height Hp and the vertical offset offset_y.

[0107] That is, the packing adjustment unit 710 replaces areas that are no longer the predetermined feature map before and after the movement with predetermined values. In other words, the packing adjustment unit 710 replaces the range of the predetermined feature map 1802 before the movement that does not include the predetermined feature map 1804 after the movement with predetermined values. The packing adjustment unit 710 does not have to replace the range of the predetermined feature map 1802 before the movement that does not include the predetermined feature map 1804 after the movement with predetermined values.

[0108] The packing adjustment unit 710 determines whether there is a feature map of a channel that has not yet been selected as a predetermined feature map (S1610). If there is a feature map of a channel that has not yet been selected as a predetermined feature map (S1610: Yes), the packing adjustment unit 710 selects that feature map as a predetermined feature map (S1602). Then, the packing adjustment unit 710 repeats the processing from S1602. If there is no feature map of a channel that has not yet been selected as a predetermined feature map (S1610: No), the packing adjustment unit 710 ends the processing.

[0109] The packing adjustment unit 710 adds padding to the feature maps of all channels included in the packing feature frame supplied from the packing unit 701 based on the offset and the adjusted padding size, and generates an adjusted packing feature frame 1910. In other words, the packing adjustment unit 710 generates a packing feature frame adjusted by a predetermined offset. The packing adjustment unit 710 may add padding to feature maps of some of the channels of the packing feature frame.

[0110] The packing adjustment unit 710 moves the feature map to align it with other adjacent feature maps. As a result, the feature map internal coding unit 104 makes the feature map values ​​in two adjacent coding blocks similar. The feature map internal coding unit 104 improves prediction efficiency when the values ​​of two adjacent coding blocks are similar compared to when the values ​​of the two adjacent coding blocks are not similar. Therefore, the feature map coding device of this embodiment achieves the effect of improving coding efficiency.

[0111] Fig. 20 shows an example of the syntax of information related to padding used by the packing adjustment unit 710. In Fig. 20, offset_x and offset_y are added to Fig. 17. offset_x represents the offset in the horizontal direction, and offset_y represents the offset in the vertical direction.

[0112] The packing adjustment unit 710 may provide at least a portion of the syntax 2000 to the feature map intra coding unit 104. The feature map intra coding unit 104 may select a coding block size based on at least a portion of the syntax 2000 provided by the packing adjustment unit 710, and may code a packing feature frame. The feature map intra coding unit 104 may code at least a portion of the syntax 2000 provided by the packing adjustment unit 710, and generate a bitstream.

[0113] The unpacking adjuster 810 makes a predetermined adjustment to the packing feature frame as follows. At least a portion of the syntax 2000 is decoded and supplied to the unpacking adjuster 810 by the feature map internal decoder 201 at the previous stage. The feature map internal decoder 201 supplies at least a portion of the syntax 2000 to the unpacking adjuster 810. The feature map internal decoder 201 may decode at least a portion of the syntax 2000 from the bitstream. The feature map internal decoder 201 may select a coding block size based on at least a portion of the syntax 2000 and decode the packing feature frame. It is now assumed that the unpacking adjuster 810 receives an offset, a predetermined feature map size, and an adjusted padding size from the feature map internal decoder 201. That is, the horizontal offset offset_x, the vertical offset offset_y, the horizontal size Wf of the feature map, the vertical size Hf of the predetermined feature map, the adjusted padding width Wp, and the adjusted padding height Hp are supplied from the feature map internal decoding unit 201.

[0114] The unpacking adjuster 810 selects a feature map of a predetermined channel from the packing feature frame supplied from the feature map internal decoder 201 as the predetermined feature map 1902 (S1802). The unpacking adjuster 810 adds the size of the predetermined feature map and the adjusted padding size to calculate an adjusted block size (S1804). That is, the unpacking adjuster 810 adds the horizontal size of the predetermined feature map and the adjusted padding width to calculate an adjusted block width Wa. Similarly, the unpacking adjuster 810 adds the vertical size of the predetermined feature map and the adjusted padding height to calculate an adjusted block height Ha.

[0115] The unpacking adjuster 810 deletes an area of ​​the padding size from an area of ​​the adjusted block size that includes a predetermined feature map (S1806). That is, the unpacking adjuster 810 deletes padding from the left side of an area of ​​the adjusted block width Wa that includes the predetermined feature map, the number of pieces of padding equal to the horizontal offset offset_x. The unpacking adjuster 810 also deletes padding from the top side of an area of ​​the adjusted block height Ha that includes the predetermined feature map, the number of pieces of padding equal to the difference between the padding width Wp and the horizontal offset offset_x. The unpacking adjuster 810 also deletes padding from the bottom side of an area of ​​the adjusted block height Ha that includes the predetermined feature map, the number of pieces of padding equal to the difference between the padding height Hp and the vertical offset offset_y.

[0116] The unpacking adjuster 810 determines whether there is a feature map of a channel that has not yet been selected as a predetermined feature map (S1808). If there is a feature map of a channel that has not yet been selected as a predetermined feature map (S1808: Yes), the unpacking adjuster 810 selects that feature map as a predetermined feature map (S1802). The unpacking adjuster 810 then repeats the process from S1802. If there is no feature map of a channel that has not yet been selected as a predetermined feature map (S1808: No), the unpacking adjuster 810 ends the process.

[0117] The unpacking adjuster 810 removes padding of the adjusted padding size from the feature maps of all channels included in the packing feature frame supplied from the feature map internal decoder 201 , and generates a packing feature frame 1500 .

[0118] In all of the above-described embodiments, the bitstream output by the feature map encoder has a specific data format that can be decoded according to the encoding method used in the embodiment, and the feature map decoder corresponding to the feature map encoder can decode the bitstream in this specific data format.

[0119] When a wired or wireless network is used to exchange bitstreams between the feature map encoder and the feature map decoder, the bitstreams may be converted into a data format suitable for the transmission mode of the communication channel before transmission. In this case, a transmitter is provided that converts the bitstream output by the feature map encoder into coded data in a data format suitable for the transmission mode of the communication channel and transmits the coded data to the network, and a receiver is provided that receives the coded data from the network, restores the coded data to a bitstream, and supplies the bitstream to the feature map decoder. The transmitter includes a memory that buffers the bitstream output by the feature map encoder, a packet processor that packetizes the bitstream, and a transmitter that transmits the packetized coded data via the network. The receiver includes a receiver that receives the packetized coded data via the network, a memory that buffers the received coded data, and a packet processor that packetizes the coded data to generate a bitstream and provides it to the feature map decoder.

[0120] The above encoding and decoding processes may be realized not only as a transmission, storage, and receiving device using hardware, but also as firmware stored in a ROM (read-only memory) or flash memory, or as software for a computer, etc. The firmware program or software program may be provided by recording it on a computer-readable recording medium, or may be provided from a server via a wired or wireless network, or may be provided as data broadcasting on terrestrial or satellite digital broadcasting.

[0121] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the components and processing steps, and that such modifications are also within the scope of the present invention. [Explanation of symbols]

[0122] 100 Feature map encoding device, 101 Neural network feature extraction unit, 102 Feature map reduction unit, 103 Feature map conversion unit, 104 Feature map internal encoding unit, 200 Feature map decoding device, 201 Feature map internal decoding unit, 202 Feature map inverse conversion unit, 203 Feature map restoration unit, 204 Neural network identification unit, 301 Convolution processing unit, 302 Activation processing unit, 303 Pooling processing unit, 322 Bottom-up processing unit, 324 Top-down processing unit, 326 Image to be subjected to feature extraction processing, 501 First feature map reduction unit, 502 First channel combination unit, 503 Second feature map reduction unit, 504 Second channel combination unit, 505 Third feature map reduction unit, 506 First padding unit, 507 Second padding unit, 508 Third padding unit, 601 8x magnification unit, 602 4x magnification unit, 603 2x magnification unit, 604 first feature map mixing unit, 605 second feature map mixing unit, 606 first unpadding unit, 607 second unpadding unit, 608 third unpadding unit, 701 packing unit, 702 feature map quantization unit, 801 feature map inverse quantization unit, 802 unpacking unit, 901 switch, 902 VVC encoding unit, 903 HEVC encoding unit, 904 AV1 encoding unit, 1001 switch, 1002 VVC decoding unit, 1003 HEVC decoding unit, 1004 AV1 decoding unit.

Claims

1. a packing unit that generates a feature frame by combining multiple feature maps into one frame; a packing adjustment unit that adds padding of a predetermined size to the feature frame; a feature map intra-coding unit that intra-codes information about the padding used by the packing adjustment unit and the feature frame to which the padding has been added; A feature map encoding device comprising:

2. a feature map internal decoding unit that internally decodes a feature frame in which a plurality of feature maps are combined into one frame and padding is added, and information about the padding; an unpacking adjuster that removes the padding from the feature frame based on information about the padding; an unpacking unit that extracts a plurality of the feature maps from the feature frame from which the padding has been removed; A feature map decoding device comprising:

Images