Feature map coding device, feature map coding method, feature map decoding device, feature map decoding method

Abstract

This provides technology for encoding and decoding feature maps. [Solution] The present invention provides a feature map encoding device comprising: a feature map reduction unit that converts a multi-scale feature map to generate a single-scale feature map; a packing unit that identifies the scan type of the single-scale feature map, packs the single-scale feature map into a frame based on the identified scan type to generate a packed feature frame; an encoding unit that encodes the scan type; and a feature map internal encoding unit that encodes the packed feature frame.

Description

Technical Field

[0001] It relates to the encoding and decoding of feature maps in a neural network.

Background Art

[0002] As a neural network technology used for image recognition such as detection of objects of various scales in an image, division of regions for each object, or tracking of an object, FPN (Feature Pyramid Network) of Non-Patent Document 1 is known. In FPN, a plurality of feature maps of various scales are generated from an image to be processed, and various image recognitions are performed using the feature maps.

[0003] FPN used for image recognition generates a plurality of feature maps from an image, and its structure utilizes CNN (Convolutional Neural Network). CNN reads an image, is composed of convolution and pooling, and can be divided into a feature amount extraction unit (backbone) that generates a feature map and an identification unit (head) that is composed of hierarchical fully connected layers and generates an output suitable for tasks such as object detection, instance segmentation, and object tracking. FPN utilizes the backbone of CNN.

[0004] The feature amount extraction unit of FPN is typically configured with a convolutional processing unit 301, an activation processing unit 302, and a pooling processing unit 303 shown in FIG. 3 as one basic unit, and has a hierarchical structure that repeats this basic unit.

[0005] Figure 4 shows the structure of the FPN. The FPN consists of a bottom-up processing unit 322 that generates a multi-scale feature map composed of multiple hierarchical layers using a CNN backbone, and a top-down processing unit 324 that aggregates features from deeper layer feature maps to shallower layer feature maps using the inverse configuration of the CNN backbone. The bottom-up processing unit 322 repeatedly performs the convolution processing unit 301, activation processing unit 302, and pooling processing unit 303, which are the basic units in Figure 3, reducing the resolution of the feature map by half each time to generate a pyramid of multiple layered feature maps. On the other hand, the top-down processing unit 324 adds feature maps with resolutions corresponding to the bottom-up processing unit 322, expanding the resolution of the feature map to the same resolution as the input image to generate a pyramid of feature maps. In other words, the FPN generates multiple feature maps for each layer from the image 326 targeted for feature extraction processing.

[0006] The convolution processing unit 301 performs convolution on the data to be processed (image or feature map) using multiple predetermined filters (kernels). In the convolution processing in the convolution processing unit 301, predetermined filtering is performed on the entire data to be processed while sliding at predetermined intervals. At this time, the sliding interval is called the stride. The convolution processing unit 301 may determine the stride based on the number of data to be processed. For example, the convolution processing unit 301 may determine the stride to be 1 if the number of data to be processed is less than a predetermined value, and determine the stride to be 2 if it is greater than or equal to the predetermined value. Multiple feature maps are generated by preparing multiple predetermined filters at each layer and generating one feature map for each filter. The unit of a feature map is called a channel. If the number (types) of predetermined filters is N (N types), then N (N channels) feature maps are generated.

[0007] The activation processing unit 302 performs an activation process that non-linearly transforms the feature map output from the convolution processing unit 301. Here, the function used for the activation process is called the activation function. The activation processing unit 302 uses the ReLU (Rectified Linear Unit) function or the sigmoid function, etc., as the activation function.

[0008] The pooling processing unit 303 is a process that downsamples the feature map output from the activation processing unit 302 by replacing local values ​​of the feature map with representative values.

[0009] On the other hand, when performing classification using a neural network, it has the capability to execute the task using feature maps of multiple channels at each layer.

[0010] In image recognition, the multi-channel feature maps of each hierarchical level are subjected to a convolution process at predetermined size intervals based on the scale of the feature maps, and the probability of the object's class is calculated for each pixel. [Prior art documents] [Non-patent literature]

[0011] [Non-Patent Document 1] Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature Pyramid Networks for Object Detection. In CVPR, 2017. [Overview of the project] [Problems that the invention aims to solve]

[0012] The amount of information in feature maps is enormous, making them unsuitable for transmission and storage. In view of the above problems, the present invention aims to provide a technology for encoding and decoding feature maps. [Means for solving the problem]

[0013] To solve the above problems, the feature map encoding device of the present invention comprises: a feature map reduction unit that converts a multi-scale feature map to generate a single-scale feature map; a packing unit that identifies the scan type of the single-scale feature map, packs the single-scale feature map into a frame based on the identified scan type to generate a packed feature frame; an encoding unit that encodes the scan type; and a feature map internal encoding unit that encodes the packed feature frame. [Effects of the Invention]

[0014] According to the present invention, feature maps can be encoded and decoded efficiently with minimal processing load. [Brief explanation of the drawing]

[0015] [Figure 1] This is a block diagram illustrating the configuration of the feature map encoding device 100. [Figure 2] This is a block diagram illustrating the configuration of the feature map decoding device 200. [Figure 3] This block diagram illustrates the basic unit of processing in each layer of the FPN. [Figure 4] This is a block diagram illustrating the structure of the FPN. [Figure 5] This is a block diagram illustrating the detailed configuration of the feature map reduction unit 102. [Figure 6] This is a block diagram illustrating the detailed configuration of the feature map restoration unit 203. [Figure 7] This is a block diagram illustrating the detailed configuration of the feature map conversion unit 103. [Figure 8] This is a block diagram illustrating the detailed configuration of the feature map inverse transform unit 202. [Figure 9] This is a block diagram illustrating the detailed configuration of the feature map internal encoding unit 104. [Figure 10] It is a block diagram for explaining the detailed configuration of the feature map internal decoding unit 201. [Figure 11] It is a diagram for explaining the number of channels, the width, and the height of the feature maps x1, x2, and x3. [Figure 12] It is a flowchart for explaining the operation of generating a packed feature frame in the packing unit 701. [Figure 13] It is a diagram for explaining flipping when packing multiple-channel feature maps into one frame. [Figure 14] It is a diagram for explaining the layers and units handled by the feature map encoding device and the feature map decoding device of the present embodiment. [Figure 15] It is a diagram for explaining the scaling coefficient and the inverse scaling coefficient. [Figure 16] It is a diagram for explaining the syntax element rearrangement_mode. [Figure 17] It is a diagram for explaining raster scan. [Figure 18] It is a diagram for explaining diagonal scan. [Figure 19] It is a diagram for explaining thinning scan. [Figure 20] It is a flowchart for explaining the operation of generating a single-scale feature map from the packed feature frame in the unpacking unit 802. [Figure 21] It is a diagram for explaining the channel rearrangement in the packing unit 701. [Figure 22] It is a diagram for explaining scan in the second embodiment. [Figure 23] It is a diagram for explaining the syntax structure in the second embodiment. [Figure 24] It is a flowchart for explaining the operation of the packing unit 701 in the second embodiment. [Figure 25]This is a flowchart illustrating the operation of the unpacking unit 802 in the second embodiment.

[0016] [Modes for carrying out the invention]

[0017] This section defines the technologies and technical terms used in this embodiment.

[0018] <Features and Feature Maps> In a convolutional neural network (CNN), a filter is used to extract features from the input layer data. The data obtained by scanning the target image (input layer data) while changing its position, and the filter coefficients are then combined to produce features or feature maps.

[0019] <Packing> The process of combining two or more frames (pictures) into a single frame (picture) by arranging them in a tile-like manner is called frame packing. In this application, packing refers to the process of combining feature maps of multiple channels into a single frame.

[0020] <Data type> A data type that represents integer values ​​is called an integer type, and a data type that represents decimal values ​​is called a decimal type.

[0021] <Layer, Unit> Figure 14 illustrates the layers and units handled by the feature map encoding and decoding devices of this embodiment. Figure 14 shows a sequence of consecutive feature frames or feature maps for all channels over time, and is referred to as a sequence layer or sequence unit. Figure 14 shows a sequence of consecutive feature maps for one channel over time, and is referred to as a sequence layer for each channel or a sequence unit for each channel. Figure 14 shows a feature frame or feature map for all channels at the same time, and is referred to as a frame layer or frame unit. Figure 14 shows a feature map for one channel at a certain time, and is referred to as a feature map layer for each channel (a feature map unit for each channel).

[0022] (First Embodiment) A feature map encoding device 100 and a feature map decoding device 200 according to a first embodiment of the present invention will be described.

[0023] Figure 1 is a block diagram of a feature map encoding device 100 according to the first embodiment. The feature map encoding device 100 of this embodiment includes a feature map reduction unit 102, a feature map conversion unit 103, and a feature map internal encoding unit 104. The feature map encoding device 100 is a device that encodes the feature map generated by the neural network feature extraction unit 101 to generate a bitstream and output it.

[0024] The neural network feature extraction unit 101 reads the image to be feature extracted, generates a feature map through convolution, activation, and pooling processes using an FPN, and supplies it to the feature map reduction unit 102. In this embodiment, a three-layer multi-scale feature map x1, x2, and x3 is generated.

[0025] The feature map reduction unit 102 converts the three-layer multi-scale feature maps x1, x2, and x3 obtained from the neural network feature extraction unit 101 into a single-layer single-scale feature map xf and supplies it to the feature map conversion unit 103. The details of the feature map reduction unit 102 will be explained in detail with reference to Figure 5.

[0026] The feature map conversion unit 103 takes the fractional single-scale feature map xf supplied from the feature map reduction unit 102, performs packing and quantization processing to convert it into an integer-type packed feature frame, and supplies it to the feature map internal encoding unit 104.

[0027] The details of the feature map conversion unit 103 will be explained in detail with reference to Figure 7.

[0028] The feature map internal encoding unit 104 encodes the integer-type packing feature frame supplied from the feature map conversion unit 103 using an image encoding standard such as VVC, HEVC, or AV1 to generate and output a bitstream. The output bitstream is supplied to the feature map decoding device 200, etc., via a network or the like.

[0029] The details of the feature map internal encoding unit 104 will be explained in detail with reference to Figure 9.

[0030] Figure 2 is a block diagram showing the configuration of a feature map decoding device 200 according to an embodiment of the present invention, corresponding to the feature map encoding device 100 in Figure 1. The feature map decoding device 200 of this embodiment includes a feature map internal decoding unit 201, a feature map inverse conversion unit 202, and a feature map reconstruction unit 203. The feature map decoding device 200 also receives a bitstream encoded by the feature map encoding device 100, decodes the bitstream to generate three-layer multi-scale feature maps x1up, x2up, and x3up, and supplies them to the neural network identification unit 204.

[0031] The feature map internal decoding unit 201 decodes the bitstream encoded by the feature map internal encoding unit 104 of the feature map encoding device 100 using an image encoding standard such as VVC, HEVC, or AV1, generates an integer-type packing feature frame, and supplies it to the feature map inverse transform unit 202.

[0032] The details of the feature internal decoding unit 201 will be explained in detail with reference to Figure 10.

[0033] The feature map inverse transform unit 202 performs inverse quantization and unpacking on the integer-type packing feature frame supplied from the feature map internal decoding unit 201, converts it into a fractional-type single-scale feature map xr, and supplies it to the feature map reconstruction unit 203.

[0034] The details of the feature map inverse transform unit 202 will be explained in detail with reference to Figure 8.

[0035] The feature map reconstruction unit 203 converts the single-scale feature map xr supplied from the feature map inverse transformation unit 202 into three-layer multi-scale feature maps x1up, x2up, and x3up, and supplies them to the neural network identification unit 204 as the output of the feature map decoding device 200.

[0036] The details of the feature map reconstruction unit 203 will be explained in detail with reference to Figure 6.

[0037] The neural network identification unit 204 performs identification processing such as identifying objects in the target image, identifying locations and landscapes, and identifying people and living things, based on the three-layer multi-scale feature maps x1up, x2up, and x3up supplied by the feature map reconstruction unit 203.

[0038] <About Feature Map Reduction and Feature Map Restoration> The feature map reduction unit 102 has the function of converting a multi-layer multi-scale feature map obtained from the neural network feature extraction unit 101 into a single-layer single-scale feature map.

[0039] The feature map reduction unit 102 is described in detail using Figure 5. The feature map reduction unit 102 consists of a first feature map reduction unit 501, a first channel coupling unit 502, a second feature map reduction unit 503, a second channel coupling unit 504, a third feature map reduction unit 505, a first padding unit 506, a second padding unit 507, a third padding unit 508, and an amplification unit 509. The feature map reduction unit 102 in Figure 5 is an example of a configuration that converts a three-layer multi-scale feature map into a single-scale feature map.

[0040] The feature map reduction unit 102 takes a three-layer multi-scale feature map, consisting of a first feature map x1, a second feature map x2, and a third feature map x3, as input, converts it into a single-layer single-scale feature map xf, and supplies it to the feature map conversion unit 103. Here, n is the index indicating the layer, Cn is the number of channels in the nth layer, Wn is the width of the feature map, and Hn is the height of the feature map. In this embodiment, the values ​​of Cn, Wn, and Hn for each layer are as shown in Figure 11. However, H and W are the width and height of the image from which feature extraction is performed, respectively.

[0041] The first padding unit 506 has the function of performing padding on the first feature map x1 and generating the first padded feature map x1pad. The first padding unit 506 determines the padding size such that the width and height of x1pad are multiples of 64. The number of channels in x1pad is the same as x1, which is 256.

[0042] The second padding unit 507 has the function of performing wrapping padding on the second feature map x2 to generate the second padded feature map x2pad. The second padding unit 507 determines the padding size such that the width and height of x2pad are multiples of 32. The number of channels in x2pad is the same as x2, which is 256.

[0043] The third padding unit 508 has the function of performing wrapping padding on the third feature map x3 to generate the third padded feature map x3pad. The third padding unit 508 determines the padding size such that the width and height of x3pad are multiples of 16. The number of channels in x3pad is the same as x3, which is 256.

[0044] In the first padding section 506, the second padding section 507, and the third padding section 508, the padding size on the left and the padding size on the right are the same, and the padding size on the top and the padding size on the bottom are the same. That is, feature maps x1, x2, and x3 are positioned at the center of x1pad, x2pad, and x3pad, respectively.

[0045] The first feature map reduction unit 501 performs convolution in the spatial and channel directions on the first padded feature map x1pad obtained from the first padding unit 506 to generate the first intermediate feature map y1. The number of channels in y1 is 192, the width is Wx1pad / 2, and the height is Hx1pad / 2. Here, Wx1pad and Hx1pad are the width and height of the first padded feature map x1pad, respectively.

[0046] The first channel merging unit 502 has the function of combining the first intermediate feature map y1 obtained from the first feature map reduction unit 501 and the second padded feature map x2pad obtained from the second padding unit 507 in the channel direction to generate an intermediate feature map y1Cx2pad. Since y1 has 192 channels and x2pad has 256 channels, the intermediate feature map y1Cx2pad has 448 channels (192 + 256).

[0047] The second feature map reduction unit 503 performs convolution in the spatial and channel directions on the intermediate feature map y1Cx2pad obtained from the first channel joining unit 502 to generate the second intermediate feature map y2. The number of channels in y2 is 192, the width is Wy1Cx2pad / 2, and the height is Hy1Cx2pad / 2. Here, Wy1Cx2pad and Hy1Cx2pad are the width and height of the intermediate feature map y1Cx2pad, respectively.

[0048] The second channel merging unit 504 has the function of combining the second intermediate feature map y2 obtained from the second feature map reduction unit 503 and the third padded feature map x3pad obtained from the third padding unit 508 in the channel direction to generate an intermediate feature map y2Cx3pad. Since the intermediate feature map y2 has 192 channels and x3pad has 256 channels, the number of channels in y1Cx2pad is 448 (192 + 256).

[0049] The third feature map reduction unit 505 performs convolution in the spatial and channel directions on the intermediate feature map y2Cx3pad obtained from the second channel joining unit 504 to generate a third intermediate feature map y3. The number of channels in y3 is 192, the width is Wy2Cx3pad / 2, and the height is Hy2Cx3pad / 2. Here, Wy2Cx3pad and Hy2Cx3pad are the width and height of the intermediate feature map y2Cx3pad, respectively.

[0050] The amplification unit 509 expands the range of signal values ​​by multiplying all elements of each channel of the feature map y3 obtained from the third feature map reduction unit 505 by a scaling factor. Let xr be the feature map with the expanded signal range. As shown in Figure 15, the scaling factor is specified on a per-channel basis. In this example, the scaling factor for channel number 1 is 1. Therefore, in this case, the signal values ​​of the elements of channel 1 are not changed. The scaling factor for channel 2 is 10, and the elements of channel 2 are multiplied by 10. The same applies to the following channels.

[0051] The scaling coefficients are assumed to be determined in advance using appropriate parameters and should not be changed during the sequence. For example, the scaling coefficients can be determined through prior machine learning.

[0052] The scaling coefficient value indicates the importance of each channel, i.e., channel importance. Channels with a large scaling coefficient, i.e., channels with high channel importance, have a wide range of signal values. Therefore, if encoding degradation occurs in the feature map internal encoding unit 104, it will have a significant impact on the image recognition result. For this reason, it is important to suppress encoding degradation in channels with high channel importance and improve the image recognition result.

[0053] The feature map reduction unit 102 supplies the single-scale feature map xf to the feature map conversion unit 103.

[0054] The feature map reconstruction unit 203 has the function of converting the single-scale feature map xr obtained from the feature map inverse transformation unit 202 into three-layer multi-scale feature maps x1up, x2up, and x3up.

[0055] The details of the feature map reconstruction unit 203 will be explained using Figure 6. The feature map reconstruction unit 203 consists of an inverse amplification unit 600, an 8x magnification unit 601, a 4x magnification unit 602, a 2x magnification unit 603, a first feature map mixing unit 604, a second feature map mixing unit 605, a first padding removal unit 606, a second padding removal unit 607, and a third padding removal unit 608.

[0056] The inverse amplification unit 600 multiplies all elements of each channel of the single-scale feature map xr obtained from the feature map inverse transformation unit 202 by an inverse scaling coefficient to reduce the range of signal values. Let this be the feature map xr' with the reduced range of signal values. This process is equivalent to restoring the feature map, whose range of signal values ​​was expanded by the amplification unit 509 of the feature map reduction unit 102, back to its original range. As shown in Figure 15, the inverse scaling coefficient is specified on a per-channel basis. The inverse scaling coefficient is the reciprocal of the scaling coefficient. Therefore, the inverse scaling coefficient for channel number 1 is 1 / 1. In this case, the signal values ​​of the elements of channel 1 are not changed. The inverse scaling coefficient for channel 2 is 1 / 10, and the elements of channel 2 are multiplied by 1 / 10. The same applies to the following channels.

[0057] The inverse scaling coefficient, like the scaling coefficient, is assumed to have appropriate parameters determined in advance and not be changed during the sequence. For example, the scaling coefficient can be determined by prior machine learning.

[0058] Channel importance can also be set using an inverse scaling factor instead of scaling. The value of the inverse scaling factor is the reciprocal of the scaling factor. Similar to the scaling factor, this indicates the importance of each channel, i.e., channel importance. Since the inverse scaling factor is the reciprocal of the scaling factor, the relative order is reversed. Therefore, channels with smaller inverse scaling factors have higher channel importance.

[0059] In Figure 15, the inverse scaling coefficient is defined as the reciprocal of the scaling coefficient, but it may be adjusted to take a different value. For example, in the process of generating integer-type packing feature frames in the feature map internal decoding unit 201, channels with a large assigned code amount contain a large amount of information. Therefore, the channel importance of channels with a large assigned code amount can be set high, and the channel importance of channels with a small assigned code amount can be set low.

[0060] The 8x magnification unit 601 has the function of expanding the feature map and reducing the number of channels by performing transposition convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr' acquired from the inverse amplification unit 600, thereby generating an intermediate feature map z1. The number of channels in z1 is 196. If the width and height of the single-scale feature map xr are xrwidth and xrheight, respectively, then the width and height of z1 are xrwidth × 8 and xrheight × 8, respectively. Here, rwidth × 8 and xrheight × 8 are the same as the width and height of the first padded feature map x1pad, which is the output of the first padding unit 506 of the feature map reduction unit 102.

[0061] The quadruple magnification unit 602 has the function of expanding features and reducing channels by performing transposition convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr obtained from the feature map inverse transformation unit 202, thereby generating an intermediate feature map z2. The number of channels in z2 is 196. The width and height of z2 are xrwidth × 4 and xrheight × 4, respectively. Here, rwidth × 4 and xrheight × 4 are the same as the width and height of the second padded feature map x2pad, which is the output of the second padding unit 507 of the feature map reduction unit 102.

[0062] The doubling unit 603 has the function of expanding the feature map and reducing the number of channels of the single-scale feature map xr obtained from the feature map inverse transformation unit 202 by performing transposition convolution in the spatial direction and convolution in the channel direction, thereby generating an intermediate feature map z3. The number of channels in z3 is 196. The width and height of z3 are xrwidth × 2 and xrheight × 2, respectively. Here, rwidth × 2 and xrheight × 2 are the same as the width and height of the third padded feature map x3pad, which is the output of the third padding unit 508 of the feature map reduction unit 102.

[0063] The first feature map mixing unit 604 has the function of generating an intermediate feature map z2up with improved quality from the intermediate feature map z2 obtained from the 4x magnification unit 602, using the intermediate feature map z1 obtained from the 8x magnification unit 601.

[0064] The second feature map mixing unit 605 has the function of generating an intermediate feature map z3up with improved quality from the intermediate feature map z3 obtained from the doubling unit 603, using the intermediate feature map z2up obtained from the first feature map mixing unit 604.

[0065] The padding removal unit 606 has the function of removing padding from the intermediate feature map z1 acquired from the 8x magnification unit 601 and generating a first output feature map x1up. The width and height of x1up are the same as the width and height of the first feature map x1 input to the feature map reduction unit 102.

[0066] The padding removal unit 607 has the function of removing padding from the intermediate feature map z2up obtained from the first feature map mixing unit 604 and generating a second output feature map x2up. The width and height of x2up are the same as the width and height of the second feature map x2 input to the feature map reduction unit 102.

[0067] The padding removal unit 608 has the function of removing padding from the intermediate feature map z3up obtained from the second feature map mixing unit 605 and generating a third output feature map x3up. The width and height of x3up are the same as the width and height of the third feature map x3 input to the feature map reduction unit 102.

[0068] In the first padding removal section 606, the second padding removal section 607, and the third padding removal section 608, similar to the first padding section 506, the second padding section 507, and the third padding section 508 of the feature map reduction section 102, the padding size on the left and the padding size on the right are set to be the same, and the padding size on the upper and lower sides are set to be the same. That is, each output feature map x1up, x2up, and x3up is assumed to be positioned at the center of each intermediate feature map z1, z2up, and z3up, respectively, and padding is removed from the top, bottom, left, and right.

[0069] <About Feature Map Transformation and Inverse Feature Map Transformation> The feature map conversion unit 103 has the function of taking the multi-channel fractional single-scale feature map xf supplied from the feature map reduction unit 102, performing packing and quantization processing, and converting it into an integer-type packed feature frame for supply to the feature map internal encoding unit 104.

[0070] Figure 7 will be used to explain the details of the feature map conversion unit 103 on the encoding side. The feature map conversion unit 103 consists of a packing unit 701 and a feature map quantization unit 702.

[0071] The packing unit 701 has the function of generating a packed feature frame by combining the input feature maps of multiple channels into a single frame. The operation of generating a packed feature frame in the packing unit 701 will be explained using the flowchart in Figure 12.

[0072] First, the packing unit 701 sorts the feature maps of multiple channels in descending order of channel importance (step S110). In this embodiment, the reciprocal of the inverse scaling coefficient is used as the channel importance. If the importance is the same, the channels are sorted in descending order of channel number. Figure 21 shows an example of sorting 1102. For example, the feature maps of multiple channels are represented as an array. The index in Figure 21 corresponds to the subscript of the array.

[0073] Next, the packing unit 701 determines the scan order for the packed feature frames (step S120). In this embodiment, one of raster scan, diagonal scan, or decimation scan can be selected. The determined scan order is transmitted to the feature map reconstruction unit via the bitstream as the syntax element rearrangement_mode, as shown in Figure 16. In the case of a raster scan, starting from the top left of the packed feature frame, the feature maps for each channel are sequentially placed in the packed feature frame, moving to the right. Once the right edge of the packed feature frame is reached, the process moves down one line and the same operation is performed. Figure 17 shows an example of the packing order in a raster scan. In the case of a diagonal scan, the starting point is the top left of the packed feature frame. Next, move one step to the right and sequentially place the feature maps of each channel into the packed feature frame in the downward left direction. Once the left edge of the packed feature frame is reached, return to the top edge of the packed feature frame, move one step to the right, and repeat the same operation. Figure 18 shows an example of the packing order in a diagonal scan. In the case of decimation scanning, each line from top to bottom of the packed feature frame is scanned from left to right, skipping one frame (first scan). After reaching the bottom right of the packed feature frame, each line from bottom to top of the packed feature frame is scanned from right to left, skipping one frame (second scan). In the first scan, the top line is designated as 0. For even-numbered lines, the starting point is the leftmost position of the packed feature frame, and for odd-numbered lines, the starting point is one position to the right of the leftmost position of the packed feature frame. That is, for even-numbered lines, the starting point is one position to the right of the packed feature frame, and for odd-numbered lines, the same operation is performed as if the starting point were the leftmost position of the packed feature frame. In the second scan, the bottom line is designated as N. For lines where N is even or odd, the starting point is one position to the left of the rightmost position of the packed feature frame, and for lines where N is even or odd, the starting point is the rightmost position of the packed feature frame. Figure 19 shows an example of the packing order in a decimation scan.

[0074] Next, the packing unit 701 places the feature maps of multiple channels into the packed feature frame according to the scan order determined in step S102 (step S130). Furthermore, the packing unit 701 performs flipping based on the in-frame position where the channel feature maps are placed. It is also possible to make it possible to select whether or not to perform flipping and transmit this selection from the encoding side to the decoding side via the bitstream.

[0075] Figure 13 illustrates the flipping process when packing multiple channel feature maps into a single frame. Flipping involves reversing the position of the elements (pixels) of each channel's feature map horizontally (left / right), vertically (up / down), or both horizontally and vertically (up / down / left / right) when packing the feature maps for each channel. In Figure 13, the four channel feature maps A (top left), B (top right), C (bottom left), and D (bottom right) are treated as a single set. No flipping is performed at position A in Figure 13. At position B, the feature map is reversed horizontally (left / right). At position C, the feature map is reversed vertically (up / down). At position D, the feature map is reversed horizontally and vertically (up / down / left / right). When the distribution of elements in each channel's feature map is similar, performing flipping based on the in-frame position where the channels are placed reduces the boundaries between each channel's feature map, improving encoding efficiency.

[0076] In this embodiment, channels are sorted in order of channel importance, i.e., scaling factor. Channels with widely separated scaling factors tend to have lower correlation. In raster scanning, horizontally adjacent channels tend to have similar scaling factors, while vertically adjacent channels tend to have widely separated scaling factors. In contrast, this embodiment allows the use of diagonal scanning. Diagonal scanning can arrange channels with similar scaling factors on average in both horizontal and vertical directions, thereby improving coding efficiency.

[0077] Furthermore, this embodiment includes a decimation scan. This alternates between channels with a wide range of element values ​​and channels with a narrow range of element values. When channels with a wide range of element values ​​and low correlation are placed next to each other, the difference in values ​​at the channel boundaries can become extremely large, potentially significantly reducing coding efficiency. By adopting this embodiment, the maximum value difference at the channel boundaries can be suppressed, thereby improving coding efficiency.

[0078] When channels are rearranged, channels with similar ranges of element values ​​tend to cluster together, resulting in a higher correlation between adjacent channels on average. Therefore, it may be advisable to configure the system to perform flipping only when channels are rearranged.

[0079] The feature map quantization unit 702 has the function of converting the elements of a decimal-type packing feature frame (feature map of all channels) into an N-bit integer type (an integer N=8 to approximately 16) within a predetermined range and outputting an integer-type packing feature frame. In this embodiment, it is assumed that it is converted into a 10-bit integer type from 0 to 1023. The feature map quantization unit 702 detects the minimum and maximum values ​​of the elements of the decimal-type packing feature frame and transmits the detected minimum and maximum values ​​of the elements of the packing feature frame (feature map of all channels) to the decoding side as metadata. In order to convert from a decimal-type packing feature frame (feature map) to an integer-type packing feature frame (feature map), a linear transformation is performed in which the minimum value of the decimal type corresponds to the minimum value of the integer type, and the maximum value of the decimal type corresponds to the maximum value of the integer type. For example, when the range of the integer type is represented by 10 bits, the minimum value of the elements of the integer-type packing feature frame (feature map) is 0, and the maximum value is 1023 (2 10 -1) is obtained. Linear quantization is performed on values ​​between the minimum and maximum values.

[0080] Next, the feature map inverse transform unit 202 has the function of performing inverse quantization and unpacking on integer-type packing feature frames decoded by VVC, HEVC, AV1, etc., supplied from the feature map internal decoding unit 201, and transforming them into a decimal-type single-scale feature map xr for supply to the feature map reconstruction unit 203.

[0081] Figure 8 will be used to explain the details of the feature map inverse transform unit 202 on the decoding side. The feature map inverse transform unit 202 is the inverse process of the feature map transform unit 103 and is composed of a feature map inverse quantization unit 801 and an unpacking unit 802.

[0082] The feature map inverse quantization unit 801 performs the inverse processing of the encoding-side feature map quantization unit 702 and has the function of converting the elements of the integer-type packing feature frame from integer type to decimal type. The feature map inverse quantization unit 801 converts the integer-type packing feature frame decoded by the feature map internal decoding unit 201 into a decimal-type packing feature frame using the minimum and maximum decimal values ​​transmitted as metadata. A linear transformation is performed to make the minimum integer value equivalent to the minimum decimal value and the maximum integer value equivalent to the maximum decimal value. For values ​​between the minimum and maximum values, linear inverse quantization is performed.

[0083] The unpacking unit 802 extracts the feature maps for each channel from the packing feature frames arranged in a single frame in the order of the raster scan, and supplies them to the feature map reconstruction unit 203 as a single-scale feature map xr.

[0084] The operation of the unpacking unit 802 will be explained using Figure 20.

[0085] First, the unpacking unit 802 decodes rearrangement_mode from the bitstream, which indicates the scan order of the packing feature frames (S210). The scan order follows Figure 16, similar to that of the packing unit 701.

[0086] The unpacking unit 802 then processes the feature maps of multiple channels one by one from the packing feature frame, according to the scan order determined in step S210. After processing all channels, this process is terminated (step S220).

[0087] If not all channels have been processed, the unpacking unit 802 acquires the next channel in scan order (step S230). For the acquired channels, the channel number is determined based on the channel importance (step S240). In the example in Figure 15, channel number 2 has the highest channel importance (channel importance 10). Therefore, the channel number of the first acquired channel is set to 2. For the subsequent acquired channels, channel numbers 3 (channel importance 8), 4 (channel importance 4), and 7 (channel importance 2) are set sequentially. If the importance is the same, the channel numbers are set in ascending order.

[0088] <About internal encoding and decoding of feature maps> Figure 9 will be used to explain the details of the feature map internal encoding unit 104. The feature map internal encoding unit 104 consists of a switch 901, a VVC encoding unit 902, an HEVC encoding unit 903, and an AV1 encoding unit 904. The switch 901 selects the encoding standard for internally encoding the feature map converted by the feature map conversion unit 103. The VVC encoding unit 902 encodes the feature map using the VVC standard and outputs a bitstream compliant with the VVC standard. The HEVC encoding unit 903 encodes the feature map using the HEVC standard and outputs a bitstream compliant with the HEVC standard. The AV1 encoding unit 904 encodes the feature map using the AV1 standard and outputs a bitstream compliant with the AV1 standard.

[0089] In the VVC, HEVC, and AV1 standards, images are divided into predetermined block sizes and then encoded.

[0090] It is also possible to implement only one of the following: VVC, HEVC, or AV1. Furthermore, it is possible to use image encoding schemes other than VVC, HEVC, and AV1.

[0091] Next, the details of the feature map internal decoding unit 201 will be explained using Figure 10. The feature map internal decoding unit 201 consists of a switch 1001, a VVC decoding unit 1002, an HEVC decoding unit 1003, and an AV1 decoding unit 1004. The switch 1001 selects the encoding standard to be internally decoded based on the information in the input bitstream that is selected for internal decoding. The VVC decoding unit 1002 decodes the feature map using the VVC standard. The HEVC decoding unit 1003 decodes the feature map using the HEVC standard. The AV1 decoding unit 1004 decodes the feature map using the AV1 standard.

[0092] In the VVC, HEVC, and AV1 standards, decoding is performed for each predetermined block size.

[0093] It is also possible to implement only one of the following: VVC, HEVC, or AV1. Furthermore, it is possible to use image encoding schemes other than VVC, HEVC, and AV1. (Second Embodiment) This embodiment differs from the first embodiment in that region division is performed in the packing section 701 and the unpacking section 802. Figure 22 shows an example of packing a 32-channel single-scale feature map into a frame having a total size of 32 channels, with 8 channels horizontally and 4 channels vertically. In 1201, one frame is divided into sub-regions of 4 channels horizontally and 2 channels vertically. The sub-regions are scanned with a raster scan, and each sub-region is scanned in the same scan order as in the first embodiment. 1202 shows the scan order of the raster scan in this embodiment, and 1203 shows the scan order of the diagonal scan.

[0094] The operation of the packing unit 701 will be explained using the flowchart in Figure 24. Processes similar to those in the first embodiment are indicated by the same reference numerals as in the flowchart in Figure 12.

[0095] Step S121 is the step of determining the size of the sub-region. The horizontal size of the sub-region is set to subpartition_width, and the vertical size of the sub-region is set to subpartition_height. If sub-region division is not performed, the horizontal size and vertical size of the sub-region are set to the horizontal size and vertical size of the frame, respectively. A flag indicating that sub-region division is not performed may be transmitted. Alternatively, a flag is_subpartioning indicating whether or not sub-region division is performed may be set. In step S121, a scan of the sub-regions is performed. The scan order of the sub-regions is a raster scan, and the process terminates when all sub-regions have been processed.

[0096] The operation of the unpacking unit 802 will be explained using the flowchart in Figure 25. Processes similar to those in the first embodiment are indicated by the same reference numerals as in the flowchart in Figure 20.

[0097] Step S211 is the step of decoding the size of the subregion. The bitstream is decoded to obtain the horizontal size of the subregion subpartition_width and the vertical size of the subregion subpartition_height.

[0098] In Step S212, a partial region scan is performed. The partial region scan order is raster scan, and the process terminates after all partial regions have been processed.

[0099] Figure 23 shows an example of the syntax structure in this embodiment. The `rearrangement_flag` in Figure 23 is a flag indicating whether or not to rearrange the channels. The execution of the processes described in this embodiment can be switched on a sequence or frame-by-frame basis. By adopting this embodiment, the frame can be divided into sub-regions, and then the channels can be rearranged. Since channels included in the same sub-region are spatially close together, channels with high correlation in both the horizontal and vertical directions can be grouped together. Furthermore, sub-regions can be configured to match processing units, for example, in internal coding. For instance, highly correlated channels can be placed within the same coding block, enabling efficient arrangement from the perspective of the coding process in internal coding.

[0100] In all the embodiments described above, the bitstream output by the feature map encoding device has a specific data format so that it can be decoded according to the encoding method used in the embodiment. Furthermore, the feature map decoding device corresponding to this feature map encoding device can decode the bitstream of this specific data format.

[0101] When a wired or wireless network is used to exchange bitstreams between a feature map encoding device and a feature map decoding device, the bitstream may be converted to a data format suitable for the transmission mode of the communication channel before transmission. In this case, a transmitting device is provided that converts the bitstream output by the feature map encoding device into encoded data in a data format suitable for the transmission mode of the communication channel and transmits it to the network, and a receiving device is provided that receives the encoded data from the network, restores it to a bitstream, and supplies it to the feature map decoding device. The transmitting device includes a memory for buffering the bitstream output by the feature map encoding device, a packet processing unit for packetizing the bitstream, and a transmitting unit for transmitting the packetized encoded data over the network. The receiving device includes a receiving unit for receiving the packetized encoded data over the network, a memory for buffering the received encoded data, and a packet processing unit for packetizing the encoded data to generate a bitstream and providing it to the feature map decoding device.

[0102] The above encoding and decoding processes may be implemented not only as hardware-based transmission, storage, and receiving devices, but also by firmware stored in ROM (read-only memory) or flash memory, or by software on a computer. The firmware program or software program may be recorded on a recording medium readable by a computer and provided, provided from a server via a wired or wireless network, or provided as data broadcasting on terrestrial or satellite digital broadcasting.

[0103] The present invention has been described above based on embodiments. The embodiments are illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of their components and processing processes, and that such modifications also fall within the scope of the present invention. [Explanation of symbols]

[0104] 100 Feature map encoding unit, 101 Neural network feature extraction unit, 102 Feature map reduction unit, 103 Feature map transformation unit, 104 Feature map internal encoding unit, 200 Feature map decoding unit, 201 Feature map internal decoding unit, 202 Feature map inverse transformation unit, 203 Feature map restoration unit, 204 Neural network identification unit, 301 Convolution processing unit, 302 Activation processing unit, 303 Pooling processing unit, 322 Bottom-up processing unit, 324 Top-down processing unit, 326 Image to be processed for feature extraction, 501 First feature map reduction unit, 502 First channel joining unit, 503 Second feature map reduction unit, 504 Second channel joining unit, 505 Third feature map reduction unit, 506 First padding unit, 507 Second padding unit, 508 Third padding unit, 509 Amplification unit, 600 601 Inverse amplification section, 602 8x amplification section, 603 2x amplification section, 604 First feature map mixing section, 605 Second feature map mixing section, 606 First padding removal section, 607 Second padding removal section, 608 Third padding removal section, 701 Packing section, 702 Feature map quantization section, 801 Feature map inverse quantization section, 802 Unpacking section, 901 Switch, 902 VVC encoding section, 903 HEVC encoding section, 904 AV1 encoding section, 1001 Switch, 1002 VVC decoding section, 1003 HEVC decoding section, 1004 AV1 decoding section.

Claims

1. A feature map reduction unit that converts multi-scale feature maps to generate single-scale feature maps, A packing unit that identifies the scan type of the single-scale feature map, packs the single-scale feature map into a frame based on the identified scan type, and generates a packed feature frame. An encoding unit for encoding the scan type, A feature map internal encoding unit that encodes the aforementioned packing feature frame, Equipped with, A feature map coding device characterized by the following features.

2. A decoding unit that decodes the scan type of a single-scale feature map, A feature map internal decoding unit that decodes the single-scale feature map packed and encoded in a frame to generate a packed feature frame, An unpacking unit that unpacks the packing feature frame based on the decoded scan type to generate the single-scale feature map, A feature map reconstruction unit that converts the single-scale feature map to generate the multi-scale feature map, Equipped with, A feature map decoding device characterized by the following features.

Images