Feature map encoding device, feature map encoding method, feature map decoding device, and feature map decoding method

Abstract

Provided is a feature map encoding device (100) characterized by comprising: a feature map reduction unit (102) that converts a multi-scale feature map to generate a single-scale feature map; a packing unit (701) that packs the single-scale feature map into a frame on the basis of information indicating a channel importance of the single-scale feature map, to generate a packing feature frame; an encoding unit that encodes a scan type; and a feature map internal encoding unit (104) that encodes the information indicating the channel importance of the single-scale feature map.

Description

Feature Map Encoding Device, Feature Map Encoding Method, Feature Map Decoding Device, Feature Map Decoding Method

[0001] It relates to the encoding and decoding of feature maps in a neural network.

[0002] As a neural network technology used for image recognition such as detecting objects of various scales in an image, dividing regions for each object, or tracking objects, FPN (Feature Pyramid Network) in Non-Patent Document 1 is known. In FPN, a plurality of feature maps of various scales are generated from the image to be processed, and various image recognitions are performed using the feature maps.

[0003] FPN used for image recognition generates a plurality of feature maps from an image, and its structure utilizes CNN (Convolutional Neural Network). CNN reads an image, is composed of convolution and pooling, and can be divided into a feature extraction unit (backbone) that generates a feature map and an identification unit (head) that is composed of a hierarchical fully connected layer and generates an output suitable for tasks such as object detection, instance segmentation, and object tracking. FPN utilizes the backbone of CNN.

[0004] The feature extraction unit of FPN is typically structured in a hierarchical manner by repeating a basic unit composed of a convolution processing unit 301, an activation processing unit 302, and a pooling processing unit 303 shown in FIG. 3.

[0005] Figure 4 shows the structure of the FPN. The FPN consists of a bottom-up processing unit 322 that generates a multi-scale feature map composed of multiple hierarchical layers using the CNN backbone, and a top-down processing unit 324 that aggregates features from deeper layer feature maps to shallower layer feature maps using the inverse configuration of the CNN backbone. The bottom-up processing unit 322 repeatedly performs the convolution processing unit 301, activation processing unit 302, and pooling processing unit 303, which are the basic units in Figure 3, reducing the resolution of the feature map by half each time to generate a pyramid of multiple layered feature maps. On the other hand, the top-down processing unit 324 adds feature maps with resolutions corresponding to the bottom-up processing unit 322, expanding the resolution of the feature map to the same resolution as the input image, and generating a pyramid of feature maps. In other words, the FPN generates multiple feature maps for each layer from the image 326 that is the target of feature extraction processing.

[0006] The convolution processing unit 301 performs convolution on the data to be processed (image or feature map) using a plurality of predetermined filters (kernels). In the convolution processing in the convolution processing unit 301, predetermined filtering is performed on the entire data to be processed while sliding at predetermined intervals. At this time, the sliding interval is called the stride. The convolution processing unit 301 may determine the stride based on the number of data to be processed. For example, the convolution processing unit 301 may determine the stride to be 1 if the number of data to be processed is less than a predetermined value, and determine the stride to be 2 if it is greater than or equal to the predetermined value. Multiple feature maps are generated by preparing a plurality of predetermined filters at each layer and generating one feature map for each filter. The unit of a feature map is called a channel. If the number (types) of predetermined filters is N (N types), then N (N channels) of feature maps are generated.

[0007] The activation processing unit 302 performs an activation process that non-linearly transforms the feature map output from the convolution processing unit 301. Here, the function used for the activation process is called the activation function. The activation processing unit 302 uses the ReLU (Rectified Linear Unit) function or the sigmoid function, etc., as the activation function.

[0008] The pooling processing unit 303 performs a process of downsampling the feature map by replacing the local values ​​of the feature map output from the activation processing unit 302 with representative values.

[0009] On the other hand, when performing classification using a neural network, it has the capability to execute the task using feature maps of multiple channels at each layer.

[0010] In image recognition, the multi-channel feature maps of each hierarchical level are subjected to a convolution process at predetermined size intervals based on the scale of the feature maps, and the probability of the object's class is calculated for each pixel.

[0011] Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature Pyramid Networks for Object Detection. In CVPR, 2017.

[0012] The amount of information in feature maps is enormous, making them unsuitable for transmission and storage. In view of the above problems, this embodiment aims to provide a technology for encoding and decoding feature maps.

[0013] To solve the above problems, the feature map encoding device of this embodiment includes a feature map reduction unit that converts a multi-scale feature map to generate a single-scale feature map, a packing unit that packs the single-scale feature map into a frame based on information indicating the channel importance of the single-scale feature map to generate a packed feature frame, an encoding unit that encodes the scan type, and an internal feature map encoding unit that encodes information indicating the channel importance of the single-scale feature map.

[0014] The feature map decoding device of this embodiment includes: a decoding unit that decodes information indicating the channel importance of a single-scale feature map; an internal feature map decoding unit that decodes the single-scale feature map packed and encoded in a frame to generate a packing feature frame; an unpacking unit that unpacks the packing feature frame based on the information indicating the channel importance of the single-scale feature map to generate the single-scale feature map; and a feature map restoration unit that converts the single-scale feature map to generate the multi-scale feature map.

[0015] According to this embodiment, feature maps can be encoded and decoded efficiently with minimal processing load.

[0016] This is a block diagram illustrating the configuration of the feature map encoding device 100. This is a block diagram illustrating the configuration of the feature map decoding device 200. This is a block diagram illustrating the basic unit processing at each layer of the FPN. This is a block diagram illustrating the structure of the FPN. This is a block diagram illustrating the detailed configuration of the feature map reduction unit 102. This is a block diagram illustrating the detailed configuration of the feature map restoration unit 203. This is a block diagram illustrating the detailed configuration of the feature map conversion unit 103. This is a block diagram illustrating the detailed configuration of the feature map inverse conversion unit 202. This is a block diagram illustrating the detailed configuration of the feature map internal encoding unit 104. This is a block diagram illustrating the detailed configuration of the feature map internal decoding unit 201. This is a diagram illustrating the number of channels, width, and height of feature maps x1, x2, and x3. This is a flowchart illustrating the operation of generating packed feature frames in the packing unit 701. This is a diagram illustrating flipping when packing multiple channel feature maps into one frame. This is a diagram illustrating the layers and units handled by the feature map encoding device and feature map decoding device of this embodiment. This is a diagram illustrating the scaling factor and the inverse scaling factor. This is a diagram illustrating the syntax element rearrangement_mode. This is a diagram illustrating raster scanning. This is a diagram illustrating vertical-first scanning. This is a diagram illustrating the syntax element rearrangement_order. This is a flowchart illustrating the operation of generating a single-scale feature map from the packing feature frame in the unpacking unit 802. This is a diagram illustrating the channel reordering in the packing unit 701. This is a diagram illustrating the syntax structure in the first embodiment. This is a diagram illustrating the syntax structure in the second embodiment. This is a flowchart illustrating the operation of the packing unit 701 in the second embodiment. This is a flowchart illustrating the operation of the unpacking unit 802 in the second embodiment. This is a diagram illustrating the syntax structure in the third embodiment.This is a flowchart illustrating the operation of the packing unit 701 in the third embodiment. This is a flowchart illustrating the operation of the unpacking unit 802 in the third embodiment.

[0017] This section defines the technologies and technical terms used in this embodiment.

[0018] <Features and Feature Maps> In a convolutional neural network (CNN), the data obtained by convolving the data of the portion scanned while changing its position within the target image (input layer data) using a filter, with the filter coefficients, is called a feature or feature map.

[0019] <Packing> Frame packing refers to the process of combining two or more frames (pictures) into a single frame (picture) by arranging them in a tile-like manner. In this application, packing refers to the process of combining feature maps of multiple channels into a single frame.

[0020] <Data Types> Data types that represent integer values ​​are designated as integer types, and data types that represent decimal values ​​are designated as decimal types.

[0021] <Layers and Units> The layers and units handled by the feature map encoding and feature map decoding devices of this embodiment will be explained using Figure 14. Figure 14 shows a sequence of consecutive feature frames or feature maps for all channels over time, and is referred to as a sequence layer or sequence unit. Figure 14 shows a sequence of consecutive feature maps for one channel over time, and is referred to as a sequence layer for each channel or a sequence unit for each channel. Figure 14 shows a feature frame or feature map for all channels at the same time, and is referred to as a frame layer or frame unit. Figure 14 shows a feature map for one channel at a certain time, and is referred to as a feature map layer for each channel (a feature map unit for each channel).

[0022] (First Embodiment) A feature map encoding device 100 and a feature map decoding device 200 according to the first embodiment of the present invention will be described.

[0023] Figure 1 is a block diagram of a feature map encoding device 100 according to the first embodiment. The feature map encoding device 100 of this embodiment includes a feature map reduction unit 102, a feature map conversion unit 103, and a feature map internal encoding unit 104. The feature map encoding device 100 is a device that encodes the feature map generated by the neural network feature extraction unit 101 to generate a bitstream and output it.

[0024] The neural network feature extraction unit 101 reads the image to be feature extracted, generates a feature map through FPN convolution, activation, and pooling processes, and supplies it to the feature map reduction unit 102. In this embodiment, a three-layer multi-scale feature map x1, x2, and x3 is generated.

[0025] The feature map reduction unit 102 converts the three-layer multi-scale feature maps x1, x2, and x3 obtained from the neural network feature extraction unit 101 into a single-layer single-scale feature map xf and supplies it to the feature map conversion unit 103. The details of the feature map reduction unit 102 will be explained in detail with reference to Figure 5.

[0026] The feature map conversion unit 103 takes the fractional single-scale feature map xf supplied from the feature map reduction unit 102, performs packing and quantization processing to convert it into an integer-type packed feature frame, and supplies it to the feature map internal encoding unit 104.

[0027] The details of the feature map conversion unit 103 will be explained in detail with reference to Figure 7.

[0028] The feature map internal encoding unit 104 encodes the integer-type packing feature frame supplied from the feature map conversion unit 103 using an image encoding standard such as VVC, HEVC, or AV1 to generate and output a bitstream. The output bitstream is supplied to the feature map decoding device 200 or the like via a network or the like.

[0029] The details of the feature map internal encoding unit 104 will be explained in detail with reference to Figure 9.

[0030] Figure 2 is a block diagram showing the configuration of a feature map decoding device 200 according to an embodiment of the present invention, corresponding to the feature map encoding device 100 in Figure 1. The feature map decoding device 200 of this embodiment includes a feature map internal decoding unit 201, a feature map inverse conversion unit 202, and a feature map reconstruction unit 203. The feature map decoding device 200 also receives a bitstream encoded by the feature map encoding device 100, decodes the bitstream to generate three-layer multi-scale feature maps x1up, x2up, and x3up, and supplies them to the neural network identification unit 204.

[0031] The feature map internal decoding unit 201 decodes the bitstream encoded by the feature map internal encoding unit 104 of the feature map encoding device 100 using an image encoding standard such as VVC, HEVC, or AV1, generates an integer-type packing feature frame, and supplies it to the feature map inverse transform unit 202.

[0032] The details of the internal feature decoding unit 201 will be explained in detail with reference to Figure 10.

[0033] The feature map inverse transformation unit 202 performs inverse quantization and unpacking on the integer-type packing feature frame supplied from the feature map internal decoding unit 201, converts it into a decimal-type single-scale feature map xr, and supplies it to the feature map reconstruction unit 203.

[0034] The details of the feature map inverse transformation unit 202 will be explained in detail with reference to Figure 8.

[0035] The feature map reconstruction unit 203 converts the single-scale feature map xr supplied from the feature map inverse transformation unit 202 into three-layer multi-scale feature maps x1up, x2up, and x3up, and supplies them to the neural network identification unit 204 as the output of the feature map decoding device 200.

[0036] The details of the feature map restoration unit 203 will be explained in detail with reference to Figure 6.

[0037] The neural network identification unit 204 performs identification processing such as identifying objects in the target image, identifying locations and landscapes, and identifying people and living things, based on the three-layer multi-scale feature maps x1up, x2up, and x3up supplied by the feature map reconstruction unit 203.

[0038] <About Feature Map Reduction and Feature Map Restoration> The feature map reduction unit 102 has the function of converting a multi-layer multi-scale feature map obtained from the neural network feature extraction unit 101 into a single-layer single-scale feature map.

[0039] The details of the feature map reduction unit 102 will be explained using Figure 5. The feature map reduction unit 102 consists of a first feature map reduction unit 501, a first channel coupling unit 502, a second feature map reduction unit 503, a second channel coupling unit 504, a third feature map reduction unit 505, a first padding unit 506, a second padding unit 507, a third padding unit 508, and an amplification unit 509. The feature map reduction unit 102 in Figure 5 is an example of a configuration that converts a three-layer multi-scale feature map into a single-scale feature map.

[0040] The feature map reduction unit 102 takes a three-layer multi-scale feature map, consisting of a first feature map x1, a second feature map x2, and a third feature map x3, as input, converts it into a single-layer single-scale feature map xf, and supplies it to the feature map conversion unit 103. Here, n is the index indicating the layer, Cn is the number of channels in the nth layer, Wn is the width of the feature map, and Hn is the height of the feature map. In this embodiment, the values ​​of Cn, Wn, and Hn for each layer are as shown in Figure 11. However, H and W are the width and height of the image from which feature extraction is performed, respectively.

[0041] The first padding unit 506 has the function of performing padding on the first feature map x1 to generate the first padded feature map x1pad. The first padding unit 506 determines the padding size such that the width and height of x1pad are multiples of 64. The number of channels in x1pad is the same as x1, which is 256.

[0042] The second padding unit 507 has a function of performing padding by folding on the second feature map x2 to generate a second padded feature map x2pad. In the second padding unit 507, the padding size is determined so that the width and height of x2pad are multiples of 32. The number of channels of x2pad is the same as that of x2 and is 256.

[0043] The third padding unit 508 has a function of performing padding by folding on the third feature map x3 to generate a third padded feature map x3pad. In the third padding unit 508, the padding size is determined so that the width and height of x3pad are multiples of 1​6. The number of channels of x3pad is the same as that of x3 and is 256.

[0044] In the first padding unit 506, the second padding unit 507, and the third padding unit 508, it is assumed that the left padding size and the right padding size are the same, and the upper padding size and the lower padding size are the same. That is, it is assumed that the feature maps x1, x2, and x3 are respectively arranged at the centers of x1pad, x2pad, and x3pad.

[0045] The first feature map reduction unit 501 performs convolution in the spatial direction and the channel direction on the first padded feature map x1pad obtained from the first padding unit 506 to generate a first intermediate feature map y1. The number of channels of y1 is 192, the width is Wx1pad / 2, and the height is Hx1pad / 2. Here, Wx1pad and Hx1pad are the width and height of the first padded feature map x1pad, respectively.

[0046] The first channel combination unit 502 has a function of combining the first intermediate feature map y1 obtained from the first feature map reduction unit 501 and the second padded feature map x2pad obtained from the second padding unit 507 in the channel direction to generate an intermediate feature map y1Cx2pad. Since the number of channels of y1 is 192 and the number of channels of x2pad is​ 256, the number of channels of the intermediate feature map y1Cx2pad is 448 (192 + 256).

[0047] The second feature map reduction unit 503 performs convolution in the spatial and channel directions on the intermediate feature map y1Cx2pad obtained from the first channel combination unit 502 to generate a second intermediate feature map y2. The number of channels of y2 is 192, the width is Wy1Cx2pad / 2, and the height is Hy1Cx2pad / 2. Here, Wy1Cx2pad and Hy1Cx2pad are the width and height of the intermediate feature map y1Cx2pad, respectively.

[0048] The second channel combination unit 504 has a function of combining the second intermediate feature map y2 obtained from the second feature map reduction unit 503 and the third padded feature map x3pad obtained from the third padding unit 508 in the channel direction to generate an intermediate feature map y2Cx3pad. Since the number of channels of the intermediate feature map y2 is 192 and the number of channels of x3pad is 256, the number of channels of y1Cx2pad is 448 (192 + 256).

[0049] The third feature map reduction unit 505 performs convolution in the spatial and channel directions on the intermediate feature map y2Cx3pad obtained from the second channel combination unit 504 to generate a third intermediate feature map y3. The number of channels of y3 is 192, the width is Wy2Cx3pad / 2, and the height is Hy2Cx3pad / 2. Here, Wy2Cx3pad and Hy2Cx3pad are the width and height of the intermediate feature map y2Cx3pad, respectively.

[0050] The amplification unit 509 multiplies all elements of each channel of the feature map y3 obtained from the third feature map reduction unit 505 by a scaling factor to expand the range of signal values. Let the feature map with the expanded signal range be xr. As shown in FIG. 15, the scaling factor is specified in units of channels. In this example, the scaling factor of channel number 1 is 1. Therefore, in this case, the elements of channel 1 have their signal values unchanged. The scaling factor of channel 2 is 10, and the elements of channel 2 are multiplied by 10. The same applies to the following channels.

[0051] The scaling coefficient value indicates the importance of each channel, i.e., channel importance. Channels with a large scaling coefficient, i.e., channels with high channel importance, have a wide range of signal values. As a result, the internal feature map encoding unit 104 can allocate a large amount of code to them, suppressing encoding degradation and improving the encoding efficiency on average.

[0052] The feature map reduction unit 102 supplies the single-scale feature map xf to the feature map conversion unit 103.

[0053] The feature map reconstruction unit 203 has the function of converting the single-scale feature map xr obtained from the feature map inverse transformation unit 202 into three-layer multi-scale feature maps x1up, x2up, and x3up.

[0054] The details of the feature map reconstruction unit 203 will be explained using Figure 6. The feature map reconstruction unit 203 consists of an inverse amplification unit 600, an 8x magnification unit 601, a 4x magnification unit 602, a 2x magnification unit 603, a first feature map mixing unit 604, a second feature map mixing unit 605, a first padding removal unit 606, a second padding removal unit 607, and a third padding removal unit 608.

[0055] The inverse amplification unit 600 multiplies all elements of each channel of the single-scale feature map xr obtained from the feature map inverse transformation unit 202 by an inverse scaling coefficient to reduce the range of signal values. Let this be the feature map xr' with the reduced range of signal values. This process is equivalent to restoring the feature map, whose range of signal values ​​was expanded by the amplification unit 509 of the feature map reduction unit 102, back to its original range. As shown in Figure 15, the inverse scaling coefficient is specified on a per-channel basis. The inverse scaling coefficient is the reciprocal of the scaling coefficient. Therefore, the inverse scaling coefficient for channel number 1 is 1 / 1. In this case, the signal values ​​of the elements of channel 1 are not changed. The inverse scaling coefficient for channel 2 is 1 / 10, and the elements of channel 2 are multiplied by 1 / 10. The same applies to the following channels.

[0056] The inverse scaling coefficient, like the scaling coefficient, is assumed to have appropriate parameters determined in advance and not be changed during the sequence. For example, the scaling coefficient can be determined by prior machine learning.

[0057] Channel importance can also be set using an inverse scaling factor instead of scaling. The value of the inverse scaling factor is the reciprocal of the scaling factor. Similar to the scaling factor, this indicates the importance of each channel, i.e., channel importance. Since the inverse scaling factor is the reciprocal of the scaling factor, the relative order is reversed. Therefore, channels with smaller inverse scaling factors have higher channel importance.

[0058] Furthermore, the scaling coefficient and inverse scaling coefficient may be predetermined, for example, by end-to-end machine learning for encoding and decoding. Also, the channel importance may be changed from a predetermined value as the encoding and decoding process progresses, such as when an image with a different trend from the pre-trained images is input. In this case, the scaling coefficient and inverse scaling coefficient are generally not inversely related, but either the scaling coefficient or the inverse scaling coefficient can be used as the channel importance during encoding and decoding. However, in order to reproduce the highly efficient encoding state achieved by machine learning, it is preferable to use the scaling coefficient during encoding and the inverse scaling coefficient during decoding. In other words, when transmitting from the encoding device to the decoding device, it is preferable to transmit the inverse scaling coefficient. In this case, the inverse scaling coefficient will be a decimal value rather than an integer value, but it may be rounded to an integer value to suppress the amount of code, or it may be left as a decimal value to improve the accuracy of the feature map. In this embodiment, the channel importance is represented by channel_importance and transmitted as a bitstream. Since channel importance (channel_importance) is information associated with each channel, it is transmitted in the number of channel instances (number_of_channels).

[0059] In Figure 15, the inverse scaling coefficient is defined as the reciprocal of the scaling coefficient, but it may be adjusted to take a different value. For example, in the process of generating integer-type packing feature frames in the feature map internal decoding unit 201, channels with a large assigned code amount contain a large amount of information. Therefore, the channel importance of channels with a large assigned code amount can be set high, and the channel importance of channels with a small assigned code amount can be set low.

[0060] The 8x magnification unit 601 has the function of expanding the feature map and reducing the number of channels by performing transposition convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr' acquired from the inverse amplification unit 600, thereby generating an intermediate feature map z1. The number of channels in z1 is 196. If the width and height of the single-scale feature map xr are xrwidth and xrheight, respectively, then the width and height of z1 are xrwidth × 8 and xrheight × 8, respectively. Here, rwidth × 8 and xrheight × 8 are the same as the width and height of the first padded feature map x1pad, which is the output of the first padding unit 506 of the feature map reduction unit 102.

[0061] The quadruple magnification unit 602 has the function of expanding features and reducing channels by performing transposition convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr obtained from the feature map inverse transformation unit 202, thereby generating an intermediate feature map z2. The number of channels in z2 is 196. The width and height of z2 are xrwidth × 4 and xrheight × 4, respectively. Here, rwidth × 4 and xrheight × 4 are the same as the width and height of the second padded feature map x2pad, which is the output of the second padding unit 507 of the feature map reduction unit 102.

[0062] The doubling unit 603 has the function of expanding the feature map and reducing the number of channels of the single-scale feature map xr obtained from the feature map inverse transformation unit 202 by performing transposition convolution in the spatial direction and convolution in the channel direction, thereby generating an intermediate feature map z3. The number of channels in z3 is 196. The width and height of z3 are xrwidth × 2 and xrheight × 2, respectively. Here, rwidth × 2 and xrheight × 2 are the same as the width and height of the third padded feature map x3pad, which is the output of the third padding unit 508 of the feature map reduction unit 102.

[0063] The first feature map mixing unit 604 has the function of generating an intermediate feature map z2up, which is an improved version of the intermediate feature map z2 obtained from the 4x magnification unit 602, using the intermediate feature map z1 obtained from the 8x magnification unit 601.

[0064] The second feature map mixing unit 605 has the function of generating an intermediate feature map z3up, which is an improved version of the intermediate feature map z3 obtained from the doubling unit 603, using the intermediate feature map z2up obtained from the first feature map mixing unit 604.

[0065] The padding removal unit 606 has the function of removing padding from the intermediate feature map z1 acquired from the 8x magnification unit 601 and generating a first output feature map x1up. The width and height of x1up are the same as the width and height of the first feature map x1 input to the feature map reduction unit 102.

[0066] The padding removal unit 607 has the function of removing padding from the intermediate feature map z2up obtained from the first feature map mixing unit 604 and generating a second output feature map x2up. The width and height of x2up are the same as the width and height of the second feature map x2 input to the feature map reduction unit 102.

[0067] The padding removal unit 608 has the function of removing padding from the intermediate feature map z3up obtained from the second feature map mixing unit 605 and generating a third output feature map x3up. The width and height of x3up are the same as the width and height of the third feature map x3 input to the feature map reduction unit 102.

[0068] In the first padding removal unit 606, the second padding removal unit 607, and the third padding removal unit 608, similar to the first padding unit 506, the second padding unit 507, and the third padding unit 508 of the feature map reduction unit 102, the padding size on the left and the padding size on the right are set to be the same, and the padding size on the upper and lower sides are set to be the same. That is, each output feature map x1up, x2up, and x3up are assumed to be positioned at the center of each intermediate feature map z1, z2up, and z3up, respectively, and padding is removed from the top, bottom, left, and right.

[0069] <About Feature Map Conversion and Inverse Feature Map Conversion> The feature map conversion unit 103 has the function of performing packing and quantization processing on the multi-channel fractional single-scale feature map xf supplied from the feature map reduction unit 102, and converting it into an integer-type packed feature frame for supply to the feature map internal encoding unit 104.

[0070] The details of the feature map conversion unit 103 on the encoding side will be explained using Figure 7. The feature map conversion unit 103 consists of a packing unit 701 and a feature map quantization unit 702.

[0071] The packing unit 701 has the function of generating a packed feature frame by combining the input feature maps of multiple channels into a single frame. The operation of generating a packed feature frame in the packing unit 701 will be explained using the flowchart in Figure 12.

[0072] First, the packing unit 701 determines the sorting order of the feature maps of multiple channels (step S110). The sorting order of the feature maps is either in descending order of channel importance or in ascending order of channel importance. In this embodiment, the reciprocal of the inverse scaling coefficient is used as the channel importance. Figure 21 shows an example of sorting 1101. For example, the feature maps of multiple channels are represented as an array. The index in Figure 21 corresponds to the subscript of the array. 1102 is the result of sorting 1101 in descending order of channel importance. 1103 is the result of sorting 1101 in ascending order of channel importance. If the importance is the same, the sorting is done in ascending order of channel number. The determined sorting order is represented by the syntax element rearrangement_order, as shown in Figure 19.

[0073] Next, the packing unit 701 sorts the feature maps of the multiple channels in descending or ascending order of channel importance based on the sorting order (step S120). In this embodiment, the reciprocal of the inverse scaling coefficient is used as the channel importance. If the importance is the same, the channels are sorted in ascending order of channel number. Figure 21 shows an example of sorting. 1102 is the result of sorting 1101.

[0074] Next, the scan order for the packed feature frames is determined (step S130). In this embodiment, either a raster scan or a vertical-first scan can be selected. The determined scan order is represented by the syntax element rearrangement_mode, as shown in Figure 16. In the case of a raster scan, starting from the top left of the packed feature frame, the feature maps of each channel are placed sequentially to the right. Once the right edge of the packed feature frame is reached, one line down is moved and the same operation is performed. Figure 17 shows an example of the packing order in a raster scan. 1201 is an example of sorting in descending order of channel importance, and 1202 is an example of sorting in ascending order of channel importance. In the case of a vertical-first scan, starting from the top left of the packed feature frame, the feature maps of each channel are placed sequentially downwards. Once the bottom edge of the packed feature frame is reached, one unit to the right is moved and the same operation is performed. Figure 18 shows an example of the packing order in a vertical-first scan. 1203 is an example of sorting by channel importance in descending order, and 1204 is an example of sorting by channel importance in ascending order.

[0075] Next, the packing unit 701 arranges the multi-channel feature maps into the packed feature frame according to the scan order determined in step S102 (step S140). The rearrangement order of the multi-channel feature maps, the scan order for the packed feature frame, rearrangement mod, and the channel importance of each channel, channel_importance, are transmitted to the feature map reconstruction unit via the bitstream (step S150).

[0076] Furthermore, the packing unit 701 performs flipping based on the in-frame position where the channel feature map is placed. The ability to select whether or not to perform flipping may be provided and transmitted from the encoding side to the decoding side via the bitstream.

[0077] Figure 13 illustrates the flipping process when packing multiple channel feature maps into a single frame. Flipping involves inverting the position of each channel's feature map elements (pixels) horizontally (left / right), vertically (up / down), or both horizontally and vertically (up / down / left / right) when packing the feature maps for each channel. In Figure 13, the four channel feature maps A (top left), B (top right), C (bottom left), and D (bottom right) are considered a single set. No flipping is performed at position A in Figure 13. At position B, the feature map is inverted horizontally (left / right). At position C, the feature map is inverted vertically (up / down). At position D, the feature map is inverted horizontally and vertically (up / down / left / right). When the distribution of elements in each channel's feature map is similar, performing flipping based on the in-frame position where the channels are placed reduces the boundaries between each channel's feature map, improving encoding efficiency.

[0078] The feature map quantization unit 702 has the function of converting the elements of a decimal-type packing feature frame (feature map of all channels) into an N-bit integer type (an integer N = approximately 8 to 16) within a predetermined range and outputting an integer-type packing feature frame. In this embodiment, it is assumed that it is converted into a 10-bit integer type from 0 to 1023. The feature map quantization unit 702 detects the minimum and maximum values ​​of the elements of the decimal-type packing feature frame and transmits the detected minimum and maximum values ​​of the elements of the packing feature frame (feature map of all channels) to the decoding side as metadata. In order to convert from a decimal-type packing feature frame (feature map) to an integer-type packing feature frame (feature map), a linear transformation is performed in which the minimum value of the decimal type corresponds to the minimum value of the integer type, and the maximum value of the decimal type corresponds to the maximum value of the integer type. For example, when the range of the integer type is represented by 10 bits, the minimum value of the elements of the integer-type packing feature frame (feature map) is 0, and the maximum value is 1023 (2 １０ -1) This is the result. Linear quantization is performed on values ​​between the minimum and maximum values.

[0079] Next, the feature map inverse transform unit 202 has the function of performing inverse quantization and unpacking on integer-type packing feature frames decoded by VVC, HEVC, AV1, etc. supplied from the feature map internal decoding unit 201, and transforming them into a decimal-type single-scale feature map xr for supply to the feature map reconstruction unit 203.

[0080] Figure 8 will be used to explain the details of the feature map inverse transformation unit 202 on the decoding side. The feature map inverse transformation unit 202 is the inverse process of the feature map transformation unit 103 and is composed of a feature map inverse quantization unit 801 and an unpacking unit 802.

[0081] The feature map inverse quantization unit 801 performs the inverse processing of the encoding-side feature map quantization unit 702 and has the function of converting the elements of the integer-type packing feature frame from integer type to decimal type. The feature map inverse quantization unit 801 converts the integer-type packing feature frame decoded by the feature map internal decoding unit 201 into a decimal-type packing feature frame using the minimum and maximum decimal values ​​transmitted as metadata. A linear transformation is performed to make the minimum integer value equivalent to the minimum decimal value and the maximum integer value equivalent to the maximum decimal value. For values ​​between the minimum and maximum values, linear inverse quantization is performed.

[0082] The unpacking unit 802 extracts the feature maps for each channel from the packing feature frames arranged in a single frame in the order of the raster scan, and supplies them to the feature map reconstruction unit 203 as a single-scale feature map xr.

[0083] The operation of the unpacking unit 802 will be explained using Figure 20.

[0084] First, the unpacking unit 802 decodes rearrangement_order, which indicates the channel rearrangement order, from the bitstream (S210). The channel rearrangement order follows Figure 19, similar to packing unit 701.

[0085] Next, rearrangement_mode, which indicates the scan order of the packing feature frames, is decoded from the bitstream (S220). The scan order follows Figure 16, similar to that of the packing unit 701.

[0086] Next, the unpacking unit 802 decodes the channel importance value channel_importance for each channel from the bitstream (S230).

[0087] In the following steps, the feature maps of multiple channels are processed one by one from the packing feature frame, according to the scan order determined in step S220. Once all channels have been processed, this process is terminated (step S240).

[0088] If not all channels have been processed, the unpacking unit 802 acquires the next channel in scan order (step S250). For the acquired channel, it determines the channel number based on the channel importance (step S260). If the channel sorting order is in descending order of channel importance, in the example of Figure 15, channel number 2 has the highest channel importance (channel importance 10). Therefore, the channel number of the first acquired channel is set to 2. For the subsequent acquired channels, channel numbers 3 (channel importance 8), 4 (channel importance 4), and 7 (channel importance 2) are set sequentially. If the importance is the same, the channel numbers are set in ascending order. If the channel sorting order is in ascending order of channel importance, for channels 1, 5, 6, and 8 (channel importance 1), which have the lowest channel importance, channel numbers 1 to 4 are set respectively. Subsequently, in ascending order of channel importance, channel numbers 7 (channel importance 2), 4 (channel importance 4), 3 (channel importance 8), and 2 (channel importance 10) are set sequentially. Figure 22 shows an example of the syntax structure in this embodiment. The rearrangement_flag in Figure 23 is a flag indicating whether or not to rearrange the channels. Whether or not to execute the processing described in this embodiment can be switched on a sequence or frame basis. channel_importance takes the value of a scaling factor or an inverse scaling factor. For high-precision adjustment, it is desirable that channel_importance have decimal precision.

[0089] In this embodiment, the sorting order of the feature maps of multiple channels and the packing order for the packed feature frames can be freely set. When the difference in values ​​at the channel boundaries becomes large, the coding efficiency decreases significantly. By adopting this embodiment, it is possible to arrange the features in a way that reduces the difference in values ​​at the channel boundaries, thereby improving the coding efficiency.

[0090] Furthermore, by adopting this embodiment, the channel importance can be shared between the feature map encoding device 100 and the feature map decoding device 200. Since the feature map can always be encoded and decoded with the appropriate channel importance while dynamically changing the channel importance, the encoding efficiency can be improved.

[0091] Here, both rearrangement_order, which indicates the channel sorting order, and rearrangement_mode, which indicates the scan order of the packing feature frames, are encoded and decoded. However, a configuration using only one of them is also acceptable. Also, here, number_of_channels is encoded and decoded before rearrangement_flag. However, the default value of number_of_channels can be shared between the encoding and decoding devices, and it can be encoded and decoded after rearrangement_mode instead of before rearrangement_flag.

[0092] <About Internal Feature Map Encoding and Decoding> Figure 9 will be used to explain the details of the feature map internal encoding unit 104. The feature map internal encoding unit 104 consists of a switch 901, a VVC encoding unit 902, a HEVC encoding unit 903, and an AV1 encoding unit 904. The switch 901 selects an encoding standard for internally encoding the feature map converted by the feature map conversion unit 103. The VVC encoding unit 902 encodes the feature map using the VVC standard and outputs a bitstream compliant with the VVC standard. The HEVC encoding unit 903 encodes the feature map using the HEVC standard and outputs a bitstream compliant with the HEVC standard. The AV1 encoding unit 904 encodes the feature map using the AV1 standard and outputs a bitstream compliant with the AV1 standard.

[0093] In the VVC, HEVC, and AV1 standards, images are divided into predetermined block sizes for encoding.

[0094] It is also possible to implement only one of the following: VVC, HEVC, or AV1. Furthermore, it is possible to use image encoding schemes other than VVC, HEVC, and AV1.

[0095] Next, the details of the feature map internal decoding unit 201 will be explained using Figure 10. The feature map internal decoding unit 201 consists of a switch 1001, a VVC decoding unit 1002, a HEVC decoding unit 1003, and an AV1 decoding unit 1004. The switch 1001 selects the encoding standard to be internally decoded based on the information in the input bitstream that is used to select internal decoding. The VVC decoding unit 1002 decodes the feature map using the VVC standard. The HEVC decoding unit 1003 decodes the feature map using the HEVC standard. The AV1 decoding unit 1004 decodes the feature map using the AV1 standard.

[0096] In the VVC, HEVC, and AV1 standards, decoding is performed for each predetermined block size.

[0097] It is also possible to implement only one of VVC, HEVC, or AV1. Furthermore, it is possible to use an image encoding method other than VVC, HEVC, or AV1. (Second Embodiment) This embodiment differs from the first embodiment in that the packing unit 701 and the unpacking unit 802 transmit the channel importance order channel_importance_order instead of the channel importance channel_importance.

[0098] The operation of the packing unit 701 will be explained using the flowchart in Figure 24. Processes similar to those in the first embodiment are indicated by the same reference numerals as in the flowchart in Figure 12.

[0099] Step S151 transmits the rearrangement_order of the feature maps of multiple channels, the rearrangement_mode of the scan order for the packed feature frames, and the channel importance order of each channel, channel_importance_order, to the feature map reconstruction unit via the bitstream. channel_importance_order assigns channel numbers sequentially in descending order of channel importance, starting with the channel with the greatest channel importance. If there are multiple channels with the same channel importance, the channel with the smaller channel number is given priority.

[0100] The operation of the unpacking unit 802 will be explained using the flowchart in Figure 25. Processes similar to those in the first embodiment are indicated by the same reference numerals as in the flowchart in Figure 20.

[0101] Step S231 decrypts channel_importance_order.

[0102] Step S261 assigns the channel numbers indicated by channel_importance_order to the acquired channels in the order they were received if the channel sorting order is in descending order of channel importance. If the channel sorting order is in ascending order of channel importance, the channel numbers indicated by channel_importance_order are assigned to the acquired channels in reverse order. Figure 23 shows an example of the syntax structure in this embodiment. The syntax structure of this embodiment includes channel_importance_order instead of channel_importance in Figure 22. channel_importance_order is a value that indicates order and can therefore be represented as an integer. Here, the scaling coefficient and inverse scaling coefficient are those that have been determined in advance, for example, by end-to-end machine learning.

[0103] By adopting this embodiment, instead of transmitting the channel importance, the order of the channel importance is transmitted. This allows for dynamic channel changes and enables the encoding and decoding of feature maps with the appropriate channel importance at all times. Since channel importance takes the form of a decimal value, it is not very informative. However, by using only the order of the channel importance, which can be represented as an integer, the feature map encoding device 100 and the feature map decoding device 200 can share information regarding the sorting of channel importance, thereby reducing the amount of code and improving encoding efficiency. (Third Embodiment) In this embodiment, channel truncation is performed on the decimal single-scale feature map xf of multiple channels supplied from the feature map reduction unit 102. Channel truncation is a method of classifying each channel into either an active channel to be encoded or an inactive channel whose encoding is omitted.

[0104] Channels that are encoded by the encoding-side feature map encoding device 100 and decoded by the decoding-side feature map decoding device 200 are designated as active channels. Conversely, channels that are not encoded by the encoding side but are generated by the decoding side are designated as inactive channels. Inactive channels are treated as feature maps with predetermined elements on the decoding side.

[0105] The operation of the packing unit 701 will be explained using the flowchart in Figure 27.

[0106] The channel importance order, channel_importance_order, is determined for each channel. channel_importance_order assigns channel numbers sequentially in descending order of channel importance, starting with the channel with the highest importance. If there are multiple channels with the same channel importance, the channel with the smaller channel number is given priority (step S301).

[0107] For each channel, it is determined whether it is an inactive channel or an active channel (step S302). If channel i is an inactive channel, the syntax element non_active_flag[i] is set to TRUE. If channel i is an active channel, the syntax element non_active_flag[i] is set to FALSE.

[0108] In a single-scale feature map xf with multiple channels and a fractional number of channels, inactive channels are excluded, i.e., only active channels are packed, to generate a packed feature frame (step S303).

[0109] The syntax elements non_active_flag and channel_importance_order are encoded. (Step S304).

[0110] The operation of the unpacking unit 802 will be explained using the flowchart in Figure 28.

[0111] For all channels i (from i=0 to i<num_feature_channel), the inactive channel flag non_active_flag[i] is decoded (steps S401, S402, S403). However, num_feature_channel is contained in a fractional single-scale feature map xf. Next, the number of active channels in the packing feature frame, num_active_channel, is calculated (step S404). The number of active channels in the packing feature frame can be calculated by subtracting the number of inactive channels, i.e., the sum of non_active_flag[i] values ​​that are FALSE, from the total number of channels.

[0112] Next, for all active channels j (from j=0 to j<num_active_channel), the channel importance order channel_importance_order[j] is decoded (S405, S406, S407). Figure 26 shows an example of the syntax structure in this embodiment. The syntax structure in this embodiment includes the syntax element non_active_channel, which indicates that channel i is an inactive channel. Since channel_importance_order is a value that indicates order, it can be represented as an integer value.

[0113] By adopting this embodiment, instead of transmitting channel importance, the order of channel importance is transmitted. This allows for dynamic channel importance changes, enabling the encoding and decoding of feature maps to always be performed with the appropriate channel importance. Since channel importance takes the form of decimal values, which are large in terms of information, the feature map encoding device 100 and the feature map decoding device 200 can share information regarding the sorting of channel importance using only the order of channel importance, which can be represented as integer values. This reduces the amount of code and improves encoding efficiency. Furthermore, the amount of code can be reduced by omitting the transmission of channels with low transmission effectiveness by designating them as inactive channels. Moreover, the amount of code can be further reduced by transmitting the order of channel importance only for active channels and omitting the transmission of the order of channel importance for inactive channels.

[0114] In all the embodiments described above, the bitstream output by the feature map encoding device has a specific data format so that it can be decoded according to the encoding method used in the embodiment. Furthermore, the feature map decoding device corresponding to this feature map encoding device can decode the bitstream of this specific data format.

[0115] When a wired or wireless network is used to exchange bitstreams between a feature map encoding device and a feature map decoding device, the bitstream may be converted to a data format suitable for the transmission mode of the communication channel before transmission. In this case, a transmitting device is provided that converts the bitstream output by the feature map encoding device into encoded data in a data format suitable for the transmission mode of the communication channel and transmits it to the network, and a receiving device is provided that receives the encoded data from the network, restores it to a bitstream, and supplies it to the feature map decoding device. The transmitting device includes a memory for buffering the bitstream output by the feature map encoding device, a packet processing unit for packetizing the bitstream, and a transmitting unit for transmitting the packetized encoded data over the network. The receiving device includes a receiving unit for receiving the packetized encoded data over the network, a memory for buffering the received encoded data, and a packet processing unit for packetizing the encoded data to generate a bitstream and providing it to the feature map decoding device.

[0116] The above encoding and decoding processes may be implemented not only as hardware-based transmission, storage, and receiving devices, but also by firmware stored in ROM (read-only memory) or flash memory, or by software on a computer. The firmware program or software program may be recorded on a recording medium readable by a computer and provided, provided from a server via a wired or wireless network, or provided as data broadcasting on terrestrial or satellite digital broadcasting.

[0117] The present invention has been described above based on embodiments. The embodiments are illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of their components and processing processes, and that such modifications also fall within the scope of the present invention.

[0118] This invention can be used in feature map encoding and decoding techniques.

[0119] 100 Feature map encoding device, 101 Neural network feature extraction unit, 102 Feature map reduction unit, 103 Feature map transformation unit, 104 Feature map internal encoding unit, 200 Feature map decoding device, 201 Feature map internal decoding unit, 202 Feature map inverse transformation unit, 203 Feature map reconstruction unit, 204 Neural network identification unit, 301 Convolution processing unit, 302 Activation processing unit, 303 Pooling processing unit, 322 Bottom-up processing unit, 324 Top-down processing unit, 326 Image to be processed for feature extraction, 501 First feature map reduction unit, 502 First channel joining unit, 503 Second feature map reduction unit, 504 Second channel joining unit, 505 Third feature map reduction unit, 506 First padding unit, 507 Second padding unit, 508 Third padding unit, 509 Amplification unit, 600 601 Reverse amplification section, 602 8x amplification section, 603 4x amplification section, 604 2x amplification section, 604 First feature map mixing section, 605 Second feature map mixing section, 606 First padding removal section, 607 Second padding removal section, 608 Third padding removal section, 701 Packing section, 702 Feature map quantization section, 801 Feature map inverse quantization section, 802 Unpacking section, 901 Switch, 902 VVC encoding section, 903 HEVC encoding section, 904 AV1 encoding section, 1001 Switch, 1002 VVC decoding section, 1003 HEVC decoding section, 1004 AV1 decoding section.

Claims

1. A feature map encoding device comprising: a feature map reduction unit that converts a multi-scale feature map to generate a single-scale feature map; a packing unit that packs the single-scale feature map into a frame based on information indicating the channel importance of the single-scale feature map to generate a packed feature frame; an encoding unit that encodes the scan type; and a feature map internal encoding unit that encodes information indicating the channel importance of the single-scale feature map.

2. A feature map encoding method comprising: a feature map reduction step of converting a multi-scale feature map to generate a single-scale feature map; a packing step of packing the single-scale feature map into a frame based on information indicating the channel importance of the single-scale feature map to generate a packed feature frame; an encoding step of encoding the scan type; and a feature map internal encoding step of encoding information indicating the channel importance of the single-scale feature map.

3. A feature map encoding program characterized by causing a computer to perform the following steps: a feature map reduction step of converting a multi-scale feature map to generate a single-scale feature map; a packing step of packing the single-scale feature map into a frame based on information indicating the channel importance of the single-scale feature map to generate a packed feature frame; an encoding step of encoding the scan type; and a feature map internal encoding step of encoding information indicating the channel importance of the single-scale feature map.

4. A feature map decoding device comprising: a decoding unit that decodes information indicating the channel importance of a single-scale feature map; an internal feature map decoding unit that decodes the single-scale feature map packed and encoded into a frame to generate a packing feature frame; an unpacking unit that unpacks the packing feature frame based on the information indicating the channel importance of the single-scale feature map to generate the single-scale feature map; and a feature map restoration unit that converts the single-scale feature map to generate a multi-scale feature map.

5. A feature map decoding method comprising: a decoding step of decoding information indicating the channel importance of a single-scale feature map; an internal feature map decoding step of decoding the single-scale feature map packed and encoded in a frame to generate a packing feature frame; an unpacking step of unpacking the packing feature frame based on the information indicating the channel importance of the single-scale feature map to generate the single-scale feature map; and a feature map restoration step of transforming the single-scale feature map to generate a multi-scale feature map.

6. A feature map decoding program characterized by causing a computer to perform the following steps: a decoding step of decoding information indicating the channel importance of a single-scale feature map; an internal feature map decoding step of decoding the single-scale feature map packed and encoded into a frame to generate a packing feature frame; an unpacking step of unpacking the packing feature frame based on the information indicating the channel importance of the single-scale feature map to generate the single-scale feature map; and a feature map restoration step of converting the single-scale feature map to generate a multi-scale feature map.

Images