Feature map encoding device, feature map encoding method, feature map decoding device, and feature map decoding method

Abstract

One embodiment of the feature map encoding device of the present application is provided with: a feature map reduction unit (102) that converts a multi-scale feature map to generate a single-scale feature map; a packing unit (701) that divides the single-scale feature map into one or more packing groups, packs the one or more packing groups into a frame, and generates a floating-point packed feature frame; a quantization unit (702) that converts elements of the floating-point packed feature frame into integer values to generate an integer packed feature frame; a feature map internal encoding unit (104) that encodes the integer packed feature frame; and an encoding unit that encodes information pertaining to the packing groups.

Description

Feature map encoding device, feature map encoding method, feature map decoding device, feature map decoding method

[0001] This paper deals with encoding and decoding of feature maps in neural networks.

[0002] As a neural network technology used for image recognition such as detecting objects at various scales in an image, dividing regions into individual objects, or tracking objects, the FPN (Feature Pyramid Network) described in Non-Patent Document 1 is known. In the FPN, multiple feature maps at various scales are generated from the image to be processed, and various image recognitions are performed using the feature maps.

[0003] The FPN used for image recognition generates multiple feature maps from an image, and its structure utilizes a CNN (Convolutional Neural Network). A CNN can be divided into a feature extraction section (backbone) that reads an image and generates a feature map through convolution and pooling, and a classification section (head) that is composed of hierarchical fully connected layers and generates output suitable for tasks such as object detection, instance segmentation, and object tracking. The FPN utilizes the CNN backbone.

[0004] The feature extraction unit of the FPN is typically configured with a hierarchical structure in which a convolution process 301, an activation process 302, and a pooling process 303 shown in FIG. 3 are configured as one basic unit, and this basic unit is repeated.

[0005] The structure of the FPN is shown in Figure 4. The FPN consists of a bottom-up processing unit 322 that uses a CNN backbone to generate a multi-scale feature map consisting of multiple hierarchical layers, and a top-down processing unit 324 that aggregates features from deep layer feature maps to shallow layer feature maps using the inverse structure of the CNN backbone. The bottom-up processing unit 322 reduces the resolution of the feature map by half by repeating the basic units of convolution processing 301, activation processing 302, and pooling processing 303 shown in Figure 3, thereby generating a pyramid of feature maps with multiple layers. Meanwhile, the top-down processing unit 324 increases the resolution of the feature map to the same resolution as the input image while adding feature maps with resolutions corresponding to those of the bottom-up processing unit 322, thereby generating a pyramid of feature maps. In other words, the FPN generates multiple feature maps for each layer from an image 326 to be subjected to feature extraction processing.

[0006] The convolution processing unit 301 performs convolution processing on the data to be processed (image or feature map) using multiple types of predetermined filters (kernels). In the convolution processing in the convolution processing unit 301, predetermined filtering is performed on all of the data to be processed while sliding at predetermined intervals. The sliding interval is called a stride. The convolution processing unit 301 may determine the stride based on the number of data to be processed. For example, the convolution processing unit 301 may determine the stride to be 1 if the number of data to be processed is smaller than a predetermined value, and may determine the stride to be 2 if the number of data to be processed is equal to or greater than the predetermined value. Multiple predetermined filters are prepared in each layer, and multiple feature maps are generated by generating one feature map for each filter. The unit of the feature map is the channel. If the number (types) of predetermined filters is N (N types), N feature maps (N channels) are generated.

[0007] The activation processing unit 302 performs activation processing to nonlinearly convert the feature map output from the convolution processing unit 301. Here, the function used in the activation processing is called an activation function. The activation processing unit 302 uses a ReLU (Rectified Linear Unit) function, a sigmoid function, or the like as the activation function.

[0008] The pooling processing unit 303 is a process for downsampling the feature map output from the activation processing unit 302 by replacing local values ​​of the feature map with representative values.

[0009] On the other hand, when performing classification using a neural network, it has the ability to perform tasks using feature maps of multiple channels at each layer.

[0010] In image classification, the feature maps of multiple channels at each layer are convolved for each predetermined size based on the scale of the feature map, and the probability of the object class is calculated for each pixel.

[0011] Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature Pyramid Networks for Object Detection.

[0012] The amount of information in a feature map is enormous, making it unsuitable for transmission and storage. In view of the above-mentioned problems, an object of this embodiment is to provide a technique for encoding and decoding a feature map.

[0013] In order to solve the above problems, one aspect of a feature map encoding device of the present application includes a feature map reduction unit that converts a multi-scale feature map to generate a single-scale feature map; a packing unit that divides the single-scale feature map into one or more packing groups and packs them into frames to generate a decimal-type packing feature frame; a quantization unit that converts elements of the decimal-type packing feature frame into integer values ​​to generate an integer-type packing feature frame; a feature map inner encoding unit that encodes the integer-type packing feature frame; and an encoding unit that encodes information related to the packing groups.

[0014] Moreover, one aspect of a feature map decoding device of the present application includes a decoding unit that decodes information related to packing groups, a feature map internal decoding unit that decodes a single-scale feature map that has been packed into a frame and encoded, and generates an integer-type packing feature frame, a dequantization unit that converts elements of the integer-type packing feature frame into decimal values, and generates a decimal-type packing feature frame, an unpacking unit that divides the decimal-type packing feature frame into one or more packing groups based on the information related to the packing groups, and unpacks the divided fractional-type packing feature frame, and generates a single-scale feature map, and a feature map restoration unit that converts the single-scale feature map, and generates a multi-scale feature map.

[0015] According to this embodiment, feature maps can be efficiently encoded and decoded with a small amount of processing.

[0016] 1 is a block diagram illustrating the configuration of a feature map encoding device 100. FIG. 2 is a block diagram illustrating the configuration of a feature map decoding device 200. FIG. 3 is a block diagram illustrating the processing that is the basic unit in each layer of an FPN. FIG. 4 is a block diagram illustrating the structure of an FPN. FIG. 5 is a block diagram illustrating the detailed configuration of a feature map reduction unit 102. FIG. 6 is a block diagram illustrating the detailed configuration of a feature map restoration unit 203. FIG. 7 is a block diagram illustrating the detailed configuration of a feature map transformation unit 103. FIG. 8 is a block diagram illustrating the detailed configuration of a feature map inverse transformation unit 202. FIG. 9 is a block diagram illustrating the detailed configuration of a feature map internal encoding unit 104. FIG. 10 is a block diagram illustrating the detailed configuration of a feature map internal decoding unit 201. FIG. 11 is a diagram illustrating the number of channels of feature maps x1, x2, and x3, the width and height of the feature maps. FIG. 12 is a diagram illustrating the state in which feature maps of multiple channels are packed into one frame. FIG. 13 is a diagram illustrating flipping when packing feature maps of multiple channels into one frame. FIG. 14 is a diagram illustrating frame division in feature map internal encoding / decoding. FIG. 15 is a diagram illustrating the scan order in feature map internal encoding / decoding. FIG. 10 is a diagram for explaining metadata for specifying a packing group. FIG. 11 is a flowchart for explaining the operation of the packing unit 701. FIG. 11 is a diagram for explaining an example in which a feature map is packed. FIG. 12 is a diagram for explaining a different example of metadata for specifying a packing group. FIG. 13 is a diagram for explaining a further different example of metadata for specifying a packing group. FIG. 14 is a flowchart for explaining the operation of the unpacking unit 802.

[0017] The technologies and technical terms used in this embodiment will be defined below.

[0018] <Features and feature maps> In a convolutional neural network (CNN), a filter is used to scan the image (input layer data) from which features are extracted while changing its position, and the results of the convolution operation between the filter coefficients are used as features or feature maps.

[0019] <Packing> Frame packing refers to combining two or more frames (pictures) into a single frame (picture) by arranging them in a tiled pattern. In this application, packing refers to combining feature maps of multiple channels into a single frame. Figure 12 shows an example of frame packing.

[0020] <Data Type> A data type that represents an integer value is called an integer type, and a data type that represents a decimal value is called a decimal type.

[0021] First Embodiment A feature map encoding device 100 and a feature map decoding device 200 according to a first embodiment of the present invention will be described.

[0022] 1 is a block diagram of a feature map coding device 100 according to a first embodiment. The feature map coding device 100 according to the embodiment includes a feature map reduction unit 102, a feature map conversion unit 103, and a feature map internal coding unit 104. The feature map coding device 100 is a device that encodes a feature map generated by a neural network feature extraction unit 101 to generate and output a bitstream.

[0023] The neural network feature extraction unit 101 reads an image from which features are to be extracted, generates a feature map by FPN convolution processing, activation processing, and pooling processing, and supplies the generated feature map to the feature map reduction unit 102. In this embodiment, a three-layer multi-scale feature map of x1, x2, and x3 is generated.

[0024] The feature map reduction unit 102 converts the three-layer multi-scale feature maps x1, x2, and x3 acquired from the neural network feature extraction unit 101 into a single-layer single-scale feature map xf and supplies it to the feature map conversion unit 103. The feature map reduction unit 102 will be described in detail with reference to FIG. 5 .

[0025] The feature map conversion unit 103 performs packing and quantization on the decimal-type single-scale feature map xf supplied from the feature map reduction unit 102, converting it into an integer-type packed feature frame, and supplies it to the feature map internal coding unit 104.

[0026] The feature map conversion unit 103 will be described in detail with reference to FIG.

[0027] The feature map internal coding unit 104 generates and outputs a bitstream by coding the integer-type packing feature frame supplied from the feature map conversion unit 103 in accordance with an image coding standard such as VVC, HEVC, or AV1. The output bitstream is supplied to the feature map decoding device 200 or the like via a network or the like.

[0028] The feature map internal coding unit 104 will be described in detail with reference to FIG.

[0029] 2 is a block diagram showing the configuration of a feature map decoding device 200 according to an embodiment of the present invention, which corresponds to the feature map encoding device 100 in FIG. 1. The feature map decoding device 200 according to this embodiment includes a feature map internal decoding unit 201, a feature map inverse conversion unit 202, and a feature map restoration unit 203. The feature map decoding device 200 receives as input a bit stream encoded by the feature map encoding device 100, decodes the bit stream, and generates three-layer multi-scale feature maps x1up, x2up, and x3up, which it supplies to a neural network identification unit 204.

[0030] The feature map internal decoding unit 201 decodes the bitstream coded by the feature map internal coding unit 104 in accordance with an image coding standard such as VVC, HEVC, or AV1, generates an integer-type packing feature map, and supplies it to the feature map inverse conversion unit 202.

[0031] The feature internal decoding unit 201 will be described in detail with reference to FIG.

[0032] The feature map inverse conversion unit 202 performs inverse quantization and unpacking on the integer-type packed feature map supplied from the feature internal decoding unit 201, converting it into a decimal-type single-scale feature map xr, and supplies it to the feature map restoration unit 203.

[0033] The feature map inverse conversion unit 202 will be described in detail with reference to Fig. 8. The feature map restoration unit 203 converts the single-scale feature map xr supplied from the feature map inverse conversion unit 202 into three-layer multi-scale feature maps x1up, x2up, and x3up, and supplies them to the neural network identification unit 204 as outputs of the feature map decoding device 200.

[0034] The feature map restoration unit 203 will be described in detail with reference to FIG.

[0035] The neural network identification unit 204 performs identification processing such as identifying objects, places, landscapes, people, and living things in the image to be identified, based on the three-layer multi-scale feature maps x1up, x2up, and x3up supplied by the feature map restoration unit 203.

[0036] <Feature Map Reduction and Feature Map Restoration> The feature map reduction unit 102 has a function of converting a multi-layered multi-scale feature map acquired from the neural network feature extraction unit 101 into a single-layered single-scale feature map.

[0037] The feature map reduction unit 102 will be described in detail using Fig. 5. The feature map reduction unit 102 is composed of a first feature map reduction unit 501, a first channel combination unit 502, a second feature map reduction unit 503, a second channel combination unit 504, a third feature map reduction unit 505, a first padding unit 506, a second padding unit 507, and a third padding unit 508. The feature map reduction unit 102 in Fig. 5 is an example of a configuration that converts a three-layer multi-scale feature map into a single-scale feature map.

[0038] The feature map reduction unit 102 receives three-layer multi-scale feature maps, i.e., a first feature map x1, a second feature map x2, and a third feature map x3, as input, converts them into a single-layer single-scale feature map xf, and supplies the resulting map to the feature map conversion unit 103. Here, n denotes an index indicating a layer, Cn denotes the number of channels in the nth layer, Wn denotes the width of the feature map, and Hn denotes the height of the feature map. In this embodiment, the values ​​of Cn, Wn, and Hn for each layer are as shown in FIG. 11 , where H and W denote the width and height of the image from which feature extraction is performed, respectively. The first padding unit 506 performs padding on the first feature map x1 to generate a first padded feature map x1pad. The first padding unit 506 determines the padding size so that the width and height of x1pad are multiples of 64. The number of channels in x1pad is 256, the same as that of x1.

[0039] The second padding unit 507 performs padding on the second feature map x2 by folding to generate a second padded feature map x2pad. The second padding unit 507 determines the padding size so that the width and height of x2pad are multiples of 32. The number of channels of x2pad is 256, the same as that of x2.

[0040] The third padding unit 508 performs padding on the third feature map x3 by folding to generate a third padded feature map x3pad. The third padding unit 508 determines the padding size so that the width and height of x3pad are multiples of 16. The number of channels of x3pad is 256, the same as that of x3.

[0041] In the first padding unit 506, the second padding unit 507, and the third padding unit 508, the padding size on the left side is the same as the padding size on the right side, and the padding size on the top side is the same as the padding size on the bottom side. That is, the feature maps x1, x2, and x3 are arranged at the centers of x1pad, x2pad, and x3pad, respectively.

[0042] The first feature map reduction unit 501 performs convolution in the spatial and channel directions on the first padded feature map x1pad acquired from the first padding unit 506 to generate a first intermediate feature map y1. The number of channels of y1 is 192, its width is Wx1pad / 2, and its height is Hx1pad / 2, where Wx1pad and Hx1pad are the width and height of the first padded feature map x1pad, respectively.

[0043] The first channel combining unit 502 has a function of combining the first intermediate feature map y1 obtained from the first feature map reduction unit 501 and the second padded feature map x2pad obtained from the second padding unit 507 in the channel direction to generate an intermediate feature map y1Cx2pad. Since the number of channels of y1 is 192 and the number of channels of x2pad is 256, the number of channels of the intermediate feature map y1Cx2pad is 448 (192 + 256).

[0044] The second feature map reduction unit 503 performs convolution in the spatial and channel directions on the intermediate feature map y1Cx2pad obtained from the first channel combination unit 502 to generate a second intermediate feature map y2. The number of channels of y2 is 192, its width is Wy1Cx2pad / 2, and its height is Hy1Cx2pad / 2, where Wy1Cx2pad and Hy1Cx2pad are the width and height of the intermediate feature map y1Cx2pad, respectively.

[0045] The second channel combining unit 504 has a function of combining the second intermediate feature map y2 obtained from the second feature map reduction unit 503 and the third padded feature map x3pad obtained from the third padding unit 508 in the channel direction to generate an intermediate feature map y2Cx3pad. Since the number of channels in the intermediate feature map y2 is 192 and the number of channels in x3pad is 256, the number of channels in y1Cx2pad is 448 (192 + 256).

[0046] The third feature map reduction unit 505 performs convolution in the spatial and channel directions on the intermediate feature map y2Cx3pad obtained from the second channel combination unit 504 to generate a third intermediate feature map y3. The number of channels of y3 is 192, its width is Wy2Cx3pad / 2, and its height is Hy2Cx3pad / 2, where Wy2Cx3pad and Hy2Cx3pad are the width and height of the intermediate feature map y2Cx3pad, respectively.

[0047] The feature map reduction unit 102 outputs the third intermediate feature map y3 as a single-scale feature map xf and supplies it to the feature map conversion unit 103.

[0048] The feature map restoration unit 203 has a function of converting the single-scale feature map xr acquired from the feature map inverse conversion unit 202 into three-layer multi-scale feature maps x1up, x2up, and x3up.

[0049] The feature map restoration unit 203 will be described in detail with reference to Fig. 6. The feature map restoration unit 203 is configured with an 8x enlargement unit 601, a 4x enlargement unit 602, a 2x enlargement unit 603, a first feature map blending unit 604, a second feature map blending unit 605, a first depadding unit 606, a second depadding unit 607, and a third depadding unit 608.

[0050] The 8x enlargement unit 601 performs transposition convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr acquired from the feature map inverse transformation unit 202 to enlarge the feature map and reduce the channels, thereby generating an intermediate feature map z1. The number of channels in z1 is 196. If the width and height of the single-scale feature map xr are xrwidth and xrheight, respectively, the width and height of z1 are xrwidth × 8 and xrheight × 8, respectively. Here, rwidth × 8 and xrheight × 8 are the same as the width and height of the first padded feature map x1pad, which is the output of the first padding unit 506 of the feature extraction unit 102.

[0051] The 4x enlargement unit 602 performs transposition convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr obtained from the feature map inverse transformation unit 202 to enlarge the features and reduce the channels, thereby generating an intermediate feature map z2. The number of channels in z2 is 196. The width and height of z2 are xrwidth×4 and xrheight×4, respectively. Here, rwidth×4 and xrheight×4 are the same as the width and height, respectively, of the second padded feature map x2pad, which is the output of the second padding unit 507 of the feature extraction unit 102.

[0052] The 2x enlargement unit 603 performs transposition convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr acquired from the feature map inverse transformation unit 202, thereby enlarging the feature map and reducing the channels, thereby generating an intermediate feature map z3. The number of channels in z3 is 196. The width and height of z3 are xrwidth×2 and xrheight×2, respectively. Here, rwidth×2 and xrheight×2 are the same as the width and height, respectively, of the third padded feature map x3pad, which is the output of the third padding unit 508 of the feature extraction unit 102.

[0053] The first feature map mixing unit 604 has a function of generating an intermediate feature map z2up by improving the quality of the intermediate feature map z2 obtained from the 4x enlargement unit 602, using the intermediate feature map z1 obtained from the 8x enlargement unit 601.

[0054] The second feature map mixing unit 605 has a function of generating an intermediate feature map z3up by using the intermediate feature map z2up obtained from the first feature map mixing unit 604, with the quality of the intermediate feature map z3 obtained from the 2x enlargement unit 603 being improved.

[0055] The padding removal unit 606 removes padding from the intermediate feature map z1 acquired from the 8x enlargement unit 601 to generate a first output feature map x1up. The width and height of x1up are the same as the width and height of the first feature map x1 input to the feature extraction unit 102.

[0056] The padding removal unit 607 removes padding from the intermediate feature map z2up acquired from the first feature map blending unit 604 to generate a second output feature map x2up. The width and height of x2up are the same as the width and height of the second feature map x2 input to the feature extraction unit 102.

[0057] The padding removal unit 608 removes padding from the intermediate feature map z3up obtained from the second feature map blending unit 605 to generate a third output feature map x3up. The width and height of x3up are the same as the width and height of the third feature map x3 input to the feature extraction unit 102.

[0058] In the first padding removal unit 606, the second padding removal unit 607, and the third padding removal unit 608, the left-side padding size is the same as the right-side padding size, and the upper-side padding size is the same as the lower-side padding size, similar to the first padding unit 506, the second padding unit 507, and the third padding unit 508 of the feature extraction unit 102. In other words, the padding on the top, bottom, left, and right sides is removed assuming that each output feature map x1up, x2up, and x3up is placed at the center of each intermediate feature map z1, z2up, and z3up, respectively.

[0059] <Feature Map Conversion and Feature Map Inverse Conversion> The feature map conversion unit 103 has a function of performing packing processing and quantization processing on the multi-channel integer-type single-scale feature map xf supplied from the feature map reduction unit 102, and converting it into an integer-type packed feature frame to be supplied to the feature map internal coding unit 104.

[0060] The feature map conversion unit 103 on the encoding side will be described in detail with reference to Fig. 7. The feature map conversion unit 103 is made up of a packing unit 701 and a feature map quantization unit 702.

[0061] The packing unit 701 has a function of generating a packed feature frame by combining input feature maps of multiple channels into one frame. FIG. 12 is a diagram illustrating a state in which feature maps of multiple channels are packed into one frame. The feature maps of each channel are sequentially arranged in one frame in raster scan order, from left to right and top to bottom. In this embodiment, the packing scan order can be specified by information such as a flag. The packing scan order will be described in detail later.

[0062] Furthermore, the packing unit 701 performs flipping based on the position in the frame where the channel feature map is arranged. Whether or not to perform flipping may be selected, and the selection may be transmitted from the encoding side to the decoding side via a bitstream.

[0063] Flipping when packing feature maps of multiple channels into one frame will be described using FIG. 13 . In flipping, when packing the feature maps of each channel, the positions of the elements (pixels) of the feature maps of each channel are flipped horizontally (left and right), vertically (up and down), or horizontally and vertically (up and down, left and right). The feature maps of four channels, A (upper left), B (upper right), C (lower left), and D (lower right) in FIG. 13 , are grouped together, and flipping is not performed at position A in FIG. 13 . At position B, the feature map is flipped horizontally (left and right). At position C, the feature map is flipped vertically (up and down). At position D, the feature map is flipped horizontally and vertically (up and down, left and right). When the distribution of elements in the feature maps of each channel is similar, flipping based on the position in the frame where the channels are placed reduces the boundaries between the feature maps of each channel, improving coding efficiency.

[0064] The feature map quantization unit 702 has a function of converting elements of a decimal-type packing feature frame (feature map of all channels) into N-bit integers (N = an integer from approximately 8 to 16) within a predetermined range and outputting an integer-type packing feature frame. In this embodiment, the elements are converted into 10-bit integers ranging from 0 to 1023. The feature map quantization unit 702 detects the minimum and maximum values ​​of the elements of the decimal-type packing feature frame and transmits the detected minimum and maximum values ​​of the elements of the packing feature frame (feature map of all channels) to the decoding side as metadata. To convert from a decimal-type packing feature frame (feature map) to an integer-type packing feature frame (feature map), a linear conversion is performed in which the minimum value of the decimal type is equivalent to the minimum value of the integer type and the maximum value of the decimal type is equivalent to the maximum value of the integer type. For example, when the integer type range is expressed in 10 bits, the minimum value of the elements of the integer-type packing feature frame (feature map) is 0 and the maximum value is 1023 (2 - 1). Values ​​between the minimum and maximum values ​​are linearly quantized.

[0065] Next, the feature map inverse transformation unit 202 has a function of performing inverse quantization processing and unpacking processing on integer-type packing feature frames decoded in VVC, HEVC, AV1, etc., supplied from the feature map internal decoding unit 201, and performing inverse transformation into a decimal-type single-scale feature map xr to be supplied to the feature map restoration unit 203.

[0066] The feature map inverse transformation unit 202 on the decoding side will be described in detail with reference to Fig. 8. The feature map inverse transformation unit 202 performs the inverse process of the feature map transformation unit 103, and is composed of a feature map inverse quantization unit 801 and an unpacking unit 802.

[0067] The feature map inverse quantization unit 801 performs the inverse process of the feature map quantization unit 702 on the encoding side, and has a function of converting elements of integer-type packing feature frames from integer to decimal type. The feature map inverse quantization unit 801 converts the integer-type packing feature frame decoded by the feature map internal decoding unit 201 into a decimal-type packing feature frame using the minimum and maximum decimal values ​​transmitted as metadata. A linear transformation is performed to convert the integer minimum value to the decimal minimum value and the integer maximum value to the decimal maximum value. Linear inverse quantization is performed on values ​​between the minimum and maximum values.

[0068] The unpacking unit 802 extracts a feature map of each channel from the packing feature frames arranged in one frame in raster scan order and supplies it to the feature map restoration unit 203 as a single-scale feature map xr.

[0069] <Feature Map Intra-Encoding / Decoding> The feature map intra-encoding unit 104 will be described in detail with reference to FIG. 9 . The feature map intra-encoding unit 104 is composed of a switch 901, a VVC encoding unit 902, an HEVC encoding unit 903, and an AV1 encoding unit 904. The switch 901 selects the encoding standard for intra-encoding the feature map converted by the feature map conversion unit 103. The VVC encoding unit 902 encodes the feature map in accordance with the VVC standard and outputs a bitstream compliant with the VVC standard. The HEVC encoding unit 903 encodes the feature map in accordance with the HEVC standard and outputs a bitstream compliant with the HEVC standard. The AV1 encoding unit 904 encodes the feature map in accordance with the AV1 standard and outputs a bitstream compliant with the AV1 standard.

[0070] In the VVC, HEVC, and AV1 standards, an image is divided into blocks of a predetermined size and then encoded.

[0071] It is also possible to implement only one of VVC, HEVC, and AV1, and it is also possible to use an image coding method other than VVC, HEVC, and AV1.

[0072] Next, the feature map internal decoding unit 201 will be described in detail with reference to FIG. 10 . The feature map internal decoding unit 201 is composed of a switch 1001, a VVC decoding unit 1002, an HEVC decoding unit 1003, and an AV1 decoding unit 1004. The switch 1001 selects the encoding standard for internal decoding based on information for selecting internal decoding within the input bitstream. The VVC decoding unit 1002 decodes the feature map in accordance with the VVC standard. The HEVC decoding unit 1003 decodes the feature map in accordance with the HEVC standard. The AV1 encoding unit 1004 decodes the feature map in accordance with the AV1 standard.

[0073] In the VVC, HEVC, and AV1 standards, decoding is performed for each predetermined block size.

[0074] It is also possible to implement only one of VVC, HEVC, and AV1, and it is also possible to use an image coding method other than VVC, HEVC, and AV1.

[0075] <Frame division for intra-encoding / decoding of feature maps> In the VVC, HEVC, and AV1 standards, one frame is divided into regions called slices or tiles, and reference across slice or tile boundaries can be prohibited. By encoding and decoding each slice or tile independently, delays in encoding and decoding processes can be reduced.

[0076] Frame division in the VVC standard will be described using FIG. 14 . FIG. 14 shows an example in which a frame 800 is divided into 16 tiles and 9 slices in rectangular slice mode. In FIG. 14 , 801 to 816 represent tiles. 817 to 825 represent slices. Tiles 801 to 816 are rectangular regions each including 20 coding tree units (CTUs), 6 horizontally and 4 vertically, for a total of 24 CTUs. A CTU is the basic encoding / decoding unit in VVC. In rectangular slice mode, a slice can be composed of multiple consecutive tiles forming a rectangular region. Slices 817, 819, 823, and 825 are each composed of a single tile. Slice 818 is composed of two tiles, tile 802 and tile 803. Similarly, slices 820, 822, and 824 are each composed of two tiles. The slice 821 is composed of four tiles: a tile 806, a tile 807, a tile 810, and a tile 811.

[0077] The scan order when tiles and slices are mixed will be described using Figure 15. In rectangular slice mode, slices within slices are scanned in raster scan order, and then tiles within each slice are scanned in raster scan order. Tiles 805 and 809 form slice 820. On the other hand, tile 806, located to the right of tile 805, belongs to slice 821, which is different from slice 820. Therefore, tile 809 is scanned next after tile 805. Similarly, tiles 806, 807, 810, and 811 form slice 821, so they are scanned in this order.

[0078] <Packing Scanning Order> The scanning order in packing described above with reference to Fig. 12 differs from the scanning order of slices and tiles in feature map intra-encoding and decoding. As soon as feature maps for slices and tiles that can be processed independently are available, the intra-encoding and decoding of feature maps can start processing the slices and tiles. Therefore, by matching the scanning order of packing with the scanning order of slices and tiles, delays can be suppressed.

[0079] In this embodiment, a single-scale feature map is divided into a plurality of packing groups, and a means for scanning the packing groups in a specified order is provided, thereby making it possible to match the scan order of packing with the scan order of slices and tiles.

[0080] 16 shows metadata encoding information for specifying a packing group. number_of_packing_group specifies the number of packing groups. start_position_x, start_position_y, packing_group_width, and packing_group_height are information set for each slice group. start_position_x and start_position_y specify the horizontal and vertical coordinates of the start position of the packing group, respectively. packing_group_width and packing_group_height specify the horizontal and vertical sizes of the packing group, respectively. That is, in this embodiment, the shape of a packing group is assumed to be rectangular.

[0081] The process of specifying the scan order of packing in the packing unit 701 will be described with reference to the flowchart in FIG.

[0082] First, the number of packing groups is determined (step S101), and all packing groups are scanned in order (steps S102 and S108).

[0083] The start position of the packing group to be processed is identified (step S103), and the number of channels included in the packing group to be processed is determined (step S104). If the horizontal size and vertical size of the channel are channel_width and channel_height, respectively, the number of channels included in the packing group, num_channel, can be calculated by packing_group_width / channel_width×packing_group_height / channel_height. If packing_group_width / channel_width or packing_group_height / channel_height has a fraction, a space will be generated in the packing group where channels cannot be packed. Spaces that cannot be packed with channels are used to store unnecessary data, so they are initialized with a predetermined value on the encoding side and truncated on the decoding side.

[0084] The following processing is performed for the num_channel channels (steps S105 and S108).

[0085] Using channel_width, channel_height, packing_width, packing_height, start_position_x, and start_position_y, the packing position of the channel to be processed is identified in raster scan order (step S106), and the channel to be processed is packed into the identified packing position (step S107).

[0086] After packing of all packing groups is completed, metadata is transmitted according to the rules in FIG.

[0087] An example of packing using this procedure is shown in Fig. 18. In Fig. 18, a single-scale feature map is divided into 16 packing loops, from packing group 901 to packing group 916. This division has the same shape as the tile division of the VVC standard in Fig. 14. Furthermore, the scan order of the packing groups in the example of Fig. 18, 901, 902, 903, 904, 905, 909, 906, 907, 910, 911, 908, 912, 913, 914, 915, and 916, is the same as the scan order of the VVC standard in Fig. 15.

[0088] 19 , metadata encoding information of min_value_of_features and max_value_of_features may be added to each Packing group. min_value_of_features and max_value_of_features are the maximum and minimum values, respectively, of the elements of the feature maps of all the channels included in the Packing group. This enables the feature map quantization unit 702 and the feature map inverse quantization unit 801 to perform the quantization process and inverse quantization process on a Packing group basis, which is more suitable for suppressing delays.

[0089] Next, the scanning order of the unpacking process in the unpacking unit 802 will be described.

[0090] First, metadata is acquired and decoded according to the rules in FIG. 16 (step S201).

[0091] The packing group is scanned the number of times specified by the decrypted metadata number_of_packing_group (steps S202 and S207).

[0092] The following process is performed the number of times specified by num_channel (steps S203 and S207). As with the packing unit 701, num_channel can be calculated by packing_group_width / channel_width×packing_group_height / channel_height.

[0093] Using channel_width, channel_height and the metadata packing_width, packing_height, start_position_x, and start_position_y, the packing position of the channel to be processed is identified in raster scan order (step S204), and the channel to be processed is packed into the identified packing position (step S206).

[0094] 20 , metadata that reuses metadata defined in the internal encoding / decoding of the feature map may be defined to specify a packing group. When uses_inner_codec_partition is 1, the shape of the packing group is determined based on the shape of the slice / tile partition defined in the internal encoding / decoding of the feature map. In this case, the size and position of the packing group can be calculated using the metadata defined in the internal encoding / decoding of the feature map, and therefore, the metadata of number_of_packing_group, start_position_x, start_position_y, packing_group_width, packing_group_height, start_position_x, and start_position_y described with reference to FIG. 16 do not need to be transmitted. However, metadata of min_value_of_features and max_value_of_features used for quantizing the feature map may be transmitted because they are not related to the shape of the packing group.

[0095] When uses_inner_codec_partition is 0, packing processing of the feature map is performed regardless of the slice / tile division defined in the feature map internal encoding / decoding. In this embodiment, when uses_inner_codec_partition is 0, packing is performed by raster scanning without dividing into packing groups, as described above with reference to FIG. 12 .

[0096] By adopting this embodiment, the scan order in feature map packing can be made to match the scan order of slices and tiles in feature map internal encoding and internal decoding, thereby eliminating bottlenecks caused by scan order mismatches and suppressing delays in encoding and decoding.

[0097] In all of the above-described embodiments, the bitstream output by the feature map encoder has a specific data format that can be decoded according to the encoding method used in the embodiment, and the feature map decoder corresponding to the feature map encoder can decode the bitstream in this specific data format.

[0098] When a wired or wireless network is used to exchange bitstreams between the feature map encoder and the feature map decoder, the bitstreams may be converted into a data format suitable for the transmission mode of the communication channel before transmission. In this case, a transmitter is provided that converts the bitstream output by the feature map encoder into coded data in a data format suitable for the transmission mode of the communication channel and transmits the coded data to the network, and a receiver is provided that receives the coded data from the network, restores the coded data to a bitstream, and supplies the bitstream to the feature map decoder. The transmitter includes a memory that buffers the bitstream output by the feature map encoder, a packet processor that packetizes the bitstream, and a transmitter that transmits the packetized coded data via the network. The receiver includes a receiver that receives the packetized coded data via the network, a memory that buffers the received coded data, and a packet processor that packetizes the coded data to generate a bitstream and provides it to the feature map decoder.

[0099] The above encoding and decoding processes may be realized not only as a transmission, storage, and receiving device using hardware, but also as firmware stored in a ROM (read-only memory), flash memory, etc., or as software for a computer, etc. The firmware program or software program may be provided by recording it on a recording medium readable by a computer, etc., or may be provided from a server via a wired or wireless network, or may be provided as data broadcasting via terrestrial or satellite digital broadcasting.

[0100] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the components and treatment processes, and that such modifications are also within the scope of the present invention.

[0101] The present invention can be used in encoding and decoding techniques for feature maps.

[0102] 100 Feature map encoding device, 101 Neural network feature extraction unit, 102 Feature map reduction unit, 103 Feature map conversion unit, 104 Feature map internal encoding unit, 200 Feature map decoding device, 201 Feature map internal decoding unit, 202 Feature map inverse conversion unit, 203 Feature map restoration unit, 204 Neural network identification unit, 301 Convolution processing unit, 302 Activation processing unit, 303 Pooling processing unit, 322 Bottom-up processing unit, 324 Top-down processing unit, 326 Image to be subjected to feature extraction processing, 501 First feature map reduction unit, 502 First channel combination unit, 503 Second feature map reduction unit, 504 Second channel combination unit, 505 Third feature map reduction unit, 506 First padding unit, 507 Second padding unit, 508 Third padding unit, 601 8x enlargement unit, 602 4x enlargement unit, 603 2x enlargement unit, 604 First feature map mixing unit, 605 Second feature map mixing unit, 606 First unpadding unit, 607 Second unpadding unit, 608 Third unpadding unit, 701 Packing unit, 702 Feature map quantization unit, 801 Feature map inverse quantization unit, 802 Unpacking unit, 901 Switch, 902 VVC encoding unit, 903 HEVC encoding unit, 904 AV1 encoding unit, 1001 Switch, 1002 VVC decoding unit, 1003 HEVC decoding unit, 1004 AV1 decoding unit.

Claims

1. A feature map encoding device comprising: a feature map reduction unit that converts a multi-scale feature map to generate a single-scale feature map; a packing unit that divides the single-scale feature map into one or more packing groups and packs them into frames to generate a decimal-type packing feature frame; a quantization unit that converts elements of the decimal-type packing feature frame into integer values ​​to generate an integer-type packing feature frame; a feature map inner encoding unit that encodes the integer-type packing feature frame; and an encoding unit that encodes information related to the packing groups.

2. A feature map encoding method comprising: a feature map reduction step of converting a multi-scale feature map to generate a single-scale feature map; a packing step of dividing the single-scale feature map into one or more packing groups and packing them into frames to generate a decimal-type packed feature frame; a quantization step of converting elements of the decimal-type packed feature frame into integer values ​​to generate an integer-type packed feature frame; a feature map inner encoding step of encoding the integer-type packed feature frame; and an encoding step of encoding information related to the packing groups.

3. A feature map encoding program that causes a computer to execute the following steps: a feature map reduction step of converting a multi-scale feature map to generate a single-scale feature map; a packing step of dividing the single-scale feature map into one or more packing groups and packing them into frames to generate a decimal-type packed feature frame; a quantization step of converting elements of the decimal-type packed feature frame into integer values ​​to generate an integer-type packed feature frame; a feature map inner encoding step of encoding the integer-type packed feature frame; and an encoding step of encoding information related to the packing groups.

4. A feature map decoding device comprising: a decoding unit that decodes information related to packing groups; a feature map internal decoding unit that decodes a single-scale feature map packed into a frame and encoded to generate an integer-type packing feature frame; a dequantization unit that converts elements of the integer-type packing feature frame into decimal values ​​to generate a decimal-type packing feature frame; an unpacking unit that divides the decimal-type packing feature frame into one or more packing groups based on the information related to the packing groups and unpacks them to generate a single-scale feature map; and a feature map restoration unit that converts the single-scale feature map to generate a multi-scale feature map.

5. A feature map decoding method comprising: a decoding step of decoding information related to packing groups; a feature map inner decoding step of decoding a single-scale feature map packed and encoded into a frame to generate an integer-type packing feature frame; a dequantization step of converting elements of the integer-type packing feature frame into decimal values ​​to generate a fractional-type packing feature frame; an unpacking step of dividing the fractional-type packing feature frame into one or more packing groups based on the information related to the packing groups and unpacking them to generate a single-scale feature map; and a feature map restoration step of converting the single-scale feature map to generate a multi-scale feature map.

6. A feature map decoding program causing a computer to execute the following steps: a decoding step of decoding information related to packing groups; a feature map inner decoding step of decoding a single-scale feature map packed and encoded into a frame to generate an integer-type packing feature frame; a dequantization step of converting elements of the integer-type packing feature frame into decimal values ​​to generate a fractional-type packing feature frame; an unpacking step of dividing the fractional-type packing feature frame into one or more packing groups based on the information related to the packing groups and unpacking them to generate a single-scale feature map; and a feature map restoration step of converting the single-scale feature map to generate a multi-scale feature map.

Images