Feature map encoding device, feature map encoding method, feature map decoding device, and feature map decoding method

Abstract

In order to solve the abovementioned problem, a feature map encoding device according to the present application is characterized by comprising: a feature map reduction unit (102) that transforms a multi-scale feature map to generate a single-scale feature map; a packing unit (701) that packs the single-scale feature map into a frame to generate a decimal-type packing feature frame; a quantization unit (702) that transforms elements of the decimal-type packing feature frame into integer values to generate an integer-type packing feature frame; and a feature map internal encoding unit (104) that encodes the integer-type packing feature frame, wherein, when the number of channels in the single-scale feature map is smaller than a prescribed minimum number of channels, the packing unit (701) adds a dummy feature map to the single-scale feature map and packs the result thereof into a frame.

Description

Feature map encoding device, feature map encoding method, feature map decoding device, feature map decoding method

[0001] This paper deals with encoding and decoding of feature maps in neural networks.

[0002] As a neural network technology used for image recognition such as detecting objects at various scales in an image, dividing regions into individual objects, or tracking objects, the FPN (Feature Pyramid Network) described in Non-Patent Document 1 is known. In the FPN, multiple feature maps at various scales are generated from the image to be processed, and various image recognitions are performed using the feature maps.

[0003] The FPN used for image recognition generates multiple feature maps from an image, and its structure utilizes a CNN (Convolutional Neural Network). A CNN can be divided into a feature extraction section (backbone) that reads an image and generates a feature map through convolution and pooling, and a classification section (head) that is composed of hierarchical fully connected layers and generates output suitable for tasks such as object detection, instance segmentation, and object tracking. The FPN utilizes the CNN backbone.

[0004] The feature extraction unit of the FPN is typically configured with a hierarchical structure in which a convolution process 301, an activation process 302, and a pooling process 303 shown in FIG. 3 are configured as one basic unit, and this basic unit is repeated.

[0005] The structure of the FPN is shown in Figure 4. The FPN consists of a bottom-up processing unit 322 that uses a CNN backbone to generate a multi-scale feature map consisting of multiple hierarchical layers, and a top-down processing unit 324 that aggregates features from deep layer feature maps to shallow layer feature maps using the inverse structure of the CNN backbone. The bottom-up processing unit 322 reduces the resolution of the feature map by half by repeating the basic units of convolution processing 301, activation processing 302, and pooling processing 303 shown in Figure 3, thereby generating a pyramid of feature maps with multiple layers. Meanwhile, the top-down processing unit 324 increases the resolution of the feature map to the same resolution as the input image while adding feature maps with resolutions corresponding to those of the bottom-up processing unit 322, thereby generating a pyramid of feature maps. In other words, the FPN generates multiple feature maps for each layer from an image 326 to be subjected to feature extraction processing.

[0006] The convolution processing unit 301 performs convolution processing on the data to be processed (image or feature map) using multiple types of predetermined filters (kernels). In the convolution processing in the convolution processing unit 301, predetermined filtering is performed on all of the data to be processed while sliding at predetermined intervals. The sliding interval is called a stride. The convolution processing unit 301 may determine the stride based on the number of data to be processed. For example, the convolution processing unit 301 may determine the stride to be 1 if the number of data to be processed is smaller than a predetermined value, and may determine the stride to be 2 if the number of data to be processed is equal to or greater than the predetermined value. Multiple predetermined filters are prepared in each layer, and multiple feature maps are generated by generating one feature map for each filter. The unit of the feature map is the channel. If the number (types) of predetermined filters is N (N types), N feature maps (N channels) are generated.

[0007] The activation processing unit 302 performs activation processing to nonlinearly convert the feature map output from the convolution processing unit 301. Here, the function used in the activation processing is called an activation function. The activation processing unit 302 uses a ReLU (Rectified Linear Unit) function, a sigmoid function, or the like as the activation function.

[0008] The pooling processing unit 303 is a process for downsampling the feature map output from the activation processing unit 302 by replacing local values ​​of the feature map with representative values.

[0009] On the other hand, when performing classification using a neural network, it has the ability to perform tasks using feature maps of multiple channels at each layer.

[0010] In image classification, the feature maps of multiple channels at each layer are convolved for each predetermined size based on the scale of the feature map, and the probability of the object class is calculated for each pixel.

[0011] Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature Pyramid Networks for Object Detection.

[0012] The amount of information in a feature map is enormous, making it unsuitable for transmission and storage. In view of the above-mentioned problems, an object of this embodiment is to provide a technique for encoding and decoding a feature map.

[0013] In order to solve the above problems, the feature map encoding device of the present application includes a feature map reduction unit that converts a multi-scale feature map to generate a single-scale feature map; a packing unit that packs the single-scale feature map into a frame to generate a decimal-type packed feature frame; a quantization unit that converts elements of the decimal-type packed feature frame into integer values ​​to generate an integer-type packed feature frame; and a feature map inner encoding unit that encodes the integer-type packed feature frame, wherein if the number of channels in the single-scale feature map is smaller than a predetermined minimum number of channels, the packing unit adds a dummy feature map to the single-scale feature map and packs it into the frame.

[0014] Also, one aspect of a feature map decoding device of the present application includes a feature map internal decoding unit that decodes a single-scale feature map that has been packed into a frame and encoded, and generates an integer-type packing feature frame; a dequantization unit that converts elements of the integer-type packing feature frame into decimal values, and generates a decimal-type packing feature frame; an unpacking unit that divides the decimal-type packing feature frame into one or more packing groups based on information related to the packing groups, and unpacks the divided decimal-type packing feature frame, and generates a single-scale feature map; and a feature map restoration unit that transforms the single-scale feature map, and generates a multi-scale feature map, wherein the unpacking unit excludes dummy feature maps when unpacking, if the number of channels in the single-scale feature map is smaller than a predetermined minimum number of channels.

[0015] According to this embodiment, feature maps can be efficiently encoded and decoded with a small amount of processing.

[0016] 1 is a block diagram illustrating the configuration of a feature map encoding device 100. FIG. 2 is a block diagram illustrating the configuration of a feature map decoding device 200. FIG. 3 is a block diagram illustrating the processing that is the basic unit in each layer of an FPN. FIG. 4 is a block diagram illustrating the structure of an FPN. FIG. 5 is a block diagram illustrating the detailed configuration of a feature map reduction unit 102. FIG. 6 is a block diagram illustrating the detailed configuration of a feature map restoration unit 203. FIG. 7 is a block diagram illustrating the detailed configuration of a feature map transformation unit 103. FIG. 8 is a block diagram illustrating the detailed configuration of a feature map inverse transformation unit 202. FIG. 9 is a block diagram illustrating the detailed configuration of a feature map internal encoding unit 104. FIG. 10 is a block diagram illustrating the detailed configuration of a feature map internal decoding unit 201. FIG. 11 is a diagram illustrating the number of channels of feature maps x1, x2, and x3, the width and height of the feature maps. FIG. 12 is a diagram illustrating the state in which feature maps of multiple channels are packed into one frame. FIG. 13 is a diagram illustrating flipping when packing feature maps of multiple channels into one frame. FIG. 14 is a diagram illustrating layers and units handled in the feature map encoding device and feature map decoding device of this embodiment. FIG. 15 is a flowchart illustrating the operation of a packing unit 701. 7 is a flowchart for explaining processing for setting the number of encoding channels in the packing unit 701. FIG. 8 is a flowchart for explaining packing processing in the packing unit 701. FIG. 9 is a flowchart for explaining operation of an unpacking unit 802. FIG. 10 is a diagram for explaining metadata for setting the minimum number of channels. FIG. 11 is a diagram for explaining packed feature frames in an embodiment.

[0017] The technologies and technical terms used in this embodiment will be defined below.

[0018] <Features and feature maps> In a convolutional neural network (CNN), a filter is used to scan the image (input layer data) from which features are extracted while changing its position, and the results of the convolution operation between the filter coefficients are used as features or feature maps.

[0019] <Packing> Frame packing refers to combining two or more frames (pictures) into a single frame (picture) by arranging them in a tiled pattern. In this application, packing refers to combining feature maps of multiple channels into a single frame. Figure 12 shows an example of frame packing.

[0020] <Data Type> A data type that represents an integer value is called an integer type, and a data type that represents a decimal value is called a decimal type.

[0021] <Layers and Units> The layers and units handled by the feature map encoding device and feature map decoding device of this embodiment will be described with reference to Figure 14. Reference numeral 181 in Figure 14 denotes a sequence of consecutive feature frames or feature maps of all channels over time, which is referred to as a sequence layer or sequence unit. Reference numeral 182 in Figure 14 denotes a sequence of consecutive feature maps of one channel over time, which is referred to as a sequence layer of each channel or a sequence unit of each channel. Reference numeral 183 in Figure 14 denotes a feature frame or feature maps of all channels at the same time, which is referred to as a frame layer or frame unit. Reference numeral 184 in Figure 14 denotes a feature map of one channel at a certain time, which is referred to as a feature map layer of each channel (a feature map unit of each channel).

[0022] First Embodiment A feature map encoding device 100 and a feature map decoding device 200 according to a first embodiment of the present invention will be described.

[0023] 1 is a block diagram of a feature map coding device 100 according to a first embodiment. The feature map coding device 100 according to the embodiment includes a feature map reduction unit 102, a feature map conversion unit 103, and a feature map internal coding unit 104. The feature map coding device 100 is a device that encodes a feature map generated by a neural network feature extraction unit 101 to generate and output a bitstream.

[0024] The neural network feature extraction unit 101 reads an image from which features are to be extracted, generates a feature map by FPN convolution processing, activation processing, and pooling processing, and supplies the generated feature map to the feature map reduction unit 102. In this embodiment, a three-layer multi-scale feature map of x1, x2, and x3 is generated.

[0025] The feature map reduction unit 102 converts the three-layer multi-scale feature maps x1, x2, and x3 acquired from the neural network feature extraction unit 101 into a single-layer single-scale feature map xf and supplies it to the feature map conversion unit 103. The feature map reduction unit 102 will be described in detail with reference to FIG. 5 .

[0026] The feature map conversion unit 103 performs packing and quantization on the decimal-type single-scale feature map xf supplied from the feature map reduction unit 102, converting it into an integer-type packed feature frame, and supplies it to the feature map internal coding unit 104.

[0027] The feature map conversion unit 103 will be described in detail with reference to FIG.

[0028] The feature map internal coding unit 104 generates and outputs a bitstream by coding the integer-type packing feature frame supplied from the feature map conversion unit 103 in accordance with an image coding standard such as VVC, HEVC, or AV1. The output bitstream is supplied to the feature map decoding device 200 or the like via a network or the like.

[0029] The feature map internal coding unit 104 will be described in detail with reference to FIG.

[0030] 2 is a block diagram showing the configuration of a feature map decoding device 200 according to an embodiment of the present invention, which corresponds to the feature map encoding device 100 in FIG. 1. The feature map decoding device 200 according to this embodiment includes a feature map internal decoding unit 201, a feature map inverse conversion unit 202, and a feature map restoration unit 203. The feature map decoding device 200 receives as input a bit stream encoded by the feature map encoding device 100, decodes the bit stream, and generates three-layer multi-scale feature maps x1up, x2up, and x3up, which it supplies to a neural network identification unit 204.

[0031] The feature map internal decoding unit 201 decodes the bitstream coded by the feature map internal coding unit 104 of the feature map coding device 100 in accordance with an image coding standard such as VVC, HEVC, or AV1, generates integer-type packing feature frames, and supplies them to the feature map inverse conversion unit 202.

[0032] The feature internal decoding unit 201 will be described in detail with reference to FIG.

[0033] The feature map inverse conversion unit 202 performs inverse quantization and unpacking on the integer-type packed feature frame supplied from the feature internal decoding unit 201, converting it into a decimal-type single-scale feature map xr, and supplies it to the feature map restoration unit 203.

[0034] The feature map inverse conversion unit 202 will be described in detail with reference to FIG.

[0035] The feature map restoration unit 203 converts the single-scale feature map xr supplied from the feature map inverse conversion unit 202 into three-layer multi-scale feature maps x1up, x2up, and x3up, and supplies them to the neural network identification unit 204 as outputs of the feature map decoding device 200.

[0036] The feature map restoration unit 203 will be described in detail with reference to FIG.

[0037] The neural network identification unit 204 performs identification processing such as identifying objects, places, landscapes, people, and living things in the image to be identified, based on the three-layer multi-scale feature maps x1up, x2up, and x3up supplied by the feature map restoration unit 203.

[0038] <Feature Map Reduction and Feature Map Restoration> The feature map reduction unit 102 has a function of converting a multi-layered multi-scale feature map acquired from the neural network feature extraction unit 101 into a single-layered single-scale feature map.

[0039] The feature map reduction unit 102 will be described in detail using Fig. 5. The feature map reduction unit 102 is composed of a first feature map reduction unit 501, a first channel combination unit 502, a second feature map reduction unit 503, a second channel combination unit 504, a third feature map reduction unit 505, a first padding unit 506, a second padding unit 507, and a third padding unit 508. The feature map reduction unit 102 in Fig. 5 is an example of a configuration that converts a three-layer multi-scale feature map into a single-scale feature map.

[0040] The feature map reduction unit 102 receives three-layer multi-scale feature maps, namely, a first feature map x1, a second feature map x2, and a third feature map x3, as input, converts them into a single-layer single-scale feature map xf, and supplies the resulting map to the feature map conversion unit 103. Here, n denotes an index indicating a layer, Cn denotes the number of channels in the nth layer, Wn denotes the width of the feature map, and Hn denotes the height of the feature map. In this embodiment, the values ​​of Cn, Wn, and Hn for each layer are as shown in FIG. 11 , where H and W denote the width and height, respectively, of the image from which feature extraction is performed.

[0041] The first padding unit 506 has a function of padding the first feature map x1 to generate a first padded feature map x1pad. The first padding unit 506 determines the padding size so that the width and height of x1pad are multiples of 64. The number of channels of x1pad is 256, the same as that of x1.

[0042] The second padding unit 507 performs padding on the second feature map x2 by folding to generate a second padded feature map x2pad. The second padding unit 507 determines the padding size so that the width and height of x2pad are multiples of 32. The number of channels of x2pad is 256, the same as that of x2.

[0043] The third padding unit 508 performs padding on the third feature map x3 by folding to generate a third padded feature map x3pad. The third padding unit 508 determines the padding size so that the width and height of x3pad are multiples of 16. The number of channels of x3pad is 256, the same as that of x3.

[0044] In the first padding unit 506, the second padding unit 507, and the third padding unit 508, the padding size on the left side is the same as the padding size on the right side, and the padding size on the top side is the same as the padding size on the bottom side. That is, the feature maps x1, x2, and x3 are arranged at the centers of x1pad, x2pad, and x3pad, respectively.

[0045] The first feature map reduction unit 501 performs convolution in the spatial and channel directions on the first padded feature map x1pad acquired from the first padding unit 506 to generate a first intermediate feature map y1. The number of channels of y1 is 192, its width is Wx1pad / 2, and its height is Hx1pad / 2, where Wx1pad and Hx1pad are the width and height of the first padded feature map x1pad, respectively.

[0046] The first channel combining unit 502 has a function of combining the first intermediate feature map y1 obtained from the first feature map reduction unit 501 and the second padded feature map x2pad obtained from the second padding unit 507 in the channel direction to generate an intermediate feature map y1Cx2pad. Since the number of channels of y1 is 192 and the number of channels of x2pad is 256, the number of channels of the intermediate feature map y1Cx2pad is 448 (192 + 256).

[0047] The second feature map reduction unit 503 performs convolution in the spatial and channel directions on the intermediate feature map y1Cx2pad obtained from the first channel combination unit 502 to generate a second intermediate feature map y2. The number of channels of y2 is 192, its width is Wy1Cx2pad / 2, and its height is Hy1Cx2pad / 2, where Wy1Cx2pad and Hy1Cx2pad are the width and height of the intermediate feature map y1Cx2pad, respectively.

[0048] The second channel combining unit 504 has a function of combining the second intermediate feature map y2 obtained from the second feature map reduction unit 503 and the third padded feature map x3pad obtained from the third padding unit 508 in the channel direction to generate an intermediate feature map y2Cx3pad. Since the number of channels in the intermediate feature map y2 is 192 and the number of channels in x3pad is 256, the number of channels in y1Cx2pad is 448 (192 + 256).

[0049] The third feature map reduction unit 505 performs convolution in the spatial and channel directions on the intermediate feature map y2Cx3pad obtained from the second channel combination unit 504 to generate a third intermediate feature map y3. The number of channels of y3 is 192, its width is Wy2Cx3pad / 2, and its height is Hy2Cx3pad / 2, where Wy2Cx3pad and Hy2Cx3pad are the width and height of the intermediate feature map y2Cx3pad, respectively.

[0050] The feature map reduction unit 102 outputs the third intermediate feature map y3 as a single-scale feature map xf and supplies it to the feature map conversion unit 103.

[0051] The feature map restoration unit 203 has a function of converting the single-scale feature map xr acquired from the feature map inverse conversion unit 202 into three-layer multi-scale feature maps x1up, x2up, and x3up.

[0052] The feature map restoration unit 203 will be described in detail with reference to Fig. 6. The feature map restoration unit 203 is configured with an 8x enlargement unit 601, a 4x enlargement unit 602, a 2x enlargement unit 603, a first feature map blending unit 604, a second feature map blending unit 605, a first depadding unit 606, a second depadding unit 607, and a third depadding unit 608.

[0053] The 8x enlargement unit 601 performs transposition convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr acquired from the feature map inverse transformation unit 202 to enlarge the feature map and reduce the channels, thereby generating an intermediate feature map z1. The number of channels in z1 is 196. If the width and height of the single-scale feature map xr are xrwidth and xrheight, respectively, the width and height of z1 are xrwidth × 8 and xrheight × 8, respectively. Here, rwidth × 8 and xrheight × 8 are the same as the width and height of the first padded feature map x1pad, which is the output of the first padding unit 506 of the feature extraction unit 102.

[0054] The 4x enlargement unit 602 performs transposition convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr obtained from the feature map inverse transformation unit 202 to enlarge the features and reduce the channels, thereby generating an intermediate feature map z2. The number of channels in z2 is 196. The width and height of z2 are xrwidth×4 and xrheight×4, respectively. Here, rwidth×4 and xrheight×4 are the same as the width and height, respectively, of the second padded feature map x2pad, which is the output of the second padding unit 507 of the feature extraction unit 102.

[0055] The 2x enlargement unit 603 performs transposition convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr acquired from the feature map inverse transformation unit 202, thereby enlarging the feature map and reducing the channels, thereby generating an intermediate feature map z3. The number of channels in z3 is 196. The width and height of z3 are xrwidth×2 and xrheight×2, respectively. Here, rwidth×2 and xrheight×2 are the same as the width and height, respectively, of the third padded feature map x3pad, which is the output of the third padding unit 508 of the feature extraction unit 102.

[0056] The first feature map mixing unit 604 has a function of generating an intermediate feature map z2up by improving the quality of the intermediate feature map z2 obtained from the 4x enlargement unit 602, using the intermediate feature map z1 obtained from the 8x enlargement unit 601.

[0057] The second feature map mixing unit 605 has a function of generating an intermediate feature map z3up by using the intermediate feature map z2up obtained from the first feature map mixing unit 604, with the quality of the intermediate feature map z3 obtained from the 2x enlargement unit 603 being improved.

[0058] The padding removal unit 606 removes padding from the intermediate feature map z1 acquired from the 8x enlargement unit 601 to generate a first output feature map x1up. The width and height of x1up are the same as the width and height of the first feature map x1 input to the feature extraction unit 102.

[0059] The padding removal unit 607 removes padding from the intermediate feature map z2up acquired from the first feature map blending unit 604 to generate a second output feature map x2up. The width and height of x2up are the same as the width and height of the second feature map x2 input to the feature extraction unit 102.

[0060] The padding removal unit 608 removes padding from the intermediate feature map z3up obtained from the second feature map blending unit 605 to generate a third output feature map x3up. The width and height of x3up are the same as the width and height of the third feature map x3 input to the feature extraction unit 102.

[0061] In the first padding removal unit 606, the second padding removal unit 607, and the third padding removal unit 608, the left-side padding size is the same as the right-side padding size, and the upper-side padding size is the same as the lower-side padding size, similar to the first padding unit 506, the second padding unit 507, and the third padding unit 508 of the feature extraction unit 102. In other words, the padding on the top, bottom, left, and right sides is removed assuming that each output feature map x1up, x2up, and x3up is placed at the center of each intermediate feature map z1, z2up, and z3up, respectively.

[0062] <Feature Map Conversion and Feature Map Inverse Conversion> The feature map conversion unit 103 has a function of performing packing processing and quantization processing on the multi-channel decimal-type single-scale feature map xf supplied from the feature map reduction unit 102, and converting it into an integer-type packed feature frame to be supplied to the feature map internal coding unit 104.

[0063] The feature map conversion unit 103 on the encoding side will be described in detail with reference to Fig. 7. The feature map conversion unit 103 is made up of a packing unit 701 and a feature map quantization unit 702.

[0064] The packing unit 701 has a function of generating a packed feature frame by combining input feature maps of multiple channels into one frame. FIG. 12 is a diagram illustrating the state in which feature maps of multiple channels are packed into one frame. The feature maps of each channel are sequentially arranged in one frame in raster scan order, from left to right and top to bottom. Details of the packing unit 701 will be described later.

[0065] Furthermore, the packing unit 701 performs flipping based on the position in the frame where the channel feature map is arranged. Whether or not to perform flipping may be selected, and the selection may be transmitted from the encoding side to the decoding side via a bitstream.

[0066] Flipping when packing feature maps of multiple channels into one frame will be described using FIG. 13 . In flipping, when packing the feature maps of each channel, the positions of the elements (pixels) of the feature maps of each channel are flipped horizontally (left and right), vertically (up and down), or horizontally and vertically (up and down, left and right). The feature maps of four channels, A (upper left), B (upper right), C (lower left), and D (lower right) in FIG. 13 , are grouped together, and flipping is not performed at position A in FIG. 13 . At position B, the feature map is flipped horizontally (left and right). At position C, the feature map is flipped vertically (up and down). At position D, the feature map is flipped horizontally and vertically (up and down, left and right). When the distribution of elements in the feature maps of each channel is similar, flipping based on the position in the frame where the channels are placed reduces the boundaries between the feature maps of each channel, improving coding efficiency.

[0067] The feature map quantization unit 702 has a function of converting elements of a decimal-type packing feature frame (feature map of all channels) into N-bit integers (N = an integer from approximately 8 to 16) within a predetermined range and outputting an integer-type packing feature frame. In this embodiment, the elements are converted into 10-bit integers ranging from 0 to 1023. The feature map quantization unit 702 detects the minimum and maximum values ​​of the elements of the decimal-type packing feature frame and transmits the detected minimum and maximum values ​​of the elements of the packing feature frame (feature map of all channels) to the decoding side as metadata. To convert from a decimal-type packing feature frame (feature map) to an integer-type packing feature frame (feature map), a linear conversion is performed in which the minimum value of the decimal type is equivalent to the minimum value of the integer type and the maximum value of the decimal type is equivalent to the maximum value of the integer type. For example, when the integer type range is expressed in 10 bits, the minimum value of the elements of the integer-type packing feature frame (feature map) is 0 and the maximum value is 1023 (2 - 1). Values ​​between the minimum and maximum values ​​are linearly quantized.

[0068] Next, the feature map inverse transformation unit 202 has a function of performing inverse quantization processing and unpacking processing on integer-type packing feature frames decoded in VVC, HEVC, AV1, etc., supplied from the feature map internal decoding unit 201, and performing inverse transformation into a decimal-type single-scale feature map xr to be supplied to the feature map restoration unit 203.

[0069] The feature map inverse transformation unit 202 on the decoding side will be described in detail with reference to Fig. 8. The feature map inverse transformation unit 202 performs the inverse process of the feature map transformation unit 103, and is composed of a feature map inverse quantization unit 801 and an unpacking unit 802.

[0070] The feature map inverse quantization unit 801 performs the inverse process of the feature map quantization unit 702 on the encoding side, and has a function of converting elements of integer-type packing feature frames from integer to decimal type. The feature map inverse quantization unit 801 converts the integer-type packing feature frame decoded by the feature map internal decoding unit 201 into a decimal-type packing feature frame using the minimum and maximum decimal values ​​transmitted as metadata. A linear transformation is performed to convert the integer minimum value to the decimal minimum value and the integer maximum value to the decimal maximum value. Linear inverse quantization is performed on values ​​between the minimum and maximum values.

[0071] The unpacking unit 802 extracts a feature map of each channel from the packing feature frames arranged in one frame in raster scan order, and supplies the extracted feature map as a single-scale feature map xr to the feature map restoration unit 203. Details of the unpacking unit 802 will be described later.

[0072] <Feature Map Intra-Encoding / Decoding> The feature map intra-encoding unit 104 will be described in detail with reference to FIG. 9 . The feature map intra-encoding unit 104 is composed of a switch 901, a VVC encoding unit 902, an HEVC encoding unit 903, and an AV1 encoding unit 904. The switch 901 selects the encoding standard for intra-encoding the feature map converted by the feature map conversion unit 103. The VVC encoding unit 902 encodes the feature map in accordance with the VVC standard and outputs a bitstream compliant with the VVC standard. The HEVC encoding unit 903 encodes the feature map in accordance with the HEVC standard and outputs a bitstream compliant with the HEVC standard. The AV1 encoding unit 904 encodes the feature map in accordance with the AV1 standard and outputs a bitstream compliant with the AV1 standard.

[0073] In the VVC, HEVC, and AV1 standards, an image is divided into blocks of a predetermined size and then encoded.

[0074] It is also possible to implement only one of VVC, HEVC, and AV1, and it is also possible to use an image coding method other than VVC, HEVC, and AV1.

[0075] Next, the feature map internal decoding unit 201 will be described in detail with reference to FIG. 10 . The feature map internal decoding unit 201 is composed of a switch 1001, a VVC decoding unit 1002, an HEVC decoding unit 1003, and an AV1 decoding unit 1004. The switch 1001 selects the encoding standard for internal decoding based on information for selecting internal decoding within the input bitstream. The VVC decoding unit 1002 decodes the feature map in accordance with the VVC standard. The HEVC decoding unit 1003 decodes the feature map in accordance with the HEVC standard. The AV1 encoding unit 1004 decodes the feature map in accordance with the AV1 standard.

[0076] In the VVC, HEVC, and AV1 standards, decoding is performed for each predetermined block size.

[0077] It is also possible to implement only one of VVC, HEVC, and AV1, and it is also possible to use an image coding method other than VVC, HEVC, and AV1.

[0078] <Division Unit for Intra-Encoding / Decoding of Feature Map> Generally, in image coding standards, one frame is divided into multiple blocks and processed sequentially. For example, in the VVC standard, a block called a CTU (Coding Tree Unit) is used as the basic division unit. The size of a CTU can be set from a minimum of 32×32 pixels to a maximum of 128×128 pixels, and the set value is transmitted as metadata. Furthermore, a CTU can be recursively divided into CBs (Coding Blocks), which are units of prediction. The size of a CB can be set from a minimum of 4×4 pixels to ¼ of the CTU size (when the CTU size is 128×128 pixels, the CB size is 32×32 pixels), and the set value of the minimum CB size is transmitted as metadata. <Explanation of Packing Unit 701> The processing of the packing unit 701 will be described using the flowchart of FIG. 15. In the packing unit 701, an encoding channel is set (step S101), and feature maps of multiple channels are packed into one frame (step S102).

[0079] 16 is a flowchart for explaining the details of step S101. First, the number of channels num_of_ch of the feature map to be packed is compared with the minimum number of channels min_num_ch (step S201). Here, the minimum number of channels min_num_ch represents the minimum number of channels of the feature map to be encoded. min_num_of_ch is set to be equal to or greater than the minimum frame size / feature map size specified in the feature map internal encoding / decoding.

[0080] If num_of_ch<min_num_ch, the number of channels of the feature map to be coded num_of_ch_to_be_coded=min_num_ch is set (step S202). Otherwise, that is, if num_of_ch>=min_num_ch, the number of channels of the feature map to be coded num_of_ch_to_be_coded=num_of_ch is set (step S203). The number of channels of the feature map to be coded num_of_ch_to_be_coded is a variable calculated in the feature map internal coding and decoding.

[0081] 17 is a flowchart for explaining the details of step S102. First, an index ch_idx for indicating the feature map to be packed is initialized to 0 (step S301). While ch_idx is smaller than num_of_ch (step S302), the ch_idx-th feature map is packed into the packed feature frame (step S303), and processing proceeds to the next ch_idx (step S304). In step S302, if ch_idx is equal to or greater than num_of_ch, processing proceeds to step S305.

[0082] While ch_idx is smaller than num_of_ch_to_be_coded (step S305), a dummy feature map is packed into the packed feature frame (step S306), and processing proceeds to the next ch_idx (step S307). In step S305, if ch_idx becomes equal to or greater than num_of_ch_to_be_coded, this procedure ends. Here, the dummy feature map is a feature map that has the same size and predetermined element values ​​as the feature map of the channel to be packed. <Details of Unpacking Unit 802> The processing of the unpacking unit 802 will be described using the flowchart in FIG. 18. First, an index ch_idx for indicating the feature map to be unpacked is initialized to 0 (step S401). While ch_idx is smaller than num_of_ch (step S402), the ch_idx-th feature map is unpacked from the packed feature frame (step S403), and the process proceeds to the next ch_idx (step S404). If ch_idx becomes equal to or greater than num_of_ch in step S402, this procedure ends.

[0083] If ch_idx is greater than or equal to num_of_ch and less than num_of_ch_to_be_coded, the packed feature frames represent dummy feature maps, so ignore them without unpacking them as feature maps.

[0084] 19 is an example of metadata related to packing. min_number_of_channels specifies the minimum number of channels, min_num_ch. number_of_channels specifies the number of channels, num_of_ch, of the feature map to be packed. packing_frame_width is the horizontal size of the packed feature frame, and packing_frame_height is the horizontal size of the packed feature frame. feature_width is the horizontal size of the packed feature map, and feature_height is the vertical size of the packed feature map. min_number_of_channels can be agreed upon in advance between the packing unit 701 and the unpacking unit 802. In this case, min_number_of_channels does not need to be transmitted as metadata.

[0085] 20 shows an example of packed feature frames when min_num_ch = 32 and num_of_ch = 16. In this example, CH1 to CH16 are packed feature maps, and DCH17 to DCH32 are dummy feature maps.

[0086] According to this embodiment, it is possible to guarantee that a packed feature frame is always composed of a feature map with at least the minimum number of channels min_num_ch, regardless of the number of channels in the feature map. It is possible to guarantee that a packed feature frame is composed of a feature map with at least the minimum number of channels min_num_ch, in other words, that the packed feature frame is at least a predetermined size. This avoids violation of the frame size limit in the intra-encoding and decoding of the feature map, and enables reliable transmission of packed feature frames.

[0087] Furthermore, the minimum number of channels min_num_ch and the number of channels num_of_ch of the feature map to be packed can be set independently of the size of the CTU, which is metadata related to the feature map internal encoding / decoding unit, and the minimum size of the CB. This allows the internal encoding / decoding unit to be changed without changing the minimum number of channels min_num_ch or the number of channels num_of_ch of the feature map to be packed, providing excellent scalability.

[0088] In all of the above-described embodiments, the bitstream output by the feature map encoder has a specific data format that can be decoded according to the encoding method used in the embodiment, and the feature map decoder corresponding to the feature map encoder can decode the bitstream in this specific data format.

[0089] When a wired or wireless network is used to exchange bitstreams between the feature map encoder and the feature map decoder, the bitstreams may be converted into a data format suitable for the transmission mode of the communication channel before transmission. In this case, a transmitter is provided that converts the bitstream output by the feature map encoder into coded data in a data format suitable for the transmission mode of the communication channel and transmits the coded data to the network, and a receiver is provided that receives the coded data from the network, restores the coded data to a bitstream, and supplies the bitstream to the feature map decoder. The transmitter includes a memory that buffers the bitstream output by the feature map encoder, a packet processor that packetizes the bitstream, and a transmitter that transmits the packetized coded data via the network. The receiver includes a receiver that receives the packetized coded data via the network, a memory that buffers the received coded data, and a packet processor that packetizes the coded data to generate a bitstream and provides it to the feature map decoder.

[0090] The above encoding and decoding processes may be realized not only as a transmission, storage, and receiving device using hardware, but also as firmware stored in a ROM (read-only memory), flash memory, etc., or as software for a computer, etc. The firmware program or software program may be provided by recording it on a computer-readable recording medium, or may be provided from a server via a wired or wireless network, or may be provided as data broadcasting on terrestrial or satellite digital broadcasting.

[0091] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the components and treatment processes, and that such modifications are also within the scope of the present invention.

[0092] The present invention can be used in encoding and decoding techniques for feature maps.

[0093] 100 Feature map encoding device, 101 Neural network feature extraction unit, 102 Feature map reduction unit, 103 Feature map conversion unit, 104 Feature map internal encoding unit, 200 Feature map decoding device, 201 Feature map internal decoding unit, 202 Feature map inverse conversion unit, 203 Feature map restoration unit, 204 Neural network identification unit, 301 Convolution processing unit, 302 Activation processing unit, 303 Pooling processing unit, 322 Bottom-up processing unit, 324 Top-down processing unit, 326 Image to be subjected to feature extraction processing, 501 First feature map reduction unit, 502 First channel combination unit, 503 Second feature map reduction unit, 504 Second channel combination unit, 505 Third feature map reduction unit, 506 First padding unit, 507 Second padding unit, 508 Third padding unit, 601 8x enlargement unit, 602 4x enlargement unit, 603 2x enlargement unit, 604 First feature map mixing unit, 605 Second feature map mixing unit, 606 First unpadding unit, 607 Second unpadding unit, 608 Third unpadding unit, 701 Packing unit, 702 Feature map quantization unit, 801 Feature map inverse quantization unit, 802 Unpacking unit, 901 Switch, 902 VVC encoding unit, 903 HEVC encoding unit, 904 AV1 encoding unit, 1001 Switch, 1002 VVC decoding unit, 1003 HEVC decoding unit, 1004 AV1 decoding unit.

Claims

1. A feature map encoding device comprising: a feature map reduction unit that converts a multi-scale feature map to generate a single-scale feature map; a packing unit that packs the single-scale feature map into a frame to generate a decimal-type packed feature frame; a quantization unit that converts elements of the decimal-type packed feature frame into integer values ​​to generate an integer-type packed feature frame; and a feature map inner encoding unit that encodes the integer-type packed feature frame, wherein if the number of channels in the single-scale feature map is smaller than a predetermined minimum number of channels, the packing unit adds a dummy feature map to the single-scale feature map and packs it into the frame.

2. A feature map encoding method comprising: a feature map reduction step of converting a multi-scale feature map to generate a single-scale feature map; a packing step of packing the single-scale feature map into a frame to generate a decimal-type packed feature frame; a quantization step of converting elements of the decimal-type packed feature frame into integer values ​​to generate an integer-type packed feature frame; and a feature map inner encoding step of encoding the integer-type packed feature frame, wherein the packing step adds a dummy feature map to the single-scale feature map and packs it into the frame if the number of channels of the single-scale feature map is smaller than a predetermined minimum number of channels.

3. A feature map encoding program that causes a computer to execute the following steps: a feature map reduction step of converting a multi-scale feature map to generate a single-scale feature map; a packing step of packing the single-scale feature map into a frame to generate a decimal-type packed feature frame; a quantization step of converting elements of the decimal-type packed feature frame into integer values ​​to generate an integer-type packed feature frame; and a feature map inner encoding step of encoding the integer-type packed feature frame, wherein the packing step adds a dummy feature map to the single-scale feature map and packs it into a frame if the number of channels of the single-scale feature map is smaller than a predetermined minimum number of channels.

4. A feature map decoding device comprising: a feature map internal decoding unit that decodes a single-scale feature map packed and encoded into a frame to generate an integer-type packing feature frame; an inverse quantization unit that converts elements of the integer-type packing feature frame into decimal values ​​to generate a decimal-type packing feature frame; an unpacking unit that divides the decimal-type packing feature frame into one or more packing groups based on information related to the packing groups and unpacks the frame to generate a single-scale feature map; and a feature map restoration unit that transforms the single-scale feature map to generate a multi-scale feature map, wherein the unpacking unit excludes dummy feature maps when the number of channels in the single-scale feature map is smaller than a predetermined minimum number of channels.

5. A feature map decoding device comprising: a feature map inner decoding step of decoding a single-scale feature map packed and encoded into a frame to generate an integer-type packing feature frame; an inverse quantization step of converting elements of the integer-type packing feature frame into decimal values ​​to generate a decimal-type packing feature frame; an unpacking step of dividing the decimal-type packing feature frame into one or more packing groups based on information related to the packing groups and unpacking to generate a single-scale feature map; and a feature map restoration step of converting the single-scale feature map to generate a multi-scale feature map, wherein the unpacking step excludes dummy feature maps when the number of channels of the single-scale feature map is smaller than a predetermined minimum number of channels.

6. A feature map decoding program that causes a computer to execute the following steps: a feature map inner decoding step of decoding a single-scale feature map that has been packed into a frame and encoded, and generating an integer-type packed feature frame; a dequantization step of converting elements of the integer-type packed feature frame into decimal values, and generating a decimal-type packed feature frame; an unpacking step of dividing the decimal-type packing feature frame into one or more packing groups based on information related to the packing groups, and unpacking the group to generate a single-scale feature map; and a feature map restoration step of converting the single-scale feature map to generate a multi-scale feature map, wherein the unpacking step excludes a dummy feature map when the number of channels in the single-scale feature map is smaller than a predetermined minimum number of channels.

Images