Feature map encoding device, feature map encoding method, feature map decoding device, and feature map decoding method

Abstract

In the present invention, elements of a fractional feature map of each channel are quantized from a fractional type to an integer type to generate an integer feature map of each channel. The integer feature maps of each channel are packed to generate one integer packing feature frame. The integer packing feature frame is encoded to generate a bit stream. Linear quantization is performed according to a quantization mode suitable for each channel at the time of quantization.

Description

Feature map encoding device, feature map encoding method, feature map decoding device, feature map decoding method

[0001] This paper deals with encoding and decoding of feature maps in neural networks.

[0002] As a neural network technology used for image recognition such as detecting objects at various scales in an image, dividing regions into individual objects, or tracking objects, the FPN (Feature Pyramid Network) described in Non-Patent Document 1 is known. In the FPN, multiple feature maps at various scales are generated from the image to be processed, and various image recognitions are performed using the feature maps.

[0003] The FPN used for image recognition generates multiple feature maps from an image, and its structure utilizes a CNN (Convolutional Neural Network). A CNN can be divided into a feature extraction section (backbone) that reads an image and generates a feature map through convolution and pooling, and a classification section (head) that is composed of hierarchical fully connected layers and generates output suitable for tasks such as object detection, instance segmentation, and object tracking. The FPN utilizes the CNN backbone.

[0004] The feature extraction unit of the FPN is typically configured with a hierarchical structure in which a convolution processing unit 301, an activation processing unit 302, and a pooling processing unit 303 shown in FIG. 3 are configured as one basic unit, and this basic unit is repeated.

[0005] The structure of the FPN is shown in Figure 4. The FPN consists of a bottom-up processor 322 that uses a CNN backbone to generate a multi-scale feature map consisting of multiple hierarchical layers, and a top-down processor 324 that aggregates features from deep feature maps to shallow feature maps using the inverse structure of the CNN backbone. The bottom-up processor 322 reduces the resolution of the feature map by half by repeating the convolution processor 301, activation processor 302, and pooling processor 303 (shown in Figure 3 ), which are basic units, to generate a pyramid of feature maps with multiple layers. Meanwhile, the top-down processor 324 adds feature maps with resolutions corresponding to the bottom-up processor 322, increasing the resolution of the feature map to the same resolution as the input image, thereby generating a pyramid of feature maps. In other words, the FPN generates multiple feature maps for each layer from an image 326 to be subjected to feature extraction processing.

[0006] The convolution processing unit 301 performs convolution processing on the data to be processed (image or feature map) using multiple types of predetermined filters (kernels). In the convolution processing in the convolution processing unit 301, predetermined filtering is performed on all of the data to be processed while sliding at predetermined intervals. The sliding interval is called a stride. The convolution processing unit 301 may determine the stride based on the number of data to be processed. For example, the convolution processing unit 301 may determine the stride to be 1 if the number of data to be processed is smaller than a predetermined value, and may determine the stride to be 2 if the number of data to be processed is equal to or greater than the predetermined value. Multiple predetermined filters are prepared in each layer, and multiple feature maps are generated by generating one feature map for each filter. The unit of the feature map is the channel. If the number (types) of predetermined filters is N (N types), N feature maps (N channels) are generated.

[0007] The activation processing unit 302 performs activation processing to nonlinearly convert the feature map output from the convolution processing unit 301. Here, the function used in the activation processing is called an activation function. The activation processing unit 302 uses a ReLU (Rectified Linear Unit) function, a sigmoid function, or the like as the activation function.

[0008] The pooling processing unit 303 is a process for downsampling the feature map output from the activation processing unit 302 by replacing local values ​​of the feature map with representative values.

[0009] On the other hand, when performing classification using a neural network, it has the ability to perform tasks using feature maps of multiple channels at each layer.

[0010] In image classification, the feature maps of multiple channels at each layer are convolved for each predetermined size based on the scale of the feature map, and the probability of the object class is calculated for each pixel.

[0011] Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature Pyramid Networks for Object Detection. In CVPR, 2017.

[0012] The amount of information in a feature map is enormous, making it unsuitable for transmission and storage. In view of the above-mentioned problems, an object of this embodiment is to provide a technique for encoding and decoding a feature map.

[0013] In order to solve the above problem, a feature map coding device according to one aspect of the present embodiment is a feature map coding device for coding a multi-scale feature map, and includes a quantization unit that converts elements of a multi-channel decimal feature map from decimal to integer to generate a multi-channel integer feature map, a packing unit that packs the multi-channel integer feature map to generate a single integer-packed feature frame, and a feature map intra-coding unit that encodes the integer-packed feature frame to generate a bitstream. The quantization unit sets two or more quantization modes, sets a minimum and maximum decimal value for each quantization mode, and performs quantization processing based on the minimum and maximum decimal value corresponding to one of the quantization modes.

[0014] Another aspect of this embodiment is a feature map decoding device that decodes a bitstream in which a multi-scale feature map is encoded, and the device includes a feature map inner decoding unit that decodes the bitstream and generates an integer-type packed feature frame, an unpacking unit that unpacks the integer-type packed feature frame to generate a multi-channel integer feature map, and an inverse quantization unit that converts elements of the multi-channel integer feature map from integer to decimal to generate a multi-channel decimal feature map, and the quantization unit decodes two or more quantization modes, decodes minimum and maximum decimal values ​​for each quantization mode, and performs inverse quantization processing based on the minimum and maximum decimal values ​​corresponding to one of the quantization modes.

[0015] Another aspect of this embodiment is a feature map encoding device for encoding a multi-scale feature map. This device includes a quantization unit that converts elements of a multi-channel decimal feature map from decimal to integer to generate a multi-channel integer feature map, a packing unit that packs the multi-channel integer feature map to generate one integer-packed feature frame, and a feature map inner encoding unit that encodes the integer-packed feature frame to generate a bitstream. The quantization unit sets two or more quantization modes for encoding, sets linear or nonlinear transformation parameters for each quantization mode for encoding, and performs quantization processing based on the linear or nonlinear transformation corresponding to one of the quantization modes.

[0016] Yet another aspect of this embodiment is a feature map decoding device that decodes a bitstream in which a multi-scale feature map is encoded, and the device includes a feature map inner decoding unit that decodes the bitstream and generates an integer-type packed feature frame, an unpacking unit that unpacks the integer-type packed feature frame to generate a multi-channel integer feature map, and an inverse quantization unit that converts elements of the multi-channel integer feature map from integer to decimal to generate a multi-channel decimal feature map, and the quantization unit decodes two or more quantization modes, decodes linear or nonlinear transformation parameters for each quantization mode, and performs inverse quantization processing based on the linear or nonlinear transformation corresponding to one of the quantization modes.

[0017] Another aspect of this embodiment is a feature map encoding device for encoding a multi-scale feature map. This device includes a quantization unit that converts elements of a multi-channel decimal feature map from decimal to integer to generate a multi-channel integer feature map, a packing unit that packs the multi-channel integer feature map to generate one integer-packed feature frame, and a feature map intra-encoding unit that encodes the integer-packed feature frame to generate a bitstream. The quantization unit sets linear or nonlinear transformation parameters for encoding, and performs quantization processing based on the linear or nonlinear transformation indicated by the parameters.

[0018] Yet another aspect of this embodiment is a feature map decoding device that decodes a bitstream in which a multi-scale feature map is encoded, and includes a feature map inner decoding unit that decodes the bitstream and generates an integer-type packed feature frame, an unpacking unit that unpacks the integer-type packed feature frame to generate a multi-channel integer feature map, and an inverse quantization unit that converts elements of the multi-channel integer feature map from integer to decimal to generate a multi-channel decimal feature map, and the quantization unit decodes linear or nonlinear transformation parameters and performs inverse quantization based on the linear or nonlinear transformation indicated by the parameters.

[0019] Another aspect of this embodiment is a feature map encoding device for encoding a multi-scale feature map. This device includes a quantization unit that converts elements of a multi-channel decimal feature map from decimal to integer to generate a multi-channel integer feature map, a packing unit that packs the multi-channel integer feature map to generate one integer-packed feature frame, and a feature map inner encoding unit that encodes the integer-packed feature frame to generate a bitstream. The quantization unit sets minimum and maximum decimal values ​​for each channel and performs quantization processing based on the minimum and maximum decimal values ​​for each channel.

[0020] Yet another aspect of this embodiment is a feature map decoding device that decodes a bitstream in which a multi-scale feature map is encoded, and includes a feature map inner decoding unit that decodes the bitstream and generates an integer-type packed feature frame, an unpacking unit that unpacks the integer-type packed feature frame to generate a multi-channel integer feature map, and an inverse quantization unit that converts elements of the multi-channel integer feature map from integers to decimal points to generate a multi-channel decimal feature map, and the quantization unit decodes minimum and maximum decimal points for each channel and performs inverse quantization based on the minimum and maximum decimal points for each channel.

[0021] Another aspect of this embodiment is a feature map encoding device for encoding a multi-scale feature map. This device includes a quantization unit that converts elements of a multi-channel fractional feature map from fractional to integer to generate a multi-channel integer feature map, a packing unit that packs the multi-channel integer feature map to generate one integer-packed feature frame, and a feature map inner encoding unit that encodes the integer-packed feature frame to generate a bitstream. The quantization unit sets scaling coefficients and shift coefficients for each channel and performs quantization processing based on the fractional scaling coefficients and shift coefficients for each channel.

[0022] Yet another aspect of this embodiment is a feature map decoding device that decodes a bitstream in which a multi-scale feature map is encoded, and includes a feature map inner decoding unit that decodes the bitstream and generates an integer-type packed feature frame, an unpacking unit that unpacks the integer-type packed feature frame to generate a multi-channel integer feature map, and an inverse quantization unit that converts elements of the multi-channel integer feature map from integer to decimal form to generate a multi-channel decimal feature map, and the quantization unit decodes scaling coefficients and shift coefficients for each channel and performs inverse quantization processing based on the scaling coefficients and shift coefficients for each channel.

[0023] According to this embodiment, feature maps can be efficiently encoded and decoded with a small amount of processing.

[0024] 1 is a block diagram illustrating the configuration of a feature map encoding device 100. FIG. 2 is a block diagram illustrating the configuration of a feature map decoding device 200. FIG. 3 is a block diagram illustrating the processing that is a basic unit in each layer of an FPN. FIG. 4 is a diagram illustrating the structure of an FPN. FIG. 5 is a block diagram illustrating the detailed configuration of a feature map reduction unit 102. FIG. 6 is a block diagram illustrating the detailed configuration of a feature map restoration unit 203. FIG. 7 is a block diagram illustrating the detailed configuration of a feature map conversion unit 103. FIG. 8 is a block diagram illustrating the detailed configuration of a feature map inverse conversion unit 202. FIG. 9 is a block diagram illustrating the detailed configuration of a feature map inner encoding unit 104. FIG. 10 is a block diagram illustrating the detailed configuration of a feature map inner decoding unit 201. FIG. 11 is a diagram illustrating the number of channels of feature maps x1, x2, and x3, the width and height of the feature maps. FIG. 12 is a diagram illustrating the state in which feature maps of multiple channels are packed into a packing feature frame. FIG. 13 is a diagram illustrating flipping when packing feature maps of multiple channels into a packing feature frame. FIG. 14 is a block diagram illustrating the detailed configuration of a feature map conversion unit 103 according to a second embodiment. FIG. 1 is a block diagram illustrating a detailed configuration of a feature map inverse transformation unit 202 according to a second embodiment. FIG. 2 is a flowchart illustrating a linear quantization processing procedure of a feature map quantization unit 721 of the feature map transformation unit 103 according to the second embodiment. FIG. 3 is a flowchart illustrating a linear quantization processing procedure of the second embodiment. FIG. 4 is a flowchart illustrating a linear inverse quantization processing procedure of a feature map inverse quantization unit 822 of the feature map inverse transformation unit 202 according to the second embodiment. FIG. 5 is a flowchart illustrating a linear inverse quantization processing procedure of the second embodiment. FIG. 6 is a diagram illustrating an example of a case in which four quantization modes of 0, 1, 2, and 3 according to the second embodiment are set, and decimal-type minimum and maximum values ​​of a feature map are set for each of the four quantization modes. FIG. 7 is a diagram illustrating an example of linear transformation used in linear quantization according to the second embodiment. FIG. 8 is a diagram illustrating an example of syntax rules for decimal-type minimum and maximum values ​​corresponding to the quantization modes according to the second embodiment. FIG. 9 is a diagram illustrating an example of syntax rules for the quantization modes of each channel according to the second embodiment.10 is a flowchart illustrating a nonlinear quantization process performed by the feature map quantization unit 721 of the feature map transformation unit 103 according to the third embodiment. FIG. 11 is a flowchart illustrating a linear inverse quantization process performed by the feature map inverse quantization unit 822 of the feature map inverse transformation unit 202 according to the third embodiment. FIG. 12 is a diagram illustrating an example of a case where four quantization modes, 0, 1, 2, and 3, are set and parameters indicating linear / nonlinear transformation are set for each of the four quantization modes according to the third embodiment. FIG. 13 is a diagram illustrating an example of nonlinear transformation used in nonlinear quantization according to the third embodiment. FIG. 14 is a diagram illustrating an example of nonlinear transformation used in nonlinear quantization according to the third embodiment. FIG. 15 is a diagram illustrating an example of nonlinear transformation used in nonlinear quantization according to the third embodiment. FIG. 16 is a diagram illustrating an example of nonlinear transformation used in nonlinear quantization according to the third embodiment. FIG. 17 is a diagram illustrating an example of nonlinear transformation used in nonlinear quantization according to the third embodiment. FIG. 18 is a diagram illustrating an example of syntax rules for parameters indicating linear / nonlinear transformation corresponding to the quantization modes according to the third embodiment. FIG. 19 is a flowchart illustrating a nonlinear quantization process performed by the feature map quantization unit 702 of the feature map transformation unit 103 according to the fourth embodiment. FIG. 19 is a flowchart illustrating a linear inverse quantization process performed by the feature map inverse quantization unit 801 of the feature map inverse transformation unit 202 according to the fourth embodiment. 10 is a diagram illustrating an example of syntax rules for nonlinear transformation parameters according to the fourth embodiment. 20 is a diagram illustrating layers and units handled by the feature map encoding device and feature map decoding device according to the present embodiment. 21 is a diagram illustrating an example of syntax rules for decimal minimum and maximum values ​​according to the first embodiment. 22 is a diagram illustrating an example of syntax rules for decimal minimum and maximum values ​​according to the first embodiment. 23 is a flowchart illustrating a linear quantization process procedure performed by the feature map quantization unit 721 of the feature map conversion unit 103 according to the fifth embodiment. 24 is a flowchart illustrating a linear inverse quantization process procedure performed by the feature map inverse quantization unit 822 of the feature map inverse conversion unit 202 according to the fifth embodiment. 25 is a diagram illustrating an example of syntax rules for decimal minimum and maximum values ​​according to the fifth embodiment. 26 is a diagram illustrating an example of syntax rules for decimal minimum and maximum values ​​according to the fifth embodiment. 27 is a flowchart illustrating a linear quantization process procedure performed by the feature map quantization unit 721 of the feature map conversion unit 103 according to the sixth embodiment.16 is a flowchart for explaining a linear quantization processing procedure according to a sixth embodiment. FIG. 17 is a flowchart for explaining a linear inverse quantization processing procedure of a feature map inverse quantization unit 822 of a feature map inverse transformation unit 202 according to a sixth embodiment. FIG. 18 is a flowchart for explaining a linear inverse quantization processing procedure according to a sixth embodiment. FIG. 19 is a diagram illustrating an example of syntax rules for decimal-type minimum and maximum values ​​according to a sixth embodiment. FIG. 20 is a diagram illustrating an example of syntax rules for decimal-type minimum and maximum values ​​according to a sixth embodiment.

[0025] The technologies and technical terms used in this embodiment will be defined below.

[0026] <Features and feature maps> In a convolutional neural network (CNN), a filter is used to scan the image (input layer data) from which features are extracted while changing its position, and the results of the convolution operation between the filter coefficients are used as features or feature maps.

[0027] <Packing> Frame packing refers to combining two or more frames (pictures) into a single frame (picture) by arranging them in a tiled pattern. In this application, packing refers to combining feature maps of multiple channels into a single frame. Figure 12 shows an example of frame packing.

[0028] <Data Type> A data type that represents an integer value is called an integer type, and a data type that represents a decimal value is called a decimal type.

[0029] <Layers and Units> The layers and units handled by the feature map encoding device and feature map decoding device of this embodiment will be described using Figure 32. Reference numeral 181 in Figure 32 denotes a sequence of packed feature frames (feature frames packed by combining feature maps from multiple channels into one frame) or feature maps of all channels that continue over time, and is referred to as a sequence layer or sequence unit. Reference numeral 182 in Figure 32 denotes a sequence of feature maps of one channel that continue over time, and is referred to as a sequence layer of each channel or a sequence unit of each channel. Reference numeral 183 in Figure 32 denotes a packed feature frame or feature maps of all channels at the same time, and is referred to as a frame layer or frame unit. Reference numeral 184 in Figure 32 denotes a feature map of one channel at a certain time, and is referred to as a feature map layer of each channel or a feature map unit of each channel.

[0030] First Embodiment A feature map encoding device 100 and a feature map decoding device 200 according to a first embodiment of the present invention will be described.

[0031] 1 is a block diagram of a feature map coding device 100 according to a first embodiment. The feature map coding device 100 according to the embodiment includes a feature map reduction unit 102, a feature map conversion unit 103, and a feature map internal coding unit 104. The feature map coding device 100 is a device that encodes a feature map generated by a neural network feature extraction unit 101 to generate and output a bitstream.

[0032] The neural network feature extraction unit 101 reads an image from which features are to be extracted, generates a feature map by FPN convolution processing, activation processing, and pooling processing, and supplies the generated feature map to the feature map reduction unit 102. In this embodiment, a three-layer multi-scale feature map of x1, x2, and x3 is generated.

[0033] The feature map reduction unit 102 converts the three-layer multi-scale feature maps x1, x2, and x3 acquired from the neural network feature extraction unit 101 into a single-layer single-scale feature map xf and supplies it to the feature map conversion unit 103. The feature map reduction unit 102 will be described in detail with reference to FIG. 5 .

[0034] The feature map conversion unit 103 performs packing and quantization on the decimal-type single-scale feature map xf supplied from the feature map reduction unit 102, converting it into an integer-type packed feature frame, and supplies this to the feature map internal coding unit 104 together with the feature map coding information used in the process.

[0035] The feature map conversion unit 103 will be described in detail with reference to FIG.

[0036] The feature map internal coding unit 104 encodes the integer-type packing feature frames supplied from the feature map conversion unit 103 in accordance with an image coding standard such as VVC, HEVC, or AV1 to generate a bitstream in which the packing feature frames are encoded, and multiplexes the feature map encoding information supplied from the feature map conversion unit 103 with the encoded bitstream to output a multiplexed bitstream. The output bitstream is supplied to the feature map decoding device 200 or the like via a network or the like.

[0037] The feature map internal coding unit 104 will be described in detail with reference to FIG.

[0038] 2 is a block diagram showing the configuration of a feature map decoding device 200 according to an embodiment of the present invention, which corresponds to the feature map encoding device 100 in FIG. 1. The feature map decoding device 200 according to this embodiment includes a feature map internal decoding unit 201, a feature map inverse conversion unit 202, and a feature map restoration unit 203. The feature map decoding device 200 receives as input a bit stream encoded by the feature map encoding device 100, decodes the bit stream, and generates three-layer multi-scale feature maps x1up, x2up, and x3up, which it supplies to a neural network identification unit 204.

[0039] The feature map internal decoding unit 201 separates the multiplexed bit stream output by the feature map internal coding unit 104 of the feature map coding device 100 into a bit stream in which feature map coding information is coded and a bit stream in which packing feature frames are coded, decodes the bit stream in which packing feature frames are coded in accordance with an image coding standard such as VVC, HEVC, or AV1, generates integer-type packing feature frames, and supplies these to the feature map inverse conversion unit 202.

[0040] The feature map internal decoding unit 201 will be described in detail with reference to FIG.

[0041] The feature map inverse conversion unit 202 performs inverse quantization and unpacking on the integer-type packed feature frame supplied from the feature map internal decoding unit 201, converting it into a decimal-type single-scale feature map xr, and supplies it to the feature map restoration unit 203.

[0042] The feature map inverse conversion unit 202 will be described in detail with reference to Fig. 8. The feature map restoration unit 203 converts the single-scale feature map xr supplied from the feature map inverse conversion unit 202 into three-layer multi-scale feature maps x1up, x2up, and x3up, and supplies them to the neural network identification unit 204 as outputs of the feature map decoding device 200.

[0043] The feature map restoration unit 203 will be described in detail with reference to FIG.

[0044] The neural network identification unit 204 performs identification processing such as identifying objects, places, landscapes, people, and living things in the image to be identified, based on the three-layer multi-scale feature maps x1up, x2up, and x3up supplied by the feature map restoration unit 203.

[0045] <Feature Map Reduction and Feature Map Restoration> The feature map reduction unit 102 has a function of converting a multi-layered multi-scale feature map acquired from the neural network feature extraction unit 101 into a single-layered single-scale feature map.

[0046] The feature map reduction unit 102 will be described in detail using Fig. 5. The feature map reduction unit 102 is composed of a first feature map reduction unit 501, a first channel combination unit 502, a second feature map reduction unit 503, a second channel combination unit 504, a third feature map reduction unit 505, a first padding unit 506, a second padding unit 507, and a third padding unit 508. The feature map reduction unit 102 in Fig. 5 is an example of a configuration that converts a three-layer multi-scale feature map into a single-scale feature map.

[0047] The feature map reduction unit 102 receives three-layer multi-scale feature maps, namely, a first feature map x1, a second feature map x2, and a third feature map x3, as input, converts them into a single-layer single-scale feature map xf, and supplies the resulting map to the feature map conversion unit 103. Here, n denotes an index indicating a layer, Cn denotes the number of channels in the nth layer, Wn denotes the width of the feature map, and Hn denotes the height of the feature map. In this embodiment, the values ​​of Cn, Wn, and Hn for each layer are as shown in FIG. 11 , where H and W denote the width and height, respectively, of the image from which feature extraction is performed.

[0048] The first padding unit 506 has a function of padding the first feature map x1 to generate a first padded feature map x1pad. The first padding unit 506 determines the padding size so that the width and height of x1pad are multiples of 64. The number of channels of x1pad is 256, the same as that of x1.

[0049] The second padding unit 507 performs padding on the second feature map x2 by folding to generate a second padded feature map x2pad. The second padding unit 507 determines the padding size so that the width and height of x2pad are multiples of 32. The number of channels of x2pad is 256, the same as that of x2.

[0050] The third padding unit 508 performs padding on the third feature map x3 by folding to generate a third padded feature map x3pad. The third padding unit 508 determines the padding size so that the width and height of x3pad are multiples of 16. The number of channels of x3pad is 256, the same as that of x3.

[0051] In the first padding unit 506, the second padding unit 507, and the third padding unit 508, the padding size on the left side is the same as the padding size on the right side, and the padding size on the top side is the same as the padding size on the bottom side. That is, the feature maps x1, x2, and x3 are arranged at the centers of x1pad, x2pad, and x3pad, respectively.

[0052] The first feature map reduction unit 501 performs convolution in the spatial and channel directions on the first padded feature map x1pad acquired from the first padding unit 506 to generate a first intermediate feature map y1. The number of channels of y1 is 192, its width is Wx1pad / 2, and its height is Hx1pad / 2, where Wx1pad and Hx1pad are the width and height of the first padded feature map x1pad, respectively.

[0053] The first channel combining unit 502 has a function of combining the first intermediate feature map y1 obtained from the first feature map reduction unit 501 and the second padded feature map x2pad obtained from the second padding unit 507 in the channel direction to generate an intermediate feature map y1Cx2pad. Since the number of channels of y1 is 192 and the number of channels of x2pad is 256, the number of channels of the intermediate feature map y1Cx2pad is 448 (192 + 256).

[0054] The second feature map reduction unit 503 performs convolution in the spatial and channel directions on the intermediate feature map y1Cx2pad obtained from the first channel combination unit 502 to generate a second intermediate feature map y2. The number of channels of y2 is 192, its width is Wy1Cx2pad / 2, and its height is Hy1Cx2pad / 2, where Wy1Cx2pad and Hy1Cx2pad are the width and height of the intermediate feature map y1Cx2pad, respectively.

[0055] The second channel combining unit 504 has a function of combining the second intermediate feature map y2 obtained from the second feature map reduction unit 503 and the third padded feature map x3pad obtained from the third padding unit 508 in the channel direction to generate an intermediate feature map y2Cx3pad. Since the number of channels in the intermediate feature map y2 is 192 and the number of channels in x3pad is 256, the number of channels in y1Cx2pad is 448 (192 + 256).

[0056] The third feature map reduction unit 505 performs convolution in the spatial and channel directions on the intermediate feature map y2Cx3pad obtained from the second channel combination unit 504 to generate a third intermediate feature map y3. The number of channels of y3 is 192, its width is Wy2Cx3pad / 2, and its height is Hy2Cx3pad / 2, where Wy2Cx3pad and Hy2Cx3pad are the width and height of the intermediate feature map y2Cx3pad, respectively.

[0057] The feature map reduction unit 102 outputs the third intermediate feature map y3 as a single-scale feature map xf and supplies it to the feature map conversion unit 103.

[0058] The feature map restoration unit 203 has a function of converting the single-scale feature map xr acquired from the feature map inverse conversion unit 202 into three-layer multi-scale feature maps x1up, x2up, and x3up.

[0059] The feature map restoration unit 203 will be described in detail with reference to Fig. 6. The feature map restoration unit 203 is configured with an 8x enlargement unit 601, a 4x enlargement unit 602, a 2x enlargement unit 603, a first feature map blending unit 604, a second feature map blending unit 605, a first depadding unit 606, a second depadding unit 607, and a third depadding unit 608.

[0060] The 8x enlargement unit 601 performs transpose convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr obtained from the feature map inverse transformation unit 202 to enlarge the feature map and reduce the channels, thereby generating an intermediate feature map z1. The number of channels in z1 is 196. If the width and height of the single-scale feature map xr are xrwidth and xrheight, respectively, the width and height of z1 are xrwidth × 8 and xrheight × 8, respectively. Here, rwidth × 8 and xrheight × 8 are the same as the width and height, respectively, of the first padded feature map x1pad, which is the output of the first padding unit 506 of the feature map reduction unit 102.

[0061] The 4x enlargement unit 602 performs transpose convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr obtained from the feature map inverse transformation unit 202 to enlarge the features and reduce the channels, thereby generating an intermediate feature map z2. The number of channels in z2 is 196. The width and height of z2 are xrwidth×4 and xrheight×4, respectively. Here, rwidth×4 and xrheight×4 are the same as the width and height, respectively, of the second padded feature map x2pad, which is the output of the second padding unit 507 of the feature map reduction unit 102.

[0062] The 2x enlargement unit 603 performs transpose convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr obtained from the feature map inverse transformation unit 202, thereby enlarging the feature map and reducing the channels, and generating an intermediate feature map z3. The number of channels in z3 is 196. The width and height of z3 are xrwidth×2 and xrheight×2, respectively. Here, rwidth×2 and xrheight×2 are the same as the width and height, respectively, of the third padded feature map x3pad, which is the output of the third padding unit 508 of the feature map reduction unit 102.

[0063] The first feature map mixing unit 604 has a function of generating an intermediate feature map z2up by improving the quality of the intermediate feature map z2 obtained from the 4x enlargement unit 602, using the intermediate feature map z1 obtained from the 8x enlargement unit 601.

[0064] The second feature map mixing unit 605 has a function of generating an intermediate feature map z3up by using the intermediate feature map z2up obtained from the first feature map mixing unit 604, with the quality of the intermediate feature map z3 obtained from the 2x enlargement unit 603 being improved.

[0065] The first padding remover 606 removes padding from the intermediate feature map z1 acquired from the 8x enlarger 601 to generate a first output feature map x1up. The width and height of x1up are the same as the width and height of the first feature map x1 input to the feature map reducer 102.

[0066] The second padding remover 607 removes padding from the intermediate feature map z2up obtained from the first feature map mixer 604 to generate a second output feature map x2up. The width and height of x2up are the same as the width and height of the second feature map x2 input to the feature map reducer 102.

[0067] The third padding remover 608 removes padding from the intermediate feature map z3up obtained from the second feature map mixer 605 to generate a third output feature map x3up. The width and height of x3up are the same as the width and height of the third feature map x3 input to the feature map reducer 102.

[0068] In the first padding remover 606, the second padding remover 607, and the third padding remover 608, the left-side padding size is the same as the right-side padding size, and the upper-side padding size is the same as the lower-side padding size, as in the first padding remover 506, the second padding remover 507, and the third padding remover 508 of the feature map reduction unit 102. In other words, the output feature maps x1up, x2up, and x3up are positioned at the center of the intermediate feature maps z1, z2up, and z3up, respectively, and padding is removed from the top, bottom, left, and right.

[0069] <Feature Map Conversion and Feature Map Inverse Conversion> The feature map conversion unit 103 has a function of performing packing processing and quantization processing on the multi-channel decimal-type single-scale feature map xf supplied from the feature map reduction unit 102, and converting it into an integer-type packed feature frame to be supplied to the feature map internal coding unit 104.

[0070] The feature map conversion unit 103 on the encoding side will be described in detail with reference to Fig. 7. The feature map conversion unit 103 is made up of a packing unit 701, a feature map quantization unit 702, and a feature map encoding information encoding unit 703.

[0071] The packing unit 701 has a function of generating a packed packed feature frame by combining input feature maps of multiple channels into one frame. Fig. 12 is a diagram illustrating the state in which feature maps of multiple channels are packed into one frame. The feature maps of each channel are sequentially arranged in one frame in raster scan order, from left to right and top to bottom.

[0072] Furthermore, the packing unit 701 performs flipping based on the position in the frame where the channel feature map is arranged. Whether or not to perform flipping may be selected, and the result may be transmitted from the encoding side to the decoding side via a bitstream. Also, the flipping process may be omitted.

[0073] Flipping when packing feature maps of multiple channels into a packing feature frame will be described using FIG. 13 . In flipping, when packing the feature maps of each channel, the positions of the elements (pixels) of the feature maps of each channel are flipped horizontally (left and right), vertically (up and down), or horizontally and vertically (up and down, left and right). The feature maps of four channels, A (upper left), B (upper right), C (lower left), and D (lower right) in FIG. 13 , are grouped together, and no flipping is performed at position A in FIG. 13 . At position B, the feature map is flipped horizontally (left and right). At position C, the feature map is flipped vertically (up and down). At position D, the feature map is flipped horizontally and vertically (up and down, left and right). When the distribution of elements in the feature maps of each channel is similar, flipping based on the position in the frame where the channels are placed reduces the boundaries between the feature maps of each channel, improving coding efficiency.

[0074] Information such as the number of channels used in packing, the number of horizontal channels of the packing feature frame, the number of vertical channels, etc. is supplied to the feature map coding information encoder 703 as elements of feature map coding information.

[0075] The feature map quantization unit 702 has a function of converting the elements of the decimal type packing feature frame (feature map of all channels) into N-bit integer type elements within a predetermined range (N = an integer from about 8 to 16) and outputting an integer type packing feature frame. In this embodiment, the elements are converted into 10-bit integer type elements ranging from 0 to 1023. The feature map quantization unit 702 detects the actual minimum value Xmin' and maximum value Xmax' (actual values) of the elements Xf(p) of the decimal type packing feature frame, and sets the decimal type minimum value Xmin and maximum value Xmax (setting values ​​for quantization) for quantization of the packing feature frame based on the detected minimum value Xmin' and maximum value Xmax' of the elements of the packing feature frame.

[0076] Note that by setting Xmin = Xmin' Xmax = Xmax', the minimum value Xmin and maximum value Xmax can be set so that all elements Xf(p) of the packing feature frame always fall within the range from the minimum value Xmin to the maximum value Xmax. However, the minimum value Xmin and maximum value Xmax will fluctuate for each frame in accordance with the minimum value Xmin' and maximum value Xmax' (actual values), which fluctuate for each frame (as time passes), and the correspondence between decimal values ​​and integer values ​​after quantization will change, resulting in temporal fluctuations, larger prediction residual values ​​during inter prediction in feature map internal encoding / decoding, which will be described later, and reduced encoding efficiency.

[0077] In practice, by setting Xmin≦Xmin′ Xmax≧Xmax′, it is possible to set the minimum value Xmin and the maximum value Xmax so that all elements Xf(p) of the packing feature frame always fall within the range from Xmin to Xmax.

[0078] By suppressing fluctuations in the minimum value Xmin and the maximum value Xmax, such as by setting a minimum value Xmin and a maximum value Xmax with a certain margin, by detecting the minimum value Xmin' and the maximum value Xmax' of a plurality of packing feature frames in sequence units or for a certain time period and setting the minimum value Xmin and the maximum value Xmax that include these, or by updating the minimum value Xmin and the maximum value Xmax so that they include the detected minimum value Xmin' and maximum value Xmax' only when the minimum value Xmin' or the maximum value Xmax' of the packing feature frames detected at a certain time period is not included in the range from the minimum value Xmin to the maximum value Xmax that was set at the previous time period, it is possible to reduce prediction residuals during inter prediction and improve coding efficiency.

[0079] The minimum value Xmin and maximum value Xmax of the elements of the set packing feature frame (feature maps of all channels) are supplied to the feature map coding information coding unit 703 as elements of feature map coding information.

[0080] By setting the minimum value Xmin and maximum value Xmax in quantization so as to increase the temporal correlation and reduce the prediction residual in the subsequent encoding process, and transmitting these values ​​from the encoding side to the decoding side, the prediction residual caused by temporal fluctuations in the minimum value Xmin and maximum value Xmax is reduced, thereby improving encoding efficiency.

[0081] To convert a decimal-type packing feature frame (feature map) into an integer-type packing feature frame (feature map), a linear transformation is performed in which the minimum value of the decimal type is equivalent to the minimum value of the integer type, and the maximum value of the decimal type is equivalent to the maximum value of the integer type. For example, when the integer type range is expressed in 10 bits, the minimum value of the elements of the integer-type packing feature frame (feature map) is 0 and the maximum value is 1023 (2 １０ -1). Values ​​between the minimum and maximum values ​​are linearly quantized. Linear quantization is performed using the following formula: Xa(p) = (Xf(p) - Xmin) / (Xmax - Xmin) where Xa(p) is a decimal variable. Furthermore, if (2 Ｎ -1), and perform fractional processing such as truncating or rounding off to obtain integer feature map Xi(p). When converting to a 10-bit integer, it is 1023 (2１０ -1) Xi(p) = Xa(p) × 1023

[0082] The feature map coded information coding unit 703 codes the feature map coded information of each channel supplied from the packing unit 701 and the feature map quantization unit 702 and transmits it to the decoding side via the feature map internal coding unit 104 .

[0083] Fig. 33A shows an example of syntax rules for the decimal minimum value Xmin and maximum value Xmax, which are elements of the feature map coding information of a packing feature frame. The syntax element quantization_min_value indicating the decimal minimum value Xmin of a packing feature frame and the syntax element quantization_max_value indicating the decimal maximum value Xmax of a packing feature frame are coded / decoded in the same frame time unit (183 in Fig. 32) (coded on the coding side and decoded on the decoding side). The syntax elements quantization_min_value and quantization_max_value are 16- to 32-bit decimal types.

[0084] Note that the coding unit for these syntax elements quantization_min_value and quantization_max_value is the same frame time unit (183 in Figure 32), but if the setting unit for the quantization parameter is a sequence unit consisting of multiple consecutive frames (181 in Figure 32), it can also be set and coded / decoded in sequence units (181 in Figure 32).

[0085] Furthermore, when sequentially encoding the syntax element quantization_min_value indicating the minimum value Xmin of the decimal type of a packing feature frame and the syntax element quantization_max_value indicating the maximum value Xmax of the decimal type of a packing feature frame in the same frame time unit, the syntax elements quantization_min_value and quantization_max_value can be encoded / decoded only when updating the minimum value Xmin and maximum value Xmax of the decimal type (using the minimum value Xmin and maximum value Xmax of the decimal type different from those of the previous frame time unit). Figure 33B shows another example of the syntax rules for the minimum value Xmin and maximum value Xmax of the decimal type. The syntax element update_quantization_param_flag is a 1-bit flag indicating whether to update the minimum value Xmin and maximum value Xmax of the decimal type of a packing feature frame. When the decimal minimum value Xmin and maximum value Xmax are updated, the syntax element update_quantization_param_flag is set to 1 for encoding / decoding, and the syntax elements quantization_min_value and quantization_max_value are also encoded / decoded. When the decimal minimum value Xmin and maximum value Xmax are not updated (the same decimal minimum value Xmin and maximum value Xmax as in the previous frame time unit are used), the syntax element update_quantization_param_flag is set to 0 for encoding / decoding; by omitting the encoding / decoding of the syntax elements quantization_min_value and quantization_max_value, the amount of code can be reduced and encoding efficiency improved when the decimal minimum value Xmin and maximum value Xmax are not updated.

[0086] Next, the feature map inverse transformation unit 202 has a function of performing inverse quantization processing and unpacking processing on integer-type packing feature frames decoded in VVC, HEVC, AV1, etc., supplied from the feature map internal decoding unit 201, and performing inverse transformation into a decimal-type single-scale feature map xr to be supplied to the feature map restoration unit 203.

[0087] The feature map coded information decoding unit 801 decodes the bit stream in which the feature map coded information supplied via the feature map internal decoding unit 201 is coded, and supplies the feature map coded information for each channel obtained as a result of the decoding to a feature map inverse quantization unit 802 and an unpacking unit 803.

[0088] The feature map inverse transformation unit 202 on the decoding side will be described in detail with reference to Fig. 8. The feature map inverse transformation unit 202 performs the inverse process of the feature map transformation unit 103, and is composed of a feature map coded information decoding unit 801, a feature map inverse quantization unit 802, and an unpacking unit 803.

[0089] The feature map inverse quantization unit 802 performs the inverse process of the feature map quantization unit 702 on the encoding side, and has the function of converting elements of integer-type packing feature frames from integer type to decimal type. The feature map inverse quantization unit 802 converts the integer-type packing feature frame decoded by the feature map internal decoding unit 201 into a decimal-type packing feature frame using the minimum and maximum decimal values ​​transmitted as metadata. A linear transformation is performed to convert the integer minimum value into the decimal minimum value and the integer maximum value into the decimal maximum value. Linear inverse quantization is performed on values ​​between the minimum and maximum values. The linear inverse transformation is performed using the following formula: Xf(p) = Xi(p) / 1023 × (Xmax - Xmin) + Xmin

[0090] The unpacking unit 803 extracts a feature map of each channel from the packing feature frames arranged in one frame in raster scan order, and supplies it to the feature map restoration unit 203 as a single-scale feature map xr.

[0091] <Feature Map Intra Coding / Decoding> The feature map intra coding unit 104 will be described in detail using FIG. 9 . The feature map intra coding unit 104 includes a switch 901, a VVC encoding unit 902, an HEVC encoding unit 903, an AV1 encoding unit 904, and a multiplexing unit 905. The switch 901 selects the coding standard for intra-coding the packing feature frame converted by the feature map conversion unit 103. The VVC encoding unit 902 encodes the feature map according to the VVC standard and outputs a bitstream compliant with the VVC standard. The HEVC encoding unit 903 encodes the packing feature frame according to the HEVC standard and outputs a bitstream compliant with the HEVC standard. The AV1 encoding unit 904 encodes the packing feature frame according to the AV1 standard and outputs a bitstream compliant with the AV1 standard.

[0092] In the VVC, HEVC, and AV1 standards, an image is divided into blocks of a predetermined size and then encoded.

[0093] It is also possible to implement only one of VVC, HEVC, and AV1, and it is also possible to use an image coding method other than VVC, HEVC, and AV1.

[0094] Furthermore, the multiplexing unit 905 multiplexes the bit stream in which the feature map coding information is coded and the bit stream in which the packing feature frame is coded, and outputs the multiplexed bit stream.

[0095] Next, the feature map internal decoding unit 201 will be described in detail with reference to Fig. 10. The feature map internal decoding unit 201 is made up of a demultiplexing unit 1005, a switch 1001, a VVC decoding unit 1002, an HEVC decoding unit 1003, and an AV1 decoding unit 1004. The switch 1001 selects the coding standard to be internally decoded based on information for selecting internal decoding within the input bitstream.

[0096] The demultiplexing unit 1005 demultiplexes the multiplexed bit stream supplied from the encoding side, and separates it into a bit stream in which feature map encoding information is encoded and a bit stream compliant with the VVC / HEVC / AV1 standard in which packing feature frames are encoded.

[0097] The VVC decoding unit 1002 decodes a bitstream in which packing feature frames are coded in accordance with the VVC standard, and generates packing feature frames.

[0098] The HEVC decoding unit 1003 decodes a bitstream in which packing feature frames are coded in accordance with the HEVC standard, and generates packing feature frames.

[0099] The AV1 decoding unit 1004 decodes a bitstream in which packing feature frames are coded in accordance with the AV1 standard, and generates packing feature frames.

[0100] In the VVC, HEVC, and AV1 standards, decoding is performed for each predetermined block size.

[0101] It is also possible to implement only one of VVC, HEVC, and AV1, and it is also possible to use an image coding method other than VVC, HEVC, and AV1.

[0102] Second Embodiment Next, a feature map encoding device 100 and a feature map decoding device 200 according to a second embodiment of the present invention will be described.

[0103] FIG. 14 is a block diagram illustrating the details of the feature map conversion unit 103 on the encoding side in the second embodiment, and is made up of a feature map quantization unit 721 and a packing unit 722.

[0104] 15 is a block diagram illustrating the details of the feature map inverse transformation unit 202 on the decoding side in the second embodiment, and is composed of an unpacking unit 821 and a feature map inverse quantization unit 822. The feature map inverse transformation unit 202 on the decoding side performs the inverse process of the feature map transformation unit 103 on the encoding side.

[0105] In the feature map encoding device 100 according to the first embodiment, the feature map quantization unit 702 of the feature map conversion unit 103 sets a minimum value Xmin and a maximum value Xmax of the decimal type of the common element for each packing feature frame indicated by 183 in Fig. 32 , that is, for each feature map of all channels, and performs linear quantization from the decimal type to an N-bit integer type (10 bits in this embodiment), thereby converting the decimal type feature map xf into an integer type feature map xi.

[0106] The feature map encoding device 100 of the second embodiment differs from the first embodiment in that the feature map quantization unit 721 of the feature map conversion unit 103 sets multiple quantization modes for each feature map of all channels, as indicated by 183 in Fig. 32 , sets a decimal minimum value Xqmin and a maximum value Xqmax for each quantization mode, and assigns a quantization mode corresponding to the minimum and maximum values ​​for each feature map of each channel, as indicated by 184 in Fig. 32 , thereby performing linear quantization that is more suitable for each channel. Note that in the feature map conversion unit 103 of the second embodiment, the feature map quantization unit 721 of Fig. 14 converts the decimal feature map of each channel from decimal to integer, and the packing unit 722 packs the integer feature map of each channel to generate an integer-type packed feature frame. Note that packing may be performed while linear quantization is being performed. Alternatively, the packing unit 722 may pack the feature maps of each channel of the decimal type to obtain a decimal type packed feature frame, and then the feature map quantization unit 721 may linearly quantize the decimal type packed feature frame from the decimal type to the integer type.

[0107] Furthermore, in the feature map decoding device 200 according to the first embodiment, the feature map inverse transform unit 202 decodes (sets) the minimum value Xmin and maximum value Xmax of the decimal type of the common element in units of packing feature frames, i.e., in units of feature maps of all channels, indicated by 183 in Fig. 32 , and performs linear inverse quantization from integer type to decimal type to convert the N-bit (10-bit in this embodiment) integer type feature map into a decimal type feature map.

[0108] The feature map decoding device 200 of the second embodiment differs from the first embodiment in that the feature map inverse transform unit 202 sets multiple quantization modes for each feature map of all channels as indicated by 183 in Fig. 32 , sets a decimal minimum value Xqmin and a maximum value Xqmax for each quantization mode, and performs linear inverse quantization that is more appropriate for each channel based on the decimal minimum value Xqmin and maximum value Xqmax corresponding to the quantization mode set for each feature map of each channel as indicated by 184 in Fig. 32 . Note that in the feature map decoding device 200 of the second embodiment, the unpacking unit 821 of Fig. 15 extracts a feature map for each channel from an integer packing feature frame, and the feature map inverse quantization unit 822 linearly inverse quantizes the feature map for each channel from integer to decimal. Note that linear inverse quantization may be performed during unpacking (extraction of the feature map for each channel). Alternatively, the feature map inverse quantization unit 822 may perform linear inverse quantization to convert the integer-type packing feature frame from the integer type to the decimal type, and the unpacking unit 821 may extract a feature map of each channel from the decimal-type packing feature frame.

[0109] The linear quantization process of the feature map quantization unit 721 of the feature map conversion unit 103 of the feature map encoding device 100 according to the second embodiment will be described with reference to the flowchart of FIG.

[0110] First, the minimum and maximum values ​​Xmin' and Xmax' of decimal type for the feature maps of all channels at the same time, as indicated by 183 in Fig. 32, and the minimum and maximum values ​​Xnmin' and Xnmax' of decimal type for the feature maps of each channel, as indicated by 184 in Fig. 32, are detected (step S1001 in Fig. 16). Here, the processing unit for detecting and setting the minimum and maximum values ​​is the same frame time unit (183 and 184 in Fig. 32), but it may also be a sequence unit consisting of multiple consecutive frames (181 and 182 in Fig. 32).

[0111] Next, based on the decimal minimum value Xnmin' and maximum value Xnmax' of the feature map Xf of each channel detected in step S1001 in Fig. 16 , a quantization mode and a decimal minimum value Xqmin and a decimal maximum value Xqmax corresponding to the quantization mode are set (step S1002 in Fig. 16 ). In the second embodiment, four quantization modes, 0, 1, 2, and 3, are set, and a decimal minimum value Xqmin and a decimal maximum value Xqmax (q = 0, 1, 2, or 3) of the feature map are set for each quantization mode. Fig. 20 shows an example in which four quantization modes, 0, 1, 2, and 3, are set, and a decimal minimum value and a decimal maximum value of the feature map are set for each quantization mode. In quantization mode 0, a decimal minimum value X0min and a maximum value X0max are set, in quantization mode 1, a decimal minimum value X1min and a maximum value X1max are set, in quantization mode 2, a decimal minimum value X2min and a maximum value X2max are set, and in quantization mode 3, a decimal minimum value X3min and a maximum value X3max are set. For example, in quantization mode 0, a decimal minimum value X0min and a maximum value X0max are set in a range that includes the values ​​of the feature map elements of all channels, in quantization mode 1, a decimal minimum value X1min and a maximum value X1max are set in a narrower range than in quantization mode 0, and in quantization modes 2 and 3, a decimal minimum value X2min and a maximum value X2max, and a decimal minimum value X3min and a maximum value X3max are set in a narrower range than in quantization mode 0 and the quantization mode. In this way, the feature maps of channels for which quantization mode 2 and quantization mode 3 can be set enable finer quantization than the feature maps of channels for which quantization mode 0 or quantization mode 1 is set, resulting in reduced quantization error and improved coding efficiency.

[0112] Next, a quantization mode is set for the feature map of each channel (step S1003 in FIG. 16). Here, by satisfying the conditions Xqmin≦Xnmin′ and Xqmax≧Xnmax′ and setting a quantization mode with a narrower range, it is possible to set a quantization mode suitable for each channel that includes the values ​​of all elements of the feature map of each channel.

[0113] For each channel, a quantization mode having a minimum value Xqmin and a maximum value Xqmax with a predetermined margin is set; the minimum value Xqmin′ and the maximum value Xqmax′ of a plurality of packing feature frames in sequence units or for a predetermined time are detected and a quantization mode having a minimum value Xqmin and a maximum value Xqmax that include these minimum values ​​is set; and only when the minimum value Xqmin′ or the maximum value Xqmax′ of packing feature frames detected at a certain time is not included in the range from the minimum value Xqmin to the maximum value Xqmax of the quantization mode set at the previous time, the quantization mode is updated to one having the minimum value Xqmin and the maximum value Xqmax so as to include the detected minimum value Xqmin′ and maximum value Xqmax′. By suppressing fluctuations in the quantization mode and the minimum value Xqmin and maximum value Xqmax of the quantization mode for each channel, temporal correlation is increased compared to the first embodiment, prediction residuals are further reduced during inter prediction in feature map intra encoding / decoding, and encoding efficiency in the temporal direction can be further improved.

[0114] Furthermore, by setting the quantization mode based on the decimal minimum value Xnmin' and maximum value Xnmax' of the feature map unit of each channel and the occurrence distribution of the element values ​​of the feature map Xf of each channel, spatial correlation is increased, prediction residuals during intra prediction in feature map internal encoding / decoding are further reduced, and encoding efficiency can be improved.

[0115] Next, the feature map of each channel is converted from decimal to integer format by linear quantization (step S1004 in FIG. 16 ). The linear quantization in step S1004 in FIG. 16 will be described using the flowchart in FIG. 17 . The linear quantization process involves linear quantization of all elements (pixels) of the feature map of each channel from decimal to 10-bit integer format (steps S1101 to S1108 in FIG. 17 ). Letting p be the index of the feature map element (pixel), the linear quantization process in steps S1102 to S1107 is repeated from 0 to (the total number of elements in the feature maps of all channels minus 1), that is, until linear quantization of all elements of the feature maps of all channels is completed. FIG. 21 shows an example of linear transformation used in linear quantization when converting each element (pixel) of the feature map from decimal format Xf(p) to 10-bit integer format Xi(p).

[0116] First, based on the quantization mode set for each channel, linear transformation is performed on each element (pixel) of the feature map for each channel using the following formula (step S1102 in FIG. 17): Xa(p)=(Xf(p)-Xqmin) / (Xqmax-Xqmin), where Xa(p) is a decimal variable and q is 0, 1, 2, or 3 that indicates the quantization mode.

[0117] Next, Xa(p) is clipped as necessary to fall within the range of 0 to 1 (steps S1103 to S1106 in FIG. 17). If the variable Xa(p) is less than 0 (step S1103 in FIG. 17), the variable Xa(p) is clipped to 0 (step S1104 in FIG. 17). If the variable Xa(p) is greater than 1 (step S1105 in FIG. 17), the variable Xa(p) is clipped to 1 (step S1106 in FIG. 17). Clipping is performed when the decimal minimum value Xqmin is set to a value greater than Xf(p) or when the decimal maximum value Xqmax is set to a value less than Xf(p). Note that clipping is not necessary if Xqmin and Xqmax are set so that Xf(p) always falls within the range from Xqmin to Xqmax.

[0118] The variable Xa(p) is (2 Ｎ-1) and truncate or round off the decimal point to obtain an integer feature map Xi(p). When converted to a 10-bit integer, the result is 1023 (2 １０ -1) (step S1107 in FIG. 17). 1 is added to the index p, and if i is smaller than the total number of elements in the feature map (i.e., until linear quantization processing is completed for all elements of the feature map for all channels), linear quantization processing from S1102 to S1107 in FIG. 17 is repeated.

[0119] Through the above processing, linear quantization processing is performed from the decimal feature map elements (pixels) Xf(p) to the 10-bit integer feature map elements (pixels) Xi(p).

[0120] Alternatively, the value may be multiplied by 1023 using the following formula and then clipped to the range from 0 to 1023: Xa(p)={(Xf(p)-Xqmin) / (Xqmax-Xqmin)}×1023

[0121] Returning to the flowchart of FIG. 16 , the number of quantization modes and the corresponding fractional minimum and maximum values ​​are coded for each packing feature frame indicated by 183 in FIG. 32 , i.e., for each feature map of all channels (step S1005 in FIG. 16 ). FIG. 22 shows an example of syntax rules for the fractional minimum and maximum values ​​for the quantization modes. First, the syntax element num_quantization_mode indicating the number of quantization modes is coded. In the second embodiment, the value of the syntax element num_quantization_mode is 4. Next, the syntax element quantization_min_value[q_mode] indicating the fractional minimum value and the syntax element quantization_max_value[q_mode] indicating the fractional maximum value for the quantization mode q_mode are coded. In the second embodiment, the values ​​of the quantization mode q_mode are 0, 1, 2, and 3. When the quantization mode q_mode is 0, the values ​​of the syntax element quantization_min_value[0] and the syntax element quantization_max_value[0] are the decimal minimum value X0min and the maximum value X0max, respectively; when the quantization mode q_mode is 1, the values ​​of the syntax element quantization_min_value[1] and the syntax element quantization_max_value[1] are the decimal minimum value X1min and the maximum value X1max, respectively; when the quantization mode q_mode is 2, the values ​​of the syntax element quantization_min_value[2] and the syntax element quantization_max_value[2] are the decimal minimum value X2min and the maximum value X2max, respectively; and when the quantization mode q_mode is 3, the values ​​of the syntax element quantization_min_value[3] and the syntax element quantization_max_value[3] are the decimal minimum value X3min and the maximum value X3max, respectively; and these syntax elements are coded respectively.

[0122] Next, the quantization mode of each channel is coded (step S1006 in FIG. 16). FIG. 23 shows an example of syntax rules for the quantization mode of each channel. The variable n indicates the channel, and the variable num_feature_channel indicates the number of channels. The value of the syntax element quantization_mode[n] indicating the quantization mode of channel n is one of the values ​​0, 1, 2, and 3 set in step S1003 in FIG. 16, and the corresponding syntax element quantization_mode[n] is coded for each channel.

[0123] Note that in steps S1005 and S1006 of FIG. 16, the coding unit is the same frame time unit (183 and 184 in FIG. 32), but if the unit for detecting and setting the minimum and maximum values ​​is a sequence unit consisting of multiple frames in succession (181 and 182 in FIG. 32), coding is performed in sequence units.

[0124] Next, the linear inverse quantization process of the feature map inverse quantization unit 822 of the feature map inverse transformation unit 202 of the feature map decoding device 200 according to the second embodiment will be described with reference to the flowchart of FIG.

[0125] First, the number of quantization modes and the corresponding decimal minimum and maximum values ​​are decoded for each packing feature frame indicated by 183 in Fig. 32, i.e., for each feature map of all channels (step S2001 in Fig. 18). According to the syntax rules in Fig. 22, the syntax element num_quantization_mode indicating the number of quantization modes is first decoded. Next, the syntax element quantization_min_value[q_mode] indicating the decimal minimum value corresponding to the quantization mode q_mode and the syntax element quantization_max_value[q_mode] indicating the decimal maximum value are decoded to obtain the decimal minimum and maximum values ​​corresponding to each quantization mode.

[0126] Next, the quantization mode of each channel is decoded (step S2002 in FIG. 18). In accordance with the syntax rules in FIG. 23, the syntax element quantization_mode [n] corresponding to each channel is decoded to obtain the quantization mode of each channel.

[0127] In steps S2001 and S2002 of Figure 18, the decoding units for the maximum and minimum values ​​are the same frame time units (183 and 184 in Figure 32), but if the coding units for the maximum and minimum values ​​are sequence units consisting of multiple frames in succession (181 and 182 in Figure 32), decoding will be performed in sequence units.

[0128] Next, the feature map of each channel is converted from integer type to decimal type by linear inverse quantization (step S2003 in FIG. 18).

[0129] The linear inverse quantization in step S2003 in Fig. 18 will be described using the flowchart in Fig. 19. In the linear quantization process, linear inverse quantization from 10-bit integer type to decimal type is performed on all elements (pixels) of the feature map of each channel (steps S2101 to S2103 in Fig. 19). Letting p be the index of the element (pixel) of the feature map, the linear inverse quantization process in S2102 is repeatedly performed from 0 to (the total number of elements in the feature maps of all channels - 1), that is, until linear inverse quantization process is completed for all elements of the feature maps of all channels.

[0130] Based on the quantization mode set for each channel, linear inverse transformation is performed on each element (pixel) of the feature map of each channel using the following equation (step S2102 in FIG. 19): Xf(p)=(Xi(p) / 1023*(Xqmax-Xqmin)+Xqmin) Next, 1 is added to index p (step S2103 in FIG. 19), and if p is smaller than the total number of elements in the feature maps of all channels (i.e., until linear inverse quantization has been completed for all elements of the feature maps of all channels), the linear inverse quantization process of S2102 in FIG. 19 is repeated.

[0131] Third Embodiment Next, a feature map encoding device 100 and a feature map decoding device 200 according to a third embodiment of the present invention will be described.

[0132] In the feature map encoding device 100 according to the second embodiment, the feature map quantization unit 721 of the feature map conversion unit 103 sets a decimal minimum value Xqmin and a decimal maximum value Xqmax for each of a plurality of quantization modes, and assigns a quantization mode according to the minimum and maximum values ​​of the feature map of each channel, thereby performing linear quantization that is more suitable for each channel.

[0133] The feature map encoding device 100 of the third embodiment differs in that the feature map quantization unit 721 of the feature map conversion unit 103 performs linear / nonlinear quantization with different characteristics for each of multiple quantization modes. In addition to the decimal minimum value Xqmin and maximum value Xqmax, the third embodiment sets turning points for nonlinear conversion and assigns a quantization mode according to the minimum value, maximum value, and characteristics (distribution?) of the feature map of each channel, thereby performing linear / nonlinear quantization that is more suitable for each channel.

[0134] Furthermore, in the feature map decoding device 200 according to the second embodiment, the feature map inverse quantization unit 822 of the feature map inverse transformation unit 202 decodes (sets) the decimal minimum value Xqmin and maximum value Xqmax set for each of the multiple quantization modes, and performs linear inverse quantization that is more appropriate for each channel, based on the decimal minimum value Xqmin and maximum value Xqmax that correspond to the quantization mode set in the feature map of each channel.

[0135] The feature map decoding device 200 of the third embodiment differs in that the feature map inverse quantization unit 822 of the feature map inverse transformer 202 performs linear / nonlinear inverse quantization with different characteristics for each of a plurality of quantization modes. In addition to the decimal minimum value Xqmin and maximum value Xqmax, the third embodiment sets turning points for nonlinear transformation and assigns a quantization mode according to the minimum value, maximum value, and characteristics (distribution?) of the feature map of each channel, thereby performing linear / nonlinear quantization that is more suitable for each channel.

[0136] The linear quantization process of the feature map quantization unit 721 of the feature map conversion unit 103 of the feature map encoding device 100 according to the third embodiment will be described with reference to the flowchart of FIG.

[0137] First, the minimum and maximum values ​​of the decimal type of the feature map Xf of each channel are detected in units of frames (feature maps of the same time) (step S1301 in FIG. 24).

[0138] Next, a quantization mode and a linear / nonlinear transformation corresponding to the quantization mode are set based on the decimal minimum and maximum values ​​of the feature map Xf of each channel detected in step S1301 of Fig. 24 (step S1302 of Fig. 24). In the third embodiment, four quantization modes, 0, 1, 2, and 3, are set, and parameters indicating the characteristics of the linear / nonlinear transformation are set for each quantization mode, i.e., the decimal minimum value Xqmin and maximum value Xqmax of the feature map, as well as an integer value XqAi corresponding to the decimal value of turning point A (hereinafter referred to as the decimal value of turning point A) XqAf, and an integer value XqBi corresponding to the decimal value of turning point B (hereinafter referred to as the decimal value of turning point B) XqBf. The setting range for the decimal values ​​of turning points A and B is Xqmin<XqAf<XqBf<Xqmax, and the setting range for the integer values ​​of turning points A and B is 0<XqAi<XqBi<1023. Figures 27A, 27B, 27C, and 27D are diagrams showing examples of nonlinear transformation. All are set within the above setting ranges.

[0139] 26 shows an example in which four quantization modes, 0, 1, 2, and 3, are set, and parameters indicating linear or nonlinear transformation are set for each of them. Linear transformation is set for quantization mode 0, and three types of nonlinear transformation are set for quantization modes 1, 2, and 3.

[0140] Next, a quantization mode is set for the feature map of each channel (step S1303 in FIG. 24).

[0141] Next, the feature map of each channel is converted from decimal to integer form by linear / non-linear conversion according to the quantization mode (step S1304 in FIG. 24).

[0142] Next, the number of quantization modes and linear / nonlinear transformation parameters corresponding to the quantization modes are coded (step S1305 in FIG. 24). FIG. 28 shows an example of syntax rules for parameters indicating linear / nonlinear transformation corresponding to the quantization modes. First, the syntax element num_quantization_mode indicating the number of quantization modes is coded. In the third embodiment, the value of the syntax element num_quantization_mode is 4. Next, for each quantization mode, the syntax element quantization_min_value[q_mode] indicating the minimum value of the decimal type corresponding to the quantization mode q_mode and the syntax element quantization_max_value[q_mode] indicating the maximum value of the decimal type are coded. In the third embodiment, the value of the quantization mode q_mode is 0, 1, 2, or 3. Furthermore, the syntax element quantization_non_linear_flag[q_mode] indicating whether or not the transformation is nonlinear is coded. 0 indicates linear transformation, and 1 indicates nonlinear transformation. In the case of a nonlinear transformation, the syntax element quantization_A_float_value[q_mode] indicating the decimal value of turning point A, the syntax element quantization_A_int_value[q_mode] indicating the integer value of turning point A, and the syntax element quantization_B_float_value[q_mode] indicating the decimal value of turning point A and the syntax element quantization_B_int_value[q_mode] indicating the integer value of turning point A are coded.

[0143] Next, the quantization mode of each channel is coded (step S1306 in Fig. 24). In accordance with the syntax rules in Fig. 23, the value of the syntax element quantization_mode[n] indicating the quantization mode of channel n is one of the values ​​0, 1, 2, and 3 set in step S1303 in Fig. 24, and the corresponding syntax element quantization_mode[n] is coded for each channel.

[0144] Next, the linear inverse quantization process of the feature map inverse quantization unit 822 of the feature map inverse transformation unit 202 of the feature map decoding device 200 according to the third embodiment will be described with reference to the flowchart of FIG.

[0145] First, the number of quantization modes and the linear / nonlinear transformation parameters corresponding to the quantization modes are decoded (step S2301 in FIG. 25). According to the syntax rules in FIG. 28, the syntax element num_quantization_mode, which indicates the number of quantization modes, is decoded. Next, the syntax element quantization_min_value[q_mode], which indicates the minimum value of the decimal type corresponding to the quantization mode q_mode, and the syntax element quantization_max_value[q_mode], which indicates the maximum value of the decimal type, are decoded to obtain the minimum and maximum values ​​of the decimal type corresponding to each quantization mode. Furthermore, the syntax element quantization_non_linear_flag[q_mode], which indicates whether or not the transformation is nonlinear, is decoded. 0 indicates linear transformation, and 1 indicates nonlinear transformation. In the case of a nonlinear transformation, the syntax element quantization_A_float_value[q_mode] indicating the decimal value of turning point A and the syntax element quantization_A_int_value[q_mode] indicating the integer value of turning point A, and the syntax element quantization_B_float_value[q_mode] indicating the decimal value of turning point B and the syntax element quantization_B_int_value[q_mode] indicating the integer value of turning point B are decoded to obtain the decimal value and integer value of turning point A of the nonlinear transformation, and the decimal value and integer value of turning point B.

[0146] Next, the quantization mode of each channel is decoded (step S2302 in FIG. 25). In accordance with the syntax rules in FIG. 23, the syntax element quantization_mode [n] corresponding to each channel is decoded to obtain the quantization mode of each channel.

[0147] Next, the integer type is converted to decimal type by linear or nonlinear conversion according to the quantization mode (step S2303 in FIG. 25).

[0148] As described above, by using an integer-to-decimal conversion (quantization mode) appropriate for the feature map of each channel depending on the task, such as object detection, instance segmentation, or object tracking, it is possible to improve the coding efficiency of the feature map while maintaining the accuracy of the neural network task. Furthermore, by setting the minimum value Xqmin and maximum value Xqmax for quantization for each channel and setting the quantization mode, it is possible to increase the temporal correlation and spatial correlation, as in the second embodiment, and improve the coding efficiency of the feature map.

[0149] Fourth Embodiment Next, a feature map encoding device 100 and a feature map decoding device 200 according to a fourth embodiment of the present invention will be described.

[0150] In the feature map encoding device 100 according to the first embodiment, the feature map quantization unit 702 of the feature map conversion unit 103 sets a minimum value Xmin and a maximum value Xmax of the decimal type of elements common to the packing feature frame, i.e., the feature maps xf of all channels, and performs linear quantization from the decimal type to an integer type with N bits (10 bits in this embodiment), thereby converting the decimal type feature map xf into an integer type feature map xi.

[0151] The feature map encoding device 100 of the fourth embodiment differs in that the feature map quantization unit 702 sets turning points for nonlinear transformation to perform nonlinear quantization that is more suitable for a sequence or frame.

[0152] Furthermore, in the feature map decoding device 200 according to the first embodiment, the feature map inverse quantization unit 802 in the feature map inverse conversion unit 202 decodes (sets) the minimum value Xmin and maximum value Xmax of the decimal type of the elements common to the packing feature frame, i.e., the feature maps of all channels, and performs linear inverse quantization from integer type to decimal type to convert the N-bit (10-bit in this embodiment) integer type feature map into a decimal type feature map.

[0153] The fourth embodiment differs in that the feature map inverse quantization unit 802 sets a turning point for nonlinear transformation to perform more suitable nonlinear quantization for a sequence or frame.

[0154] The nonlinear quantization process of the feature map quantization unit 702 of the feature map encoding device 100 according to the fourth embodiment will be described with reference to the flowchart of FIG.

[0155] First, the minimum and maximum values ​​of the decimal type of the feature map Xf of each channel are detected in units of frames (feature maps of the same time) (step S1201 in FIG. 29).

[0156] Next, the nonlinear transformation to be performed by the feature map quantization unit 702 is set (step S1202 in FIG. 29 ). Here, based on the minimum and maximum decimal values ​​of the detected feature map, the minimum and maximum decimal values ​​Xqmin and Xqmax of the feature map to be encoded and decoded are set. Here, in the fourth embodiment, q is set to 0. In addition, the decimal value of turning point A (hereinafter referred to as the decimal value of turning point A) XqAf and the corresponding integer value (hereinafter referred to as the integer value of turning point A) XqAi, and the decimal value of turning point B (hereinafter referred to as the decimal value of turning point B) XqBf and the corresponding integer value (hereinafter referred to as the integer value of turning point B) XqBi are set. Note that the setting range of the decimal values ​​of turning points A and B is Xqmin<XqAf<XqBf<Xqmax, and the setting range of the integer values ​​of turning points A and B is 0<XqAi<XqBi<1023. 27A, 27B, 27C, and 27D are diagrams showing examples of nonlinear transformation, all of which are set within the above-mentioned setting range.

[0157] Next, the feature map of each channel is converted from decimal to integer form by nonlinear conversion based on the set linear / nonlinear conversion parameters Xqmin, Xqmax, XqAf, and XqBf (step S1203 in FIG. 29).

[0158] Next, parameters related to the linear / nonlinear transformation are coded (step S12045 in FIG. 29 ). FIG. 31 shows an example of syntax rules for nonlinear transformation parameters. First, the minimum value min_float_value and maximum value max_float_value of the feature map are coded. Next, the syntax element num_non_linear_points, which indicates the number of turning points of the nonlinear transformation, is coded. In the fourth embodiment, the value of the syntax element num_non_linear_points is 2. Next, if num_non_linear_points is 1 or greater, the syntax element quantization_min_value[point], which indicates the minimum value of the decimal type of turning points of the nonlinear transformation, and the syntax element quantization_max_value[point], which indicates the maximum value of the decimal type of turning points of the nonlinear transformation, are coded for the number of turning points of the nonlinear transformation.

[0159] Next, the nonlinear inverse quantization process of the feature map inverse quantization unit 802 of the feature map decoding device 200 according to the fourth embodiment will be described with reference to the flowchart of FIG.

[0160] First, parameters related to linear / nonlinear transformation are decoded (step S2201 in FIG. 30 ). According to the syntax rules in FIG. 31 , the minimum value min_float_value and maximum value max_float_value of the feature map are decoded first. Next, the syntax element num_non_linear_points, which indicates the number of turning points of the nonlinear transformation, is decoded. In the fourth embodiment, the value of the syntax element num_non_linear_points is 2. Next, if num_non_linear_points is 1 or greater, the syntax element quantization_float_value[point], which indicates the decimal value of the turning point of the nonlinear transformation, and the syntax element quantization_int_value[point], which indicates the integer value, are decoded for the number of turning points of the nonlinear transformation.

[0161] Next, the integer type is converted to the decimal type by nonlinear conversion based on the decoded linear / nonlinear conversion parameters Xqmin, Xqmax, XqAf, and XqBf (step S2202 in FIG. 30).

[0162] Note that if the syntax element num_non_linear_points, which indicates the number of turning points of the nonlinear transformation, is set to 0, in step S1203 of FIG. 29 and step S2202 of FIG. 30, a linear transformation / inverse linear transformation is performed based on the minimum value min_float_value and the maximum value max_float_value of the feature map to be encoded / decoded.

[0163] As described above, by using integer-to-decimal conversion (quantization mode) suitable for the feature maps of all channels depending on the task, such as object detection, instance segmentation, or object tracking, it is possible to improve the coding efficiency of the feature maps while maintaining the accuracy of the neural network task. Furthermore, by setting the minimum value Xmin and maximum value Xmax for quantization in the same way as in the first embodiment, it is possible to increase the temporal correlation, thereby improving the coding efficiency of the feature maps in the same way as in the first embodiment.

[0164] Fifth Embodiment Next, a feature map encoding device 100 and a feature map decoding device 200 according to a fifth embodiment of the present invention will be described.

[0165] In the feature map encoding device 100 according to the second embodiment, the feature map quantization unit 721 of the feature map conversion unit 103 sets multiple quantization modes in units of packing feature frames, i.e., in units of feature maps for all channels, as indicated by 183 in Fig. 32 , sets a decimal minimum value Xqmin and a decimal maximum value Xqmax for each quantization mode, and assigns a quantization mode according to the minimum and maximum values ​​for the feature map units of each channel, as indicated by 184 in Fig. 32 , thereby performing linear quantization appropriate for each channel.

[0166] The feature map coding device 100 of the fifth embodiment differs from the feature map coding device 100 of the second embodiment in that the feature map quantization unit 721 of the feature map conversion unit 103 sets a decimal minimum value Xnmin and a decimal maximum value Xnmax for each feature map of each channel, as indicated by 184 in Fig. 32, and performs linear quantization that is appropriate for each channel. In all other respects, the feature map coding device 100 is the same as the feature map coding device 100 of the second embodiment.

[0167] Furthermore, in the feature map decoding device 200 according to the second embodiment, the feature map inverse transform unit 202 sets a plurality of quantization modes in units of packing feature frames, i.e., in units of feature maps for all channels, as indicated by 183 in Fig. 32 , and sets a decimal minimum value Xqmin and a maximum value Xqmax for each quantization mode. Linear inverse quantization that is more appropriate for each channel is performed based on the decimal minimum value Xqmin and maximum value Xqmax that correspond to the quantization mode set in units of feature maps for each channel, as indicated by 184 in Fig. 32 .

[0168] The feature map decoding device 200 of the fifth embodiment differs from the feature map decoding device 200 of the second embodiment in that the feature map inverse quantization unit 822 of the feature map inverse transformer 202 sets a decimal minimum value Xnmin and a maximum value Xnmax for each feature map of each channel as indicated by 184 in Fig. 32 , and performs linear inverse quantization that is more appropriate for each channel based on the decimal minimum value Xnmin and maximum value Xnmax set for each feature map of each channel as indicated by 184 in Fig. 32 . Other than that, the feature map decoding device 200 is similar to the feature map decoding device 200 of the second embodiment.

[0169] The linear quantization process of the feature map quantization unit 721 of the feature map conversion unit 103 of the feature map encoding device 100 according to the fifth embodiment will be described with reference to the flowchart of FIG.

[0170] First, the minimum value Xnmin' and maximum value Xnmax' of the decimal type in the feature map unit of each channel, indicated by 184 in Fig. 32, are detected (step S1401 in Fig. 34). Furthermore, the occurrence distribution of the element values ​​of the feature map Xf of each channel may be detected.

[0171] Next, the decimal minimum value Xnmin and maximum value Xnmax of the feature map Xf of each channel are set based on the decimal minimum value Xnmin' and maximum value Xnmax' of the feature map Xf of each channel detected in step S1401 in Fig. 34 (step S1402 in Fig. 34). Here, by setting the minimum value Xnmin and maximum value Xnmax in the range Xnmin≦Xnmin' Xnmax≧Xnmax', it is possible to set the decimal minimum value Xnmin and maximum value Xnmax in a range that includes the values ​​of all elements of the feature map of each channel.

[0172] For each channel n, a minimum value Xnmin and a maximum value Xnmax that have a predetermined margin are set; the minimum value Xnmin' and the maximum value Xnmax' of a plurality of packing feature frames are detected in sequence units or at a predetermined time, and a minimum value Xnmin and a maximum value Xnmax that include these are set; and only when the minimum value Xnmin' or the maximum value Xnmax' of packing feature frames detected at a certain time is not included in the range from the minimum value Xnmin to the maximum value Xnmax set at the previous time, the minimum value Xnmin and the maximum value Xnmax are updated so as to include the detected minimum value Xnmin' or maximum value Xnmax'. In this way, fluctuations in the minimum value Xnmin and the maximum value Xnmax for each channel are suppressed, resulting in a higher temporal correlation than in the first and second embodiments, a further reduction in prediction residuals during inter prediction in feature map intra encoding / decoding, and further improvement in encoding efficiency.

[0173] Furthermore, by setting the minimum value Xnmin' and maximum value Xnmax' of decimal type in feature map units for each channel and the minimum value Xnmin and maximum value Xnmax based on the occurrence distribution of element values ​​of the feature map Xf for each channel, the spatial correlation between channels is increased, the prediction residual during intra prediction in feature map internal encoding / decoding is further reduced, and encoding efficiency can be improved.

[0174] Here, the minimum and maximum values ​​are set in the same frame time unit (183 and 184 in Figure 32), but the minimum and maximum values ​​may also be set in a sequence unit consisting of multiple consecutive frames (181 and 182 in Figure 32).

[0175] Next, the feature map of each channel is converted from decimal to integer format by linear quantization (step S1404 in FIG. 34 ). The linear quantization in step S1404 in FIG. 34 will be described using the flowchart in FIG. 17 . The linear quantization process involves linear quantization of all elements (pixels) of the feature map of each channel from decimal to 10-bit integer format (steps S1101 to S1108 in FIG. 17 ). Letting p be the index of the feature map element (pixel), the linear quantization process in S1102 to S1107 is repeated from 0 to (the total number of elements in the feature maps of all channels minus 1), that is, until linear quantization of all elements of the feature maps of all channels is completed. FIG. 21 shows an example of linear transformation used in linear quantization when converting each element (pixel) of the feature map from decimal format Xf(p) to 10-bit integer format Xi(p).

[0176] First, based on the minimum value Xnmin and maximum value Xnmax set for each channel n, linear transformation is performed on each element (pixel) of the feature map for each channel using the following equation (step S1102 in FIG. 17 ): Here, the minimum value Xqmin for each quantization group in the second embodiment in FIG. 17 is replaced with the minimum value Xnmin for each channel in the fifth embodiment, and the minimum value Xqmax for each quantization group in the second embodiment in FIG. 17 is replaced with the minimum value Xnmax for each channel in the fifth embodiment: Xa(p)=(Xf(p)-Xnmin) / (Xnmax-Xnmin), where Xa(p) is a decimal variable, and n is a variable indicating each channel.

[0177] Next, Xa(p) is clipped as necessary to fall within the range of 0 to 1 (steps S1103 to S1106 in FIG. 17). If the variable Xa(p) is less than 0 (step S1103 in FIG. 17), the variable Xa(p) is clipped to 0 (step S1104 in FIG. 17). If the variable Xa(p) is greater than 1 (step S1105 in FIG. 17), the variable Xa(p) is clipped to 1 (step S1106 in FIG. 17). Clipping is performed when the decimal minimum value Xnmin is set to a value greater than Xf(p) or when the decimal maximum value Xnmax is set to a value less than Xf(n). Note that clipping is not necessary if Xnmin and Xnmax are set so that Xf(p) always falls within the range from Xnmin to Xnmax.

[0178] The variable Xa(p) is (2 Ｎ -1) to obtain an integer feature map Xi(p). When converted to a 10-bit integer, it is multiplied by 1023 (2 １０ -1) (step S1107 in FIG. 17). 1 is added to index p, and if i is smaller than the total number of elements in the feature map (that is, until linear quantization processing is completed for all elements of the feature maps of all channels), linear quantization processing from S1102 to S1107 in FIG. 17 is repeated. Through the above processing, linear quantization processing is performed from decimal feature map elements (pixels) Xf(p) to 10-bit integer feature map elements (pixels) Xi(p).

[0179] Alternatively, the value may be multiplied by 1023 using the following formula and then clipped to the range from 0 to 1023: Xa(p)={(Xf-Xnmin) / (Xnmax-Xnmin)}×1023

[0180] Returning to the flowchart in Fig. 34, the decimal minimum and maximum values ​​of each channel are encoded in units of the feature map of each channel indicated by 184 in Fig. 32 (step S1405 in Fig. 34). Fig. 36A shows an example of syntax rules for the decimal minimum and maximum values ​​of each channel, which are elements of the feature map encoding information. First, the syntax element num_feature_channel, which indicates the number of channels, is encoded. In the fifth embodiment, the value of the syntax element num_feature_channel is 64.

[0181] Next, the decimal minimum value Xnmin and maximum value Xnmax for each channel are coded. The variable n indicates the channel. For all channels, the syntax element quantization_min_value[n] indicating the minimum value Xnmin for each channel n and the syntax element quantization_max_value[n] indicating the maximum value Xnmax for each channel n are coded.

[0182] Next, the linear inverse quantization process of the feature map inverse quantization unit 822 of the feature map inverse transformation unit 202 of the feature map decoding device 200 according to the fifth embodiment will be described with reference to the flowchart of FIG.

[0183] For each channel feature map indicated by 184 in Fig. 32, the number of quantization modes and the corresponding decimal minimum and maximum values ​​are decoded (step S2401 in Fig. 35). According to the syntax rules in Fig. 36A, first, the syntax element num_feature_channel indicating the number of quantization modes is decoded. Next, for all channels, the syntax element quantization_min_value[n] indicating the decimal minimum value corresponding to each channel n and the syntax element quantization_max_value[n] indicating the decimal maximum value are decoded to obtain the decimal minimum value Xnmin and maximum value Xnmax corresponding to each channel n.

[0184] Next, the feature map of each channel is converted from integer type to decimal type by linear inverse quantization (step S2403 in FIG. 35).

[0185] The linear inverse quantization in step S2403 in Fig. 35 will be described using the flowchart in Fig. 19. In the linear quantization process, linear inverse quantization from 10-bit integer type to decimal type is performed on all elements (pixels) of the feature map of each channel (steps S2101 to S2103 in Fig. 19). Letting p be the index of the element (pixel) of the feature map, the linear inverse quantization process in S2102 is repeatedly performed from 0 to (the total number of elements in the feature maps of all channels - 1), that is, until linear inverse quantization process is completed for all elements of the feature maps of all channels.

[0186] Based on the quantization mode set for each channel, linear inverse transformation is performed on each element (pixel) of the feature map for each channel using the following equation (step S2102 in FIG. 19). Here, the minimum value Xqmin for each quantization group in the second embodiment in FIG. 19 is replaced with the minimum value Xnmin for each channel in the fifth embodiment, and the minimum value Xqmax for each quantization group in the second embodiment in FIG. 19 is replaced with the minimum value Xnmax for each channel in the fifth embodiment. Xf(p)=(Xi(p) / 1023×(Xnmax−Xnmin)+Xnmin

[0187] Next, 1 is added to index p (step S2103 in FIG. 19 ), and if p is smaller than the total number of elements in the feature maps of all channels (i.e., until linear inverse quantization processing of all elements in the feature maps of all channels is completed), the linear inverse quantization processing of S2102 in FIG. 19 is repeatedly performed.

[0188] In step S1405 of FIG. 34, the syntax element quantization_min_value[n] indicating the minimum value Xnmin of each channel n and the syntax element quantization_max_value[n] indicating the maximum value Xnmax of each channel n are coded in the same frame time unit (184 in FIG. 32). However, if the setting unit for the minimum value Xnmin and maximum value Xnmax of each channel n is set to a sequence unit consisting of multiple consecutive frames (182 in FIG. 32), the syntax elements quantization_min_value[n] and quantization_max_value[n] are coded / decoded in sequence units (coding on the coding side and decoding on the decoding side).

[0189] Furthermore, when sequentially encoding the syntax element quantization_min_value[n] indicating the minimum value Xnmin of each channel n and the syntax element quantization_max_value[n] indicating the maximum value Xnmax of each channel n in the same frame time unit, it is also possible to encode / decode the syntax elements quantization_min_value[n] and quantization_max_value[n] of channel n, which are updated only when updating the decimal-type minimum value Xnmin and maximum value Xnmax for each channel (using decimal-type minimum value Xnmin and maximum value Xnmax different from those in the previous frame time unit). Figure 33B shows another example of the syntax rules for the decimal-type minimum value Xnmin and maximum value Xnmax. The syntax element update_quantization_param_flag[n] is a 1-bit flag indicating whether to update the decimal-type minimum value Xnmin and maximum value Xnmax of channel n. When the decimal minimum value Xnmin and maximum value Xnmax are updated, the syntax element update_quantization_param_flag[n] is set to 1 for encoding / decoding, and the syntax elements quantization_min_value[n] and quantization_max_value[n] are also encoded / decoded. When the decimal minimum value Xmin and maximum value Xmax are not updated (the same decimal minimum value Xmin and maximum value Xmax as in the previous frame time unit are used), the syntax element update_quantization_param_flag[n] is set to 0 for encoding / decoding. By omitting the encoding / decoding of the syntax elements quantization_min_value[n] and quantization_max_value[n], the amount of code can be reduced and encoding efficiency improved when the decimal minimum value Xmin and maximum value Xmax are not updated.

[0190] Sixth Embodiment Next, a feature map encoding device 100 and a feature map decoding device 200 according to a sixth embodiment of the present invention will be described.

[0191] In the feature map encoding device 100 according to the fifth embodiment, the feature map quantization unit 721 of the feature map conversion unit 103 sets a decimal minimum value Xnmin and a decimal maximum value Xnmax for each feature map of each channel, as indicated by 184 in FIG. 32 , and performs linear quantization that is more suitable for each channel.

[0192] The feature map coding device 100 of the sixth embodiment differs from the feature map coding device 100 of the fifth embodiment in that the feature map quantization unit 721 of the feature map conversion unit 103 sets a scaling coefficient αn and a shift coefficient βn for each feature map of each channel, as indicated by 184 in Fig. 32 , and performs linear quantization appropriate for each channel based on the scaling coefficient αn and the shift coefficient βn. Other than that, the feature map coding device 100 of the sixth embodiment is similar to the feature map coding device 100 of the fifth embodiment.

[0193] Furthermore, in the feature map decoding device 200 of the fifth embodiment, the feature map inverse transform unit 202 sets a decimal minimum value Xnmin and a maximum value Xnmax for each feature map of each channel as indicated by 184 in Fig. 32, and performs linear inverse quantization that is more appropriate for each channel based on the decimal minimum value Xnmin and maximum value Xnmax set for each feature map of each channel as indicated by 184 in Fig. 32. Other than that, the feature map decoding device 200 is similar to the feature map decoding device 200 of the sixth embodiment.

[0194] The feature map decoding device 200 of the sixth embodiment differs from the feature map decoding device 200 of the fifth embodiment in that the feature map inverse quantization unit 822 of the feature map inverse transformation unit 202 sets a scaling coefficient αn and a shift coefficient βn for each feature map of each channel as indicated by 184 in Fig. 32 , and performs linear inverse quantization appropriate for each channel based on the scaling coefficient αn and shift coefficient βn set for each feature map of each channel as indicated by 184 in Fig. 32 . Other than that, the feature map decoding device 200 of the sixth embodiment is the same as the feature map decoding device 200 of the fifth embodiment.

[0195] The linear quantization process of the feature map quantization unit 721 of the feature map conversion unit 103 of the feature map encoding device 100 according to the fifth embodiment will be described with reference to the flowchart of FIG.

[0196] First, the minimum value Xnmin' and maximum value Xnmax' of the decimal type in the feature map unit of each channel, indicated by 184 in Fig. 32, are detected (step S1501 in Fig. 37). Furthermore, the occurrence distribution of the element values ​​of the feature map Xf of each channel may be detected.

[0197] Next, a scaling coefficient αn and a shift coefficient βn of the feature map of each channel are set based on the decimal minimum value Xnmin' and maximum value Xnmax' of the feature map Xf of each channel detected in step S1501 of FIG. 37 (step S1502 of FIG. 37). For example, the scaling coefficient αn and the shift coefficient βn of each channel can be derived using the following equations: αn=1 / (Xnmax-Xnmin) βn=Xnmin / (Xnmax-Xnmin) Note that by setting the minimum values ​​Xnmin and Xnmin in the same way as in the fifth embodiment, the scaling coefficient αn and the shift coefficient βn can be set within a range that includes the values ​​of all elements of the feature map of each channel.

[0198] As in the fifth embodiment, for each channel n, a minimum value Xnmin and a maximum value Xnmax that have a margin of a predetermined rate are set; minimum values ​​Xnmin′ and maximum values ​​Xnmax′ of a plurality of packing feature frames in sequence units or for a predetermined time are detected and a minimum value Xnmin and a maximum value Xnmax that include these minimum values ​​Xnmin and maximum values ​​Xnmax are set; and only when the minimum value Xnmin′ or maximum value Xnmax′ of packing feature frames detected at a certain time is not included in the range from the minimum value Xnmin to the maximum value Xnmax set at the previous time, the minimum value Xnmin and maximum value Xnmax are updated so as to include the detected minimum value Xnmin′ or maximum value Xnmax′. In this way, fluctuations in the minimum value Xnmin and the maximum value Xnmax are suppressed and the scaling coefficient αn and the shift coefficient βn are set for each channel, thereby achieving a higher temporal correlation than in the first and second embodiments, further reducing prediction residuals during inter prediction in feature map intra encoding / decoding, and further improving encoding efficiency.

[0199] Furthermore, by setting the minimum value Xnmin' and maximum value Xnmax' of decimal type in feature map units for each channel and the occurrence distribution of element values ​​of the feature map Xf for each channel, and by setting the scaling coefficient αn and shift coefficient βn, spatial correlation is increased, prediction residuals during intra prediction in feature map internal encoding / decoding are further reduced, and encoding efficiency can be improved.

[0200] Here, the setting processing unit for the scaling coefficient αn and the shift coefficient βn is the same frame time unit (183 and 184 in Figure 32), but the setting processing unit for the scaling coefficient αn and the shift coefficient βn may also be a sequence unit consisting of multiple frames that are consecutive in time (181 and 182 in Figure 32).

[0201] Next, the feature map of each channel is converted from decimal to integer format by linear quantization (step S1504 in FIG. 37 ). The linear quantization in step S1504 in FIG. 37 will be described using the flowchart in FIG. 38 . The linear quantization process involves linear quantization of all elements (pixels) of the feature map of each channel from decimal to 10-bit integer format (steps S1601 to S1608 in FIG. 38 ). Let p be the index of the feature map element (pixel), and the linear quantization process in steps S1602 to S1607 is repeated from 0 to (the total number of elements in the feature maps of all channels minus 1), that is, until linear quantization of all elements of the feature maps of all channels is completed. FIG. 21 shows an example of linear transformation used in linear quantization when converting each element (pixel) of the feature map from decimal format Xf(p) to 10-bit integer format Xi(p).

[0202] First, based on the scaling coefficient αn and shift coefficient βn set for each channel n, linear transformation is performed on each element (pixel) of the feature map for each channel using the following formula (step S1602 in FIG. 38): Xa(p) = αn × Xf(p) + βn, where Xa(p) is a decimal variable and n is a variable indicating each channel.

[0203] Next, Xa(p) is clipped as necessary to fall within the range of 0 to 1 (steps S1603 to S1606 in FIG. 38). If the variable Xa(p) is less than 0 (step S1603 in FIG. 38), the variable Xa(p) is clipped to 0 (step S1604 in FIG. 38). If the variable Xa(p) is greater than 1 (step S1605 in FIG. 38), the variable Xa(p) is clipped to 1 (step S1606 in FIG. 38). Clipping is performed when the decimal minimum value Xnmin is set to a value greater than Xf(p) or when the decimal maximum value Xnmax is set to a value less than Xf(n). Note that clipping is not necessary if Xnmin and Xnmax are set so that Xf(p) always falls within the range from Xnmin to Xnmax.

[0204] The variable Xa(p) is (2 Ｎ -1) to obtain an integer feature map Xi(p). When converted to a 10-bit integer, it is multiplied by 1023 (2 １０ -1) (step S1607 in FIG. 38). 1 is added to index p, and if i is smaller than the total number of elements in the feature map (that is, until linear quantization of all elements in the feature maps of all channels is completed), linear quantization processing from S1602 to S1607 in FIG. 38 is repeated. Through the above processing, linear quantization processing is performed from decimal feature map elements (pixels) Xf(p) to 10-bit integer feature map elements (pixels) Xi(p).

[0205] Alternatively, the value may be multiplied by 1023 using the following formula and then clipped to the range from 0 to 1023: Xa(p)={(αn×Xf(p)+βn)}×1023

[0206] Returning to the flowchart in Fig. 37, the scaling coefficient αn and shift coefficient βn of each channel are encoded in units of the feature map of each channel indicated by 184 in Fig. 32 (step S1505 in Fig. 37). Fig. 41A shows an example of syntax rules for the scaling coefficient αn and shift coefficient βn of each channel. First, the syntax element num_feature_channel, which indicates the number of channels, is encoded. In the fifth embodiment, the value of the syntax element num_feature_channel is 64.

[0207] Next, the scaling coefficient αn and shift coefficient βn for each channel are coded. The variable n indicates the channel. For all channels, the syntax element quantization_scaling_param[n] indicating the scaling coefficient αn for each channel n and the syntax element quantization_shift_param[n] indicating the shift coefficient βn for each channel n are coded.

[0208] Next, the linear inverse quantization process of the feature map inverse quantization unit 822 of the feature map inverse transformation unit 202 of the feature map decoding device 200 according to the fifth embodiment will be described with reference to the flowchart of FIG.

[0209] The number of quantization modes and the scaling coefficients αn and shift coefficients βn corresponding to the quantization modes are decoded for each feature map of each channel indicated by 184 in Fig. 32 (step S2501 in Fig. 39 ). According to the syntax rules in Fig. 41A , the syntax element num_feature_channel indicating the number of quantization modes is first decoded. Next, for all channels, the syntax element quantization_scaling_param[n] indicating the scaling coefficient αn corresponding to each channel n and the syntax element quantization_shift_param[n] indicating the shift coefficient βn are decoded to obtain the scaling coefficients αn and shift coefficients βn corresponding to each channel n.

[0210] In step S2501 of Figure 39, the decoding unit for the scaling coefficient αn and shift coefficient βn of each channel is the same frame time unit (184 in Figure 32), but if the setting processing unit for quantization is a sequence unit consisting of multiple frames that are consecutive in time (182 in Figure 32), the scaling coefficient αn and shift coefficient βn of each channel are decoded in sequence units.

[0211] Next, the feature map of each channel is converted from integer type to decimal type by linear inverse quantization (step S2503 in FIG. 39).

[0212] The linear inverse quantization in step S2503 in Fig. 39 will be described using the flowchart in Fig. 40. In the linear quantization process, linear inverse quantization from 10-bit integer type to decimal type is performed on all elements (pixels) of the feature map of each channel (steps S2601 to S2603 in Fig. 40). Letting p be the index of the element (pixel) of the feature map, the linear inverse quantization process in S2602 is repeatedly performed from 0 to (the total number of elements in the feature maps of all channels - 1), that is, until linear inverse quantization process is completed for all elements of the feature maps of all channels.

[0213] Based on the quantization mode set for each channel, linear inverse transformation is performed for each element (pixel) of the feature map for each channel using the following formula (step S2602 in FIG. 40): Xf(p)=(Xi(p) / 1023-βn) / αn

[0214] Next, 1 is added to index p (step S2603 in FIG. 40), and if p is smaller than the total number of elements in the feature maps of all channels (i.e., until linear inverse quantization processing of all elements in the feature maps of all channels is completed), the linear inverse quantization processing of S2602 in FIG. 40 is repeatedly performed.

[0215] In step S1505 of Fig. 37 , the scaling coefficient αn and shift coefficient βn of each channel are coded in the same frame time unit (184 in Fig. 32 ). However, if the scaling coefficient αn and shift coefficient βn of each channel are set in sequence units (182 in Fig. 32 ) consisting of multiple consecutive frames, the scaling coefficient αn and shift coefficient βn of each channel are coded in sequence units. Furthermore, when sequentially coding the syntax element quantization_scaling_param[n] indicating the scaling coefficient αn of each channel n and the syntax element quantization_shift_param[n] indicating the shift coefficient βn of each channel n in the same frame time unit, it is also possible to code / decode the syntax elements quantization_scaling_param[n] and quantization_shift_param[n] of channel n, which are updated only when the scaling coefficient αn and shift coefficient βn are updated for each channel (the scaling coefficient αn and shift coefficient βn are different from those of the previous frame time unit). 33B shows another example of syntax rules for the scaling coefficient αn and the shift coefficient βn. The syntax element update_quantization_param_flag[n] is a 1-bit flag indicating whether or not to update the scaling coefficient αn and the shift coefficient βn of channel n. When updating the scaling coefficient αn and the shift coefficient βn, the syntax element update_quantization_param_flag[n] is set to 1 and encoding / decoding is performed, and the syntax elements quantization_scaling_param[n] and quantization_shift_param[n] are also encoded / decoded.If the scaling coefficient αn and shift coefficient βn are not updated (the same scaling coefficient αn and shift coefficient βn as in the previous frame time unit are used), the syntax element update_quantization_param_flag[n] is set to 0 for encoding / decoding, and the encoding / decoding of the syntax elements quantization_scaling_param[n] and quantization_shift_param[n] is omitted. This reduces the amount of code and improves coding efficiency when the scaling coefficient αn and shift coefficient βn are not updated.

[0216] In all of the above-described embodiments, the bitstream output by the feature map encoder has a specific data format that can be decoded according to the encoding method used in the embodiment, and the feature map decoder corresponding to the feature map encoder can decode the bitstream in this specific data format.

[0217] When a wired or wireless network is used to exchange bitstreams between the feature map encoder and the feature map decoder, the bitstreams may be converted into a data format suitable for the transmission mode of the communication channel before transmission. In this case, a transmitter is provided that converts the bitstream output by the feature map encoder into coded data in a data format suitable for the transmission mode of the communication channel and transmits the coded data to the network, and a receiver is provided that receives the coded data from the network, restores the coded data to a bitstream, and supplies the bitstream to the feature map decoder. The transmitter includes a memory that buffers the bitstream output by the feature map encoder, a packet processor that packetizes the bitstream, and a transmitter that transmits the packetized coded data via the network. The receiver includes a receiver that receives the packetized coded data via the network, a memory that buffers the received coded data, and a packet processor that packetizes the coded data to generate a bitstream and provides it to the feature map decoder.

[0218] The above encoding and decoding processes may be realized not only as a transmission, storage, and receiving device using hardware, but also as firmware stored in a ROM (read-only memory), flash memory, etc., or as software for a computer, etc. The firmware program or software program may be provided by recording it on a computer-readable recording medium, or may be provided from a server via a wired or wireless network, or may be provided as data broadcasting on terrestrial or satellite digital broadcasting.

[0219] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the components and processing steps, and that such modifications are also within the scope of the present invention.

[0220] It can be used to encode and decode feature maps in neural networks.

[0221] 100 Feature map encoding device, 101 Neural network feature extraction unit, 102 Feature map reduction unit, 103 Feature map conversion unit, 104 Feature map internal encoding unit, 200 Feature map decoding device, 201 Feature map internal decoding unit, 202 Feature map inverse conversion unit, 203 Feature map restoration unit, 204 Neural network identification unit, 301 Convolution processing unit, 302 Activation processing unit, 303 Pooling processing unit, 322 Bottom-up processing unit, 324 Top-down processing unit, 326 Image to be subjected to feature extraction processing, 501 First feature map reduction unit, 502 First channel combination unit, 503 Second feature map reduction unit, 504 Second channel combination unit, 505 Third feature map reduction unit, 506 First padding unit, 507 Second padding unit, 508 Third padding unit, 601 8x enlargement unit, 602 4x enlargement unit, 603 2x enlargement unit, 604 First feature map blending unit, 605 Second feature map blending unit, 606 First unpadding unit, 607 Second unpadding unit, 608 Third unpadding unit, 701 Packing unit, 702 Feature map quantization unit, 802 Feature map inverse quantization unit, 803 Unpacking unit, 901 Switch, 902 VVC encoding unit, 903 HEVC encoding unit, 904 AV1 encoding unit, 1001 Switch, 1002 VVC decoding unit, 1003 HEVC decoding unit, 1004 AV1 decoding unit, 721 Feature map quantization unit, 722 Packing unit, 821 Unpacking unit, 822 Feature map inverse quantization unit.

Claims

1. A feature map coding device for coding multi-scale feature maps, comprising: a quantization unit that converts elements of a multi-channel decimal feature map from decimal to integer to generate a multi-channel integer feature map; a packing unit that packs the multi-channel integer feature map to generate a single packed integer feature frame; and a feature map internal coding unit that encodes the packed integer feature frame to generate a bitstream, wherein the quantization unit sets two or more quantization modes, sets minimum and maximum decimal values ​​for each quantization mode, and performs quantization processing based on the minimum and maximum decimal values ​​corresponding to one of the quantization modes.

2. A feature map decoding device that decodes a bitstream in which a multi-scale feature map has been coded, comprising: a feature map internal decoding unit that decodes the bitstream and generates an integer-type packed feature frame; an unpacking unit that unpacks the integer-type packed feature frame to generate a multi-channel integer-type feature map; and an inverse quantization unit that converts elements of the multi-channel integer-type feature map from integer to decimal type to generate a multi-channel decimal-type feature map, wherein the quantization unit decodes two or more quantization modes, decodes the minimum and maximum decimal values ​​for each quantization mode, and performs inverse quantization processing based on the minimum and maximum decimal values ​​corresponding to one of the quantization modes.

3. A feature map coding device for coding multi-scale feature maps, comprising: a quantization unit that converts elements of a multi-channel decimal feature map from decimal to integer to generate a multi-channel integer feature map; a packing unit that packs the multi-channel integer feature map to generate a single integer-packed feature frame; and a feature map internal coding unit that encodes the integer-packed feature frame to generate a bitstream, wherein the quantization unit sets two or more quantization modes for coding, sets linear or nonlinear transformation parameters for each quantization mode for coding, and performs quantization processing based on the linear or nonlinear transformation corresponding to one of the quantization modes.

4. A feature map decoding device that decodes a bitstream in which a multi-scale feature map is encoded, comprising: a feature map internal decoding unit that decodes the bitstream and generates an integer-type packed feature frame; an unpacking unit that unpacks the integer-type packed feature frame to generate a multi-channel integer-type feature map; and an inverse quantization unit that converts elements of the multi-channel integer-type feature map from integer to decimal type to generate a multi-channel decimal-type feature map, wherein the quantization unit decodes two or more quantization modes, decodes linear or nonlinear transformation parameters for each quantization mode, and performs inverse quantization processing based on the linear or nonlinear transformation corresponding to one of the quantization modes.

5. A feature map coding device for coding multi-scale feature maps, comprising: a quantization unit that converts elements of a multi-channel decimal feature map from decimal to integer to generate a multi-channel integer feature map; a packing unit that packs the multi-channel integer feature map to generate a single integer-packed feature frame; and a feature map internal coding unit that encodes the integer-packed feature frame to generate a bitstream, wherein the quantization unit sets linear or nonlinear transformation parameters for coding, and performs quantization processing based on the linear or nonlinear transformation indicated by the parameters.

6. A feature map decoding device that decodes a bitstream in which a multi-scale feature map is encoded, comprising: a feature map internal decoding unit that decodes the bitstream and generates an integer-type packed feature frame; an unpacking unit that unpacks the integer-type packed feature frame to generate a multi-channel integer-type feature map; and an inverse quantization unit that converts elements of the multi-channel integer-type feature map from integer to decimal type to generate a multi-channel decimal-type feature map, wherein the quantization unit decodes linear or nonlinear transformation parameters and performs inverse quantization processing based on the linear or nonlinear transformation indicated by the parameters.

7. A feature map coding device for coding multi-scale feature maps, comprising: a quantization unit that converts elements of a multi-channel decimal feature map from decimal to integer to generate a multi-channel integer feature map; a packing unit that packs the multi-channel integer feature map to generate a single integer packed feature frame; and a feature map internal coding unit that encodes the integer packed feature frame to generate a bitstream, wherein the quantization unit sets minimum and maximum decimal values ​​for each channel and performs quantization processing based on the minimum and maximum decimal values ​​for each channel.

8. A feature map decoding device that decodes a bitstream in which a multi-scale feature map is encoded, comprising: a feature map internal decoding unit that decodes the bitstream and generates an integer-type packed feature frame; an unpacking unit that unpacks the integer-type packed feature frame to generate a multi-channel integer-type feature map; and an inverse quantization unit that converts elements of the multi-channel integer-type feature map from integer to decimal type to generate a multi-channel decimal-type feature map, wherein the quantization unit decodes minimum and maximum decimal values ​​for each channel and performs inverse quantization processing based on the minimum and maximum decimal values ​​for each channel.

9. A feature map coding device for coding multi-scale feature maps, comprising: a quantization unit that converts elements of a multi-channel decimal feature map from decimal to integer to generate a multi-channel integer feature map; a packing unit that packs the multi-channel integer feature map to generate a single integer-packed feature frame; and a feature map internal coding unit that encodes the integer-packed feature frame to generate a bitstream, wherein the quantization unit sets scaling coefficients and shift coefficients for each channel and performs quantization processing based on the decimal-type scaling coefficients and shift coefficients for each channel.

10. A feature map decoding device that decodes a bitstream in which a multi-scale feature map is encoded, comprising: a feature map internal decoding unit that decodes the bitstream and generates an integer-type packed feature frame; an unpacking unit that unpacks the integer-type packed feature frame to generate a multi-channel integer-type feature map; and an inverse quantization unit that converts elements of the multi-channel integer-type feature map from integer to decimal type to generate a multi-channel decimal-type feature map, wherein the quantization unit decodes scaling coefficients and shift coefficients for each channel and performs inverse quantization processing based on the scaling coefficients and shift coefficients for each channel.

Images