Feature map encoding device, feature map encoding method, feature map decoding device, and feature map decoding method

Abstract

To provide a device for efficiently encoding / decoding a feature map with a small processing amount.SOLUTION: In a feature map encoding device including a feature map reducing section, a feature map converting section, and a feature map internal encoding section, the feature map converting section 103 includes a range adjusting section that adjusts, for feature maps of a plurality of channels supplied from the feature map reducing section, feature amounts included in the feature maps on the basis of a parameter including a scaling coefficient and an offset coefficient, and a packing section that combines the feature maps of the plurality of channels into one frame to generate a packed feature frame, and the range adjusting section selects, for each element included in the parameter, whether encoding is performed in units of frames or in units of channels.SELECTED DRAWING: Figure 15

Description

[Technical Field]

[0001] Coding and decoding of feature maps in neural networks. [Background technology]

[0002] A well-known neural network technology used for image recognition, such as detecting objects at various scales in an image, dividing regions into individual objects, or tracking objects, is the FPN (Feature Pyramid Network) described in Non-Patent Document 1. In FPN, multiple feature maps at various scales are generated from the image to be processed, and various image recognitions are performed using the feature maps.

[0003] FPNs used for image recognition generate multiple feature maps from images, and their structure utilizes a CNN (Convolutional Neural Network). CNNs can be divided into a feature extraction section (backbone) that reads an image and generates a feature map using convolution and pooling, and a classification section (head) that is composed of hierarchical fully connected layers and generates output suitable for tasks such as object detection, instance segmentation, and object tracking. FPNs utilize the CNN backbone.

[0004] The feature extraction unit of the FPN is typically configured with a hierarchical structure in which a convolution process 301, an activation process 302, and a pooling process 303 shown in FIG. 3 are configured as one basic unit, and this basic unit is repeated.

[0005] Figure 4 shows the structure of the FPN. The FPN consists of a bottom-up processor 322 that uses a CNN backbone to generate a multi-scale feature map consisting of multiple hierarchical layers, and a top-down processor 324 that aggregates features from deep-layer feature maps to shallow-layer feature maps using the inverse structure of the CNN backbone. The bottom-up processor 322 reduces the resolution of the feature map by half by repeating the basic units of convolution 301, activation 302, and pooling 303 shown in Figure 3, generating a pyramid of feature maps with multiple layers. Meanwhile, the top-down processor 324 adds feature maps with resolutions corresponding to those of the bottom-up processor 322, increasing the resolution of the feature map to the same resolution as the input image, generating a pyramid of feature maps. In other words, the FPN generates multiple feature maps for each layer from the image 326 to be subjected to feature extraction processing.

[0006] The convolution processing unit 301 performs convolution processing on the data to be processed (image or feature map) using multiple types of predetermined filters (kernels). In the convolution processing in the convolution processing unit 301, predetermined filtering is performed on all of the data to be processed while sliding at predetermined intervals. The sliding interval is called the stride. The convolution processing unit 301 may determine the stride based on the number of data to be processed. For example, the convolution processing unit 301 may determine the stride to be 1 if the number of data to be processed is smaller than a predetermined value, and may determine the stride to be 2 if the number is equal to or greater than the predetermined value. Multiple feature maps are generated by preparing multiple predetermined filters at each layer and generating one feature map for each filter. The unit of the feature map is the channel. If the number (types) of predetermined filters is N (N types), N feature maps (N channels) are generated.

[0007] The activation processing unit 302 performs activation processing to nonlinearly convert the feature map output from the convolution processing unit 301. Here, the function used in the activation processing is called an activation function. The activation processing unit 302 uses a ReLU (Rectified Linear Unit) function, a sigmoid function, or the like as the activation function.

[0008] The pooling processing unit 303 is a process for downsampling the feature map output from the activation processing unit 302 by replacing local values ​​of the feature map with representative values.

[0009] On the other hand, when performing classification using a neural network, it has the ability to perform tasks using feature maps of multiple channels at each layer.

[0010] In image classification, the feature maps of multiple channels at each layer are convolved for each predetermined size based on the scale of the feature map, and the probability of the object class is calculated for each pixel. [Prior art documents] [Non-patent literature]

[0011] [Non-Patent Document 1] Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature Pyramid Networks for Object Detection. In CVPR, 2017. Summary of the Invention [Problem to be solved by the invention]

[0012] The amount of information in a feature map is enormous, making it unsuitable for transmission and storage. In view of the above problems, the present invention has an object to provide a technology for encoding and decoding feature maps. [Means for solving the problem]

[0013] In order to solve the above problem, a feature map coding device disclosed herein includes a range adjustment unit that adjusts feature amounts included in feature maps of multiple channels based on parameters including scaling coefficients and offset coefficients, and a packing unit that combines the feature maps of the multiple channels into a single frame to generate a packed feature frame, where the range adjustment unit selects whether to encode on a frame-by-frame basis or a channel-by-channel basis for each element included in the parameters.

[0014] Moreover, the feature map decoding device of the present disclosure includes an unpacking unit that extracts feature maps of multiple channels from a packing feature frame, and a range restoration unit that restores feature amounts included in the feature maps of the multiple channels based on parameters including a scaling coefficient and an offset coefficient, and the range restoration unit selects whether to perform decoding on a frame-by-frame basis or a channel-by-channel basis for each element included in the parameters. [Effects of the Invention]

[0015] According to the present invention, feature maps can be efficiently coded and decoded with a small amount of processing. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a block diagram illustrating the configuration of a feature map encoding device 100. [Figure 2] FIG. 2 is a block diagram illustrating the configuration of a feature map decoding device 200. [Figure 3] FIG. 1 is a block diagram for explaining the basic unit of processing in each layer of the FPN. [Figure 4] FIG. 1 is a block diagram illustrating the structure of an FPN. [Figure 5] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map reduction unit 102. [Figure 6] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map restoration unit 203. [Figure 7] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map conversion unit 103. [Figure 8] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map inverse conversion unit 202. [Figure 9] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map internal coding unit 104. [Figure 10] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map internal decoding unit 201. [Figure 11] FIG. 10 is a diagram for explaining the number of channels of feature maps x1, x2, and x3, the width of the feature maps, and the height of the feature maps. [Figure 12] FIG. 10 is a diagram illustrating a state in which feature maps of multiple channels are packed into one frame. [Figure 13] FIG. 10 is a diagram illustrating flipping when packing feature maps of multiple channels into one frame. [Figure 14] 1 is a diagram illustrating layers and units handled by a feature map encoding device and a feature map decoding device according to an embodiment of the present invention. FIG. [Figure 15] FIG. 10 is a block diagram for explaining another example of the feature map conversion unit 103 and the feature map inverse conversion unit 202. [Figure 16] 10 is an example of syntax that is encoded by the range adjustment unit 710 or decoded by the range restoration unit 810. [Figure 17] 10 is an example of syntax that is encoded by the range adjustment unit 710 or decoded by the range restoration unit 810. [Figure 18] 10 is a flowchart illustrating the operation of a range adjustment section 710. [Figure 19] 10 is an example of syntax that is encoded by the range adjustment unit 710 or decoded by the range restoration unit 810. [Figure 20] 10 is a flowchart illustrating the operation of a range adjustment section 710. [Figure 21]10 is a flowchart illustrating the operation of a range restoration unit 810. [Figure 22] 10 is a flowchart illustrating the operation of a range restoration unit 810. DETAILED DESCRIPTION OF THE INVENTION

[0017] The technologies and technical terms used in this embodiment will be defined below.

[0018] <Features and feature maps> In a convolutional neural network (CNN), a filter is used to scan the image (input layer data) from which features are extracted, and the results of the convolution operation between the data and the filter coefficients are used as features or a feature map.

[0019] <Packing> Frame packing refers to combining two or more frames (pictures) into a single frame (picture) by arranging them in a tiled pattern. In this application, packing refers to combining feature maps from multiple channels into a single frame. Figure 12 shows an example of frame packing.

[0020] <Data Type> A data type that represents an integer value is called an integer type, and a data type that represents a decimal value is called a decimal type.

[0021] <Layer, Unit> The layers and units handled by the feature map encoding device and feature map decoding device of this embodiment will be described using Figure 14. Reference numeral 181 in Figure 14 denotes a sequence of consecutive feature frames or feature maps for all channels over time, referred to as a sequence layer or sequence unit. Reference numeral 182 in Figure 14 denotes a sequence of consecutive feature maps for one channel over time, referred to as a sequence layer for each channel or a sequence unit for each channel. Reference numeral 183 in Figure 14 denotes a feature frame or feature maps for all channels at the same time, referred to as a frame layer or frame unit. Reference numeral 184 in Figure 14 denotes a feature map for one channel at a certain time, referred to as a feature map layer for each channel (a feature map unit for each channel).

[0022] (First embodiment) A feature map encoding device 100 and a feature map decoding device 200 according to a first embodiment of the present invention will be described.

[0023] 1 is a block diagram of a feature map encoding device 100 according to a first embodiment. The feature map encoding device 100 according to the embodiment includes a feature map reduction unit 102, a feature map conversion unit 103, and a feature map internal encoding unit 104. The feature map encoding device 100 encodes a feature map generated by a neural network feature extraction unit 101 to generate and output a bitstream.

[0024] The neural network feature extraction unit 101 reads an image from which features are to be extracted, generates a feature map by FPN convolution processing, activation processing, and pooling processing, and supplies the generated feature map to the feature map reduction unit 102. In this embodiment, a three-layer multi-scale feature map of x1, x2, and x3 is generated.

[0025] The feature map reduction unit 102 converts the three-layer multi-scale feature maps x1, x2, and x3 acquired from the neural network feature extraction unit 101 into a single-layer single-scale feature map xf, and supplies it to the feature map conversion unit 103. The feature map reduction unit 102 will be described in detail with reference to FIG. 5.

[0026] The feature map conversion unit 103 performs packing and quantization on the decimal-type single-scale feature map xf supplied from the feature map reduction unit 102, converting it into an integer-type packed feature frame, and supplies it to the feature map internal coding unit 104.

[0027] The feature map conversion unit 103 will be described in detail with reference to FIG.

[0028] The feature map intra-encoding unit 104 encodes the integer-type packing feature frames supplied from the feature map conversion unit 103 in accordance with an image encoding standard such as VVC, HEVC, or AV1 to generate and output a bitstream. The output bitstream is supplied to the feature map decoding device 200 or the like via a network or the like.

[0029] The feature map internal coding unit 104 will be described in detail with reference to FIG.

[0030] 2 is a block diagram showing the configuration of a feature map decoding device 200 according to an embodiment of the present invention, which corresponds to the feature map encoding device 100 in FIG. 1. The feature map decoding device 200 according to this embodiment includes a feature map internal decoding unit 201, a feature map inverse conversion unit 202, and a feature map restoration unit 203. The feature map decoding device 200 receives as input a bit stream encoded by the feature map encoding device 100, decodes the bit stream, and generates three-layer multi-scale feature maps x1up, x2up, and x3up, which it supplies to a neural network classification unit 204.

[0031] The feature map internal decoding unit 201 decodes the bitstream coded by the feature map internal coding unit 104 of the feature map coding device 100 in accordance with an image coding standard such as VVC, HEVC, or AV1, generates integer-type packing feature frames, and supplies them to the feature map inverse conversion unit 202.

[0032] The feature internal decoding unit 201 will be described in detail with reference to FIG.

[0033] The feature map inverse conversion unit 202 performs inverse quantization and unpacking on the integer-type packed feature frame supplied from the feature internal decoding unit 201, converting it into a decimal-type single-scale feature map xr, and supplies it to the feature map restoration unit 203.

[0034] The feature map inverse conversion unit 202 will be described in detail with reference to FIG.

[0035] The feature map restoration unit 203 converts the single-scale feature map xr supplied from the feature map inverse conversion unit 202 into three-layer multi-scale feature maps x1up, x2up, and x3up, and supplies them to the neural network identification unit 204 as the output of the feature map decoding device 200.

[0036] The feature map restoration unit 203 will be described in detail with reference to FIG.

[0037] The neural network classification unit 204 performs classification processing such as classification of objects, places, landscapes, people, and living things in the classification target image based on the three-layer multi-scale feature maps x1up, x2up, and x3up supplied by the feature map restoration unit 203.

[0038] <Feature map reduction and restoration> The feature map reduction unit 102 has a function of converting the multi-layered multi-scale feature map acquired from the neural network feature extraction unit 101 into a single-layered single-scale feature map.

[0039] The feature map reduction unit 102 will be described in detail using Figure 5. The feature map reduction unit 102 is composed of a first feature map reduction unit 501, a first channel combination unit 502, a second feature map reduction unit 503, a second channel combination unit 504, a third feature map reduction unit 505, a first padding unit 506, a second padding unit 507, and a third padding unit 508. The feature map reduction unit 102 in Figure 5 is an example of a configuration that converts a three-layer multi-scale feature map into a single-scale feature map.

[0040] The feature map reduction unit 102 receives three-layer multi-scale feature maps, namely, a first feature map x1, a second feature map x2, and a third feature map x3, as input, converts them into a single-layer single-scale feature map xf, and supplies the result to the feature map conversion unit 103. Here, n denotes an index indicating a layer, Cn denotes the number of channels in the nth layer, Wn denotes the width of the feature map, and Hn denotes the height of the feature map. In this embodiment, the values ​​of Cn, Wn, and Hn for each layer are as shown in FIG. 11, where H and W denote the width and height, respectively, of the image from which features are extracted.

[0041] The first padding unit 506 has a function of padding the first feature map x1 to generate a first padded feature map x1pad. The first padding unit 506 determines the padding size so that the width and height of x1pad are multiples of 64. The number of channels of x1pad is 256, the same as that of x1.

[0042] The second padding unit 507 performs padding on the second feature map x2 by folding to generate a second padded feature map x2pad. The second padding unit 507 determines the padding size so that the width and height of x2pad are multiples of 32. The number of channels of x2pad is 256, the same as that of x2.

[0043] The third padding unit 508 performs padding on the third feature map x3 by folding to generate a third padded feature map x3pad. The third padding unit 508 determines the padding size so that the width and height of x3pad are multiples of 16. The number of channels in x3pad is 256, the same as that of x3.

[0044] In the first padding unit 506, the second padding unit 507, and the third padding unit 508, the left padding size is the same as the right padding size, and the top padding size is the same as the bottom padding size. That is, the feature maps x1, x2, and x3 are positioned at the centers of x1pad, x2pad, and x3pad, respectively.

[0045] The first feature map reduction unit 501 performs convolution in the spatial and channel directions on the first padded feature map x1pad obtained from the first padding unit 506 to generate a first intermediate feature map y1. The number of channels of y1 is 192, its width is Wx1pad / 2, and its height is Hx1pad / 2, where Wx1pad and Hx1pad are the width and height of the first padded feature map x1pad, respectively.

[0046] The first channel combining unit 502 has a function of combining the first intermediate feature map y1 obtained from the first feature map reduction unit 501 and the second padded feature map x2pad obtained from the second padding unit 507 in the channel direction to generate an intermediate feature map y1Cx2pad. Since y1 has 192 channels and x2pad has 256 channels, the intermediate feature map y1Cx2pad has 448 channels (192 + 256).

[0047] The second feature map reduction unit 503 generates a second intermediate feature map y2 by performing convolution in the spatial and channel directions on the intermediate feature map y1Cx2pad obtained from the first channel combination unit 502. The number of channels of y2 is 192, its width is Wy1Cx2pad / 2, and its height is Hy1Cx2pad / 2, where Wy1Cx2pad and Hy1Cx2pad are the width and height of the intermediate feature map y1Cx2pad, respectively.

[0048] The second channel combining unit 504 has a function of combining the second intermediate feature map y2 obtained from the second feature map reduction unit 503 and the third padded feature map x3pad obtained from the third padding unit 508 in the channel direction to generate an intermediate feature map y2Cx3pad. Since the number of channels in the intermediate feature map y2 is 192 and the number of channels in x3pad is 256, the number of channels in y1Cx2pad is 448 (192 + 256).

[0049] The third feature map reduction unit 505 generates a third intermediate feature map y3 by performing convolution in the spatial and channel directions on the intermediate feature map y2Cx3pad obtained from the second channel combination unit 504. The number of channels of y3 is 192, its width is Wy2Cx3pad / 2, and its height is Hy2Cx3pad / 2, where Wy2Cx3pad and Hy2Cx3pad are the width and height of the intermediate feature map y2Cx3pad, respectively.

[0050] The feature map reduction unit 102 outputs the third intermediate feature map y3 as a single-scale feature map xf and supplies it to the feature map conversion unit 103.

[0051] The feature map restoration unit 203 has a function of converting the single-scale feature map xr acquired from the feature map inverse conversion unit 202 into three-layer multi-scale feature maps x1up, x2up, and x3up.

[0052] The feature map restoration unit 203 will be described in detail using Fig. 6. The feature map restoration unit 203 is composed of an 8x enlargement unit 601, a 4x enlargement unit 602, a 2x enlargement unit 603, a first feature map blending unit 604, a second feature map blending unit 605, a first de-padding unit 606, a second de-padding unit 607, and a third de-padding unit 608.

[0053] The 8x enlargement unit 601 performs transposition convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr obtained from the feature map inverse transformation unit 202 to enlarge the feature map and reduce the channels, thereby generating an intermediate feature map z1. The number of channels in z1 is 196. If the width and height of the single-scale feature map xr are xrwidth and xrheight, respectively, the width and height of z1 are xrwidth × 8 and xrheight × 8, respectively. Here, rwidth × 8 and xrheight × 8 are the same as the width and height, respectively, of the first padded feature map x1pad, which is the output of the first padding unit 506 of the feature extraction unit 102.

[0054] The 4x enlargement unit 602 performs transposition convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr obtained from the feature map inverse transformation unit 202 to enlarge the features and reduce the channels, generating an intermediate feature map z2. The number of channels in z2 is 196. The width and height of z2 are xrwidth×4 and xrheight×4, respectively. Here, rwidth×4 and xrheight×4 are the same as the width and height, respectively, of the second padded feature map x2pad, which is the output of the second padding unit 507 of the feature extraction unit 102.

[0055] The 2x enlargement unit 603 performs transposition convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr obtained from the feature map inverse transformation unit 202 to enlarge the feature map and reduce the channels, thereby generating an intermediate feature map z3. The number of channels in z3 is 196. The width and height of z3 are xrwidth×2 and xrheight×2, respectively. Here, rwidth×2 and xrheight×2 are the same as the width and height, respectively, of the third padded feature map x3pad, which is the output of the third padding unit 508 in the feature extraction unit 102.

[0056] The first feature map mixing unit 604 has a function of generating an intermediate feature map z2up by improving the quality of the intermediate feature map z2 obtained from the 4x enlargement unit 602, using the intermediate feature map z1 obtained from the 8x enlargement unit 601.

[0057] The second feature map mixing unit 605 has a function of generating an intermediate feature map z3up that improves the quality of the intermediate feature map z3 obtained from the 2x enlargement unit 603, by using the intermediate feature map z2up obtained from the first feature map mixing unit 604.

[0058] The padding removal unit 606 removes padding from the intermediate feature map z1 acquired from the 8x enlargement unit 601 to generate a first output feature map x1up. The width and height of x1up are the same as the width and height of the first feature map x1 input to the feature extraction unit 102.

[0059] The padding removal unit 607 has a function of removing padding from the intermediate feature map z2up acquired from the first feature map blending unit 604 to generate a second output feature map x2up. The width and height of x2up are the same as the width and height of the second feature map x2 input to the feature extraction unit 102.

[0060] The padding removal unit 608 has a function of removing padding from the intermediate feature map z3up acquired from the second feature map blending unit 605 to generate a third output feature map x3up. The width and height of x3up are the same as the width and height of the third feature map x3 input to the feature extraction unit 102.

[0061] In the first padding removal unit 606, the second padding removal unit 607, and the third padding removal unit 608, the left padding size is the same as the right padding size, and the top padding size is the same as the bottom padding size, similar to the first padding unit 506, the second padding unit 507, and the third padding unit 508 of the feature extraction unit 102. In other words, the padding on the top, bottom, left, and right sides is removed by assuming that each output feature map x1up, x2up, and x3up is placed at the center of each intermediate feature map z1, z2up, and z3up, respectively.

[0062] <Feature map transformation and feature map inverse transformation> The feature map conversion unit 103 has a function of performing packing and quantization on the multi-channel decimal-type single-scale feature map xf supplied from the feature map reduction unit 102, and converting it into an integer-type packed feature frame to be supplied to the feature map internal coding unit 104.

[0063] The feature map conversion unit 103 on the encoding side will be described in detail with reference to Fig. 7. The feature map conversion unit 103 is made up of a packing unit 701 and a feature map quantization unit 702.

[0064] The packing unit 701 has a function of generating a packed feature frame by combining input feature maps of multiple channels into one frame. Fig. 12 is a diagram explaining the state in which feature maps of multiple channels are packed into one frame. The feature maps of each channel are sequentially arranged in one frame in raster scan order, from left to right and top to bottom.

[0065] Furthermore, the packing unit 701 performs flipping based on the position in the frame where the channel feature map is placed. Whether or not to perform flipping can be selected, and the selection may be transmitted from the encoding side to the decoding side via a bitstream.

[0066] Flipping is performed when packing feature maps of multiple channels into one frame, as shown in FIG. 13. In flipping, the positions of the elements (pixels) of each channel's feature map are flipped horizontally (left and right), vertically (up and down), or horizontally and vertically (up and down, left and right) when packing the feature maps of each channel. The feature maps of four channels, A (upper left), B (upper right), C (lower left), and D (lower right) in FIG. 13, are grouped together as one set. No flipping is performed at position A in FIG. 13. At position B, the feature map is flipped horizontally (left and right). At position C, the feature map is flipped vertically (up and down). At position D, the feature map is flipped horizontally and vertically (up and down, left and right). When the distribution of elements in the feature maps of each channel is similar, flipping based on the position within the frame where the channels are placed reduces the boundaries between the feature maps of each channel, improving coding efficiency.

[0067] The feature map quantization unit 702 has a function of converting elements of a decimal-type packing feature frame (feature map of all channels) into N-bit integer type elements (N=an integer from about 8 to 16) within a predetermined range, and outputting an integer-type packing feature frame. In this embodiment, the elements are converted into 10-bit integer type elements ranging from 0 to 1023. The feature map quantization unit 702 detects the minimum and maximum values ​​of the elements of the decimal-type packing feature frame, and transmits the detected minimum and maximum values ​​of the elements of the packing feature frame (feature map of all channels) to the decoding side as metadata. To convert from a decimal-type packing feature frame (feature map) to an integer-type packing feature frame (feature map), a linear conversion is performed in which the minimum value of the decimal type is equivalent to the minimum value of the integer type, and the maximum value of the decimal type is equivalent to the maximum value of the integer type. For example, when the integer type range is expressed in 10 bits, the minimum value of the elements of the integer-type packing feature frame (feature map) is 0 and the maximum value is 1023 (2 10 -1). Values ​​between the minimum and maximum values ​​are linearly quantized.

[0068] Next, the feature map inverse transform unit 202 has a function of performing inverse quantization and unpacking on the integer-type packing feature frame decoded by VVC, HEVC, AV1, etc., supplied from the feature map internal decoding unit 201, and performing inverse transform into a decimal-type single-scale feature map xr to be supplied to the feature map restoration unit 203.

[0069] The feature map inverse transformation unit 202 on the decoding side will be described in detail with reference to Fig. 8. The feature map inverse transformation unit 202 performs the inverse process of the feature map transformation unit 103, and is made up of a feature map inverse quantization unit 801 and an unpacking unit 802.

[0070] The feature map inverse quantization unit 801 performs the inverse process of the feature map quantization unit 702 on the encoding side, and has a function of converting elements of integer-type packing feature frames from integer type to decimal type. The feature map inverse quantization unit 801 converts the integer-type packing feature frame decoded by the feature map internal decoding unit 201 into a decimal-type packing feature frame using the minimum and maximum decimal values ​​transmitted as metadata. A linear conversion is performed to convert the integer minimum value to the decimal minimum value and the integer maximum value to the decimal maximum value. Linear inverse quantization is performed on values ​​between the minimum and maximum values.

[0071] The unpacking unit 802 extracts a feature map for each channel from the packing feature frames arranged in one frame in raster scan order and supplies it to the feature map restoration unit 203 as a single-scale feature map xr.

[0072] <Feature map inner encoding and decoding> The feature map internal coding unit 104 will be described in detail using Figure 9. The feature map internal coding unit 104 is made up of a switch 901, a VVC encoding unit 902, an HEVC encoding unit 903, and an AV1 encoding unit 904. The switch 901 selects the encoding standard for internally coding the feature map converted by the feature map conversion unit 103. The VVC encoding unit 902 encodes the feature map in the VVC standard and outputs a bitstream that complies with the VVC standard. The HEVC encoding unit 903 encodes the feature map in the HEVC standard and outputs a bitstream that complies with the HEVC standard. The AV1 encoding unit 904 encodes the feature map in the AV1 standard and outputs a bitstream that complies with the AV1 standard.

[0073] In the VVC, HEVC, and AV1 standards, images are divided into blocks of a specified size and then encoded.

[0074] It is also possible to implement only one of VVC, HEVC, and AV1, and it is also possible to use image coding methods other than VVC, HEVC, and AV1.

[0075] Next, the feature map internal decoding unit 201 will be described in detail with reference to Fig. 10. The feature map internal decoding unit 201 is made up of a switch 1001, a VVC decoding unit 1002, an HEVC decoding unit 1003, and an AV1 decoding unit 1004. The switch 1001 selects the encoding standard for internal decoding based on information for selecting internal decoding in the input bitstream. The VVC decoding unit 1002 decodes the feature map in accordance with the VVC standard. The HEVC decoding unit 1003 decodes the feature map in accordance with the HEVC standard. The AV1 encoding unit 1004 decodes the feature map in accordance with the AV1 standard.

[0076] In the VVC, HEVC, and AV1 standards, decoding is performed for each predetermined block size.

[0077] It is also possible to implement only one of VVC, HEVC, and AV1, and it is also possible to use image coding methods other than VVC, HEVC, and AV1.

[0078] <Feature map range adjustment> 15(a) is a block diagram illustrating another example of the feature map conversion unit 103. The feature map conversion unit 103 is made up of a range adjustment unit 710, a packing unit 701, and a feature map quantization unit 702.

[0079] The range adjustment unit 710 adjusts, as described below, the values ​​of feature quantities included in the channel-by-channel feature maps supplied from the feature map reduction unit 102. The packing unit 701 combines the feature maps of multiple channels supplied from the range adjustment unit 710 into one frame and generates the result as a packing feature frame. The feature map quantization unit 702 converts the decimal-type packing feature frame supplied from the packing unit 701 into an integer-type packing feature frame.

[0080] The range adjustment unit 710 adjusts the values ​​by multiplying all values ​​of the feature quantity in a specified channel by a specified scaling coefficient and adding a specified offset coefficient. When the range adjustment unit 710 adjusts the values, the range of values ​​included in the specified channel is also adjusted. In other words, the range adjustment unit 710 adjusts the distribution of values. The range adjustment unit 710 adjusts the values ​​for each of multiple channels.

[0081] Hereinafter, the scaling coefficient and offset coefficient used by the range adjustment unit 710 to adjust the value of the feature amount will be collectively referred to as parameters.

[0082] The range adjustment unit 710 may set the parameters to predetermined values ​​or to pre-stored values. The range adjustment unit 710 may set the parameters based on feature maps of multiple channels included in one frame generated as a packing feature frame by the packing unit 701. The range adjustment unit 710 may set the parameters based on statistical values ​​of feature quantities of multiple channels. The range adjustment unit 710 may select a frame or multiple channels as a reference unit. The range adjustment unit 710 may calculate statistical values ​​of feature quantities in channel units and in a reference unit, and set parameters for each channel based on the ratio or difference between the statistical values ​​in channel units and the reference unit. The range adjustment unit 710 may calculate an average value or variance of feature quantities in channel units, and use these as statistical values ​​for the channel units. The range adjustment unit 710 may calculate an average value or variance of feature quantities in reference units, and use these as statistical values ​​for the reference unit. The range adjustment unit 710 may set the parameters so that the statistical values ​​for each channel approximately match the statistical values ​​for the reference unit. The range adjustment unit 710 may set an offset coefficient based on the average value per channel and the average value in the reference unit. The range adjustment unit 710 may set an offset coefficient based on the difference between the average value per channel and the average value in the reference unit. The range adjustment unit 710 may set a scaling coefficient based on the variance value per channel and the variance value in the reference unit. The range adjustment unit 710 may set a scaling coefficient based on the ratio between the variance value per channel and the variance value in the reference unit.

[0083] The range adjustment unit 710 adjusts parameters so that the range of values ​​included in the feature map for the first channel approximately matches the range of values ​​included in the feature map for the second channel. This results in the distribution of the range of values ​​included in the feature maps for each channel approximately matching, improving prediction efficiency. Therefore, the feature map encoding device of this embodiment has the effect of improving encoding efficiency.

[0084] Any one of the range adjustment unit 710, the feature map conversion unit 103, the internal encoding unit 104 of the feature map, and the feature map encoding device 100 encodes a syntax including parameters. The syntax will be described later. The syntax encoded by the range adjustment unit 710 may be determined, and the syntax may be encoded by a configuration other than the range adjustment unit 710. Hereinafter, an example in which the range adjustment unit 710 encodes a syntax including parameters as metadata in units of frames and transmits it to the decoding side will be described. The range adjustment unit 710 may repeat the following frame unit processing until the sequence ends. The range adjustment unit 710 may be configured to encode a part of the syntax.

[0085] <Syntax> FIG. 16 is an example of the syntax 1600 encoded by the range adjustment unit 710. In the syntax 1600, channel_size represents the number of channels of the feature map, scale_coef represents the scaling coefficient, and offset_coef represents the offset coefficient. Here, a parameter with [i] attached indicates that it is a value per channel. Note that the description "for(i = 0; i < M; i++)" indicates that with the initial value i = 0, while satisfying the condition i < maximum value M, the syntax enclosed in square brackets is processed and 1 is added to i. Also, the square brackets indicate a predetermined unit for processing. When using this syntax 1600, the range adjustment unit 710 encodes the number of channels, the scaling coefficient, and the offset coefficient for each channel in units of frames.

[0086] <Modified Example 1 of Syntax> An example different from the syntax 1600 in Fig. 16 will be described. Fig. 17 shows an example of syntax 1700 encoded by the range adjustment unit 710. In syntax 1700, coding_skip_flag represents a flag indicating whether to skip parameter encoding, and scale_frame_flag represents a flag indicating whether to encode scaling coefficients on a frame-by-frame basis. Here, parameters marked with [i] represent channel-by-channel values, and parameters not marked with [i] represent frame-by-frame values. Note that the notation "if(x)" indicates that the syntax enclosed in curly brackets is processed if x is true, and indicates that if "else" is present and x is false, the syntax enclosed in curly brackets below it is processed, and "!x" indicates the negation of x.

[0087] A case where this syntax 1700 is used will be described with reference to FIG. 18. First, the range adjustment unit 710 determines whether or not to skip encoding of parameters on a frame-by-frame basis (S1002). The range adjustment unit 710 may set the scaling coefficient to 1 and the offset coefficient to 0 at the beginning of processing or the beginning of a sequence. The range adjustment unit 710 determines the value as true if the same parameters as those of a frame preceding the frame to be processed (hereinafter referred to as the frame to be processed) are used. When the range adjustment unit 710 calculates parameters for the frame to be processed, if the parameters are substantially the same as those of the previous frame, it may adjust the values ​​of the frame to be processed using the parameters of the previous frame and determine to skip encoding of the parameters. The range adjustment unit 710 encodes coding_skip_flag, which indicates whether or not to skip encoding of parameters (S1002).

[0088] When coding_skip_flag is true (S1004: YES), the range adjustment unit 710 does not encode the parameters. When coding_skip_flag is false (S1004: NO), the range adjustment unit 710 selects the unit for encoding the parameters (S1006). The range adjustment unit 710 may select whether to encode each element included in the parameters frame by frame or channel by channel. For example, the range adjustment unit 710 may select to encode the scaling coefficients of the parameters frame by frame and the offset coefficients of the parameters channel by channel. Alternatively, the range adjustment unit 710 may select to encode the scaling coefficients of the parameters channel by channel and the offset coefficients of the parameters frame by frame. The range adjustment unit 710 may encode one of the scaling coefficients and the offset coefficients frame by frame and the other channel by channel. In other words, the range adjustment unit 710 may encode the scaling coefficients and the offset coefficients in the same or different units. The range adjustment unit 710 may select whether the scaling coefficient and the offset coefficient are coded in the same or different units on a frame-by-frame basis or on a sequence-by-sequence basis. The range adjustment unit 710 may code the scaling coefficient and the offset coefficient in the same or different units on a sequence-by-sequence basis. Here, it is assumed that the range adjustment unit 710 codes the scaling coefficient and the offset coefficient in different units on a sequence-by-sequence basis. In other words, the units in which the range adjustment unit 710 codes the parameters are always different between the scaling coefficient and the offset coefficient.

[0089] The range adjustment unit 710 may select a unit for encoding the parameters based on the feature map. The range adjustment unit 710 may compare a per-channel statistical value with a reference unit statistical value to select a unit for encoding the parameters. The range adjustment unit 710 may calculate a difference between an average value included in the per-channel statistical value and an average value included in the reference unit statistical value as a first evaluation value, calculate a ratio between a variance value included in the per-channel statistical value and an variance value included in the reference unit statistical value as a second evaluation value, and select a unit for encoding the parameters based on the first evaluation value and the second evaluation value. The range adjustment unit 710 may compare the first evaluation value with the second evaluation value, and if the influence of the first evaluation value is large, select to encode the offset coefficients on a per-channel basis and the scaling coefficients on a per-frame basis. The range adjustment unit 710 may compare the first evaluation value with the second evaluation value, and if the influence of the second evaluation value is large, select to encode the scaling coefficients on a per-channel basis and the offset coefficients on a per-frame basis.

[0090] The range adjustment unit 710 encodes scale_frame_flag, which indicates whether or not the scaling coefficient is to be encoded on a frame-by-frame basis (S1006).

[0091] The range adjustment unit 710 encodes channel_size, which indicates the number of channels (S1008).

[0092] If scale_frame_flag is true (S1010: YES), the range adjustment unit 710 encodes the scaling coefficients on a frame-by-frame basis and encodes the offset coefficients on a channel-by-channel basis (S1012).If scale_frame_flag is false (S1010: NO), the range adjustment unit 710 encodes the scaling coefficients on a channel-by-channel basis and encodes the offset coefficients on a frame-by-frame basis (S1014).

[0093] The range adjustment unit 710 selects the unit for encoding the parameters. Preferably, the range adjustment unit 710 selects whether to encode the parameters frame by frame or channel by channel, for each element included in the parameters. This reduces the amount of code required for syntax including parameters related to feature adjustment, while improving prediction efficiency. Therefore, the feature map encoding device of this embodiment effectively improves encoding efficiency.

[0094] The range adjustment unit 710 does not need to encode the coding_skip_flag that indicates whether or not to skip parameter encoding. In other words, the range adjustment unit 710 may omit the processes of S1002 and S1004 in Fig. 18. The range adjustment unit 710 may encode the syntax for each predetermined frame.

[0095] The range adjustment unit 710 may encode a flag indicating whether or not to encode the parameters on a frame-by-frame basis, instead of scale_frame_flag indicating whether or not to encode the scaling coefficients on a frame-by-frame basis. That is, the range adjustment unit 710 may encode a flag indicating whether or not to encode both the scaling coefficients and the offset coefficients on a frame-by-frame basis. The range adjustment unit 710 may encode both the scaling coefficients and the offset coefficients on a frame-by-frame basis, or may encode both on a channel-by-channel basis.

[0096] Range adjustment section 710 may not encode channel_size indicating the number of channels, and may omit the process of S1008 in Fig. 18. A component other than range adjustment section 710 may encode channel_size.

[0097] <Syntax variation 2> An example different from syntax 1600 in Fig. 16 and syntax 1700 in Fig. 17 will be described. Fig. 19 shows an example of syntax 1900 encoded by range adjustment unit 710. In syntax 1900, offset_frame_flag is a flag indicating whether to encode the offset coefficient on a frame-by-frame basis, param_precision is information indicating the precision of the scaling coefficient scale_coef and offset coefficient offset_coef included in the parameters, select_channel_flag is a flag indicating whether to specify a channel for the parameters, coding_channel_size is the number of channels for encoding the parameters, scale_channel_id is the channel number for the scaling coefficients, and offset_channel_id is the channel number for the offset coefficients. Note that the notation "if(x || y)" indicates that the syntax enclosed in curly brackets is processed if at least one of x and y is true.

[0098] A case where this syntax 1900 is used will be described with reference to Fig. 20. The range adjustment unit 710 may perform the above-mentioned processing for some of the syntax included in the syntax 1700. First, the range adjustment unit 710 determines whether or not to skip parameter coding on a frame-by-frame basis, and encodes coding_skip_flag indicating whether or not to skip parameter coding (S1102).

[0099] If coding_skip_flag is true (S1104: YES), the range adjustment unit 710 does not encode the parameters. If coding_skip_flag is false (S1104: NO), the range adjustment unit 710 selects the unit for encoding the parameters (S1106). Now, it is assumed that the range adjustment unit 710 encodes the scaling coefficient and the offset coefficient in different units on a frame-by-frame basis. In other words, the range adjustment unit 710 changes the unit for encoding the parameters for each frame. The range adjustment unit 710 encodes scale_frame_flag, which indicates whether the scaling coefficient is to be encoded on a frame-by-frame basis, and offset_frame_flag, which indicates whether the offset coefficient is to be encoded on a frame-by-frame basis (S1106).

[0100] The range adjustment unit 710 encodes param_precision, which indicates the precision of the parameters (S1108). The range adjustment unit 710 may encode the number of bits of the scaling coefficients and offset coefficients included in the parameters as the precision of the parameters. The range adjustment unit 710 may set the precision of the parameters and a specified precision, and encode the difference between the precision of the parameters and the specified precision. For example, the range adjustment unit 710 may set the precision of the parameters to 10 bits, the specified precision to 8 bits, and encode the difference, 2. The range adjustment unit 710 encodes the parameters based on the precision of the parameters. For example, the range adjustment unit 710 may set the specified precision to 8 bits, and encode the parameter precision as 1 bit if the parameter to be encoded is in the range from 0 to 511 when multiplied by 2 to the power of 10. The range adjustment unit 710 may set the precision of the parameters based on the feature amount included in the frame to be processed. For example, the range adjustment unit 710 may set the specified precision to 8 bits, and encode the parameter precision as 1 bit if the feature amount contained in the frame to be processed is in the range of 0 to 511 when multiplied by 2 to the power of 10.

[0101] If at least one of scale_frame_flag and offset_frame_flag is false (S1110: YES), range adjustment unit 710 encodes select_channel_flag, which indicates whether to specify a channel for the parameter, and coding_channel_size, which indicates the number of channels for encoding the parameter (S1112).

[0102] If scale_frame_flag is true (S1114: YES), the range adjustment unit 710 encodes the scaling coefficients on a frame-by-frame basis (S1116). If scale_frame_flag is false (S1114: NO) and select_channel_flag is true (S1118: YES), the range adjustment unit 710 encodes the channel number scale_channel_id of the scaling coefficients (S1120). If scale_frame_flag is false (S1114: NO), the range adjustment unit 710 encodes the scaling coefficients on a channel-by-channel basis (S1122). That is, if select_channel_flag is true, the range adjustment unit 710 encodes the scaling coefficients of the channel specified by scale_channel_id, and if select_channel_flag is false, the range adjustment unit 710 encodes the scaling coefficients of all channels. The range adjustment unit 710 can reduce the amount of code by encoding the scaling coefficients by specifying some of the channels included in the frame.

[0103] If offset_frame_flag is true (S1124: YES), the range adjustment unit 710 encodes the offset coefficient on a frame-by-frame basis (S1126). If offset_frame_flag is false (S1124: NO) and select_channel_flag is true (S1128: YES), the range adjustment unit 710 encodes the channel number offset_channel_id of the offset coefficient (S1130). If offset_frame_flag is false (S1124: NO), the range adjustment unit 710 encodes the offset coefficient on a channel-by-channel basis (S1132). That is, if select_channel_flag is true, the range adjustment unit 710 encodes the offset coefficient of the channel specified by offset_channel_id, and if select_channel_flag is false, the range adjustment unit 710 encodes the offset coefficient of all channels. The range adjustment unit 710 can reduce the amount of code by encoding the offset coefficient by specifying some of the channels included in the frame.

[0104] The range adjustment unit 710 selects the unit for encoding the parameters. Preferably, the range adjustment unit 710 selects whether to encode the parameters frame by frame or channel by channel, for each element included in the parameters. More preferably, the range adjustment unit 710 encodes the precision of the parameters to reduce the amount of coding for the parameters. Even more preferably, the range adjustment unit 710 encodes the parameters by specifying a channel number to reduce the amount of coding for the parameters. This makes it possible to improve prediction efficiency while suppressing the amount of coding for syntax including parameters related to feature adjustment. Therefore, the feature map encoding device of this embodiment has the effect of improving encoding efficiency.

[0105] The range adjustment unit 710 does not need to encode the coding_skip_flag that indicates whether or not to skip parameter encoding. In other words, the range adjustment unit 710 may omit the processes of S1102 and S1104 in Fig. 20. The range adjustment unit 710 may encode parameters for each predetermined frame.

[0106] The range adjustment unit 710 may encode the precision of the scaling coefficient (scale_coef) and the precision of the offset coefficient (offset_coef) included in the parameters, instead of the precision of the scaling coefficient (scale_coef) and the offset coefficient (offset_coef). Alternatively, the range adjustment unit 710 may not encode the precision of the scaling coefficient (scale_coef) and the offset coefficient (offset_coef), and may omit the process of S1108 in FIG. 20 .

[0107] Instead of encoding select_channel_flag, which indicates whether or not to specify a channel for the parameters, the range adjustment unit 710 may encode a flag indicating whether or not to specify a channel for the scaling coefficient and a flag indicating whether or not to specify a channel for the offset coefficient.

[0108] When select_channel_flag, which indicates whether a parameter channel is specified, is true, the range adjustment unit 710 may encode the scaling coefficients of channels not specified by the scaling coefficient channel number scale_channel_id.When select_channel_flag, which indicates whether a parameter channel is specified, is true, the range adjustment unit 710 may encode the offset coefficients of channels not specified by the offset coefficient channel number offset_channel_id.

[0109] When select_channel_flag, which indicates whether or not a parameter channel is specified, is false, range adjustment unit 710 may not encode coding_channel_size, which indicates the number of channels for encoding the parameters. In other words, range adjustment unit 710 may omit the coding_channel_size encoding process of S1112 in FIG. 20 .

[0110] Range adjustment unit 710 may not encode select_channel_flag, which indicates whether a parameter channel is specified. In this case, range adjustment unit 710 may not encode the number of channels for encoding parameters, coding_channel_size, the channel number of the scaling coefficient, scale_channel_id, and the channel number of the offset coefficient, offset_channel_id. In other words, range adjustment unit 710 may omit encoding select_channel_flag in S1112 and the processes of S1118, S1120, S1128, and S1130 in FIG. 20 .

[0111] Range adjustment unit 710 may encode the channel number of the parameter instead of encoding the channel number of the scaling coefficient, scale_channel_id, and the channel number of the offset coefficient, offset_channel_id. That is, range adjustment unit 710 may encode the channel number of the parameter in S1112 of Fig. 20 and omit the processes of S1118, S1120, S1128, and S1130. Range adjustment unit 710 may encode the scaling coefficient of the channel specified by the channel number of the parameter in S1122, and encode the offset coefficient of the channel specified by the channel number of the parameter in S1132.

[0112] The range adjustment unit 710 does not need to encode the coding_channel_size, which indicates the number of channels for encoding parameters. That is, the range adjustment unit 710 may omit the coding_channel_size encoding process in S1112 of Fig. 20. A component other than the range adjustment unit 710 may encode the number of all channels in the feature map.

[0113] <Range recovery of feature maps> 15(b) is a block diagram illustrating another example of the feature map inverse transformation unit 202. The feature map inverse transformation unit 202 performs the inverse process of the feature map transformation unit 103, and is configured with a feature map inverse quantization unit 801, an unpacking unit 802, and a range restoration unit 810.

[0114] The feature map inverse quantization unit 801 converts the integer-type packing feature frame decoded by the feature map internal decoding unit 201 into a decimal-type packing feature frame using the decimal-type maximum and minimum values ​​transmitted as metadata. The unpacking unit 802 extracts a feature map for each channel in raster scan order from the packing feature frame supplied from the feature map inverse quantization unit 801. The range restoration unit 810 restores the feature values ​​included in the feature maps for multiple channels supplied from the unpacking unit 802, as described below.

[0115] The range restoration unit 810 performs the reverse process of the range adjustment unit 710. That is, the range restoration unit 810 restores the values ​​by subtracting a predetermined offset coefficient from all values ​​of feature quantities in a predetermined channel and dividing the values ​​by a predetermined scaling coefficient. The range restoration unit 810 restores values ​​for each of multiple channels.

[0116] Hereinafter, the scaling coefficients and offset coefficients used by the range restoration unit 810 to restore the feature values ​​will be collectively referred to as parameters, just like the range adjustment unit 710.

[0117] Any of the range restoration unit 810, the feature map inverse conversion unit 202, the feature map internal decoding unit 201, and the feature map decoding device 200 decodes the syntax including the parameters. The syntax may be decoded by a configuration other than the range restoration unit 810, and the range restoration unit 810 may perform processing based on the decoded syntax. Below, an example will be described in which the range restoration unit 810 decodes the syntax including the parameters on a frame-by-frame basis. The range restoration unit 810 may repeat the following frame-by-frame processing until the sequence ends. The range restoration unit 810 may be configured to decode part of the syntax.

[0118] <Syntax> 16 is an example of syntax 1600 decoded by the range restoration unit 810. When this syntax 1600 is used, the range restoration unit 810 decodes the number of channels, and the scaling coefficient and offset coefficient for each channel on a frame-by-frame basis.

[0119] <Syntax Variation 1> A description will be given of an example other than the syntax 1600. Fig. 17 shows an example of syntax 1700 decoded by the range restoration unit 810.

[0120] A case where this syntax 1700 is used will be described with reference to Fig. 21. First, the range restoration unit 810 decodes coding_skip_flag, which indicates whether or not parameter coding has been skipped (S1302).

[0121] When coding_skip_flag is true (S1304: YES), the range restoration unit 810 sets the parameters decoded in a frame prior to the current frame as the parameters of the current frame (S1306). At the beginning of processing or the beginning of a sequence, the range restoration unit 810 may set the scaling coefficient to 1 and the offset coefficient to 0. If there are no parameters decoded in a frame prior to the current frame, the range restoration unit 810 may set the scaling coefficient to 1 and the offset coefficient to 0.

[0122] If coding_skip_flag is false (S1304: NO), range restoration unit 810 decodes scale_frame_flag, which indicates whether or not scaling coefficients are to be decoded on a frame-by-frame basis (S1308).

[0123] The range restoration unit 810 decodes the channel_size indicating the number of channels (S1310).

[0124] If scale_frame_flag is true (S1312: YES), the range restoration unit 810 decodes the scaling coefficients on a frame-by-frame basis and decodes the offset coefficients on a channel-by-channel basis (S1314).If scale_frame_flag is false (S1312: NO), the range restoration unit 810 decodes the scaling coefficients on a channel-by-channel basis and decodes the offset coefficients on a frame-by-frame basis (S1316).

[0125] The range restoration unit 810 decodes a bitstream that has been coded by selecting a unit for coding parameters on the coding side. Preferably, the range restoration unit 810 selects whether to decode frame-by-frame or channel-by-channel for each element included in the parameters. This reduces the amount of code required for syntax including parameters for adjusting features, thereby reducing the amount of computation required for decoding. Therefore, the feature map decoding device of this embodiment has the effect of improving computational efficiency.

[0126] When some syntax elements are not coded on the coding side, the range restoration unit 810 does not need to decode those syntax elements.

[0127] If the syntax does not include coding_skip_flag, range restoration section 810 may omit the processes of S1302, S1304, and S1306 in FIG.

[0128] If the syntax does not include channel_size, range restoration section 810 may omit the process of S1310 in FIG.

[0129] <Syntax variation 2> A description will be given of an example different from the syntax 1600 and the syntax 1700. Fig. 18 shows an example of syntax 1900 decoded by the range restoration unit 810.

[0130] A case where this syntax 1900 is used will be described with reference to Fig. 22. The range restoration unit 810 may perform the above-mentioned processing for some of the syntax included in the syntax 1700. First, the range restoration unit 810 decodes coding_skip_flag, which indicates whether parameter coding has been skipped (S1402).

[0131] If coding_skip_flag is true (S1404: YES), range restoration unit 810 sets the parameters decoded in the frame prior to the frame to be processed as the parameters of the frame to be processed (S1406).

[0132] When coding_skip_flag is false (S1404: NO), range restoration unit 810 decodes scale_frame_flag, which indicates whether or not to decode scaling coefficients on a frame-by-frame basis, and offset_frame_flag, which indicates whether or not to decode offset coefficients on a frame-by-frame basis (S1408).

[0133] The range restoration unit 810 decodes param_precision, which indicates the precision of the parameters (S1410). The range restoration unit 810 may set a specified precision and perform decoding by adding the precision of the parameters to the specified precision. For example, if the range restoration unit 810 sets the specified precision to 8 bits and decodes the precision of the parameters to 2, the sum of these, 10 bits, may be used as the precision of the parameters.

[0134] If at least one of scale_frame_flag and offset_frame_flag is false (S1412: YES), range restoration unit 810 decodes select_channel_flag, which indicates whether or not to specify a channel, and coding_channel_size, which indicates the number of channels for which parameters are to be decoded (S1414).

[0135] If scale_frame_flag is true (S1416: YES), the range restoration unit 810 decodes the scaling coefficients on a frame-by-frame basis (S1418). If scale_frame_flag is false (S1416: NO) and select_channel_flag is true (S1420: YES), the range restoration unit 810 decodes the channel number scale_channel_id of the scaling coefficients (S1422). If scale_frame_flag is false (S1416: NO), the range adjustment unit 710 decodes the scaling coefficients on a channel-by-channel basis (S1424). That is, if select_channel_flag is true, the range restoration unit 810 decodes the scaling coefficients of the channel specified by scale_channel_id, and if select_channel_flag is false, the range restoration unit 810 decodes the scaling coefficients of all channels.

[0136] If offset_frame_flag is true (S1426: YES), the range restoration unit 810 decodes the offset coefficients on a frame-by-frame basis (S1428). If offset_frame_flag is false (S1426: NO) and select_channel_flag is true (S1430: YES), the range adjustment unit 710 decodes the channel number offset_channel_id of the offset coefficients (S1432). If offset_frame_flag is false (S1426: NO), the range adjustment unit 710 decodes the scaling coefficients on a channel-by-channel basis (S1434). That is, if select_channel_flag is true, the range restoration unit 810 decodes the offset coefficients of the channel specified by offset_channel_id, and if select_channel_flag is false, the range restoration unit 810 decodes the offset coefficients of all channels.

[0137] The range restoration unit 810 decodes a bitstream that has been coded by selecting a unit for coding parameters on the coding side. Preferably, the range restoration unit 810 selects whether to decode frame-by-frame or channel-by-channel for each element included in the parameters. More preferably, the range restoration unit 810 decodes the precision of the parameters and reduces the amount of coding for the parameters. Even more preferably, the range restoration unit 810 decodes the parameters by specifying a channel number and reduces the amount of coding for the parameters. This reduces the amount of coding for syntax including parameters for adjusting features, thereby reducing the amount of computation required for decoding. Therefore, the feature map decoding device of this embodiment has the effect of improving computational efficiency.

[0138] When some syntax elements are not coded on the coding side, the range restoration unit 810 does not need to decode those syntax elements.

[0139] If the syntax does not include coding_skip_flag, range restoration section 810 may omit the processes of S1402, S1404, and S1406 in FIG.

[0140] If the syntax does not include param_precision, range restoration section 810 may omit the process of S1410 in FIG.

[0141] When select_channel_flag is true, the range restoration unit 810 may decode scaling coefficients of channels not specified by the scaling coefficient channel number scale_channel_id.When select_channel_flag is true, the range restoration unit 810 may decode offset coefficients of channels not specified by the offset coefficient channel number offset_channel_id.When select_channel_flag is true, the range restoration unit 810 may set scaling coefficients of channels not specified by the scaling coefficient channel number scale_channel_id to 1 and offset coefficients of channels not specified by the offset coefficient channel number offset_channel_id to 0.

[0142] When select_channel_flag, which indicates whether a parameter channel is specified, is false, the range restoration unit 810 may not decode coding_channel_size, which indicates the number of channels for encoding the parameters. That is, the range restoration unit 810 may omit the coding_channel_size decoding process of S1414 in Fig. 22. The range restoration unit 810 may set the number of channels to the number of all channels in the feature map.

[0143] If the syntax does not include select_channel_flag, range restoration section 810 may omit the process of decoding select_channel_flag in S1414, and the processes of S1420, S1422, S1430, and S1432 in FIG.

[0144] If the syntax does not include scale_channel_id and offset_channel_id but does include a parameter channel number, range restoration unit 810 may decode the parameter channel number in S1414 of Fig. 22 and omit the processes of S1420, S1422, S1430, and S1432. Range restoration unit 810 may decode the scaling coefficient of the channel specified by the parameter channel number in S1424, and decode the offset coefficient of the channel specified by the parameter channel number in S1434.

[0145] If the syntax does not include coding_channel_size, range restoration unit 810 may omit the coding_channel_size decoding process in S1414 of FIG.

[0146] In all of the above-described embodiments, the bitstream output by the feature map encoder has a specific data format that can be decoded according to the encoding method used in the embodiment, and the feature map decoder corresponding to the feature map encoder can decode the bitstream in this specific data format.

[0147] When a wired or wireless network is used to exchange bitstreams between the feature map encoder and the feature map decoder, the bitstreams may be converted into a data format suitable for the transmission mode of the communication channel before transmission. In this case, a transmitter is provided that converts the bitstream output by the feature map encoder into coded data in a data format suitable for the transmission mode of the communication channel and transmits the coded data to the network, and a receiver is provided that receives the coded data from the network, restores the coded data to a bitstream, and supplies the bitstream to the feature map decoder. The transmitter includes a memory that buffers the bitstream output by the feature map encoder, a packet processor that packetizes the bitstream, and a transmitter that transmits the packetized coded data via the network. The receiver includes a receiver that receives the packetized coded data via the network, a memory that buffers the received coded data, and a packet processor that packetizes the coded data to generate a bitstream and provides it to the feature map decoder.

[0148] The above encoding and decoding processes may be realized not only as a transmission, storage, and receiving device using hardware, but also as firmware stored in a ROM (read-only memory) or flash memory, or as software for a computer, etc. The firmware program or software program may be provided by recording it on a computer-readable recording medium, or may be provided from a server via a wired or wireless network, or may be provided as data broadcasting on terrestrial or satellite digital broadcasting.

[0149] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the components and treatment processes, and that such modifications are also within the scope of the present invention. [Explanation of symbols]

[0150] 100 Feature map encoding device, 101 Neural network feature extraction unit, 102 Feature map reduction unit, 103 Feature map conversion unit, 104 Feature map internal encoding unit, 200 Feature map decoding device, 201 Feature map internal decoding unit, 202 Feature map inverse conversion unit, 203 Feature map restoration unit, 204 Neural network identification unit, 301 Convolution processing unit, 302 Activation processing unit, 303 Pooling processing unit, 322 Bottom-up processing unit, 324 Top-down processing unit, 326 Image to be subjected to feature extraction processing, 501 First feature map reduction unit, 502 First channel combination unit, 503 Second feature map reduction unit, 504 Second channel combination unit, 505 Third feature map reduction unit, 506 First padding unit, 507 Second padding unit, 508 Third padding unit, 601 8x magnification unit, 602 4x magnification unit, 603 2x magnification unit, 604 first feature map mixing unit, 605 second feature map mixing unit, 606 first unpadding unit, 607 second unpadding unit, 608 third unpadding unit, 701 packing unit, 702 feature map quantization unit, 801 feature map inverse quantization unit, 802 unpacking unit, 901 switch, 902 VVC encoding unit, 903 HEVC encoding unit, 904 AV1 encoding unit, 1001 switch, 1002 VVC decoding unit, 1003 HEVC decoding unit, 1004 AV1 decoding unit.

Claims

1. a range adjustment unit that adjusts feature amounts included in a feature map of a plurality of channels based on parameters including a scaling coefficient and an offset coefficient; a packing unit that combines the feature maps of the plurality of channels into one frame to generate a packing feature frame, the range adjustment unit selects whether to encode the data on a frame-by-frame basis or on a channel-by-channel basis for each element included in the parameters. Feature map encoder.

2. an unpacking unit for extracting multi-channel feature maps from the packed feature frame; a range restoration unit that restores feature amounts included in the feature maps of the plurality of channels based on parameters including a scaling coefficient and an offset coefficient; the range restoration unit selects whether to decode on a frame-by-frame basis or on a channel-by-channel basis for each element included in the parameters. Feature map decoder.

Images