Feature map encoding device, feature map encoding method, feature map decoding device, and feature map decoding method

Abstract

A feature map encoding device according to one embodiment of the present application comprises: a neural network feature amount extraction unit (101) that generates a multi-scale feature map from an image; a feature map reduction unit (102) that converts the multi-scale feature map into a single-scale feature map; a feature map conversion unit (103) that converts the single-scale feature map into a packing feature frame; a distribution characteristic calculation unit (111) that calculates distribution characteristics of the multi-scale feature map and distribution characteristics of the single-scale feature map on a frame-by-frame basis; and a feature map internal encoding unit (104) that encodes the packing feature frame and the distribution characteristics.

Description

Feature Map Encoding Device, Feature Map Encoding Method, Feature Map Decoding Device, Feature Map Decoding Method

[0001] It relates to the encoding and decoding of feature maps in a neural network.

[0002] As a neural network technology used for image recognition such as detecting objects of various scales in an image, dividing regions for each object, or tracking objects, the FPN (Feature Pyramid Network) of Non-Patent Document 1 is known. In FPN, a plurality of feature maps of various scales are generated from the image to be processed, and various image recognitions are performed using the feature maps.

[0003] The FPN used for image recognition generates a plurality of feature maps from an image, and its structure utilizes a CNN (Convolutional Neural Network). A CNN reads an image, is composed of convolution and pooling, and can be divided into a feature amount extraction unit (backbone) that generates a feature map and a discrimination unit (head) that is composed of a hierarchical fully connected layer and generates an output suitable for tasks such as object detection, instance segmentation, and object tracking. FPN utilizes the backbone of CNN.

[0004] The feature amount extraction unit of FPN is typically structured with a convolutional processing unit 301, an activation processing unit 302, and a pooling processing unit 303 shown in FIG. 3 as one basic unit, and this basic unit is repeated in a hierarchical structure.

[0005] Figure 4 shows the structure of the FPN. The FPN consists of a bottom-up processing unit 322 that generates a multi-scale feature map composed of multiple hierarchical layers using the CNN backbone, and a top-down processing unit 324 that aggregates features from deeper layer feature maps to shallower layer feature maps using the inverse configuration of the CNN backbone. The bottom-up processing unit 322 repeatedly performs the convolution processing unit 301, activation processing unit 302, and pooling processing unit 303, which are the basic units in Figure 3, reducing the resolution of the feature map by half each time to generate a pyramid of multiple layered feature maps. On the other hand, the top-down processing unit 324 adds feature maps with resolutions corresponding to the bottom-up processing unit 322, expanding the resolution of the feature map to the same resolution as the input image, and generating a pyramid of feature maps. In other words, the FPN generates multiple feature maps for each layer from the image 326 that is the target of feature extraction processing.

[0006] The convolution processing unit 301 performs convolution on the data to be processed (image or feature map) using a plurality of predetermined filters (kernels). In the convolution processing in the convolution processing unit 301, predetermined filtering is performed on the entire data to be processed while sliding at predetermined intervals. At this time, the sliding interval is called the stride. The convolution processing unit 301 may determine the stride based on the number of data to be processed. For example, the convolution processing unit 301 may determine the stride to be 1 if the number of data to be processed is less than a predetermined value, and determine the stride to be 2 if it is greater than or equal to the predetermined value. Multiple feature maps are generated by preparing a plurality of predetermined filters at each layer and generating one feature map for each filter. The unit of a feature map is called a channel. If the number (types) of predetermined filters is N (N types), then N (N channels) of feature maps are generated.

[0007] The activation processing unit 302 performs an activation process that non-linearly transforms the feature map output from the convolution processing unit 301. Here, the function used for the activation process is called the activation function. The activation processing unit 302 uses the ReLU (Rectified Linear Unit) function or the sigmoid function, etc., as the activation function.

[0008] The pooling processing unit 303 performs a process of downsampling the feature map by replacing the local values ​​of the feature map output from the activation processing unit 302 with representative values.

[0009] On the other hand, when performing classification using a neural network, it has the capability to execute the task using feature maps of multiple channels at each layer.

[0010] In image recognition, the multi-channel feature maps of each hierarchical level are subjected to a convolution process at predetermined size intervals based on the scale of the feature maps, and the probability of the object's class is calculated for each pixel.

[0011] Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature Pyramid Networks for Object Detection. In CVPR, 2017.

[0012] The amount of information in feature maps is enormous, making them unsuitable for transmission and storage. In view of the above problems, this embodiment aims to provide a technology for encoding and decoding feature maps.

[0013] To solve the above problems, one embodiment of the feature map encoding device of the present invention comprises: a neural network feature extraction unit that generates a multi-scale feature map from an image; a feature map reduction unit that converts the multi-scale feature map into a single-scale feature map; a feature map conversion unit that converts the single-scale feature map into a packing feature frame; a distribution characteristic calculation unit that calculates the distribution characteristics of the multi-scale feature map and the single-scale feature map on a frame-by-frame basis; and a feature map internal encoding unit that encodes the packing feature frame and the distribution characteristics.

[0014] Furthermore, one embodiment of the feature map decoding device of the present invention comprises: an internal feature map decoding unit that decodes packing feature frames and distribution characteristics on a frame-by-frame basis; an inverse feature map conversion unit that converts the packing feature frames into a single-scale feature map; a first feature map refinement unit that refines the single-scale feature map using the distribution characteristics; a feature map restoration unit that converts the refined single-scale feature map into a multi-scale feature map; and a second feature map refinement unit that refines the multi-scale feature map using the distribution characteristics.

[0015] According to this embodiment, feature maps can be encoded and decoded efficiently with minimal processing load.

[0016] This is a block diagram illustrating the configuration of the feature map encoding device 100. This is a block diagram illustrating the configuration of the feature map decoding device 200. This is a block diagram illustrating the basic unit processing at each layer of the FPN. This is a diagram illustrating the structure of the FPN. This is a block diagram illustrating the detailed configuration of the feature map reduction unit 102. This is a block diagram illustrating the detailed configuration of the feature map restoration unit 203. This is a block diagram illustrating the detailed configuration of the feature map conversion unit 103. This is a block diagram illustrating the detailed configuration of the feature map inverse conversion unit 202. This is a block diagram illustrating the detailed configuration of the feature map internal encoding unit 104. This is a block diagram illustrating the detailed configuration of the feature map internal decoding unit 201. This is a diagram illustrating the number of channels, width, and height of feature maps x1, x2, and x3. This is a diagram illustrating the state in which multiple channel feature maps are packed into one frame. This is a diagram illustrating flipping when packing multiple channel feature maps into one frame. This is a diagram illustrating the layers and units handled by the feature map encoding device and feature map decoding device of this embodiment. This is a flowchart illustrating the processing of the feature map encoding device 100 when feature map refinement is performed on a frame-by-frame basis. This is a flowchart illustrating the processing of the feature map decoding device 200 when feature map refinement is performed on a frame-by-frame basis. This is a diagram illustrating the effects of feature map refinement. This is a diagram showing the syntax of the distribution characteristics. This is a diagram showing the syntax of the distribution characteristics. This is a diagram showing the syntax of the bitstream used in modification 6 of this embodiment. This is a diagram showing the syntax of the bitstream used in modification 7 of this embodiment.

[0017] This section defines the technologies and technical terms used in this embodiment.

[0018] <Features and Feature Maps> In a convolutional neural network (CNN), the data obtained by convolving the data of the portion scanned while changing its position within the target image (input layer data) using a filter, with the filter coefficients, is called a feature or feature map.

[0019] <Packing> Frame packing is the process of combining two or more frames (pictures) into a single frame (picture) by arranging them in a tile-like manner. In this application, packing refers to the process of combining feature maps of multiple channels into a single frame. Figure 12 shows an example of frame packing.

[0020] <Data Types> Data types that represent integer values ​​are designated as integer types, and data types that represent decimal values ​​are designated as decimal types.

[0021] <Layers and Units> The layers and units handled by the feature map encoding and feature map decoding devices of this embodiment will be explained using Figure 14. Figure 14 shows a sequence of consecutive feature frames or feature maps for all channels over time, and is referred to as a sequence layer or sequence unit. Figure 14 shows a sequence of consecutive feature maps for one channel over time, and is referred to as a sequence layer for each channel or a sequence unit for each channel. Figure 14 shows a feature frame or feature map for all channels at the same time, and is referred to as a frame layer or frame unit. Figure 14 shows a feature map for one channel at a certain time, and is referred to as a feature map layer for each channel (a feature map unit for each channel).

[0022] (First Embodiment) A feature map encoding device 100 and a feature map decoding device 200 according to the first embodiment of the present invention will be described.

[0023] Figure 1 is a block diagram of a feature map encoding device 100 according to the first embodiment. The feature map encoding device 100 of this embodiment includes a feature map reduction unit 102, a feature map conversion unit 103, a feature map internal encoding unit 104, and a distribution characteristic calculation unit 111. The feature map encoding device 100 is a device that encodes the feature map generated by the neural network feature extraction unit 101 to generate a bitstream and output it.

[0024] The neural network feature extraction unit 101 reads the image to be feature extracted, generates a feature map through FPN convolution, activation, and pooling processes, and supplies it to the feature map reduction unit 102. In this embodiment, a three-layer multi-scale feature map x1, x2, and x3 is generated.

[0025] The feature map reduction unit 102 converts the three-layer multi-scale feature maps x1, x2, and x3 obtained from the neural network feature extraction unit 101 into a single-layer single-scale feature map xf and supplies it to the feature map conversion unit 103. The details of the feature map reduction unit 102 will be explained in detail with reference to Figure 5.

[0026] The feature map conversion unit 103 takes the fractional single-scale feature map xf supplied from the feature map reduction unit 102, performs packing and quantization processing to convert it into an integer-type packed feature frame, and supplies it to the feature map internal encoding unit 104.

[0027] The details of the feature map conversion unit 103 will be explained in detail with reference to Figure 7.

[0028] The feature map internal encoding unit 104 encodes the integer-type packing feature frame supplied from the feature map conversion unit 103 using an image encoding standard such as VVC, HEVC, or AV1 to generate and output a bitstream. The output bitstream is supplied to the feature map decoding device 200 or the like via a network or the like.

[0029] The distribution characteristic calculation unit 111 obtains feature map values ​​for each layer from the neural network feature extraction unit 101 and calculates the distribution characteristics of the feature map for each layer. The distribution characteristic calculation unit 111 obtains feature map values ​​for each channel from the feature map reduction unit 102 and calculates the distribution characteristics of the feature map for each channel. The distribution characteristic calculation unit 111 outputs the calculated distribution characteristics to the feature map internal encoding unit 104.

[0030] The details of the feature map internal encoding unit 104 will be explained in detail with reference to Figure 9.

[0031] Figure 2 is a block diagram showing the configuration of a feature map decoding device 200 according to an embodiment of the present invention, corresponding to the feature map encoding device 100 in Figure 1. The feature map decoding device 200 of this embodiment includes a feature map internal decoding unit 201, a feature map inverse conversion unit 202, a feature map reconstruction unit 203, a first feature map refinement unit 211, and a second feature map refinement unit 212. The feature map decoding device 200 also receives a bitstream encoded by the feature map encoding device 100, decodes the bitstream to generate three-layer multi-scale feature maps x1up, x2up, and x3up, and supplies them to the neural network identification unit 204.

[0032] The feature map internal decoding unit 201 decodes the bitstream encoded by the feature map internal encoding unit 104 of the feature map encoding device 100 using an image encoding standard such as VVC, HEVC, or AV1, generates an integer-type packing feature frame, and supplies it to the feature map inverse transform unit 202.

[0033] The details of the feature map internal decoding unit 201 will be explained in detail with reference to Figure 10.

[0034] The feature map inverse transform unit 202 performs inverse quantization and unpacking on the integer-type packing feature frame supplied from the feature map internal decoding unit 201, converts it into a fractional-type single-scale feature map xr, and supplies it to the first feature map refinement unit 211. The details of the feature map inverse transform unit 202 will be explained in detail with reference to Figure 8.

[0035] The first feature map refinement unit 211 refines the feature map that has been inverted by the feature map inverse transformation unit 202, and outputs the refined feature map to the feature map restoration unit 203.

[0036] The feature map reconstruction unit 203 converts the single-scale feature map xr supplied from the first feature map refinement unit 211 into three-layer multi-scale feature maps x1up, x2up, and x3up, and supplies them to the second feature map refinement unit 212. The details of the feature map reconstruction unit 203 will be explained in detail with reference to Figure 6.

[0037] The second feature map refinement unit 212 refines the feature map restored by the feature map restoration unit 203 and outputs it to the neural network identification unit 204 as the output of the feature map decoding device 200.

[0038] The neural network identification unit 204 performs identification processing such as identifying objects in the target image, identifying locations and landscapes, and identifying people and living things, based on the three-layer multi-scale feature maps x1up, x2up, and x3up supplied by the second feature map refinement unit 212.

[0039] <About Feature Map Reduction and Feature Map Restoration> The feature map reduction unit 102 has the function of converting a multi-layer multi-scale feature map obtained from the neural network feature extraction unit 101 into a single-layer single-scale feature map.

[0040] The feature map reduction unit 102 will be explained in detail using Figure 5. The feature map reduction unit 102 consists of a first feature map reduction unit 501, a first channel merging unit 502, a second feature map reduction unit 503, a second channel merging unit 504, a third feature map reduction unit 505, a first padding unit 506, a second padding unit 507, and a third padding unit 508. The feature map reduction unit 102 in Figure 5 is an example of a configuration that converts a three-layer multi-scale feature map into a single-scale feature map.

[0041] The feature map reduction unit 102 takes a three-layer multi-scale feature map, consisting of a first feature map x1, a second feature map x2, and a third feature map x3, as input, converts it into a single-layer single-scale feature map xf, and supplies it to the feature map conversion unit 103. Here, n is the index indicating the layer, Cn is the number of channels in the nth layer, Wn is the width of the feature map, and Hn is the height of the feature map. In this embodiment, the values ​​of Cn, Wn, and Hn for each layer are as shown in Figure 11. However, H and W are the width and height of the image from which feature extraction is performed, respectively.

[0042] The first padding unit 506 has a function of performing padding on the first feature map x1 and generating a first padded feature map x1pad. In the first padding unit 506, the padding size is determined so that the width and height of x1pad are multiples of 64. The number of channels of x1pad is the same as that of x1 and is 256.

[0043] The second padding unit 507 has a function of performing padding by folding on the second feature map x2 and generating a second padded feature map x2pad. In the second padding unit 507, the padding size is determined so that the width and height of x2pad are multiples of 32. The number of channels of x2pad is the same as that of x2 and is 256.

[0044] The third padding unit 508 has a function of performing padding by folding on the third feature map x3 and generating a third padded feature map x3pad. In the third padding unit 508, the padding size is determined so that the width and height of x3pad are multiples of 16. The number of channels of x3pad is the same as that of x3 and is 256.

[0045] In the first padding unit 506, the second padding unit 507, and the third padding unit 508, it is assumed that the left padding size and the right padding size are the same, and the upper padding size and the lower padding size are the same. That is, it is assumed that the feature maps x1, x2, and x3 are respectively arranged at the centers of x1pad, x2pad, and x3pad.

[0046] The first feature map reduction unit 501 performs convolution in the spatial direction and the channel direction on the first padded feature map x1pad obtained from the first padding unit 506, and generates a first intermediate feature map y1. The number of channels of y1 is 192, the width is Wx1pad / 2, and the height is Hx1pad / 2. Here, Wx1pad and Hx1pad are the width and height of the first padded feature map x1pad, respectively.

[0047] The first channel combiner 502 has a function of combining the first intermediate feature map y1 obtained from the first feature map reduction unit 501 and the second padded feature map x2pad obtained from the second padding unit 507 in the channel direction to generate an intermediate feature map y1Cx2pad. Since the number of channels of y1 is 192 and the number of channels of x2pad is 256, the number of channels of the intermediate feature map y1Cx2pad is 448 (192 + 256).

[0048] The second feature map reduction unit 503 performs convolution on the intermediate feature map y1Cx2pad obtained from the first channel combiner 502 in the spatial and channel directions to generate a second intermediate feature map y2. The number of channels of y[2] is 192, the width is Wy1Cx2pad / 2, and the height is Hy1Cx2pad / 2. Here, Wy1Cx2pad and Hy1Cx2pad are the width and height of the intermediate feature map y1Cx2pad, respectively.

[0049] The second channel combiner 504 has a function of combining the second intermediate feature map y2 obtained from the second feature map reduction unit 503 and the third padded feature map x3pad obtained from the third padding unit 508 in the channel direction to generate an intermediate feature map y2Cx3pad. Since the number of channels of the intermediate feature map y2 is 192 and the number of channels of x3pad is 256, the number of channels of y1Cx2pad is 448 (192 + 256). <​​​​​

[0052] The feature map reconstruction unit 203 has the function of converting the single-scale feature map xr obtained from the first feature map refinement unit 211 into three-layer multi-scale feature maps x1up, x2up, and x3up.

[0053] The details of the feature map restoration unit 203 will be explained using Figure 6. The feature map restoration unit 203 consists of an 8x magnification unit 601, a 4x magnification unit 602, a 2x magnification unit 603, a first feature map mixing unit 604, a second feature map mixing unit 605, a first padding removal unit 606, a second padding removal unit 607, and a third padding removal unit 608.

[0054] The 8x magnification unit 601 has the function of expanding the feature map and reducing the number of channels of the single-scale feature map xr obtained from the first feature map refinement unit 211 by performing transposition convolution in the spatial direction and convolution in the channel direction, thereby generating an intermediate feature map z1. The number of channels in z1 is 196. If the width and height of the single-scale feature map xr are xrwidth and xrheight, respectively, then the width and height of z1 will be xrwidth × 8 and xrheight × 8, respectively. Here, rwidth × 8 and xrheight × 8 are the same as the width and height of the first padded feature map x1pad, which is the output of the first padding unit 506 of the feature map reduction unit 102.

[0055] The quadruple magnification unit 602 has the function of expanding features and reducing channels by performing transposition convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr obtained from the first feature map refinement unit 211, thereby generating an intermediate feature map z2. The number of channels in z2 is 196. The width and height of z2 are xrwidth × 4 and xrheight × 4, respectively. Here, rwidth × 4 and xrheight × 4 are the same as the width and height of the second padded feature map x2pad, which is the output of the second padding unit 507 of the feature map reduction unit 102.

[0056] The doubling unit 603 has the function of expanding the feature map and reducing the number of channels of the single-scale feature map xr obtained from the first feature map refinement unit 211 by performing transposition convolution in the spatial direction and convolution in the channel direction, thereby generating an intermediate feature map z3. The number of channels in z3 is 196. The width and height of z3 are xrwidth × 2 and xrheight × 2, respectively. Here, rwidth × 2 and xrheight × 2 are the same as the width and height of the third padded feature map x3pad, which is the output of the third padding unit 508 of the feature map reduction unit 102.

[0057] The first feature map mixing unit 604 has the function of generating an intermediate feature map z2up, which is an improved version of the intermediate feature map z2 obtained from the 4x magnification unit 602, using the intermediate feature map z1 obtained from the 8x magnification unit 601.

[0058] The second feature map mixing unit 605 has the function of generating an intermediate feature map z3up, which is an improved version of the intermediate feature map z3 obtained from the doubling unit 603, using the intermediate feature map z2up obtained from the first feature map mixing unit 604.

[0059] The padding removal unit 606 has the function of removing padding from the intermediate feature map z1 acquired from the 8x magnification unit 601 and generating a first output feature map x1up. The width and height of x1up are the same as the width and height of the first feature map x1 input to the feature map reduction unit 102.

[0060] The padding removal unit 607 has the function of removing padding from the intermediate feature map z2up obtained from the first feature map mixing unit 604 and generating a second output feature map x2up. The width and height of x2up are the same as the width and height of the second feature map x2 input to the feature map reduction unit 102.

[0061] The padding removal unit 608 has the function of removing padding from the intermediate feature map z3up obtained from the second feature map mixing unit 605 and generating a third output feature map x3up. The width and height of x3up are the same as the width and height of the third feature map x3 input to the feature map reduction unit 102.

[0062] In the first padding removal unit 606, the second padding removal unit 607, and the third padding removal unit 608, similar to the first padding unit 506, the second padding unit 507, and the third padding unit 508 of the feature map reduction unit 102, the padding size on the left and the padding size on the right are set to be the same, and the padding size on the upper and lower sides are set to be the same. That is, each output feature map x1up, x2up, and x3up are assumed to be positioned at the center of each intermediate feature map z1, z2up, and z3up, respectively, and padding is removed from the top, bottom, left, and right.

[0063] <About Feature Map Conversion and Inverse Feature Map Conversion> The feature map conversion unit 103 has the function of performing packing and quantization processing on the multi-channel fractional single-scale feature map xf supplied from the feature map reduction unit 102, and converting it into an integer-type packed feature frame for supply to the feature map internal encoding unit 104.

[0064] The details of the feature map conversion unit 103 on the encoding side will be explained using Figure 7. The feature map conversion unit 103 consists of a packing unit 701 and a feature map quantization unit 702.

[0065] The packing unit 701 has the function of generating a packed feature frame by combining the input feature maps of multiple channels into a single frame. Figure 12 is a diagram illustrating the state in which the feature maps of multiple channels have been packed into one frame. The feature maps of each channel are sequentially placed in one frame from left to right, from top to bottom, in the order of the raster scan.

[0066] Furthermore, the packing unit 701 performs flipping based on the in-frame position where the channel feature map is placed. The ability to select whether or not to perform flipping may be provided and transmitted from the encoding side to the decoding side via the bitstream.

[0067] Figure 13 illustrates the flipping process when packing multiple channel feature maps into a single frame. Flipping involves inverting the position of each channel's feature map elements (pixels) horizontally (left / right), vertically (up / down), or both horizontally and vertically (up / down / left / right) when packing the feature maps for each channel. In Figure 13, the four channel feature maps A (top left), B (top right), C (bottom left), and D (bottom right) are considered a single set. No flipping is performed at position A in Figure 13. At position B, the feature map is inverted horizontally (left / right). At position C, the feature map is inverted vertically (up / down). At position D, the feature map is inverted horizontally and vertically (up / down / left / right). When the distribution of elements in each channel's feature map is similar, performing flipping based on the in-frame position where the channels are placed reduces the boundaries between each channel's feature map, improving encoding efficiency.

[0068] The feature map quantization unit 702 has the function of converting the elements of a decimal-type packing feature frame (feature map of all channels) into an N-bit integer type (an integer N = approximately 8 to 16) within a predetermined range and outputting an integer-type packing feature frame. In this embodiment, it is assumed that it is converted into a 10-bit integer type from 0 to 1023. The feature map quantization unit 702 detects the minimum and maximum values ​​of the elements of the decimal-type packing feature frame and transmits the detected minimum and maximum values ​​of the elements of the packing feature frame (feature map of all channels) to the decoding side as metadata. In order to convert from a decimal-type packing feature frame (feature map) to an integer-type packing feature frame (feature map), a linear transformation is performed in which the minimum value of the decimal type corresponds to the minimum value of the integer type, and the maximum value of the decimal type corresponds to the maximum value of the integer type. For example, when the range of the integer type is represented by 10 bits, the minimum value of the elements of the integer-type packing feature frame (feature map) is 0, and the maximum value is 1023 (2 １０-1) This is the result. Linear quantization is performed on values ​​between the minimum and maximum values.

[0069] Next, the feature map inverse transform unit 202 has the function of performing inverse quantization and unpacking on integer-type packing feature frames decoded by VVC, HEVC, AV1, etc. supplied from the feature map internal decoding unit 201, and transforming them into a decimal-type single-scale feature map xr for supply to the first feature map refinement unit 211.

[0070] Figure 8 will be used to explain the details of the feature map inverse transformation unit 202 on the decoding side. The feature map inverse transformation unit 202 is the inverse process of the feature map transformation unit 103 and is composed of a feature map inverse quantization unit 801 and an unpacking unit 802.

[0071] The feature map inverse quantization unit 801 performs the inverse processing of the encoding-side feature map quantization unit 702 and has the function of converting the elements of the integer-type packing feature frame from integer type to decimal type. The feature map inverse quantization unit 801 converts the integer-type packing feature frame decoded by the feature map internal decoding unit 201 into a decimal-type packing feature frame using the minimum and maximum decimal values ​​transmitted as metadata. A linear transformation is performed to make the minimum integer value equivalent to the minimum decimal value and the maximum integer value equivalent to the maximum decimal value. For values ​​between the minimum and maximum values, linear inverse quantization is performed.

[0072] In the unpacking unit 802, feature maps for each channel are extracted from the packing feature frames arranged in a single frame in the order of raster scan, and supplied to the first feature map refinement unit 211 as a single-scale feature map xr.

[0073] <About Internal Feature Map Encoding and Decoding> Figure 9 will be used to explain the details of the feature map internal encoding unit 104. The feature map internal encoding unit 104 consists of a switch 901, a VVC encoding unit 902, a HEVC encoding unit 903, and an AV1 encoding unit 904. The switch 901 selects an encoding standard for internally encoding the feature map converted by the feature map conversion unit 103. The VVC encoding unit 902 encodes the feature map using the VVC standard and outputs a bitstream compliant with the VVC standard. The HEVC encoding unit 903 encodes the feature map using the HEVC standard and outputs a bitstream compliant with the HEVC standard. The AV1 encoding unit 904 encodes the feature map using the AV1 standard and outputs a bitstream compliant with the AV1 standard.

[0074] In the VVC, HEVC, and AV1 standards, images are divided into predetermined block sizes for encoding.

[0075] It is also possible to implement only one of the following: VVC, HEVC, or AV1. Furthermore, it is possible to use image encoding schemes other than VVC, HEVC, and AV1.

[0076] The feature map internal encoding unit 104 acquires and encodes the distribution characteristics calculated by the distribution characteristic calculation unit 111, and outputs them together with the feature map as a bitstream.

[0077] Next, the details of the feature map internal decoding unit 201 will be explained using Figure 10. The feature map internal decoding unit 201 consists of a switch 1001, a VVC decoding unit 1002, a HEVC decoding unit 1003, and an AV1 decoding unit 1004. The switch 1001 selects the encoding standard to be internally decoded based on the information in the input bitstream that is used to select internal decoding. The VVC decoding unit 1002 decodes the feature map using the VVC standard. The HEVC decoding unit 1003 decodes the feature map using the HEVC standard. The AV1 decoding unit 1004 decodes the feature map using the AV1 standard.

[0078] In the VVC, HEVC, and AV1 standards, decoding is performed for each predetermined block size.

[0079] It is also possible to implement only one of the following: VVC, HEVC, or AV1. Furthermore, it is possible to use image encoding schemes other than VVC, HEVC, and AV1.

[0080] The feature map internal decoding unit 201 decodes the distribution characteristics contained in the bitstream and outputs the decoded distribution characteristics to the first feature map refinement unit 211 and the second feature map refinement unit 212.

[0081] <Feature Map Refinement Process> The processing of the feature map encoding device 100 when feature map refinement is performed on a frame-by-frame basis will be explained using Figure 15. The distribution characteristic calculation unit 111 obtains feature map values ​​for each layer from the neural network feature extraction unit 101 and calculates the distribution characteristics of the feature map for each layer (S12). The distribution characteristic calculation unit 111 calculates the mean and standard deviation of the feature map as distribution characteristics. The distribution characteristic calculation unit 111 obtains feature map values ​​for each channel from the feature map reduction unit 102 and calculates the distribution characteristics of the feature map for each channel (S14).

[0082] The feature map internal encoding unit 104 acquires and encodes the distribution characteristics calculated by the distribution characteristics calculation unit 111, and outputs them together with the feature map as a bitstream (S16). The feature map encoding device 100 repeats the above process on a frame-by-frame basis.

[0083] The processing of the feature map decoding device 200 when feature map refinement is performed on a frame-by-frame basis will be explained with reference to Figure 16. The feature map internal decoding unit 201 decodes the distribution characteristics contained in the bitstream (S22).

[0084] The first feature map refinement unit 211 calculates the distribution characteristics of each channel included in the feature map that has been inverted by the feature map inverse transformation unit 202 (S24). The first feature map refinement unit 211 calculates the mean and standard deviation of the feature map as distribution characteristics.

[0085] The first feature map refinement unit 211 refines each channel element included in the feature map that has been inverted by the feature map inverse transformation unit 202 (S26). Specifically, the first feature map refinement unit 211 subtracts the average value calculated by the first feature map refinement unit 211 from each channel element included in the feature map, divides it by the standard deviation calculated by the first feature map refinement unit 211, multiplies it by the standard deviation for each channel decoded by the feature map internal decoding unit 201, and adds the average value for each channel decoded by the feature map internal decoding unit 201. The first feature map refinement unit 211 outputs the refined feature map to the feature map restoration unit 203.

[0086] The second feature map refinement unit 212 calculates the distribution characteristics for each layer included in the feature map restored by the feature map restoration unit 203 (S28). The second feature map refinement unit 212 calculates the mean and standard deviation of the feature map as distribution characteristics.

[0087] The second feature map refinement unit 212 refines each element of each layer included in the feature map restored by the feature map restoration unit 203 (S30). Specifically, the second feature map refinement unit 212 subtracts the average value calculated by the second feature map refinement unit 212 from each element of each layer included in the feature map, divides it by the standard deviation calculated by the second feature map refinement unit 212, multiplies it by the standard deviation of each layer decoded by the feature map internal decoding unit 201, and adds the average value of each layer decoded by the feature map internal decoding unit 201. The second feature map refinement unit 212 outputs the refined feature map to the neural network identification unit 204. The feature map decoding device 200 repeats the above process on a frame-by-frame basis.

[0088] The effect of feature map refinement processing will be explained using the example in Figure 17. Figure 17, 401 is xf, which is the probability distribution of elements contained in a predetermined channel (or layer) of the single-scale feature map in the feature map encoding device 100. Figure 17, 402 is xr, which is the probability distribution of elements contained in a predetermined channel (or layer) of the decoded single-scale feature map in the feature map encoding device 200. Figure 17, 403 is xr', which is the probability distribution of elements contained in a predetermined channel (or layer) of the refined single-scale feature map.

[0089] Generally, since the packing feature frame encoded by the feature map internal encoding unit 104 undergoes quantization, 402 is degraded in encoding compared to 401. Due to the encoding degradation of 402, the distribution characteristics of 402 deviate from those of 401. By adopting the configuration of this embodiment, it is possible to obtain the difference between the distribution characteristics of 401 before quantization and the distribution characteristics of 402 after quantization, and to correct 402 using this difference, thereby restoring 403 which has a probability distribution close to that of 401.

[0090] By using the distribution characteristics of the feature map of the original signal calculated and transmitted on the encoding side, and the distribution characteristics of the feature map of the decoded signal calculated on the decoding side, the feature map of the decoded signal can be restored to a value close to that of the original signal's feature map, thereby improving encoding efficiency.

[0091] <Syntax> The distribution characteristics calculated by the distribution characteristics calculation unit 111 are encoded by the feature map internal encoding unit 104 using a predetermined syntax, and the information regarding the distribution characteristics is decoded by the feature map internal decoding unit 201 using a predetermined syntax.

[0092] The syntax for distribution characteristics will be explained using Figure 18. In syntax 1000, layer_size represents the number of layers in the feature map, av_layer represents the average value for each layer in the feature map, dev_layer represents the standard deviation for each layer in the feature map, channel_size represents the number of channels in the feature map, av_channel represents the average value for each channel in the feature map, and dev_channel represents the standard deviation for each channel in the feature map. Here, parameters preceded by [i] indicate that they are values ​​per layer or per channel. The notation "for (i=0; i<M; i++)" indicates that, starting with an initial value i=0, the syntax enclosed in curly braces will be processed and 1 added to i as long as the condition i < maximum value M is met. The curly braces also indicate a predetermined unit for processing. The feature map internal encoding unit 104 encodes the number of layers and the distribution characteristics for each layer, and the number of channels and the distribution characteristics for each channel.

[0093] The feature map internal encoding unit 104 may encode the number of layers and the number of channels separately from the distribution characteristics. The feature map internal encoding unit 104 may encode the distribution characteristics for each layer and the distribution characteristics for each channel. The feature map internal encoding unit 104 may encode either the distribution characteristics for each layer or the distribution characteristics for each channel. The feature map internal encoding unit 104 may encode the distribution characteristics of some layers of the feature map. The feature map internal encoding unit 104 may encode the distribution characteristics of some channels of the feature map. The feature map internal encoding unit 104 may encode a portion of the distribution characteristics of the feature map. The feature map internal decoding unit 201 may decode a portion of the distribution characteristics encoded by the feature map internal encoding unit 104. The first feature map refinement unit 211 and the second feature map refinement unit 212 may perform refinement processing using a portion of the distribution characteristics decoded by the feature map internal decoding unit 201.

[0094] <Modification 1> The feature map encoding device 100 may encode the distribution characteristics in predetermined units instead of frame units. The feature map encoding device 100 may repeat the process shown in Figure 15 in predetermined units instead of frame units.

[0095] The feature map encoding device 100 may encode the distribution characteristics in predetermined frames, but may not encode the distribution characteristics in frames other than the predetermined frames. The feature map encoding device 100 may perform the processing shown in Figure 15 in predetermined frames, but may not perform the processing shown in Figure 15 in frames other than the predetermined frames.

[0096] The distribution characteristic calculation unit 111 may calculate the distribution characteristics in a predetermined frame. The feature map internal encoding unit 104 may encode the distribution characteristics calculated in the predetermined frame.

[0097] The feature map decoding device 200 may refine the feature map at predetermined intervals rather than on a frame-by-frame basis. The feature map decoding device 200 may repeat the process shown in Figure 16 at predetermined intervals instead of on a frame-by-frame basis.

[0098] The feature map decoding device 200 may refine the feature map in a predetermined frame, but may not refine the feature map in frames other than the predetermined frame. The feature map decoding device 200 may perform the processing shown in Figure 16 in a predetermined frame, but may not perform the processing shown in Figure 16 in frames other than the predetermined frame.

[0099] The feature map internal decoding unit 201 may decode the distribution characteristics in a predetermined frame. The first feature map refinement unit 211 refines the feature map in a predetermined frame, but does not need to refine the feature map in frames other than the predetermined frame. The first feature map refinement unit 211 outputs the feature map to the feature map restoration unit 203 regardless of whether refinement processing is performed. The second feature map refinement unit 212 refines the feature map in a predetermined frame, but does not need to refine the feature map in frames other than the predetermined frame. The second feature map refinement unit 212 outputs the feature map to the neural network identification unit 204 regardless of whether refinement processing is performed.

[0100] The syntax for distribution characteristics when refining a feature map in predetermined units will be explained using Figure 19. In syntax 1100, refinement_flag is a flag indicating whether or not to perform refinement processing. The feature map internal encoding unit 104 encodes the distribution characteristics by setting refinement_flag=1 if refinement processing is to be performed. The feature map internal encoding unit 104 does not encode the distribution characteristics by setting refinement_flag=0 if refinement processing is not to be performed. The feature map internal decoding unit 201 decodes the distribution characteristics when refinement_flag=1. The feature map internal decoding unit 201 does not decode the distribution characteristics when refinement_flag=0.

[0101] The distribution characteristic calculation unit 111 may encode the feature map as refinement_flag=0 if it does not want to refine the feature map in a predetermined frame.

[0102] In this way, the syntax includes a refinement_flag that indicates whether or not to perform refinement. If refinement is required, information regarding refinement is transmitted; if refinement is not required, information regarding refinement is not transmitted. This allows refinement to be achieved with a minimum amount of code, improving encoding efficiency.

[0103] In the image encoding standard targeted by the feature map internal encoding unit 104 and the feature map internal decoding unit 201, refinement_flag = 1 may be set for feature maps that become intraframes, and refinement_flag = 0 may be set for feature maps that become interframes. By referencing the refined intraframe, the quality of the interframe is also improved, so the refinement process can be achieved with the minimum amount of encoding.

[0104] <Modification 2> The feature map decoding device 200 may refine the feature map in a predetermined frame. Alternatively, the feature map decoding device 200 may refine the feature map in frames other than the predetermined frame using the distribution characteristics of the predetermined frame. The feature map decoding device 200 may perform the processing shown in Figure 16 in the predetermined frame, and in frames other than the predetermined frame, perform the processing shown in Figure 16 except for S22.

[0105] The syntax may be syntax 1100. In syntax 1100, refinement_flag may be a flag indicating whether or not to decode the distribution characteristics.

[0106] In this way, the syntax includes a refinement_flag indicating whether or not to decode the distribution characteristics. If the change in distribution characteristics is greater than a predetermined value, information regarding the refinement process is transmitted. If the change in distribution characteristics is less than or equal to the predetermined value, information regarding the refinement process is not transmitted. This allows the refinement process to be achieved with a minimum amount of code, improving encoding efficiency.

[0107] The processing of the feature map decoding device 200 will be explained using Figure 16. The feature map internal decoding unit 201 decodes the distribution characteristics included in the bitstream when refinement_flag = 1 (S22). When refinement_flag = 0, the feature map internal decoding unit 201 does not decode the distribution characteristics as they are not included in the bitstream.

[0108] The first feature map refinement unit 211 stores the channel-specific distribution characteristics included in the decoded bitstream when refinement_flag = 1, and uses the stored channel-specific distribution characteristics for refinement when refinement_flag = 0.

[0109] The second feature map refinement unit 212 stores the distribution characteristics of each layer included in the decoded bitstream when refinement_flag = 1, and uses the stored distribution characteristics of each layer for the refinement process when refinement_flag = 0.

[0110] The first feature map refinement unit 211 calculates the distribution characteristics of each channel included in the feature map that has been inverted by the feature map inverse transformation unit 202 (S24). The first feature map refinement unit 211 calculates the mean and standard deviation of the feature map as distribution characteristics.

[0111] The first feature map refinement unit 211 refines each channel element included in the feature map that has been inverted by the feature map inverse transformation unit 202 (S26). In other words, when refinement_flag = 1, the first feature map refinement unit 211 subtracts the average value calculated by the first feature map refinement unit 211 from each channel element included in the feature map, divides it by the standard deviation calculated by the first feature map refinement unit 211, multiplies it by the standard deviation for each channel decoded by the feature map internal decoding unit 201, and adds the average value for each channel decoded by the feature map internal decoding unit 201. If refinement_flag = 0, the first feature map refinement unit 211 subtracts the average value calculated by the first feature map refinement unit 211 from each element of each channel included in the feature map, divides it by the standard deviation calculated by the first feature map refinement unit 211, multiplies it by the standard deviation for each channel stored in the first feature map refinement unit 211, and adds the average value for each channel stored in the first feature map refinement unit 211.

[0112] The first feature map refinement unit 211 outputs the refined feature map to the feature map restoration unit 203.

[0113] The second feature map refinement unit 212 calculates the distribution characteristics for each layer included in the feature map restored by the feature map restoration unit 203 (S28). The second feature map refinement unit 212 calculates the mean and standard deviation of the feature map as distribution characteristics.

[0114] The second feature map refinement unit 212 refines each element of each layer included in the feature map restored by the feature map restoration unit 203 (S30). Specifically, when refinement_flag = 1, the second feature map refinement unit 212 subtracts the average value calculated by the second feature map refinement unit 212 from each element of each layer included in the feature map, divides it by the standard deviation calculated by the second feature map refinement unit 212, multiplies it by the standard deviation of each layer decoded by the feature map internal decoding unit 201, and adds the average value of each layer decoded by the feature map internal decoding unit 201. If refinement_flag = 0, the second feature map refinement unit 212 subtracts the average value calculated by the second feature map refinement unit 212 from each element of each layer included in the feature map, divides it by the standard deviation calculated by the second feature map refinement unit 212, multiplies it by the standard deviation for each layer stored in the second feature map refinement unit 212, and adds the average value for each layer stored in the second feature map refinement unit 212.

[0115] The second feature map refinement unit 212 outputs the refined feature map to the neural network identification unit 204. The feature map decoding device 200 repeats the above process on a frame-by-frame basis.

[0116] <Modification 3> The feature map encoding device 100 may encode the distribution characteristics in the reference frame but not in the non-reference frame. The feature map encoding device 100 may perform the processing shown in Figure 15 in the reference frame but not in the non-reference frame.

[0117] The feature map decoder 200 may refine the feature map in the reference frame but not in the non-reference frame. The feature map decoder 200 may perform the processing shown in Figure 16 in the reference frame but not in the non-reference frame.

[0118] The feature map decoder 200 may refine the feature map in the reference frame. Alternatively, the feature map decoder 200 may refine the feature map in the non-reference frame using the distribution characteristics of the reference frame. The feature map decoder 200 may perform the processing shown in Figure 16 in the reference frame and perform the processing shown in Figure 16 in the non-reference frame, excluding step S22.

[0119] The feature map decoder 200 may use the distribution characteristics of multiple reference frames when refining the feature map in a non-referenced frame using the distribution characteristics of the reference frame. For example, in a non-referenced frame that references two reference frames, the feature map decoder 200 may use the average of the two distribution characteristics of the two reference frames as the distribution characteristic of the non-referenced frame.

[0120] The feature map decoder 200 may use the distribution characteristics of a non-referenced frame obtained by apportioning the distribution characteristics of multiple reference frames according to their distance from the non-referenced frame. For example, if the distance D1 between the first reference frame and the non-referenced frame is 1, and the distance D2 between the second reference frame and the non-referenced frame is 2, the feature map decoder 200 may use the distribution characteristics of the non-referenced frame obtained by multiplying the distribution characteristics of the first reference frame by D2, multiplying the distribution characteristics of the second reference frame by D1, and dividing by (D1 + D2).

[0121] The feature map encoding device 100 may encode the distribution characteristics in a portion of the reference frame, but may not encode the distribution characteristics in the frames excluding the portion of the reference frame. The feature map encoding device 100 may perform the processing shown in Figure 15 in a portion of the reference frame, but may not perform the processing shown in Figure 15 in the frames excluding the portion of the reference frame.

[0122] The feature map decoding device 200 may refine the feature map using the distribution characteristics of previously decoded reference frames in frames excluding a portion of the reference frame.

[0123] <Modification 4> The first feature map refinement unit 211 does not need to perform refinement processing when refinement_flag = 0. The first feature map refinement unit 211 may perform refinement processing using already stored distribution characteristics if the decoded bitstream does not contain distribution characteristics when refinement_flag = 1. The first feature map refinement unit 211 may perform refinement processing using predetermined distribution characteristics if the decoded bitstream does not contain distribution characteristics when refinement_flag = 1. The first feature map refinement unit 211 does not need to perform refinement processing if the decoded bitstream does not contain distribution characteristics when refinement_flag = 1.

[0124] The second feature map refinement unit 212 does not need to perform refinement processing if refinement_flag = 0. If refinement_flag = 1 and the decoded bitstream does not contain distribution characteristics, the second feature map refinement unit 212 may use already stored distribution characteristics to perform refinement processing. If refinement_flag = 1 and the decoded bitstream does not contain distribution characteristics, the second feature map refinement unit 212 may use predetermined distribution characteristics to perform refinement processing. If refinement_flag = 1 and the decoded bitstream does not contain distribution characteristics, the second feature map refinement unit 212 does not need to perform refinement processing.

[0125] In this way, by defining the operation of the refinement process regardless of whether or not the bitstream contains distribution characteristics, error tolerance can be improved.

[0126] <Modification 5> The distribution characteristic calculation unit 111 may obtain the values ​​of the feature map from the neural network feature extraction unit 101 and calculate the distribution characteristics of the feature map by combining multiple layers. The distribution characteristic calculation unit 111 may calculate one mean value and one standard deviation from the value of the feature map of multiple layers. The distribution characteristic calculation unit 111 may calculate one mean value av2 and one standard deviation dev2 for one frame of the feature map obtained from the neural network feature extraction unit 101.

[0127] The distribution characteristic calculation unit 111 may obtain feature map values ​​from the feature map reduction unit 102 and calculate the distribution characteristics of the feature map by combining multiple channels. The distribution characteristic calculation unit 111 may calculate one mean value and one standard deviation from the feature map values ​​of multiple channels. The distribution characteristic calculation unit 111 may calculate one mean value av1 and one standard deviation dev1 for one frame of the feature map obtained from the feature map reduction unit 102. The distribution characteristic calculation unit 111 may calculate either a first distribution characteristic consisting of a mean value av1 and standard deviation dev1 based on the feature map obtained from the feature map reduction unit 102, or a second distribution characteristic consisting of a mean value av2 and standard deviation dev2 based on the feature map obtained from the neural network feature extraction unit 101. The feature map internal encoding unit 104 may acquire and encode either the first distribution characteristic or the second distribution characteristic calculated by the distribution characteristic calculation unit 111, and output it as a bitstream together with the feature map. The feature map internal encoding unit 104 may encode the first distribution characteristic and then encode the second distribution characteristic, or encode the second distribution characteristic and then encode the first distribution characteristic.

[0128] These features enable refinement processing even with fewer distribution characteristic parameters, thereby improving coding efficiency.

[0129] <Modification 6> The feature map internal encoding unit 104 encodes the distribution characteristics calculated by the distribution characteristics calculation unit 111, encodes the feature map converted by the feature map conversion unit 103 as internal encoding using a predetermined encoding standard, and outputs it as a bitstream.

[0130] The syntax of the bitstream output by the feature map internal encoding unit 104 will be explained using Figure 20. Here, a predetermined unit in the structure of the bitstream is defined as a unit type, and the unit type is indicated by a capital letter followed by parentheses (), and the parameters included in that unit type are indicated by curly braces {}. In syntax 1200, the unit types are VMPS, RSD, CVD, and FSPS, FPPS, FPH, and FSEI included in RSD. A header indicating the unit type is attached to the beginning of each unit type (not shown). In syntax 1200, VMPS (Vision Model Parameter Set) is a set of parameters common to the bitstream, including the parameter image_width which indicates the width of the image to be extracted and the parameter image_height which indicates the height of the image to be extracted. RSD (ReStoration_Data) is the feature map parameters used in the feature map inverse transformation unit 202, feature map reconstruction unit 203, first feature map refinement unit 211, and second feature map refinement unit 212. CVD (Coded Video_Data) is the internally encoded feature map data inner_codec_data.

[0131] This section explains the syntax of RSD. FSPS (Feature Sequence Parameter Set) is a set of parameters common to feature map sequences, including the parameter layer_size, which indicates the number of feature map layers, and the parameter channel_size, which indicates the number of feature maps in each layer.

[0132] An FPPS (Feature Picture Parameter Set) is a set of parameters common to one or more frames (pictures) that include distribution characteristic parameters. Here, the distribution characteristic consists of one mean value and one standard deviation for each frame. Therefore, the syntax consists of a flag refinement_flag indicating whether or not to perform refinement, distribution characteristics av1 and dev1 used in the refinement process in the first feature map refinement unit 211, and distribution characteristics av2 and dev2 used in the refinement process in the second feature map refinement unit 212.

[0133] FPH (Feature Picture Header) is a set of frame (picture) parameters that includes the parameter pic_order_cnt, which corresponds to the POC (Picture Order Count) indicating the display order of frames (pictures) within the CVD. FSEI (Feature Supplemental Enhancement Information) is a set of auxiliary parameters that includes the parameter fcm_level, which indicates the complexity of the feature map.

[0134] The feature map internal decoding unit 201 may decode the distribution characteristic parameters contained in the FPPS and supply those distribution characteristic parameters to the first feature map refinement unit 211 and the second feature map refinement unit 212. The first feature map refinement unit 211 and the second feature map refinement unit 212 may refine the feature map of the subsequent CVD using its distribution characteristic parameters.

[0135] The feature map internal encoding unit 104 may encode the FPH with distribution characteristic parameters included but without including distribution characteristic parameters in the FPPS. The feature map internal decoding unit 201 may decode the distribution characteristic parameters included in the FPH and supply those distribution characteristic parameters to the first feature map refinement unit 211 and the second feature map refinement unit 212. The first feature map refinement unit 211 and the second feature map refinement unit 212 may refine the feature map of the subsequent CVD using its distribution characteristic parameters.

[0136] The feature map internal encoding unit 104 may encode the POC and distribution characteristic parameters into the FPH. The feature map internal encoding unit 104 can specify the CVD to which the distribution characteristic parameters are applied by encoding the POC and distribution characteristic parameters in association. The feature map internal decoding unit 201 may decode the POC and distribution characteristic parameters included in the FPH and supply the distribution characteristic parameters to the first feature map refinement unit 211 and the second feature map refinement unit 212. The first feature map refinement unit 211 and the second feature map refinement unit 212 may refine the feature map of the CVD having the corresponding POC using the distribution characteristic parameters. The feature map internal decoding unit 201 may refer to the POC included in the CVD, select the feature map of the CVD that has the same value as the POC included in the FPH, and supply the selected feature map to the first feature map refinement unit 211.

[0137] As a result, the bitstream output from the feature map encoding device 100 undergoes a decoding process, including refinement, by the feature map decoding device 200, thereby improving the accuracy of the feature map. Furthermore, it becomes possible to specify the feature map to which the distribution characteristic parameters should be applied and perform the refinement process, thereby improving the accuracy of the feature map.

[0138] <Modification 7> A modified version of the RSD syntax is shown in Figure 21 as syntax 1300. Since FSPS and FPH are the same as syntax 1200 in Figure 20, their explanation is omitted. FSEI is an auxiliary group of parameters and therefore does not necessarily need to be included in the syntax.

[0139] Here, the refinement process in the first feature map refinement unit 211 is called the first refinement process, and the refinement process in the second feature map refinement unit 212 is called the second refinement process. Until now, when at least one of the first or second refinement process is performed, the flag refinement_flag = 1 is used to indicate whether or not to perform the refinement process, and when neither the first nor the second refinement process is performed, refinement_flag = 0 is used. In the modified example 7, the syntax distinguishes between the first refinement process and the second refinement process. FPPS includes the distribution characteristic parameters used in the first refinement process and the distribution characteristic parameters used in the second refinement process, respectively. Specifically, FPPS includes a flag refinement_flag1 indicating whether or not to perform a first refinement process, distribution characteristics av1 and dev1 used in the first refinement process, and a flag refinement_flag2 indicating whether or not to perform a second refinement process, distribution characteristics av2 and dev2 used in the second refinement process.

[0140] If the first refinement process is not performed, the flag refinement_flag1 = 0, which indicates whether or not to perform the first refinement process, is set, and the distribution characteristic parameters av1 and dev1 used in the first refinement process are not included in the syntax 1300. In other words, if the first refinement process is not performed, the feature map internal encoding unit 104 does not encode the distribution characteristic parameters used in the first refinement process. Also, if the first refinement process is not performed, the first feature map refinement unit 211 does not decode the distribution characteristic parameters used in the first refinement process. If the first refinement process is not performed, the distribution characteristic calculation unit 111 does not need to calculate the distribution characteristic parameters used in the first refinement process.

[0141] If the second refinement process is not performed, the flag refinement_flag2 = 0, which indicates whether or not to perform the second refinement process, is set, and the distribution characteristic parameters av2 and dev2 used in the second refinement process are not included in the syntax 1300. In other words, if the second refinement process is not performed, the feature map internal encoding unit 104 does not encode the distribution characteristic parameters used in the second refinement process. Also, if the second refinement process is not performed, the second feature map refinement unit 212 does not decode the distribution characteristic parameters used in the second refinement process. If the second refinement process is not performed, the distribution characteristic calculation unit 111 does not need to calculate the distribution characteristic parameters used in the second refinement process.

[0142] As a result, the bitstream output from the feature map encoding device 100 is decoded by the feature map decoding device 200, including a refinement process, thereby improving the accuracy of the feature map. Furthermore, it becomes possible to select whether or not to encode the distribution characteristic parameters depending on whether or not the first refinement process and the second refinement process are performed, and the coding amount can be reduced and coding efficiency improved when the distribution characteristic parameters are not encoded.

[0143] In all the embodiments described above, the bitstream output by the feature map encoding device has a specific data format so that it can be decoded according to the encoding method used in the embodiment. Furthermore, the feature map decoding device corresponding to this feature map encoding device can decode the bitstream of this specific data format.

[0144] When a wired or wireless network is used to exchange bitstreams between a feature map encoding device and a feature map decoding device, the bitstream may be converted to a data format suitable for the transmission mode of the communication channel before transmission. In this case, a transmitting device is provided that converts the bitstream output by the feature map encoding device into encoded data in a data format suitable for the transmission mode of the communication channel and transmits it to the network, and a receiving device is provided that receives the encoded data from the network, restores it to a bitstream, and supplies it to the feature map decoding device. The transmitting device includes a memory for buffering the bitstream output by the feature map encoding device, a packet processing unit for packetizing the bitstream, and a transmitting unit for transmitting the packetized encoded data over the network. The receiving device includes a receiving unit for receiving the packetized encoded data over the network, a memory for buffering the received encoded data, and a packet processing unit for packetizing the encoded data to generate a bitstream and providing it to the feature map decoding device.

[0145] The above encoding and decoding processes may be implemented not only as hardware-based transmission, storage, and receiving devices, but also by firmware stored in ROM (read-only memory) or flash memory, or by software on a computer. The firmware program or software program may be recorded on a recording medium readable by a computer and provided, provided from a server via a wired or wireless network, or provided as data broadcasting on terrestrial or satellite digital broadcasting.

[0146] The present invention has been described above based on embodiments. The embodiments are illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of their components and processing processes, and that such modifications also fall within the scope of the present invention.

[0147] This invention can be used in feature map encoding and decoding techniques.

[0148] 100 Feature map encoding device, 101 Neural network feature extraction unit, 102 Feature map reduction unit, 103 Feature map conversion unit, 104 Feature map internal encoding unit, 111 Distribution characteristic calculation unit, 200 Feature map decoding device, 201 Feature map internal decoding unit, 202 Feature map inverse conversion unit, 203 Feature map reconstruction unit, 204 Neural network identification unit, 211 First feature map refinement unit, 212 Second feature map refinement unit, 301 Convolution processing unit, 302 Activation processing unit, 303 Pooling processing unit, 322 Bottom-up processing unit, 324 Top-down processing unit, 326 Image to be processed for feature extraction, 501 First feature map reduction unit, 502 First channel joining unit, 503 Second feature map reduction unit, 504 Second channel joining unit, 505 506 Third feature map reduction section, 507 First padding section, 508 Second padding section, 601 8x enlargement section, 602 4x enlargement section, 603 2x enlargement section, 604 First feature map mixing section, 605 Second feature map mixing section, 606 First padding removal section, 607 Second padding removal section, 608 Third padding removal section, 701 Packing section, 702 Feature map quantization section, 801 Feature map inverse quantization section, 802 Unpacking section, 901 Switch, 902 VVC encoding section, 903 HEVC encoding section, 904 AV1 encoding section, 1001 Switch, 1002 VVC decoding section, 1003 HEVC decoding section, 1004 AV1 decoding unit.

Claims

1. A feature map encoding device comprising: a neural network feature extraction unit that generates a multi-scale feature map from an image; a feature map reduction unit that converts the multi-scale feature map into a single-scale feature map; a feature map conversion unit that converts the single-scale feature map into a packing feature frame; a distribution characteristic calculation unit that calculates the distribution characteristics of the multi-scale feature map and the single-scale feature map on a frame-by-frame basis; and a feature map internal encoding unit that encodes the packing feature frame and the distribution characteristics.

2. A feature map encoding method comprising: a neural network feature extraction step of generating a multi-scale feature map from an image; a feature map reduction step of converting the multi-scale feature map into a single-scale feature map; a feature map conversion step of converting the single-scale feature map into a packing feature frame; a distribution characteristic calculation step of calculating the distribution characteristics of the multi-scale feature map and the single-scale feature map on a frame-by-frame basis; and a feature map internal encoding step of encoding the packing feature frame and the distribution characteristics.

3. A feature map encoding program characterized by causing a computer to perform the following steps: a neural network feature extraction step of generating a multi-scale feature map from an image; a feature map reduction step of converting the multi-scale feature map into a single-scale feature map; a feature map conversion step of converting the single-scale feature map into a packing feature frame; a distribution characteristic calculation step of calculating the distribution characteristics of the multi-scale feature map and the single-scale feature map on a frame-by-frame basis; and a feature map internal encoding step of encoding the packing feature frame and the distribution characteristics.

4. A feature map decoding device comprising: an internal feature map decoding unit that decodes packing feature frames and distribution characteristics on a frame-by-frame basis; an inverse feature map conversion unit that converts the packing feature frames into a single-scale feature map; a first feature map refinement unit that refines the single-scale feature map using the distribution characteristics; a feature map restoration unit that converts the refined single-scale feature map into a multi-scale feature map; and a second feature map refinement unit that refines the multi-scale feature map using the distribution characteristics.

5. A feature map decoding method comprising: a feature map internal decoding step of decoding a packing feature frame and distribution characteristics on a frame-by-frame basis; a feature map inverse transformation step of converting the packing feature frame into a single-scale feature map; a first feature map refinement step of refining the single-scale feature map using the distribution characteristics; a feature map restoration step of converting the refined single-scale feature map into a multi-scale feature map; and a second feature map refinement step of refining the multi-scale feature map using the distribution characteristics.

6. A feature map decoding program characterized by causing a computer to perform the following steps: an internal feature map decoding step of decoding packing feature frames and distribution characteristics on a frame-by-frame basis; an inverse feature map transformation step of converting the packing feature frames into a single-scale feature map; a first feature map refinement step of refining the single-scale feature map using the distribution characteristics; a feature map restoration step of converting the refined single-scale feature map into a multi-scale feature map; and a second feature map refinement step of refining the multi-scale feature map using the distribution characteristics.

Images