Feature map encoding device, feature map encoding method, feature map decoding device, and feature map decoding method

Abstract

To provide a device for efficiently encoding / decoding a feature map with a small processing amount.SOLUTION: The feature map decoding device 200 includes a feature map internal decoding unit for decoding the first single-scale feature maps that have been packed into frames and encoded to generate integer-type packed feature frames, and a feature map internal decoding unit for converting elements of the integer-type packed feature frames into decimal values to generate decimal-type packed feature frames. A feature map inverse transformation unit for generating a second single-scale feature map by dividing the second single-scale feature map into one or more packing groups based on information about the packing groups and unpacking the second single-scale feature map, a single-scale feature map refinement unit for generating a third single-scale feature map by calculating a distribution characteristic of the second single-scale feature map and refining the second single-scale feature map, and a feature map restoration unit for generating a multi-scale feature map by transforming the third single-scale feature map.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] Coding and decoding of feature maps in neural networks. [Background technology]

[0002] A well-known neural network technology used for image recognition, such as detecting objects at various scales in an image, dividing regions into individual objects, or tracking objects, is the FPN (Feature Pyramid Network) described in Non-Patent Document 1. In FPN, multiple feature maps at various scales are generated from the image to be processed, and various image recognitions are performed using the feature maps.

[0003] FPNs used for image recognition generate multiple feature maps from images, and their structure utilizes a CNN (Convolutional Neural Network). CNNs can be divided into a feature extraction section (backbone) that reads an image and generates a feature map using convolution and pooling, and a classification section (head) that is composed of hierarchical fully connected layers and generates output suitable for tasks such as object detection, instance segmentation, and object tracking. FPNs utilize the CNN backbone.

[0004] The feature extraction unit of the FPN is typically configured with a hierarchical structure in which a convolution process 301, an activation process 302, and a pooling process 303 shown in FIG. 3 are configured as one basic unit, and this basic unit is repeated.

[0005] Figure 4 shows the structure of the FPN. The FPN consists of a bottom-up processor 322 that uses a CNN backbone to generate a multi-scale feature map consisting of multiple hierarchical layers, and a top-down processor 324 that aggregates features from deep-layer feature maps to shallow-layer feature maps using the inverse structure of the CNN backbone. The bottom-up processor 322 reduces the resolution of the feature map by half by repeating the basic units of convolution 301, activation 302, and pooling 303 shown in Figure 3, generating a pyramid of feature maps with multiple layers. Meanwhile, the top-down processor 324 adds feature maps with resolutions corresponding to those of the bottom-up processor 322, increasing the resolution of the feature map to the same resolution as the input image, generating a pyramid of feature maps. In other words, the FPN generates multiple feature maps for each layer from the image 326 to be subjected to feature extraction processing.

[0006] The convolution processing unit 301 performs convolution processing on the data to be processed (image or feature map) using multiple types of predetermined filters (kernels). In the convolution processing in the convolution processing unit 301, predetermined filtering is performed on all of the data to be processed while sliding at predetermined intervals. The sliding interval is called the stride. The convolution processing unit 301 may determine the stride based on the number of data to be processed. For example, the convolution processing unit 301 may determine the stride to be 1 if the number of data to be processed is smaller than a predetermined value, and may determine the stride to be 2 if the number is equal to or greater than the predetermined value. Multiple feature maps are generated by preparing multiple predetermined filters at each layer and generating one feature map for each filter. The unit of the feature map is the channel. If the number (types) of predetermined filters is N (N types), N feature maps (N channels) are generated.

[0007] The activation processing unit 302 performs activation processing to nonlinearly convert the feature map output from the convolution processing unit 301. Here, the function used in the activation processing is called an activation function. The activation processing unit 302 uses a ReLU (Rectified Linear Unit) function, a sigmoid function, or the like as the activation function.

[0008] The pooling processing unit 303 is a process for downsampling the feature map output from the activation processing unit 302 by replacing local values ​​of the feature map with representative values.

[0009] On the other hand, when performing classification using a neural network, it has the ability to perform tasks using feature maps of multiple channels at each layer.

[0010] In image classification, the feature maps of multiple channels at each layer are convolved for each predetermined size based on the scale of the feature map, and the probability of the object class is calculated for each pixel. [Prior art documents] [Non-patent literature]

[0011] [Non-Patent Document 1] Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature Pyramid Networks for Object Detection. In CVPR, 2017. Summary of the Invention [Problem to be solved by the invention]

[0012] The amount of information in a feature map is enormous, making it unsuitable for transmission and storage. In view of the above problems, the present invention has an object to provide a technology for encoding and decoding feature maps. [Means for solving the problem]

[0013] In order to solve the above problems, the feature map encoding unit of the present application comprises a feature map reduction unit that converts a multi-scale feature map to generate a single-scale feature map; a single-scale feature map distribution property calculation unit that calculates distribution properties of the single-scale feature map; a packing unit that packs the single-scale feature map into a frame to generate a decimal-type packed feature frame; a quantization unit that converts elements of the decimal-type packing feature frame into integer values ​​to generate an integer-type packed feature frame; a single-scale feature map distribution property encoding unit that encodes the distribution property; and a feature map inner encoding unit that encodes the integer-type packed feature frame.

[0014] a feature map inner decoding unit that decodes the single-scale feature map packed into a frame and encoded to generate an integer-type packing feature frame; a dequantization unit that converts elements of the integer-type packing feature frame into decimal values ​​to generate a decimal-type packing feature frame; an unpacking unit that divides the decimal-type packing feature frame into one or more packing groups based on information about the packing groups and unpacks them to generate a second single-scale feature map; a single-scale feature map refinement unit that calculates a second distribution characteristic of the second single-scale feature map and refines the second single-scale feature map based on the first distribution characteristic and the second distribution characteristic to generate a third single-scale feature map; and a feature map restoration unit that converts the third single-scale feature map to generate a multi-scale feature map. The present invention is characterized by comprising: [Effects of the Invention]

[0015] According to the present invention, feature maps can be efficiently coded and decoded with a small amount of processing. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a block diagram illustrating the configuration of a feature map encoding device 100. [Figure 2] FIG. 2 is a block diagram illustrating the configuration of a feature map decoding device 200. [Figure 3] FIG. 1 is a block diagram for explaining the basic unit of processing in each layer of the FPN. [Figure 4] FIG. 1 is a block diagram illustrating the structure of an FPN. [Figure 5] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map reduction unit 102. [Figure 6] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map restoration unit 203. [Figure 7] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map conversion unit 103. [Figure 8] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map inverse conversion unit 202. [Figure 9] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map internal coding unit 104. [Figure 10] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map internal decoding unit 201. [Figure 11] FIG. 10 is a diagram for explaining the number of channels of feature maps x1, x2, and x3, the width of the feature maps, and the height of the feature maps. [Figure 12] FIG. 10 is a diagram illustrating a state in which feature maps of multiple channels are packed into one frame. [Figure 13] FIG. 10 is a diagram illustrating flipping when packing feature maps of multiple channels into one frame. [Figure 14] 1 is a diagram illustrating layers and units handled by a feature map encoding device and a feature map decoding device according to an embodiment of the present invention. FIG. [Figure 15] 2 is a flowchart for explaining the operation of a single-scale feature map distribution characteristic calculation unit 105 in FIG. 1. [Figure 16] 3 is a flowchart illustrating the operation of the single-scale feature map refinement unit 206 of FIG. 2. [Figure 17] FIG. 4 is a diagram illustrating metadata related to refinement of a feature map according to the first embodiment. [Figure 18] FIG. 1 is a block diagram illustrating a configuration of a feature map encoding device 100 according to a second embodiment. [Figure 19] FIG. 10 is a block diagram illustrating a configuration of a feature map decoding device 200 according to a second embodiment. [Figure 20] 19 is a flowchart for explaining the operation of the multi-scale feature map distribution property calculation unit 107 of FIG. 18. [Figure 21] 20 is a flowchart illustrating the operation of the multi-scale feature map refinement unit 208 of FIG. 19. [Figure 22] FIG. 10 is a diagram illustrating the operation of refining a feature map. [Figure 23] FIG. 10 is a diagram illustrating metadata related to feature map refinement according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] The technologies and technical terms used in this embodiment will be defined below.

[0018] <Features and feature maps> In a convolutional neural network (CNN), a filter is used to scan the image (input layer data) from which features are extracted, and the results of the convolution operation between the data and the filter coefficients are used as features or a feature map.

[0019] <Packing> Frame packing refers to combining two or more frames (pictures) into a single frame (picture) by arranging them in a tiled pattern. In this application, packing refers to combining feature maps from multiple channels into a single frame. Figure 12 shows an example of frame packing.

[0020] <Data Type> A data type that represents an integer value is called an integer type, and a data type that represents a decimal value is called a decimal type.

[0021] <Layer, Unit> The layers and units handled by the feature map encoding device and feature map decoding device of this embodiment will be described using Figure 14. Reference numeral 181 in Figure 14 denotes a sequence of consecutive feature frames or feature maps for all channels over time, referred to as a sequence layer or sequence unit. Reference numeral 182 in Figure 14 denotes a sequence of consecutive feature maps for one channel over time, referred to as a sequence layer for each channel or a sequence unit for each channel. Reference numeral 183 in Figure 14 denotes a feature frame or feature maps for all channels at the same time, referred to as a frame layer or frame unit. Reference numeral 184 in Figure 14 denotes a feature map for one channel at a certain time, referred to as a feature map layer for each channel (a feature map unit for each channel).

[0022] (First embodiment) A feature map encoding device 100 and a feature map decoding device 200 according to a first embodiment of the present invention will be described.

[0023] 1 is a block diagram of a feature map encoding device 100 according to a first embodiment. The feature map encoding device 100 of this embodiment includes a feature map reduction unit 102, a feature map conversion unit 103, a feature map internal encoding unit 104, a single-scale feature map distribution characteristic calculation unit 105, and a single-scale feature map distribution characteristic encoding unit 106. The feature map encoding device 100 encodes a feature map generated by a neural network feature extraction unit 101 to generate and output a bitstream.

[0024] The neural network feature extraction unit 101 reads an image from which features are to be extracted, generates a feature map by FPN convolution processing, activation processing, and pooling processing, and supplies the generated feature map to the feature map reduction unit 102. In this embodiment, a three-layer multi-scale feature map of x1, x2, and x3 is generated.

[0025] The feature map reduction unit 102 converts the three-layer multi-scale feature maps x1, x2, and x3 acquired from the neural network feature extraction unit 101 into a single-layer single-scale feature map xf, and supplies it to the feature map conversion unit 103. The feature map reduction unit 102 will be described in detail with reference to FIG. 5.

[0026] The feature map conversion unit 103 performs packing and quantization on the decimal-type single-scale feature map xf supplied from the feature map reduction unit 102, converting it into an integer-type packed feature frame, and supplies it to the feature map internal coding unit 104.

[0027] The feature map conversion unit 103 will be described in detail with reference to FIG.

[0028] The feature map intra-encoding unit 104 encodes the integer-type packing feature frames supplied from the feature map conversion unit 103 in accordance with an image encoding standard such as VVC, HEVC, or AV1 to generate and output a bitstream. The output bitstream is supplied to the feature map decoding device 200 or the like via a network or the like.

[0029] The single-scale feature map distribution characteristic calculation unit 105 calculates the standard deviation σorg of all elements of all channels of the small number of single-scale feature maps xf supplied from the feature map reduction unit 102. Details of the single-scale feature map distribution characteristic calculation unit 105 will be described later with reference to FIG. 15. In this embodiment, the standard deviation is used as the distribution characteristic of the single-scale feature map, but other distribution characteristics such as the variance and the mean may also be used, or a combination of these may also be used.

[0030] The single-scale feature map distribution characteristic encoding unit 106 encodes σorg supplied from the single-scale feature map distribution characteristic calculation unit 105 to generate and output a bitstream. The output bitstream is supplied to the feature map decoding device 200 and the like via a network or the like.

[0031] The feature map internal coding unit 104 will be described in detail with reference to FIG.

[0032] 2 is a block diagram showing the configuration of a feature map decoding device 200 according to an embodiment of the present invention, which corresponds to the feature map encoding device 100 in FIG. 1. The feature map decoding device 200 according to this embodiment includes a feature map internal decoding unit 201, a feature map inverse conversion unit 202, a feature map restoration unit 203, a single-scale feature map distribution characteristic decoding unit 205, and a single-scale feature map refinement unit 206. The feature map decoding device 200 receives as input a bitstream encoded by the feature map encoding device 100, decodes the bitstream, and generates three-layer multi-scale feature maps x1up, x2up, and x3up, which it supplies to a neural network identification unit 204.

[0033] The feature map internal decoding unit 201 decodes the bitstream coded by the feature map internal coding unit 104 of the feature map coding device 100 in accordance with an image coding standard such as VVC, HEVC, or AV1, generates integer-type packing feature frames, and supplies them to the feature map inverse conversion unit 202. The feature internal decoding unit 201 will be described in detail with reference to FIG. 10 .

[0034] The feature map inverse conversion unit 202 performs inverse quantization and unpacking on the integer-type packed feature frame supplied from the feature internal decoding unit 201, converting it into a decimal-type single-scale feature map xr, and supplies it to the single-scale feature map refinement unit 206.

[0035] The feature map inverse conversion unit 202 will be described in detail with reference to FIG.

[0036] The feature map restoration unit 203 converts the refined single-scale feature map xr' supplied from the single-scale feature map refinement unit 206 into three-layer multi-scale feature maps x1up, x2up, and x3up, and supplies them to the neural network classification unit 204 as outputs of the feature map decoding device 200. The feature map restoration unit 203 will be described in detail with reference to FIG. 6.

[0037] The neural network classification unit 204 performs classification processing such as classification of objects, places, landscapes, people, and living things in the classification target image based on the three-layer multi-scale feature maps x1up, x2up, and x3up supplied by the feature map restoration unit 203.

[0038] The single-scale feature map distribution characteristic decoding unit 205 decodes the bit stream encoded by the single-scale feature map distribution characteristic encoding unit 106 of the feature map encoding device 100, generates σ org, and supplies it to the single-scale feature map refinement unit 206.

[0039] The single-scale feature map refinement unit 206 performs a refinement process on the decimal-type single-scale feature map xr supplied from the feature map inverse conversion unit 202, using σorg supplied from the single-scale feature map distribution property decoding unit 205, to generate a refined single-scale feature map xr'. The refined single-scale feature map xr' is supplied to the feature map restoration unit 203. Details of the single-scale feature map refinement unit 206 will be described later using FIG. 16.

[0040] <Feature map refinement> Figure 17 shows an example of metadata related to single-scale feature map refinement. refines_single_scale_feature is a flag indicating whether single-scale feature map refinement is enabled or disabled. When refines_single_scale_feature=1, single-scale feature map refinement is enabled and subsequent metadata is transmitted. When refines_single_scale_feature=0, single-scale feature map refinement is disabled. sigma_single_scale_org indicates the standard deviation σorg of all elements of all channels of the single-scale feature map xf, calculated by the single-scale feature map distribution characteristic calculation unit 105 of the feature map encoding device 100.

[0041] In this embodiment, Refinement SEI (Supplemental Enhancement Information) containing only parameters related to refinement, i.e., supplemental information defining refinement, is transmitted independently from other metadata. If the feature map decoding device 200 is unable to acquire the Refinement SEI due to network delays or the like, or if a simple feature map decoding device 200 is unable to interpret the Refinement SEI, the feature map can be decoded without any problems even if the refinement process is skipped, thereby improving the degree of freedom in decoding.

[0042] The operation of the single-scale feature map distribution characteristic calculation unit 105 will be explained using FIG. 15. The user can select whether to enable or disable single-scale feature map refinement. If single-scale feature map refinement is enabled (Yes in step S101), the single-scale feature map distribution characteristic calculation unit 105 calculates the standard deviation σorg of all elements in all channels of the single-scale feature map xf (step S102). The calculated σorg is supplied to the single-scale feature map distribution characteristic coding unit 106 and output as a bitstream (step S103).

[0043] The operation of the single-scale feature map refinement unit 206 will be described with reference to Fig. 16. When single-scale feature map refinement is enabled, that is, when refines_single_scale_feature = 1 (when step S201 is Yes), the single-scale feature map refinement unit 206 obtains σorg decoded by the single-scale feature map distribution characteristic decoding unit 205 (step S202). The single-scale feature map refinement unit 206 calculates the standard deviation σdec of all elements of all channels of the single-scale feature map xr (step S203), and calculates the refinement coefficient α = σorg / σdec (step S204).

[0044] The single-scale feature map refinement unit 206 calculates a refined single-scale feature map xr' by multiplying all elements of all channels of xr by α (step S205).

[0045] The effect of single-scale feature maps will be explained using Figure 22. 401 is xf, i.e., the probability distribution of all elements of all channels of the single-scale feature map (first single-scale feature map) in the feature map encoding device 100. 402 is xr, i.e., the probability distribution of all elements of all channels of the decoded single-scale feature map (second single-scale feature map) in the feature map encoding device 200. 403 is xr', i.e., the probability distribution of all elements of all channels of the refined single-scale feature map (third single-scale feature map).

[0046] In general, the packing feature frame coded by the feature map intra coding unit 104 is quantized, and therefore 402 suffers from coding degradation compared to 401. Due to the coding degradation of 402, the distribution characteristics of 402 vary from the distribution characteristics of 401. By adopting the configuration of this embodiment, it is possible to obtain the difference between the distribution characteristics of 401 before quantization and the distribution characteristics of 402 after quantization, and this difference can be used to correct 402, thereby restoring 403, which has a probability distribution close to 401.

[0047] In particular, since the coding degradation of 401 and 402 is due to rounding processing, the error between 401 and 402 takes a value close to 0 on average. Therefore, the average value of 401 and 402 does not fluctuate much due to coding degradation. The refinement processing in this embodiment can be realized using only a linear term using standard deviation and does not have a constant term, so maximum refinement accuracy can be obtained with simple processing.

[0048] As described above, by adopting this configuration, the feature map of the decoded signal can be restored to values ​​close to those of the feature map of the original signal based on the standard deviation of the feature map of the original signal calculated and transmitted on the encoding side and the distribution characteristics of the feature map of the decoded signal calculated on the decoding side, thereby improving coding efficiency.

[0049] <Feature map reduction and restoration> The feature map reduction unit 102 has a function of converting the multi-layered multi-scale feature map acquired from the neural network feature extraction unit 101 into a single-layered single-scale feature map.

[0050] The feature map reduction unit 102 will be described in detail using Figure 5. The feature map reduction unit 102 is composed of a first feature map reduction unit 501, a first channel combination unit 502, a second feature map reduction unit 503, a second channel combination unit 504, a third feature map reduction unit 505, a first padding unit 506, a second padding unit 507, and a third padding unit 508. The feature map reduction unit 102 in Figure 5 is an example of a configuration that converts a three-layer multi-scale feature map into a single-scale feature map.

[0051] The feature map reduction unit 102 receives three-layer multi-scale feature maps, namely, a first feature map x1, a second feature map x2, and a third feature map x3, as input, converts them into a single-layer single-scale feature map xf, and supplies the result to the feature map conversion unit 103. Here, n denotes an index indicating a layer, Cn denotes the number of channels in the nth layer, Wn denotes the width of the feature map, and Hn denotes the height of the feature map. In this embodiment, the values ​​of Cn, Wn, and Hn for each layer are as shown in FIG. 11, where H and W denote the width and height, respectively, of the image from which features are extracted.

[0052] The first padding unit 506 has a function of padding the first feature map x1 to generate a first padded feature map x1pad. The first padding unit 506 determines the padding size so that the width and height of x1pad are multiples of 64. The number of channels of x1pad is 256, the same as that of x1.

[0053] The second padding unit 507 performs padding on the second feature map x2 by folding to generate a second padded feature map x2pad. The second padding unit 507 determines the padding size so that the width and height of x2pad are multiples of 32. The number of channels of x2pad is 256, the same as that of x2.

[0054] The third padding unit 508 performs padding on the third feature map x3 by folding to generate a third padded feature map x3pad. The third padding unit 508 determines the padding size so that the width and height of x3pad are multiples of 16. The number of channels in x3pad is 256, the same as that of x3.

[0055] In the first padding unit 506, the second padding unit 507, and the third padding unit 508, the left padding size is the same as the right padding size, and the top padding size is the same as the bottom padding size. That is, the feature maps x1, x2, and x3 are positioned at the centers of x1pad, x2pad, and x3pad, respectively.

[0056] The first feature map reduction unit 501 performs convolution in the spatial and channel directions on the first padded feature map x1pad obtained from the first padding unit 506 to generate a first intermediate feature map y1. The number of channels of y1 is 192, its width is Wx1pad / 2, and its height is Hx1pad / 2, where Wx1pad and Hx1pad are the width and height of the first padded feature map x1pad, respectively.

[0057] The first channel combining unit 502 has a function of combining the first intermediate feature map y1 obtained from the first feature map reduction unit 501 and the second padded feature map x2pad obtained from the second padding unit 507 in the channel direction to generate an intermediate feature map y1Cx2pad. Since y1 has 192 channels and x2pad has 256 channels, the intermediate feature map y1Cx2pad has 448 channels (192 + 256).

[0058] The second feature map reduction unit 503 generates a second intermediate feature map y2 by performing convolution in the spatial and channel directions on the intermediate feature map y1Cx2pad obtained from the first channel combination unit 502. The number of channels of y2 is 192, its width is Wy1Cx2pad / 2, and its height is Hy1Cx2pad / 2, where Wy1Cx2pad and Hy1Cx2pad are the width and height of the intermediate feature map y1Cx2pad, respectively.

[0059] The second channel combining unit 504 has a function of combining the second intermediate feature map y2 obtained from the second feature map reduction unit 503 and the third padded feature map x3pad obtained from the third padding unit 508 in the channel direction to generate an intermediate feature map y2Cx3pad. Since the number of channels in the intermediate feature map y2 is 192 and the number of channels in x3pad is 256, the number of channels in y1Cx2pad is 448 (192 + 256).

[0060] The third feature map reduction unit 505 generates a third intermediate feature map y3 by performing convolution in the spatial and channel directions on the intermediate feature map y2Cx3pad obtained from the second channel combination unit 504. The number of channels of y3 is 192, its width is Wy2Cx3pad / 2, and its height is Hy2Cx3pad / 2, where Wy2Cx3pad and Hy2Cx3pad are the width and height of the intermediate feature map y2Cx3pad, respectively.

[0061] The feature map reduction unit 102 outputs the third intermediate feature map y3 as a single-scale feature map xf and supplies it to the feature map conversion unit 103.

[0062] The feature map restoration unit 203 has a function of converting the single-scale feature map xr acquired from the feature map inverse conversion unit 202 into three-layer multi-scale feature maps x1up, x2up, and x3up.

[0063] The feature map restoration unit 203 will be described in detail using Fig. 6. The feature map restoration unit 203 is composed of an 8x enlargement unit 601, a 4x enlargement unit 602, a 2x enlargement unit 603, a first feature map blending unit 604, a second feature map blending unit 605, a first de-padding unit 606, a second de-padding unit 607, and a third de-padding unit 608.

[0064] The 8x enlargement unit 601 performs transposition convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr obtained from the feature map inverse transformation unit 202 to enlarge the feature map and reduce the channels, thereby generating an intermediate feature map z1. The number of channels in z1 is 196. If the width and height of the single-scale feature map xr are xrwidth and xrheight, respectively, the width and height of z1 are xrwidth × 8 and xrheight × 8, respectively. Here, rwidth × 8 and xrheight × 8 are the same as the width and height, respectively, of the first padded feature map x1pad, which is the output of the first padding unit 506 of the feature extraction unit 102.

[0065] The 4x enlargement unit 602 performs transposition convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr obtained from the feature map inverse transformation unit 202 to enlarge the features and reduce the channels, generating an intermediate feature map z2. The number of channels in z2 is 196. The width and height of z2 are xrwidth×4 and xrheight×4, respectively. Here, rwidth×4 and xrheight×4 are the same as the width and height, respectively, of the second padded feature map x2pad, which is the output of the second padding unit 507 of the feature extraction unit 102.

[0066] The 2x enlargement unit 603 performs transposition convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr obtained from the feature map inverse transformation unit 202 to enlarge the feature map and reduce the channels, thereby generating an intermediate feature map z3. The number of channels in z3 is 196. The width and height of z3 are xrwidth×2 and xrheight×2, respectively. Here, rwidth×2 and xrheight×2 are the same as the width and height, respectively, of the third padded feature map x3pad, which is the output of the third padding unit 508 in the feature extraction unit 102.

[0067] The first feature map mixing unit 604 has a function of generating an intermediate feature map z2up by improving the quality of the intermediate feature map z2 obtained from the 4x enlargement unit 602, using the intermediate feature map z1 obtained from the 8x enlargement unit 601.

[0068] The second feature map mixing unit 605 has a function of generating an intermediate feature map z3up that improves the quality of the intermediate feature map z3 obtained from the 2x enlargement unit 603, by using the intermediate feature map z2up obtained from the first feature map mixing unit 604.

[0069] The padding removal unit 606 removes padding from the intermediate feature map z1 acquired from the 8x enlargement unit 601 to generate a first output feature map x1up. The width and height of x1up are the same as the width and height of the first feature map x1 input to the feature extraction unit 102.

[0070] The padding removal unit 607 has a function of removing padding from the intermediate feature map z2up acquired from the first feature map blending unit 604 to generate a second output feature map x2up. The width and height of x2up are the same as the width and height of the second feature map x2 input to the feature extraction unit 102.

[0071] The padding removal unit 608 has a function of removing padding from the intermediate feature map z3up acquired from the second feature map blending unit 605 to generate a third output feature map x3up. The width and height of x3up are the same as the width and height of the third feature map x3 input to the feature extraction unit 102.

[0072] In the first padding removal unit 606, the second padding removal unit 607, and the third padding removal unit 608, the left padding size is the same as the right padding size, and the top padding size is the same as the bottom padding size, similar to the first padding unit 506, the second padding unit 507, and the third padding unit 508 of the feature extraction unit 102. In other words, the padding on the top, bottom, left, and right sides is removed by assuming that each output feature map x1up, x2up, and x3up is placed at the center of each intermediate feature map z1, z2up, and z3up, respectively.

[0073] <Feature map transformation and feature map inverse transformation> The feature map conversion unit 103 has a function of performing packing and quantization on the multi-channel decimal-type single-scale feature map xf supplied from the feature map reduction unit 102, and converting it into an integer-type packed feature frame to be supplied to the feature map internal coding unit 104.

[0074] The feature map conversion unit 103 on the encoding side will be described in detail with reference to Fig. 7. The feature map conversion unit 103 is made up of a packing unit 701 and a feature map quantization unit 702.

[0075] The packing unit 701 has a function of generating a packed feature frame by combining input feature maps of multiple channels into one frame. Fig. 12 is a diagram explaining the state in which feature maps of multiple channels are packed into one frame. The feature maps of each channel are sequentially arranged in one frame in raster scan order, from left to right and top to bottom.

[0076] Furthermore, the packing unit 701 performs flipping based on the position in the frame where the channel feature map is placed. Whether or not to perform flipping can be selected, and the selection may be transmitted from the encoding side to the decoding side via a bitstream.

[0077] Flipping is performed when packing feature maps of multiple channels into one frame, as shown in FIG. 13. In flipping, the positions of the elements (pixels) of each channel's feature map are flipped horizontally (left and right), vertically (up and down), or horizontally and vertically (up and down, left and right) when packing the feature maps of each channel. The feature maps of four channels, A (upper left), B (upper right), C (lower left), and D (lower right) in FIG. 13, are grouped together as one set. No flipping is performed at position A in FIG. 13. At position B, the feature map is flipped horizontally (left and right). At position C, the feature map is flipped vertically (up and down). At position D, the feature map is flipped horizontally and vertically (up and down, left and right). When the distribution of elements in the feature maps of each channel is similar, flipping based on the position within the frame where the channels are placed reduces the boundaries between the feature maps of each channel, improving coding efficiency.

[0078] The feature map quantization unit 702 has a function of converting elements of a decimal-type packing feature frame (feature map of all channels) into N-bit integer type elements (N=an integer from about 8 to 16) within a predetermined range, and outputting an integer-type packing feature frame. In this embodiment, the elements are converted into 10-bit integer type elements ranging from 0 to 1023. The feature map quantization unit 702 detects the minimum and maximum values ​​of the elements of the decimal-type packing feature frame, and transmits the detected minimum and maximum values ​​of the elements of the packing feature frame (feature map of all channels) to the decoding side as metadata. To convert from a decimal-type packing feature frame (feature map) to an integer-type packing feature frame (feature map), a linear conversion is performed in which the minimum value of the decimal type is equivalent to the minimum value of the integer type, and the maximum value of the decimal type is equivalent to the maximum value of the integer type. For example, when the integer type range is expressed in 10 bits, the minimum value of the elements of the integer-type packing feature frame (feature map) is 0 and the maximum value is 1023 (2 10 -1). Values ​​between the minimum and maximum values ​​are linearly quantized.

[0079] Next, the feature map inverse transform unit 202 has a function of performing inverse quantization and unpacking on the integer-type packing feature frame decoded by VVC, HEVC, AV1, etc., supplied from the feature map internal decoding unit 201, and performing inverse transform into a decimal-type single-scale feature map xr to be supplied to the feature map restoration unit 203.

[0080] The feature map inverse transformation unit 202 on the decoding side will be described in detail with reference to Fig. 8. The feature map inverse transformation unit 202 performs the inverse process of the feature map transformation unit 103, and is made up of a feature map inverse quantization unit 801 and an unpacking unit 802.

[0081] The feature map inverse quantization unit 801 performs the inverse process of the feature map quantization unit 702 on the encoding side, and has a function of converting elements of integer-type packing feature frames from integer type to decimal type. The feature map inverse quantization unit 801 converts the integer-type packing feature frame decoded by the feature map internal decoding unit 201 into a decimal-type packing feature frame using the minimum and maximum decimal values ​​transmitted as metadata. A linear conversion is performed to convert the integer minimum value to the decimal minimum value and the integer maximum value to the decimal maximum value. Linear inverse quantization is performed on values ​​between the minimum and maximum values.

[0082] The unpacking unit 802 extracts a feature map for each channel from the packing feature frames arranged in one frame in raster scan order and supplies it to the feature map restoration unit 203 as a single-scale feature map xr.

[0083] <Feature map inner encoding and decoding> The feature map internal coding unit 104 will be described in detail using Figure 9. The feature map internal coding unit 104 is made up of a switch 901, a VVC encoding unit 902, an HEVC encoding unit 903, and an AV1 encoding unit 904. The switch 901 selects the encoding standard for internally coding the feature map converted by the feature map conversion unit 103. The VVC encoding unit 902 encodes the feature map in the VVC standard and outputs a bitstream that complies with the VVC standard. The HEVC encoding unit 903 encodes the feature map in the HEVC standard and outputs a bitstream that complies with the HEVC standard. The AV1 encoding unit 904 encodes the feature map in the AV1 standard and outputs a bitstream that complies with the AV1 standard.

[0084] In the VVC, HEVC, and AV1 standards, images are divided into blocks of a specified size and then encoded.

[0085] It is also possible to implement only one of VVC, HEVC, and AV1, and it is also possible to use image coding methods other than VVC, HEVC, and AV1.

[0086] Next, the feature map internal decoding unit 201 will be described in detail with reference to Fig. 10. The feature map internal decoding unit 201 is made up of a switch 1001, a VVC decoding unit 1002, an HEVC decoding unit 1003, and an AV1 decoding unit 1004. The switch 1001 selects the encoding standard for internal decoding based on information for selecting internal decoding in the input bitstream. The VVC decoding unit 1002 decodes the feature map in accordance with the VVC standard. The HEVC decoding unit 1003 decodes the feature map in accordance with the HEVC standard. The AV1 encoding unit 1004 decodes the feature map in accordance with the AV1 standard.

[0087] In the VVC, HEVC, and AV1 standards, decoding is performed for each predetermined block size.

[0088] It is also possible to implement only one of VVC, HEVC, and AV1, and it is also possible to use image coding methods other than VVC, HEVC, and AV1. (Second embodiment) In the first embodiment, we have described a feature map encoding device 100 and a feature map decoding device 200 that refine single-scale feature maps. Similarly, refinement can also be performed on multi-scale feature maps.

[0089] 18 and 19 are block diagrams of a feature map encoding device 100 and a feature map decoding device 200 according to the second embodiment, respectively.

[0090] The feature map encoding device 100 in Figure 18 differs from the feature map encoding device 100 in Figure 1 in that it includes a multi-scale feature map distribution characteristic calculation unit 107 and a multi-scale feature map distribution characteristic encoding unit 108 instead of the single-scale feature map distribution characteristic calculation unit 105 and the single-scale feature map distribution characteristic encoding unit 106 of the feature map encoding device 100 in Figure 1. The feature map decoding device 200 in Figure 19 differs from the feature map decoding device 200 in Figure 2 in that it includes a multi-scale feature map distribution characteristic decoding unit 207 and a multi-scale feature map refinement unit 208 instead of the single-scale feature map distribution characteristic decoding unit 205 and the single-scale feature map refinement unit 206.

[0091] FIG. 23 shows an example of metadata according to this embodiment. refines_multi_scale_features is a flag indicating whether multi-scale feature map refinement is enabled or disabled. When refines_multi_scale_features=1, multi-scale feature map refinement is enabled, and subsequent metadata is transmitted. When refines_multi_scale_features=0, multi-scale feature map refinement is disabled. In this embodiment, refines_multi_scale_features is shared by all layers. In other words, it is possible to set whether refinement is enabled or disabled for all layers, and enable / disable control is not performed for each layer. However, a configuration in which enable / disable is switched on a layer-by-layer basis by setting refines_multi_scale_features for each layer is also possible. i is a variable indicating the layer. sigma_multi_scale_org[i] denotes the standard deviation σ(n)org of all elements in all channels of the multi-scale feature map xn (n=i+1, n=1, 2, 3) of layer i, calculated by the multi-scale feature map distribution characteristic calculation unit 107 of the feature map encoding device 100.

[0092] The operation of the multi-scale feature map distribution characteristic calculation unit 107 will be described with reference to FIG. 20. The user can select whether to enable or disable multi-scale feature map refinement. If multi-scale feature map refinement is enabled (if step S301 is Yes), the multi-scale feature map distribution characteristic calculation unit 107 calculates the standard deviation σ(n)org of all elements of all channels of the single-scale feature map xn in layer n in order (S302) for all multi-scale feature map layers (step S303). The calculated σ(n)org is supplied to the multi-scale feature map distribution characteristic coding unit 108 and output as a bitstream (step S103).

[0093] The operation of the multi-scale feature map refinement unit 208 will be described with reference to Fig. 21. When multi-scale feature map refinement is enabled, that is, when refines_multi_scale_features = 1 (Yes in step S401), the multi-scale feature map refinement unit 208 acquires σ(n)org decoded by the multi-scale feature map distribution characteristic decoding unit 207 for all multi-scale feature map layers in order (No in S402) (step S403). For each multi-scale feature map xnup (n = 1, 2, 3), the multi-scale feature map refinement unit 208 calculates the standard deviation σ(n)dec of all elements of all channels (step S404) and calculates the refinement coefficient α(n) = σ(n)org / σ(n)dec (step S405).

[0094] The refined multi-scale feature map x1up' is calculated by multiplying all elements of all channels of x1up by α(1). The same calculation is performed for n=2 and 3 to calculate x2up' and x3up' (step S406). Here, xnup' corresponds to x(n)refine in Figure 21.

[0095] By adopting this embodiment, refinement can be performed not only on single-scale feature maps but also on each layer of multi-scale feature maps, which prevents degradation in coding of packed feature maps and improves coding efficiency.

[0096] By combining the first and second embodiments, it is possible to create a configuration that refines both single-scale feature maps and multi-scale feature maps.

[0097] In all of the above-described embodiments, the bitstream output by the feature map encoder has a specific data format that can be decoded according to the encoding method used in the embodiment, and the feature map decoder corresponding to the feature map encoder can decode the bitstream in this specific data format.

[0098] When a wired or wireless network is used to exchange bitstreams between the feature map encoder and the feature map decoder, the bitstreams may be converted into a data format suitable for the transmission mode of the communication channel before transmission. In this case, a transmitter is provided that converts the bitstream output by the feature map encoder into coded data in a data format suitable for the transmission mode of the communication channel and transmits the coded data to the network, and a receiver is provided that receives the coded data from the network, restores the coded data to a bitstream, and supplies the bitstream to the feature map decoder. The transmitter includes a memory that buffers the bitstream output by the feature map encoder, a packet processor that packetizes the bitstream, and a transmitter that transmits the packetized coded data via the network. The receiver includes a receiver that receives the packetized coded data via the network, a memory that buffers the received coded data, and a packet processor that packetizes the coded data to generate a bitstream and provides it to the feature map decoder.

[0099] The above encoding and decoding processes may be realized not only as a transmission, storage, and receiving device using hardware, but also as firmware stored in a ROM (read-only memory) or flash memory, or as software for a computer, etc. The firmware program or software program may be provided by recording it on a computer-readable recording medium, or may be provided from a server via a wired or wireless network, or may be provided as data broadcasting on terrestrial or satellite digital broadcasting.

[0100] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the components and treatment processes, and that such modifications are also within the scope of the present invention. [Explanation of symbols]

[0101] 100 Feature map encoding device, 101 Neural network feature extraction unit, 102 Feature map reduction unit, 103 Feature map conversion unit, 104 Feature map internal encoding unit, 105 Single-scale feature map distribution characteristic calculation unit, 106 Single-scale feature map distribution characteristic encoding unit, 107 Multi-scale feature map distribution characteristic calculation unit 107, 108 Multi-scale feature map distribution characteristic encoding unit 108, 200 Feature map decoding device, 201 Feature map internal decoding unit, 202 Feature map inverse conversion unit, 203 Feature map restoration unit, 204 Neural network identification unit, 205 Single-scale feature map distribution characteristic decoding unit, 206 Single-scale feature map refinement unit, 207 Multi-scale feature map distribution characteristic decoding unit, 208 Multi-scale feature map refinement unit 208, 301 Convolution processing unit, 302 Activation processing unit, 303 Pooling processing unit, 322 Bottom-up processing unit, 324 Top-down processing unit, 326 Image to be subjected to feature extraction processing, 501 First feature map reduction unit, 502 First channel combining unit, 503 Second feature map reduction unit, 504 Second channel combining unit, 505 Third feature map reduction unit, 506 First padding unit, 507 Second padding unit, 508 Third padding unit, 601 8x enlargement unit, 602 4x enlargement unit, 603 2x enlargement unit, 604 First feature map mixing unit, 605 Second feature map mixing unit, 606 First unpadding unit, 607 Second unpadding unit, 608 Third unpadding unit, 701 Packing unit, 702 Feature map quantization unit, 801 Feature map inverse quantization unit, 802 Unpacking unit, 901 Switch, 902 VVC encoding unit, 903 HEVC encoding unit, 904 AV1 encoding unit, 1001 switch, 1002 VVC decoding unit, 1003 HEVC decoding unit, 1004 AV1 decoding unit.

Claims

1. a feature map reducer that converts the multi-scale feature maps to generate single-scale feature maps; a single-scale feature map distribution characteristic calculation unit that calculates a distribution characteristic of the single-scale feature map; a packing unit that packs the single-scale feature map into a frame to generate a fractional packed feature frame; a quantization unit that converts elements of the minority-type packing feature frame into integer values ​​to generate an integer-type packing feature frame; a single-scale feature map distribution characteristic encoding unit that encodes the distribution characteristic; a feature map inner coding unit that codes the integer-type packing feature frame; Equipped with A feature map encoding device characterized by:

2. a single-scale feature map distribution characteristic decoding unit that decodes a first distribution characteristic of the first single-scale feature map; a feature map internal decoder that decodes the frame-packed and encoded single-scale feature map to generate an integer-packed feature frame; an inverse quantization unit that converts elements of the integer-type packing feature frame into decimal values ​​to generate a decimal-type packing feature frame; an unpacking unit that divides the fractional packing feature frame into one or more packing groups based on information about the packing groups and unpacks the fractional packing feature frame to generate a second single-scale feature map; a single-scale feature map refinement unit that calculates a second distribution characteristic of the second single-scale feature map, and generates a third single-scale feature map by refining the second single-scale feature map based on the first distribution characteristic and the second distribution characteristic; a feature map restoration unit that transforms the third single-scale feature map to generate a multi-scale feature map; Equipped with A feature map decoding device characterized by:

Images