Feature map encoding device, feature map encoding method, feature map decoding device, and feature map decoding method

Abstract

A feature map encoding device (100) determines, from feature maps of a plurality of channels, an active channel to be encoded, generates one packing feature frame by packing only the feature map of the active channel, and generates a bit stream by encoding the packing feature frame.

Description

Feature Map Encoding Device, Feature Map Encoding Method, Feature Map Decoding Device, Feature Map Decoding Method

[0001] It relates to the encoding and decoding of feature maps in a neural network.

[0002] As a neural network technology used for image recognition such as detecting objects of various scales in an image, dividing regions for each object, or tracking objects, FPN (Feature Pyramid Network) in Non-Patent Document 1 is known. In FPN, a plurality of feature maps of various scales are generated from the image to be processed, and various image recognitions are performed using the feature maps.

[0003] FPN used for image recognition generates a plurality of feature maps from an image, and its structure uses CNN (Convolutional Neural Network). CNN reads an image, is composed of convolution and pooling, and can be divided into a feature extraction unit (backbone) that generates feature maps and an identification unit (head) that is composed of a hierarchical fully connected layer and generates outputs suitable for tasks such as object detection, instance segmentation, and object tracking. FPN uses the backbone of CNN.

[0004] The feature extraction unit of FPN is typically structured with a convolutional processing unit 301, an activation processing unit 302, and a pooling processing unit 303 shown in FIG. 3 as one basic unit, and this basic unit is repeated in a hierarchical structure.

[0005] Figure 4 shows the structure of the FPN. The FPN consists of a bottom-up processing unit 322 that generates a multi-scale feature map composed of multiple hierarchical layers using the CNN backbone, and a top-down processing unit 324 that aggregates features from deeper layer feature maps to shallower layer feature maps using the inverse configuration of the CNN backbone. The bottom-up processing unit 322 repeatedly performs the convolution processing unit 301, activation processing unit 302, and pooling processing unit 303, which are the basic units in Figure 3, reducing the resolution of the feature map by half each time to generate a pyramid of multiple layered feature maps. On the other hand, the top-down processing unit 324 adds feature maps with resolutions corresponding to the bottom-up processing unit 322, expanding the resolution of the feature map to the same resolution as the input image, and generating a pyramid of feature maps. In other words, the FPN generates multiple feature maps for each layer from the image 326 that is the target of feature extraction processing.

[0006] The convolution processing unit 301 performs convolution on the data to be processed (image or feature map) using a plurality of predetermined filters (kernels). In the convolution processing in the convolution processing unit 301, predetermined filtering is performed on the entire data to be processed while sliding at predetermined intervals. At this time, the sliding interval is called the stride. The convolution processing unit 301 may determine the stride based on the number of data to be processed. For example, the convolution processing unit 301 may determine the stride to be 1 if the number of data to be processed is less than a predetermined value, and determine the stride to be 2 if it is greater than or equal to the predetermined value. Multiple feature maps are generated by preparing a plurality of predetermined filters at each layer and generating one feature map for each filter. The unit of a feature map is called a channel. If the number (types) of predetermined filters is N (N types), then N (N channels) of feature maps are generated.

[0007] The activation processing unit 302 performs an activation process that non-linearly transforms the feature map output from the convolution processing unit 301. Here, the function used for the activation process is called the activation function. The activation processing unit 302 uses the ReLU (Rectified Linear Unit) function or the sigmoid function, etc., as the activation function.

[0008] The pooling processing unit 303 performs a process of downsampling the feature map by replacing the local values ​​of the feature map output from the activation processing unit 302 with representative values.

[0009] On the other hand, when performing classification using a neural network, it has the capability to execute the task using feature maps of multiple channels at each layer.

[0010] In image recognition, the multi-channel feature maps of each hierarchical level are subjected to a convolution process at predetermined size intervals based on the scale of the feature maps, and the probability of the object's class is calculated for each pixel.

[0011] Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature Pyramid Networks for Object Detection. In CVPR, 2017.

[0012] The amount of information in feature maps is enormous, making them unsuitable for transmission and storage. In view of the above problems, this embodiment aims to provide a technology for encoding and decoding feature maps.

[0013] To solve the above problems, a feature map encoding device in one aspect of this embodiment is a feature map encoding device that encodes a multi-scale feature map, comprising: a packing unit that packs multiple channel feature maps to generate a single packing feature frame; a feature map encoding information encoding unit that encodes feature map encoding information to obtain a bitstream of feature map encoding information; and a feature map internal encoding unit that encodes the packing feature frame to generate a bitstream, wherein the packing unit determines the active channel to be encoded from the multiple channel feature maps, and packs only the feature map of the active channel to generate a single packing feature frame. Another aspect of this embodiment is a feature map encoding device. This device is a feature map encoding device for encoding multi-scale feature maps, comprising: a packing unit that packs multiple channel feature maps to generate a single packing feature frame; a feature map analysis and conversion unit that calculates distribution characteristic information of the packing feature frame and converts the data type of the packing feature frame; a feature map encoding information encoding unit that encodes feature map encoding information to obtain a bitstream of feature map encoding information; and a feature map internal encoding unit that encodes the packing feature frame to generate a bitstream. The packing unit determines the active channel to be encoded from the multiple channel feature maps, packs only the feature map of the active channel to generate a single packing feature frame, and the feature map encoding information includes the distribution characteristic information.

[0014] Another aspect of this embodiment is a feature map decoding device. This device is a feature map decoding device that decodes a bitstream on which a multi-scale feature map is encoded, and comprises: an internal feature map decoding unit that decodes the bitstream and generates a packing feature frame; a feature map encoded information decoding unit that decodes the bitstream and obtains feature map encoded information; and an unpacking unit that unpacks the packing feature frame and generates a multi-channel fractional feature map, wherein the unpacking unit generates inactive channels that are not to be decoded. Another aspect of this embodiment is a feature map decoding device. This device is a feature map decoding device that decodes a bitstream on which a multi-scale feature map is encoded, and comprises: an internal feature map decoding unit that decodes the bitstream and generates a packing feature frame; a feature map encoded information decoding unit that decodes the bitstream and obtains feature map encoded information; a feature map refining unit that calculates distribution characteristic information of the packing feature frame, refines the packing feature frame based on the distribution characteristic information contained in the decoded encoded information and obtains a refined packing feature frame; and an unpacking unit that unpacks the refined packing feature frame and generates a multi-channel fractional feature map, wherein the unpacking unit generates inactive channels that are not to be decoded.

[0015] According to this embodiment, feature maps can be encoded and decoded efficiently with minimal processing load.

[0016] This is a block diagram illustrating the configuration of the feature map encoding device 100. This is a block diagram illustrating the configuration of the feature map decoding device 200. This is a block diagram illustrating the basic unit processing at each layer of the FPN. This is a diagram illustrating the structure of the FPN. This is a block diagram illustrating the detailed configuration of the feature map reduction unit 102. This is a block diagram illustrating the detailed configuration of the feature map restoration unit 203. This is a block diagram illustrating the detailed configuration of the feature map conversion unit 103. This is a block diagram illustrating the detailed configuration of the feature map inverse conversion unit 202. This is a block diagram illustrating the detailed configuration of the feature map internal encoding unit 104. This is a block diagram illustrating the detailed configuration of the feature map internal decoding unit 201. This is a diagram illustrating the number of channels, width, and height of feature maps x1, x2, and x3. This is a diagram illustrating an example of the positional relationship of packing feature frames. This is a diagram illustrating flipping when packing multiple channel feature maps into one frame. This is a diagram illustrating the layers and units handled by the feature map encoding device and feature map decoding device of this embodiment. This is a flowchart illustrating the feature map encoding processing procedure according to the first embodiment. This is a flowchart illustrating the feature map decoding procedure according to the first embodiment. This is an example of the syntax rules for active channel-related feature map coding information according to the first embodiment. This is another example of the syntax rules for active channel-related feature map coding information according to the first embodiment. This is a flowchart illustrating the coding procedure for active channel-related feature map coding information according to the first embodiment. This is a flowchart illustrating the decoding procedure for active channel-related feature map coding information according to the first embodiment. This is a diagram illustrating an example of a state in which multiple channel feature maps are packed into one frame. This is an example of the syntax rules for active channel-related feature map coding information according to the second embodiment. This is a flowchart illustrating the coding procedure for active channel-related feature map coding information according to the second embodiment.This is a flowchart illustrating the decoding process for active channel-related feature map coding information according to the second embodiment. This is a block diagram illustrating the configuration of the feature map conversion unit 103 according to the third embodiment. This is a block diagram illustrating the configuration of the feature map inverse conversion unit 202 according to the third embodiment. This is a flowchart illustrating the operation of the feature map analysis conversion unit 704 according to the third embodiment. This is a flowchart illustrating the operation of the feature map refinement unit 804 according to the third embodiment. This is a diagram illustrating the syntax elements of distribution characteristic information related to the feature map refinement process of the third embodiment. This is a flowchart illustrating the operation of the feature map analysis conversion unit 704 according to the third embodiment. This is a flowchart illustrating the operation of the feature map refinement unit 804 according to the third embodiment. This is a diagram illustrating the feature map refinement process. This is another example of the syntax rules for active channel-related feature map coding information according to the first embodiment.

[0017] This section defines the technologies and technical terms used in this embodiment.

[0018] <Features and Feature Maps> In a convolutional neural network (CNN), the data obtained by convolving the data of the portion scanned while changing its position within the target image (input layer data) using a filter, with the filter coefficients, is called a feature or feature map.

[0019] <Packing> Frame packing refers to the process of combining two or more frames (pictures) into a single frame (picture) by arranging them in a tile-like manner. In this application, packing refers to the process of combining feature maps of multiple channels into a single frame.

[0020] <Data Types> Data types that represent integer values ​​are designated as integer types, and data types that represent decimal values ​​are designated as decimal types.

[0021] <Layers and Units> The layers and units handled by the feature map encoding and feature map decoding devices of this embodiment will be explained using Figure 14. Figure 14 shows a sequence of consecutive feature frames or feature maps for all channels over time, and is referred to as a sequence layer or sequence unit. Figure 14 shows a sequence of consecutive feature maps for one channel over time, and is referred to as a sequence layer for each channel or a sequence unit for each channel. Figure 14 shows a feature frame or feature map for all channels at the same time, and is referred to as a frame layer or frame unit. Figure 14 shows a feature map for one channel at a certain time, and is referred to as a feature map layer for each channel (a feature map unit for each channel).

[0022] (First Embodiment) A feature map encoding device 100 and a feature map decoding device 200 according to the first embodiment of the present invention will be described.

[0023] Figure 1 is a block diagram of a feature map encoding device 100 according to the first embodiment. The feature map encoding device 100 of this embodiment includes a feature map reduction unit 102, a feature map conversion unit 103, and a feature map internal encoding unit 104. The feature map encoding device 100 is a device that encodes the feature map generated by the neural network feature extraction unit 101 to generate a bitstream and output it.

[0024] The neural network feature extraction unit 101 reads the image to be feature extracted, generates a feature map through FPN convolution, activation, and pooling processes, and supplies it to the feature map reduction unit 102. In this embodiment, a three-layer multi-scale feature map x1, x2, and x3 is generated.

[0025] The feature map reduction unit 102 converts the three-layer multi-scale feature maps x1, x2, and x3 obtained from the neural network feature extraction unit 101 into a single-layer single-scale feature map xf and supplies it to the feature map conversion unit 103. The details of the feature map reduction unit 102 will be explained in detail with reference to Figure 5.

[0026] The feature map conversion unit 103 takes the fractional single-scale feature map xf supplied from the feature map reduction unit 102, performs packing and quantization processing to convert it into an integer-type packed feature frame, and supplies it to the feature map internal encoding unit 104.

[0027] The details of the feature map conversion unit 103 will be explained in detail with reference to Figure 7.

[0028] The feature map internal encoding unit 104 encodes the integer-type packing feature frame supplied from the feature map conversion unit 103 using an image encoding standard such as VVC, HEVC, or AV1 to generate and output a bitstream. The output bitstream is supplied to the feature map decoding device 200 or the like via a network or the like.

[0029] The details of the feature map internal encoding unit 104 will be explained in detail with reference to Figure 9.

[0030] Figure 2 is a block diagram showing the configuration of a feature map decoding device 200 according to an embodiment of the present invention, corresponding to the feature map encoding device 100 in Figure 1. The feature map decoding device 200 of this embodiment includes a feature map internal decoding unit 201, a feature map inverse conversion unit 202, and a feature map reconstruction unit 203. The feature map decoding device 200 also receives a bitstream encoded by the feature map encoding device 100, decodes the bitstream to generate three-layer multi-scale feature maps x1up, x2up, and x3up, and supplies them to the neural network identification unit 204.

[0031] The feature map internal decoding unit 201 decodes the bitstream encoded by the feature map internal encoding unit 104 of the feature map encoding device 100 using an image encoding standard such as VVC, HEVC, or AV1, generates an integer-type packing feature frame, and supplies it to the feature map inverse transform unit 202.

[0032] The details of the internal feature decoding unit 201 will be explained in detail with reference to Figure 10.

[0033] The feature map inverse transformation unit 202 performs inverse quantization and unpacking on the integer-type packing feature frame supplied from the feature map internal decoding unit 201, converts it into a decimal-type single-scale feature map xr, and supplies it to the feature map reconstruction unit 203.

[0034] The details of the feature map inverse transformation unit 202 will be explained in detail with reference to Figure 8.

[0035] The feature map reconstruction unit 203 converts the single-scale feature map xr supplied from the feature map inverse transformation unit 202 into three-layer multi-scale feature maps x1up, x2up, and x3up, and supplies them to the neural network identification unit 204 as the output of the feature map decoding device 200.

[0036] The details of the feature map restoration unit 203 will be explained in detail with reference to Figure 6.

[0037] The neural network identification unit 204 performs identification processing such as identifying objects in the target image, identifying locations and landscapes, and identifying people and living things, based on the three-layer multi-scale feature maps x1up, x2up, and x3up supplied by the feature map reconstruction unit 203.

[0038] <About Feature Map Reduction and Feature Map Restoration> The feature map reduction unit 102 has the function of converting a multi-layer multi-scale feature map obtained from the neural network feature extraction unit 101 into a single-layer single-scale feature map.

[0039] The feature map reduction unit 102 will be explained in detail using Figure 5. The feature map reduction unit 102 consists of a first feature map reduction unit 501, a first channel merging unit 502, a second feature map reduction unit 503, a second channel merging unit 504, a third feature map reduction unit 505, a first padding unit 506, a second padding unit 507, and a third padding unit 508. The feature map reduction unit 102 in Figure 5 is an example of a configuration that converts a three-layer multi-scale feature map into a single-scale feature map.

[0040] The feature map reduction unit 102 takes as input three layers of multi-scale feature maps, namely the first feature map x1, the second feature map x2, and the third feature map x3, and converts them into a single-layer single-scale feature map xf, which is then supplied to the feature map conversion unit 103. Here, let the index indicating the layer be n, the number of channels of the n-th layer be Cn, the width of the feature map be Wn, and the height of the feature map be Hn. In the present embodiment, the values of Cn, Wn, and Hn for each layer are as shown in FIG. 11. However, H and W are respectively the width and height of the image for which feature extraction is performed.

[0041] The first padding unit 506 has a function of performing padding on the first feature map x1 to generate the first padded feature map x1pad. In the first padding unit 506, the padding size is determined such that the width and height of x1pad are multiples of 64. The number of channels of x1pad is the same as that of x1, which is 256.

[0042] The second padding unit 507 has a function of performing padding on the second feature map x2 by folding to generate the second padded feature map x2pad. In the second padding unit 507, the padding size is determined such that the width and height of x2pad are multiples of 32. The number of channels of x2pad is the same as that of x2, which is 256.

[0043] The third padding unit 508 has a function of performing padding on the third feature map x3 by folding to generate the third padded feature map x3pad. In the third padding unit 508, the padding size is determined such that the width and height of x3pad are multiples of 16. The number of channels of x3pad is the same as that of x3, which is 256.

[0044] In the first padding unit 506, the second padding unit 507, and the third padding unit 508, the left padding size and the right padding size are the same, and the upper padding size and the lower padding size are the same. That is, the feature maps x1, x2, and x3 are respectively arranged at the centers of x1pad, x2pad, and x3pad.

[0045] The first feature map reduction unit 501 performs convolution in the spatial and channel directions on the first padded feature map x1pad obtained from the first padding unit 506 to generate the first intermediate feature map y1. The number of channels of y1 is 192, the width is Wx1pad / 2, and the height is Hx1pad / 2. Here, Wx1pad and Hx1pad are the width and height of the first padded feature map x1pad, respectively.

[0046] The first channel combination unit 502 has a function of combining the first intermediate feature map y1 obtained from the first feature map reduction unit 501 and the second padded feature map x2pad obtained from the second padding unit 507 in the channel direction to generate the intermediate feature map y1Cx2pad. Since the number of channels of y1 is 192 and the number of channels of x2pad is 256, the number of channels of the intermediate feature map y1Cx2pad is 448 (192 + 256).

[0047] The second feature map reduction unit 503 performs convolution in the spatial and channel directions on the intermediate feature map y1Cx2pad obtained from the first channel combination unit 502 to generate the second intermediate feature map y2. The number of channels of y2 is 192, the width is Wy1Cx2pad / 2, and the height is Hy1Cx2pad / 2. Here, Wy1Cx2pad and Hy1Cx2pad are the width and height of the intermediate feature map y1Cx2pad, respectively.

[0048] The second channel combination unit 504 has a function of combining the second intermediate feature map y2 obtained from the second feature map reduction unit 503 and the third padded feature map x3pad obtained from the third padding unit 508 in the channel direction to generate the intermediate feature map y2Cx3pad. Since the number of channels of the intermediate feature map y2 is 192 and the number of channels of x3pad is 256, the number of channels of y1Cx2pad is 448 (192 + 256).

[0049] The third feature map reduction unit 505 performs convolution in the spatial and channel directions on the intermediate feature map y2Cx3pad obtained from the second channel coupling unit 504 to generate a third intermediate feature map y3. The number of channels in y3 is 192, the width is Wy2Cx3pad / 2, and the height is Hy2Cx3pad / 2. Here, Wy2Cx3pad and Hy2Cx3pad are the width and height of the intermediate feature map y2Cx3pad, respectively.

[0050] The feature map reduction unit 102 outputs the third intermediate feature map y3 as a single-scale feature map xf and supplies it to the feature map conversion unit 103.

[0051] The feature map reconstruction unit 203 has the function of converting the single-scale feature map xr obtained from the feature map inverse transformation unit 202 into three-layer multi-scale feature maps x1up, x2up, and x3up.

[0052] The details of the feature map restoration unit 203 will be explained using Figure 6. The feature map restoration unit 203 consists of an 8x magnification unit 601, a 4x magnification unit 602, a 2x magnification unit 603, a first feature map mixing unit 604, a second feature map mixing unit 605, a first padding removal unit 606, a second padding removal unit 607, and a third padding removal unit 608.

[0053] The 8x magnification unit 601 has the function of expanding the feature map and reducing the number of channels of the single-scale feature map xr obtained from the feature map inverse transformation unit 202 by performing transposition convolution in the spatial direction and convolution in the channel direction, thereby generating an intermediate feature map z1. The number of channels in z1 is 196. If the width and height of the single-scale feature map xr are xrwidth and xrheight, respectively, then the width and height of z1 will be xrwidth × 8 and xrheight × 8, respectively. Here, rwidth × 8 and xrheight × 8 are the same as the width and height of the first padded feature map x1pad, which is the output of the first padding unit 506 of the feature map reduction unit 102.

[0054] The quadruple magnification unit 602 has the function of expanding features and reducing channels by performing transposition convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr obtained from the feature map inverse transformation unit 202, thereby generating an intermediate feature map z2. The number of channels in z2 is 196. The width and height of z2 are xrwidth × 4 and xrheight × 4, respectively. Here, rwidth × 4 and xrheight × 4 are the same as the width and height of the second padded feature map x2pad, which is the output of the second padding unit 507 of the feature map reduction unit 102.

[0055] The doubling unit 603 has the function of expanding the feature map and reducing the number of channels of the single-scale feature map xr obtained from the feature map inverse transformation unit 202 by performing transposition convolution in the spatial direction and convolution in the channel direction, thereby generating an intermediate feature map z3. The number of channels in z3 is 196. The width and height of z3 are xrwidth × 2 and xrheight × 2, respectively. Here, rwidth × 2 and xrheight × 2 are the same as the width and height of the third padded feature map x3pad, which is the output of the third padding unit 508 of the feature map reduction unit 102.

[0056] The first feature map mixing unit 604 has the function of generating an intermediate feature map z2up, which is an improved version of the intermediate feature map z2 obtained from the 4x magnification unit 602, using the intermediate feature map z1 obtained from the 8x magnification unit 601.

[0057] The second feature map mixing unit 605 has the function of generating an intermediate feature map z3up, which is an improved version of the intermediate feature map z3 obtained from the doubling unit 603, using the intermediate feature map z2up obtained from the first feature map mixing unit 604.

[0058] The padding removal unit 606 has the function of removing padding from the intermediate feature map z1 acquired from the 8x magnification unit 601 and generating a first output feature map x1up. The width and height of x1up are the same as the width and height of the first feature map x1 input to the feature map reduction unit 102.

[0059] The padding removal unit 607 has the function of removing padding from the intermediate feature map z2up obtained from the first feature map mixing unit 604 and generating a second output feature map x2up. The width and height of x2up are the same as the width and height of the second feature map x2 input to the feature map reduction unit 102.

[0060] The padding removal unit 608 has the function of removing padding from the intermediate feature map z3up obtained from the second feature map mixing unit 605 and generating a third output feature map x3up. The width and height of x3up are the same as the width and height of the third feature map x3 input to the feature map reduction unit 102.

[0061] In the first padding removal unit 606, the second padding removal unit 607, and the third padding removal unit 608, similar to the first padding unit 506, the second padding unit 507, and the third padding unit 508 of the feature map reduction unit 102, the padding size on the left and the padding size on the right are set to be the same, and the padding size on the upper and lower sides are set to be the same. That is, each output feature map x1up, x2up, and x3up are assumed to be positioned at the center of each intermediate feature map z1, z2up, and z3up, respectively, and padding is removed from the top, bottom, left, and right.

[0062] <About Feature Map Conversion and Inverse Feature Map Conversion> The feature map conversion unit 103 has the function of performing packing and quantization processing on the multi-channel fractional single-scale feature map xf supplied from the feature map reduction unit 102, and converting the resulting integer-type packed feature frame into an integer-type packed feature frame for supply to the feature map internal encoding unit 104.

[0063] The details of the feature map conversion unit 103 on the encoding side will be explained using Figure 7. The feature map conversion unit 103 consists of a packing unit 701, a feature map quantization unit 702, and a feature map encoding information encoding unit 703.

[0064] The packing unit 701 has the function of generating a packed packing feature frame by combining the input feature maps of multiple channels into a single frame.

[0065] In this embodiment, for a single-scale feature map xf of multiple channels supplied by the feature map reduction unit 102, the active channels to be encoded and the inactive channels to be omitted from encoding are determined, and only the active channels are packed into a packing feature frame and encoded.

[0066] Channels that are encoded by the encoding-side feature map encoding device 100 and decoded by the decoding-side feature map decoding device 200 are designated as active channels. Conversely, channels that are not encoded by the encoding side but are generated by the decoding side are designated as inactive channels. Inactive channels are treated as feature maps with predetermined elements on the decoding side.

[0067] Furthermore, the packing unit 701 performs flipping based on the in-frame position where the channel feature map is placed. The ability to select whether or not to perform flipping may be provided and transmitted from the encoding side to the decoding side via the bitstream.

[0068] Figure 13 illustrates the flipping process when packing multiple channel feature maps into a single frame. Flipping involves inverting the position of each channel's feature map elements (pixels) horizontally (left / right), vertically (up / down), or both horizontally and vertically (up / down / left / right) when packing the feature maps for each channel. In Figure 13, the four channel feature maps A (top left), B (top right), C (bottom left), and D (bottom right) are considered a single set. No flipping is performed at position A in Figure 13. At position B, the feature map is inverted horizontally (left / right). At position C, the feature map is inverted vertically (up / down). At position D, the feature map is inverted horizontally and vertically (up / down / left / right). When the distribution of elements in each channel's feature map is similar, performing flipping based on the in-frame position where the channels are placed reduces the boundaries between each channel's feature map, improving encoding efficiency.

[0069] The packing unit 701 supplies the packing feature frame to the feature map quantization unit 702, and also supplies information such as the total number of channels, the number of active channels, and an index for identifying the active channels as feature map encoded information to the feature map encoded information encoding unit 703.

[0070] The feature map quantization unit 702 has the function of converting the elements of a decimal-type packing feature frame (feature map of all channels) into an N-bit integer type (an integer N = approximately 8 to 16) within a predetermined range and outputting an integer-type packing feature frame. In this embodiment, it is assumed that it is converted into a 10-bit integer type from 0 to 1023. The feature map quantization unit 702 determines the minimum and maximum values ​​of the elements of the decimal-type packing feature frame (feature map of all channels) and supplies the determined minimum and maximum values ​​of the elements of the decimal-type packing feature frame (feature map) to the feature map coding information coding unit 703 as one of the feature map coding information. In order to convert from a decimal-type packing feature frame (feature map) to an integer-type packing feature frame (feature map), a linear transformation is performed in which the minimum value of the decimal type corresponds to the minimum value of the integer type, and the maximum value of the decimal type corresponds to the maximum value of the integer type. For example, when the range of the integer type is represented by 10 bits, the minimum value of the elements of the integer-type packing feature frame (feature map) is 0, and the maximum value is 1023 (2 １０ -1) This is the result. Linear quantization is performed on values ​​between the minimum and maximum values.

[0071] The feature map coding information encoding unit 703 encodes the feature map coding information for each channel supplied from the packing unit 701 and the feature map quantization unit 702, and transmits it to the decoding side through the feature map internal encoding unit 104. The encoding procedure for the feature map coding information for each channel will be described later.

[0072] Next, the feature map inverse transform unit 202 has the function of performing inverse quantization and unpacking on the integer-type packing feature frame decoded by VVC, HEVC, AV1, etc. supplied from the feature map internal decoding unit 201, and then inversely transforming it into a decimal-type single-scale feature map xr for supply to the feature map reconstruction unit 203.

[0073] The details of the feature map inverse transformation unit 202 on the decoding side will be explained using Figure 8. The feature map inverse transformation unit 202 is the inverse process of the feature map transformation unit 103 and consists of a feature map encoded information decoding unit 801, a feature map inverse quantization unit 802, and an unpacking unit 803.

[0074] The feature map encoding information decoding unit 801 decodes the bitstream containing the feature map encoding information supplied through the feature map internal decoding unit 201, and supplies the resulting feature map encoding information for each channel to the feature map inverse quantization unit 802 and the unpacking unit 803. The decoding procedure for the feature map encoding information for each channel will be described later.

[0075] The feature map inverse quantization unit 802 performs the inverse processing of the encoding-side feature map quantization unit 702 and has the function of converting the elements of the integer-type packing feature frame from integer type to decimal type. The feature map inverse quantization unit 802 converts the integer-type packing feature frame decoded by the feature map internal decoding unit 201 into a decimal-type packing feature frame using the minimum and maximum decimal values ​​transmitted as metadata. A linear transformation is performed to make the minimum integer value equivalent to the minimum decimal value and the maximum integer value equivalent to the maximum decimal value. For values ​​between the minimum and maximum values, linear inverse quantization is performed.

[0076] The unpacking unit 803 extracts the feature maps of each active channel from the packing feature frames arranged in one frame in a predetermined raster scan order. Furthermore, it generates inactive channels. The decoded active channels and the generated inactive channels are supplied to the feature map reconstruction unit 203 as a single-scale feature map xr. At this time, the feature maps of each active channel are extracted sequentially from the top left of the packing feature frame, from left to right, and from top to bottom, in a predetermined raster scan order. For channels that are not decoded, inactive channels are generated. Active channels are arranged in the order indicated by the index, with the decoded active channel being placed for channels whose index has been decoded, and the inactive channel being placed for channels whose index has not been decoded.

[0077] <About Internal Feature Map Encoding and Decoding> Figure 9 will be used to explain the details of the feature map internal encoding unit 104. The feature map internal encoding unit 104 consists of a switch 901, a VVC encoding unit 902, a HEVC encoding unit 903, an AV1 encoding unit 904, and a multiplexing unit 905. The switch 901 selects an encoding standard for internally encoding the feature map converted by the feature map conversion unit 103. The VVC encoding unit 902 encodes the packing feature frame using the VVC standard and outputs a bitstream compliant with the VVC standard. The HEVC encoding unit 903 encodes the packing feature frame using the HEVC standard and outputs a bitstream compliant with the HEVC standard. The AV1 encoding unit 904 encodes the packing feature frame using the AV1 standard and outputs a bitstream compliant with the AV1 standard.

[0078] In the VVC, HEVC, and AV1 standards, images are divided into predetermined block sizes for encoding.

[0079] It is also possible to implement only one of the following: VVC, HEVC, or AV1. Furthermore, it is possible to use image encoding schemes other than VVC, HEVC, and AV1.

[0080] Furthermore, the multiplexing unit 905 multiplexes the bitstream containing the encoded feature map information and the bitstream containing the encoded packing feature frame, and outputs it as a multiplexed bitstream.

[0081] Next, the details of the feature map internal decoding unit 201 will be explained using Figure 10. The feature map internal decoding unit 201 consists of a multiplexing / decoupling unit 1005, a switch 1001, a VVC decoding unit 1002, a HEVC decoding unit 1003, and an AV1 decoding unit 1004. The switch 1001 selects the encoding standard to be internally decoded based on the information in the input bitstream that is selected for internal decoding.

[0082] The multiplexing and separation unit 1005 multiplexes and separates the multiplexed bitstream supplied from the encoding side, separating the bitstream on which the feature map encoding information is encoded from the VVC / HEVC / AV1 standard-compliant bitstream on which the packing feature frame is encoded.

[0083] The VVC decoding unit 1002 decodes the bitstream in which the packing feature frame is encoded according to the VVC standard and generates a packing feature frame. The HEVC decoding unit 1003 decodes the bitstream in which the packing feature frame is encoded according to the HEVC standard and generates a packing feature frame. The AV1 decoding unit 1004 decodes the bitstream in which the packing feature frame is encoded according to the AV1 standard and generates a packing feature frame.

[0084] In the VVC, HEVC, and AV1 standards, decoding is performed for each predetermined block size.

[0085] It is also possible to implement only one of the following: VVC, HEVC, or AV1. Furthermore, it is possible to use image encoding schemes other than VVC, HEVC, and AV1.

[0086] <Regarding the Feature Map Encoding / Decoding Process Procedure> Next, the feature map encoding process procedure performed by the feature map reduction unit 102, the feature map conversion unit 103, and the feature map internal encoding unit 104 of the encoded map encoding unit 100 will be explained. Figure 15 is a flowchart illustrating the feature map encoding process procedure according to the first embodiment.

[0087] First, the packing unit 701 of the feature map conversion unit 103 evaluates the importance of the feature map for each channel and determines the active and inactive channels (step S1001). By omitting the encoding of inactive channels, which are channels that are not important, encoding efficiency is improved, and encoding degradation of high-importance channels that have a significant impact on the image recognition result is suppressed.

[0088] Next, the packing unit 701 of the feature map conversion unit 103 determines the order of the feature maps of the active channels (step S1002). The order of the feature maps of the active channels can be arbitrarily determined based on the importance of each channel, the correlation between channels, etc. Figure 21 shows an example of the active channel feature maps packed into one frame. The total number of channels, the number of active channels, and the number of inactive channels can be arbitrarily set according to the importance of the feature maps, etc. For example, channels whose importance exceeds a predetermined threshold can be set as active channels, and the other channels can be set as inactive channels. Alternatively, a representative importance can be calculated for all channels, and channels whose importance exceeds this representative importance can be set as active channels, and the other channels can be set as inactive channels. Representative importance can be calculated using statistical information such as the mean or median. In this embodiment, the total number of channels is 64, the number of active channels is 48, and the number of inactive channels is 16. The numbers shown in Figure 21 indicate an index for identifying active channels. Channels 2, 8, 15, 17, 21, 24, 26, 32, 36, 38, 39, 41, 44, 45, 49, and 62, which are not shown in Figure 21, are inactive channels. The encoding / decoding of the active channel index will be described later.

[0089] Next, the packing unit 701 packs the feature maps of each channel according to the determined order of the feature maps of the active channels to generate a packing feature frame (step S1003 in Figure 15). Inactive channels that are not to be encoded are discarded. If there are gaps even after placing all the feature maps of the active channels in the packing feature frame, a predetermined value (such as 0 or the average value) is set for the elements of the gaps.

[0090] The packing unit 701 supplies the packing feature frame to the feature map quantization unit 702, and also supplies information such as the total number of channels, the number of active channels, and the index of the active channels to the feature map coding information encoding unit 703 as feature map coding information.

[0091] Next, the feature map quantization unit 702 of the feature map conversion unit 103 quantizes the packing feature frame and converts the packing feature frame from a decimal type to an integer type (step S1004 in Figure 15).

[0092] Next, the feature map encoding information encoding unit 703 of the feature map conversion unit 103 encodes feature map encoding information such as the total number of channels, the number of active channels, the index of the active channels, and information regarding quantization, and generates a bitstream with the encoded feature map encoding information (step S1005 in Figure 15). The encoding of feature map encoding information will be described later.

[0093] Next, the VVC encoding unit 902, HEVC encoding unit 903, or AV1 encoding unit 904 of the feature map internal encoding unit 104 encodes the packing feature frame and generates a bitstream in which the packing feature frame has been encoded (step S1006 in Figure 15).

[0094] Next, the multiplexing unit 905 of the feature map internal encoding unit 104 multiplexes the bitstream in which the feature map encoding information is encoded with the bitstream in which the feature frames are encoded to generate a multiplexed bitstream (step S1007 in Figure 15).

[0095] Next, the feature map decoding process performed by the feature map internal decoding unit 201 and the feature map inverse transformation unit 202 of the coded map decoding unit 200 will be described. Figure 16 is a flowchart illustrating the feature map decoding process according to the first embodiment.

[0096] In the multiplexing / decoupling unit 1005 of the feature map internal decoding unit 201, the multiplexed bitstream output on the encoding side is multiplexed / decoupled, separating the bitstream in which the feature map encoding information is encoded from the bitstream in which the packing feature frame is encoded (step S2001 in Figure 16).

[0097] Next, the VVC decoding unit 1002, HEVC decoding unit 1003, or AV1 decoding unit 1004 of the feature map internal decoding unit 201 decodes the bitstream in which the packing feature frame is encoded and generates a packing feature frame (step S2002 in Figure 16).

[0098] Next, the feature map encoding information decoding unit 801 of the feature map inverse transform unit 202 decodes the bitstream on which the feature map encoding information has been encoded to generate feature map encoding information such as the total number of channels, the number of active channels, the index of the active channels, and information regarding quantization (step S2003 in Figure 16). The decoding of the feature map encoding information will be described later.

[0099] Next, the feature map inverse quantization unit 802 of the feature map inverse transformation unit 202 converts the packing feature frame from integer type to decimal type (step S2004 in Figure 16).

[0100] Next, the unpacking unit 803 of the feature map inverse transform unit 202 unpacks the packed feature frame and generates a feature map for each active channel (step S2005 in Figure 16).

[0101] Next, the unpacking unit 803 of the feature map inverse transform unit 202 generates a feature map for the inactive channel (step S2006 in Figure 16). Specifically, in step S2003, channels whose index of the active channel was not decoded are designated as inactive channels, and a predetermined value such as 0 or the average value is set for each element (each pixel) of the feature map of the inactive channel to generate the feature map for the inactive channel.

[0102] <Regarding the encoding / decoding process of feature map encoding information related to active channels in the first embodiment> In the first embodiment, in addition to information regarding the total number of channels, information regarding the number of active channels is encoded / decoded, and the channel index is encoded / decoded for each active channel. On the decoding side, the index of each inactive channel is generated and implicitly set.

[0103] The encoding / decoding of active channel-related feature map coding information in the first embodiment, performed by the feature map coding information encoding unit 703 and the feature map coding information decoding unit 801, will be described in detail below.

[0104] Figure 17 shows an example of the syntax rules for active channel-related feature map coding information on a frame-by-frame basis (183 in Figure 14) according to the first embodiment. Figure 18 also shows an example of the syntax rules for encoding / decoding active channel-related feature map coding information using a feature_channel_header, which is prepared for each channel feature map (184 in Figure 14) according to the first embodiment.

[0105] The syntax element `num_feature_channel` indicates the total number of channels (the sum of active and inactive channels), while the syntax element `num_active_feature_channel` indicates the number of active channels.

[0106] The variable n is an integer variable that increases by 1 from 0 to num_active_feature_channel-1 (number of active channels - 1). The value of variable n indicates a predetermined raster scan order. Figure 12 shows an example of the positional relationship between variable n and packing feature frames. The numbers shown in Figure 12 indicate the value of variable n.

[0107] The syntax element feature_channel_index[n] where n is 0 to num_active_feature_channel-1 (number of active channels - 1) indicates the index of the active channel that will be placed in the nth position in a given raster scan order. The syntax element feature_channel_index[n] where n is num_active_feature_channel (number of active channels) to num_feature_channel-1 (total number of channels - 1) indicates the index of the remaining inactive channels excluding the active channels.

[0108] When encoding / decoding each syntax element, arithmetic coding is used, where symbols with a higher probability of occurrence are assigned shorter codes, and symbols with a lower probability of occurrence are assigned longer codes.

[0109] Figure 33 shows another example of the syntax rules for active channel-related feature map coding information on a frame-by-frame basis (183 in Figure 14) according to the first embodiment. In the example in Figure 33, the syntax element num_non_active_feature_channel, which indicates the number of inactive channels, is transmitted instead of the syntax element num_active_feature_channel, which indicates the number of active channels. The number of active channels, num_active_feature_channel, is calculated by the difference between the syntax element num_feature_channel, which indicates the total number of channels, and the syntax element num_non_active_feature_channel, which indicates the number of inactive channels.

[0110] Next, the encoding process procedure for active channel-related feature map encoding information according to the first embodiment will be described. Figure 19 is a flowchart illustrating the encoding process procedure for active channel-related feature map encoding information according to the first embodiment.

[0111] First, the syntax element num_feature_channel, which indicates the total number of channels, is encoded (step S1101 in Figure 19). In this embodiment, the value of the syntax element num_feature_channel is 64.

[0112] Next, the syntax element num_active_feature_channel, which indicates the number of active channels, is encoded (step S1102 in Figure 19). In this embodiment, the value of the syntax element num_active_feature_channel is 48.

[0113] Let n be the index of each channel. In a predetermined raster scan order, the encoding process of the syntax element feature_channel_index[n] is repeatedly performed (step S1104 in Figure 19) until n is from 0 to num_active_feature_channel-1 (number of active channels - 1), that is, until the encoding process of the indices of all active channels is completed, and the variable n is incremented by 1 (steps S1103 to S1105 in Figure 19). In the example shown in Figure 21, the elements 48, 56, 59, 58, 23, 7, 6, 35, 20, 1, 42, 27, 0, 30, 16, 63, 9, 54, 14, 43, 61, 11, 60, 55, 46, 51, 50, 22, 13, 3, 10, 18, 40, 52, 28, 31, 37, 5, 34, 33, 47, 12, 53, 4, 29, 19, 57, and 25 shown in Figure 21 are set to the active channel index feature_channel_index[n] (where n ranges from 0 to num_active_feature_channel-1).

[0114] Next, the decoding procedure for feature map coding information related to the active channel according to the first embodiment will be described. Figure 20 is a flowchart illustrating the decoding procedure for feature map coding information related to the active channel according to the first embodiment.

[0115] First, the syntax element num_feature_channel, which indicates the total number of channels, is decoded (step S1201 in Figure 20). Next, the syntax element num_active_feature_channel, which indicates the number of active channels, is decoded (step S1202 in Figure 20).

[0116] Next, the index of each channel is set to n, and the decoding process of the syntax element feature_channel_index[n] is repeatedly performed in a predetermined raster scan order, from n = 0 to num_active_feature_channel-1 (number of active channels - 1), that is, until the decoding process of the feature map coding information associated with all active channels is completed (step S1204 in Figure 20), and the variable n is incremented by 1 (steps S1203 to S1205 in Figure 20).

[0117] Next, let n be the index of each channel, and in a predetermined raster scan order, the index of the inactive channel, feature_channel_index[n], is set for all inactive channels excluding the active channel, from n from num_active_feature_channel to num_feature_channel-1 (total number of channels - 1) (step S1207 in Figure 20), and the variable n is incremented by 1 (steps S1206 to S1208 in Figure 20). Here, the index of the inactive channel, feature_channel_index[n] (where n is from num_active_feature_channel to num_feature_channel-1), is set to a unique value other than the index of the decoded active channel. In the example in Figure 21, the inactive channel indices 2, 8, 15, 17, 21, 24, 26, 32, 36, 38, 39, 41, 44, 45, 49, and 62, which are not shown in Figure 21, are set to feature_channel_index[n] (where n ranges from num_active_feature_channel to num_feature_channel-1).

[0118] In step S1207, each element (each pixel) of the feature map of the channel for which the index of the inactive channel has been set is assigned a predetermined value such as 0 or the average value.

[0119] In the first embodiment, the syntax element feature_channel_index[n] indicating the index of each active channel in a predetermined raster scan order is encoded, and the decoding side decodes the syntax element feature_channel_index[n] indicating the index of each active channel in a predetermined raster scan order, thereby allowing the decoding side to identify the active channel based on the index of each active channel. Furthermore, the syntax element num_active_feature_channel indicating the number of active channels is encoded, and the decoding side decodes the syntax element num_active_feature_channel, thereby reducing the amount of encoding required for the syntax element feature_channel_index[n] by omitting the encoding of the inactive channel index on the encoding side and generating the inactive channel index on the decoding side. Even in a configuration where the syntax element num_non_active_feature_channel, which indicates the number of active channels, is encoded as num_non_active_feature_channel, and the number of active channels is calculated using the total number of channels num_feature_channels and the number of inactive channels num_non_active_feature_channel, the effect of reducing the coding amount of the syntax element feature_channel_index[n] by omitting the encoding of the index of inactive channels is similarly achieved.

[0120] (Second Embodiment) A feature map encoding device 100 and a feature map decoding device 200 according to a second embodiment of the present invention will be described.

[0121] The configuration of the feature map encoding device 100 according to the second embodiment is the same as that of the feature map encoding device 100 according to the first embodiment, and the configuration of the feature map decoding device 200 according to the second embodiment is the same as that of the feature map decoding device 200 according to the first embodiment.

[0122] The second embodiment differs from the first embodiment in its syntax and the encoding / decoding procedure for the associated active channel-related feature map encoding information.

[0123] <Regarding the encoding / decoding process of feature map encoding information related to active channels in the second embodiment> In the second embodiment, information regarding the total number of channels is encoded / decoded, and a flag indicating whether each channel is an active channel is encoded / decoded. If it is an active channel, the channel index is encoded / decoded. Furthermore, on the decoding side, if it is an inactive channel, the channel index is generated to implicitly set it.

[0124] The encoding / decoding process of active channel-related feature map encoding information in the second embodiment, performed by the feature map encoding information encoding unit 703 and the feature map encoding information decoding unit 801, will be described in detail below.

[0125] Figure 22 shows an example of the syntax rules for active channel-related feature map encoding information on a frame-by-frame basis (183 in Figure 14) according to the second embodiment. The syntax rules shown in Figure 22 according to the second embodiment differ from the syntax rules shown in Figure 17 according to the first embodiment in that the syntax elements active_channel_flag[n] and feature_channel_index[n] are encoded instead of the syntax element num_active_feature_channel indicating the number of active channels, and the loop repetition conditions are different.

[0126] Similar to the first embodiment, the syntax element num_feature_channel represents the total number of channels (the sum of active and inactive channels).

[0127] The variable n is an integer variable, and the variable SigFlag indicates that the syntax element active_channel_flag[n] is being encoded / decoded. When the variable SigFlag is 1 (true), it increases by 1 from 0 to num_feature_channel-1 (total number of channels - 1).

[0128] The syntax element active_channel_flag[n] indicates a flag that shows whether the nth channel in a given raster scan order is an active channel. An active_channel_flag[n] of 1 indicates that the nth channel is an active channel, and an active_channel_flag[n] of 0 indicates that the nth channel is an inactive channel. Once active_channel_flag[n] is set to 0, the variable SigFlag is set to 0, and active_channel_flag[n] is no longer encoded / decoded. If active_channel_flag[n] is not encoded / decoded, active_channel_flag[n] is implicitly set to 0, indicating an inactive channel. Also, similar to the first embodiment, the syntax element feature_channel_index[n] indicates the index of the active channel that is placed in the nth position in a given raster scan order.

[0129] When encoding / decoding each syntax element, arithmetic coding is used, where symbols with a higher probability of occurrence are assigned shorter codes, and symbols with a lower probability of occurrence are assigned longer codes.

[0130] Next, the coding procedure for feature map coding information related to the active channel according to the second embodiment will be described. Figure 23 is a flowchart illustrating the coding procedure for feature map coding information related to the active channel according to the second embodiment.

[0131] First, the syntax element num_feature_channel, which indicates the total number of channels, is encoded (step S1301 in Figure 23). In this embodiment, the value of the syntax element num_feature_channel is 64.

[0132] Next, the variable SigFlag is set to 1 (true) (step S1302 in Figure 23).

[0133] Next, let n be the index of each channel, and in a predetermined raster scan order, the encoding process of the syntax element active_channel_flag[n] and the syntax element feature_channel_index[n] is repeatedly performed for each channel, where the variable SigFlag is 1 (true) and n is from 0 to num_feature_channel-1 (total number of channels - 1), thereby incrementing the variable n by 1 (steps S1303 to S1308 in Figure 23).

[0134] The processes in steps S1303 to S1308 of Figure 23 will be explained in more detail. If the variable SigFlag is 1 (true) and n is between 0 and num_feature_channel-1 (total number of channels - 1) (YES in step S1303 of Figure 23), the syntax element active_channel_flag[n] is encoded (step S1304 of Figure 23). Furthermore, if the syntax element active_channel_flag[n] is 1 (true) (YES in step S1305 of Figure 23), the syntax element feature_channel_index[n] is encoded (step S1306 of Figure 23). On the other hand, if the syntax element active_channel_flag[n] is 0 (false) (NO in step S1305 of Figure 23), the syntax element feature_channel_index[n] is not encoded, and the variable SigFlag is set to 0 (false) (step S1307 of Figure 23).

[0135] If the variable SigFlag is 0 (false), or if the value of the variable n reaches num_feature_channel (the total number of channels) (NO in step S1303 in Figure 23), the iterative processing of steps S1303 to S1308 is terminated, and this encoding process is ended. As a result, the encoding process for subsequent syntax elements active_channel_flag[n] is not performed, and the syntax element active_channel_flag[n] is implicitly set to 0 (false).

[0136] Next, the decoding procedure for active channel-related feature map coding information according to the second embodiment will be described. Figure 23 is a flowchart illustrating the decoding procedure for active channel-related feature map coding information according to the second embodiment.

[0137] First, the syntax element num_feature_channel, which indicates the total number of channels, is decoded (step S1401 in Figure 24).

[0138] Next, the variable SigFlag is set to 1 (true) (step S1402 in Figure 24).

[0139] Next, let n be the index of each channel, and in a predetermined raster scan order, the decoding process of the syntax elements active_channel_flag[n] and feature_channel_index[n] is repeatedly performed for the variable SigFlag to be 1 (true) and n from 0 to num_feature_channel-1 (total number of channels - 1), and the variable n is incremented by 1 (steps S1403 to S1408 in Figure 24).

[0140] The processes in steps S1403 to S1408 of Figure 24 will be explained in more detail. If the variable SigFlag is 1 (true) and n is between 0 and num_feature_channel-1 (total number of channels - 1) (YES in step S1403 of Figure 24), the syntax element active_channel_flag[n] is decoded (step S1404 of Figure 24). Furthermore, if the syntax element active_channel_flag[n] is 1 (true) (YES in step S1405 of Figure 24), the syntax element feature_channel_index[n] is decoded (step S1406 of Figure 24). On the other hand, if the syntax element active_channel_flag[n] is 0 (false) (NO in step S1405 of Figure 24), the syntax element feature_channel_index[n] is not decoded, and the variable SigFlag is set to 0 (false) (step S1407 of Figure 24).

[0141] If the variable SigFlag is 0 (false), or if the value of the variable n reaches num_feature_channel (the total number of channels) (NO in step S1403 in Figure 24), the iterative processing of steps S1403 to S1408 is terminated. As a result, the decoding process for subsequent syntax elements active_channel_flag[n] is not performed, and the syntax element active_channel_flag[n] is implicitly set to 0 (false).

[0142] Next, the index of each channel is set to n, and in a predetermined raster scan order, the index of the inactive channel, feature_channel_index[n], is set for all remaining inactive channels excluding the active channel, up to n = num_feature_channel-1 (total number of channels - 1) (step S1410 in Figure 23), and the variable n is incremented by 1 (steps S1409 to S1411 in Figure 23). Here, similar to step S1207 in the first embodiment, the index of the inactive channel, feature_channel_index[n], is set to a non-repeating value other than the index of the decoded active channel.

[0143] In step S1410, each element (each pixel) of the feature map of the channel for which the index of the inactive channel has been set is assigned a predetermined value such as 0 or the average value.

[0144] Similar to the first embodiment, in the second embodiment as well, the encoding side encodes the syntax element active_channel_flag[n] in a predetermined raster scan order, and if the syntax element active_channel_flag[n] is 1 (true), encodes the syntax element feature_channel_index[n] indicating the index of each active channel. The decoding side decodes the syntax element active_channel_flag[n] in a predetermined raster scan order, and if the syntax element active_channel_flag[n] is 1 (true), decodes the syntax element feature_channel_index[n] indicating the index of each active channel. This allows the decoding side to identify the active channel based on the index of each active channel. Furthermore, by encoding the syntax element active_channel_flag[n] and decoding it on the decoding side, if the syntax element active_channel_flag[n] is 0 (false) on the encoding side, the encoding of the inactive channel index is omitted and the inactive channel index is generated on the decoding side, thereby reducing the amount of code in the syntax element feature_channel_index[n].

[0145] Furthermore, if the syntax element active_channel_flag[n] becomes 0, the encoding / decoding of the syntax element active_channel_flag[n] for subsequent channels is omitted, and the channel is implicitly determined to be inactive, thereby reducing the amount of code in the syntax element active_channel_flag[n].

[0146] (Third Embodiment) A feature map encoding device 100 and a feature map decoding device 200 according to a third embodiment of the present invention will be described.

[0147] The configuration of the feature map encoding device 100 according to the third embodiment is the same as that of the feature map encoding device 100 according to the first embodiment, and the configuration of the feature map decoding device 200 according to the third embodiment is the same as that of the feature map decoding device 200 according to the first embodiment. However, the differences are that in the feature map conversion unit 103, the feature map quantization unit 702 of the first embodiment is changed to a feature map analysis conversion unit described later in the third embodiment, and in the feature map inverse conversion unit 202, the feature map inverse quantization unit 802 of the first embodiment is changed to a feature map refinement unit described later in the third embodiment.

[0148] <Regarding Feature Map Conversion and Inverse Feature Map Conversion in the Third Embodiment> The feature map conversion unit 103 of the third embodiment has the function of performing packing processing and distribution characteristic calculation processing on the multi-channel decimal single-scale feature map xf supplied from the feature map reduction unit 102, and converting the generated integer-type packing feature frame into an integer-type packing feature frame for supply to the feature map internal encoding unit 104.

[0149] Using Figure 25, the details of the feature map conversion unit 103 on the encoding side of the third embodiment will be explained. The feature map conversion unit 103 consists of a packing unit 701, a feature map analysis conversion unit 704, and a feature map encoding information encoding unit 703. The difference from the feature map conversion unit 103 of the first embodiment shown in Figure 7 is that the feature map quantization unit 702 of the first embodiment is replaced in the feature map analysis conversion unit 704, which will be described later, in the third embodiment.

[0150] The processing of the packing unit 701 in the third embodiment is the same as that of the packing unit 701 in the first embodiment. However, in the packing unit 701 of the third embodiment, the generated packing feature frame is supplied to the feature map analysis and conversion unit 704.

[0151] The feature map analysis and conversion unit 704 analyzes the elements of the fractional packing feature frame supplied from the packing unit 701 to calculate distribution characteristic information, and also performs data type conversion processing of the packing feature frame, converting the elements of the fractional packing feature frame (feature map of all channels) into an N-bit integer type within a predetermined range (an integer N = approximately 8 to 16), and has the function of outputting an integer-type packing feature frame. In this embodiment, it is converted to a 10-bit integer type from 0 to 1023.

[0152] The feature map analysis and conversion unit 704 calculates the standard deviation σorg and mean value morg as distribution characteristic information for the elements of the packing feature map. Calculating the distribution characteristic information for the elements of the packing feature map is equivalent to calculating the distribution characteristic information for the elements of the feature map of the active channels, excluding the inactive channels. Details of the distribution characteristic calculation method of the feature map analysis and conversion unit 704 will be described later with reference to Figure 30. The calculated distribution characteristic information, i.e., the standard deviation and mean value of the packing feature map, is supplied to the encoded map encoded information encoding unit 703 as feature map encoded information. In this embodiment, the standard deviation and mean value are used as the distribution characteristics of the packing feature map, but other distribution characteristics such as variance may be used, or a combination of them may be used.

[0153] The feature map coding information encoding unit 703 encodes feature map coding information that includes the total number of channels, the number of active channels, and an index for identifying active channels supplied from the packing unit 701, as well as distribution characteristic information such as the standard deviation σorg and mean value morg supplied from the feature map analysis and conversion unit 704, and transmits it to the decoding side through the feature map internal encoding unit 104. The encoding procedure for feature map coding information will be described later.

[0154] On the other hand, the feature map inverse conversion unit 202 of the third embodiment has the function of performing refinement and unpacking processes on integer-type packing feature frames decoded by VVC, HEVC, AV1, etc. supplied from the feature map internal decoding unit 201, and inversely converting the resulting decimal-type single-scale feature map xr into a decimal-type single-scale feature map xr for supply to the feature map reconstruction unit 203.

[0155] Using Figure 26, the details of the feature map inverse transformation unit 202 on the decoding side of the third embodiment will be explained. The feature map inverse transformation unit 202 of the third embodiment is the inverse process of the feature map transformation unit 103 of the third embodiment and is composed of a feature map encoded information decoding unit 801, a feature map refinement unit 804, and an unpacking unit 803. The difference between the feature map inverse transformation unit 202 of the first embodiment shown in Figure 8 and the feature map inverse transformation unit 202 of the third embodiment is that the feature map inverse quantization unit 802 of the first embodiment is changed to a feature map refinement unit 804 in the third embodiment.

[0156] The feature map coding information decoding unit 801 decodes the bitstream containing the feature map coding information supplied through the feature map internal decoding unit 201, and supplies the resulting feature map coding information to the feature map refinement unit 804 and the unpacking unit 803. The decoded feature map coding information also includes distribution characteristic information such as the standard deviation σorg and the mean value morg. The decoding procedure for feature map coding information will be described later.

[0157] The feature map refinement unit 804 refines the mapping feature frame decoded by the feature map internal decoding unit 201 using distribution characteristic information such as the standard deviation σorg and mean value morg supplied from the feature map encoding information decoding unit 801, thereby generating a refined mapping feature frame. The refined mapping feature frame is then supplied to the unpacking unit 803. Details of the feature map refinement unit 804 will be described later with reference to Figure 31.

[0158] The processing of the unpacking unit 803 in the third embodiment is the same as that of the unpacking unit 803 in the first embodiment.

[0159] <Regarding the Feature Map Encoding / Decoding Processing Procedure of the Third Embodiment> Next, the feature map encoding processing procedure performed by the feature map reduction unit 102, feature map conversion unit 103, and feature map internal encoding unit 104 of the encoding map encoding unit 100 of the third embodiment will be described. Figure 27 is a flowchart illustrating the feature map encoding processing procedure according to the third embodiment. Steps with the same symbols as those in Figure 15 are the same processing as in Figure 15. That is, steps S1001, S1002, S1003, S1005, S1006, and S1007 in Figure 27 are the same processing as steps S1001, S1002, S1003, S1005, S1006, and S1007 in Figure 15, respectively. However, the feature map encoding information encoded in step S1005 of Figure 27 also includes distribution characteristic information such as the standard deviation σorg and the mean value morg.

[0160] In the third embodiment, the feature map analysis and conversion unit 704 of the feature map conversion unit 103 calculates distribution characteristic information such as the standard deviation σorg and mean value morg of the packing feature frame, and uses the calculated distribution characteristic information to convert the packing feature frame from decimal type to integer type (step S1008 in Figure 27).

[0161] Next, the feature map decoding process performed in the feature map internal decoding unit 201 and the feature map inverse transform unit 202 of the coded map decoding unit 200 of the third embodiment will be described. Figure 28 is a flowchart illustrating the feature map decoding process according to the first embodiment. Steps with the same code as those in Figure 16 are the same processes as in Figure 16. That is, steps S2001, S2002, S2003, S2005, and S2006 in Figure 28 are the same processes as steps S2001, S2002, S2003, S2005, and S2006 in Figure 16, respectively. However, the feature map coded information decoded in step S2003 of Figure 28 also includes distribution characteristic information such as the standard deviation σorg and the mean value morg.

[0162] In the feature map refinement unit 804 of the feature map inverse transformation unit 202 of the third embodiment, distribution characteristic information such as the standard deviation σorg and mean value morg of the decoded packing feature frame is calculated, and a refinement process is performed using the calculated distribution characteristic information and the decoded distribution characteristic information to convert the packing feature frame from integer type to decimal type (step S2007 in Figure 28).

[0163] <Calculation / encoding / decoding process of distribution characteristic information of feature map in the third embodiment> Figure 29 shows an example of syntax rules for distribution characteristic information encoded in the feature map encoded information encoding unit 703 and step S1005 in Figure 27, and decoded in the feature map encoded information decoding unit 801 and step S2003 in Figure 28. In the third embodiment, distribution characteristic information is encoded / decoded on a picture-by-picture basis. The syntax element feature_refine_flag is a flag that indicates whether to enable or disable the feature map refinement process. When feature_refine_flag is 1, the feature map refinement process is enabled, and the syntax elements feature_std and feature_mean are encoded / decoded. When feature_refine_flag is 0, the feature map refinement process is disabled. The syntax element feature_std indicates the standard deviation σorg of the elements of the packing feature frame calculated by the feature map encoded information encoding unit 703. The syntax element feature_mean represents the average value (morg) of the elements of the packing feature frame calculated by the feature map coding information coding unit 703.

[0164] The feature map distribution characteristic calculation encoding process of the third embodiment will be explained using Figure 30. This operation corresponds to step S1008 in Figure 27. The user can choose whether to enable or disable the feature map analysis and transformation process on the encoding side and the feature map refinement process on the decoding side. If the feature map refinement process on the decoding side is enabled, the value of the syntax element feature_std is set to 1, and if the feature map refinement process on the decoding side is enabled, the value of the syntax element feature_std is set to 0 and encoded (step S1501). If the feature map analysis and transformation process on the encoding side and the feature map refinement process on the decoding side are enabled (Yes in step S1502), the feature map analysis and transformation unit 704 calculates the distribution characteristics such as the standard deviation σorg and mean value morg of the elements of the packing feature frame, and encodes the syntax element feature_std indicating the calculated standard deviation σorg and the syntax element feature_mean indicating the mean value morg (step S1503).

[0165] The feature map refinement process of the third embodiment will be explained using Figure 31. This operation corresponds to step S2007 in Figure 28. First, the syntax element feature_std is decoded (step S1601). If the feature map refinement process is enabled (Yes in step S1602), the feature map refinement unit 804 decodes the bitstream and obtains the syntax element feature_std indicating the standard deviation σorg and the syntax element feature_std indicating the mean value morg in the feature map encoding information decoding unit (step S1603). The feature map refinement unit 804 calculates the standard deviation σdec and mean value mdec for all elements of all channels of the single-scale feature map xr (step S1604).

[0166] Furthermore, the feature map refinement unit 804 refines all elements of the decoded packing feature frame xr using the decoded distribution characteristics such as the standard deviation σorg and mean morg, and the calculated distribution characteristics such as the standard deviation σdec and mean mdec, to calculate the refined packing feature frame xr' (step S1605). The refinement process is performed according to the following formula.

[0167] xr'[i] = {(xr[i] - mdec) / σdec} * σorg + morph Figure 32 is used to explain the effect of single-scale feature maps. Figure 32 shows that 401 is xf, which is the probability distribution of all elements of the active channels of the single-scale feature map (first single-scale feature map) in the feature map encoding device 100, and all elements of the mapping feature frame. Figure 402 shows xr, which is the probability distribution of all elements of the active channels of the decoded single-scale feature map (second single-scale feature map) in the feature map encoding device 200, and all elements of the decoded mapping feature frame. 403 is xr', which is the probability distribution of all elements of all channels of the refined single-scale feature map (third single-scale feature map), and all elements of the refined packing feature frame.

[0168] Generally, the packing feature frame encoded by the feature map internal encoding unit 104 is quantized by the internal encoding process, so 402 is encoded degraded compared to 401. Due to the encoding degradation of 402, the distribution characteristics of 402 change from the distribution characteristics of 401. By adopting the configuration of this embodiment, the difference between the distribution characteristics of 401 before encoding and the distribution characteristics of 402 after decoding can be obtained, and 402 can be corrected using this difference, so that 403 with a probability distribution close to 401 can be restored.

[0169] Based on the above, by adopting this configuration, the feature map of the decoded signal can be restored to a value close to the feature map of the original signal based on the distribution characteristics of the feature map of the original signal calculated and transmitted on the encoding side and the distribution characteristics of the feature map of the decoded signal calculated on the decoding side, thereby improving encoding efficiency.

[0170] Furthermore, by calculating the distribution characteristics on the encoding side and performing the refinement process on the decoding side using only the active channels and excluding the inactive channels, the accuracy of the refinement process can be improved, thereby increasing the encoding efficiency.

[0171] In addition, in the third embodiment, the bitstream on which the distribution characteristic information generated by the feature map encoding information encoding unit 703 has been encoded may be supplied to the multiplexing unit 905 of the feature map internal encoding unit 104 for multiplexing. In that case, on the encoding side, the multiplexed bitstream is separated by the multiplexing / decoupling unit 1005 of the feature map decoding unit 201, and the bitstream on which the distribution characteristic information has been encoded is supplied to the feature map encoding information decoding unit 801 of the feature map inverse conversion unit 202.

[0172] In all the embodiments described above, the bitstream output by the feature map encoding device has a specific data format so that it can be decoded according to the encoding method used in the embodiment. Furthermore, the feature map decoding device corresponding to this feature map encoding device can decode the bitstream of this specific data format.

[0173] When a wired or wireless network is used to exchange bitstreams between a feature map encoding device and a feature map decoding device, the bitstream may be converted to a data format suitable for the transmission mode of the communication channel before transmission. In this case, a transmitting device is provided that converts the bitstream output by the feature map encoding device into encoded data in a data format suitable for the transmission mode of the communication channel and transmits it to the network, and a receiving device is provided that receives the encoded data from the network, restores it to a bitstream, and supplies it to the feature map decoding device. The transmitting device includes a memory for buffering the bitstream output by the feature map encoding device, a packet processing unit for packetizing the bitstream, and a transmitting unit for transmitting the packetized encoded data over the network. The receiving device includes a receiving unit for receiving the packetized encoded data over the network, a memory for buffering the received encoded data, and a packet processing unit for packetizing the encoded data to generate a bitstream and providing it to the feature map decoding device.

[0174] The above encoding and decoding processes may be implemented not only as hardware-based transmission, storage, and receiving devices, but also by firmware stored in ROM (read-only memory) or flash memory, or by software on a computer. The firmware program or software program may be recorded on a recording medium readable by a computer and provided, provided from a server via a wired or wireless network, or provided as data broadcasting on terrestrial or satellite digital broadcasting.

[0175] The present invention has been described above based on embodiments. The embodiments are illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of their components and processing processes, and that such modifications also fall within the scope of the present invention.

[0176] This invention can be used in feature map encoding and decoding techniques.

[0177] 100 Feature map encoding device, 101 Neural network feature extraction unit, 102 Feature map reduction unit, 103 Feature map transformation unit, 104 Feature map internal encoding unit, 200 Feature map decoding device, 201 Feature map internal decoding unit, 202 Feature map inverse transformation unit, 203 Feature map restoration unit, 204 Neural network identification unit, 301 Convolution processing unit, 302 Activation processing unit, 303 Pooling processing unit, 322 Bottom-up processing unit, 324 Top-down processing unit, 326 Image to be processed for feature extraction, 501 First feature map reduction unit, 502 First channel joining unit, 503 Second feature map reduction unit, 504 Second channel joining unit, 505 Third feature map reduction unit, 506 First padding unit, 507 Second padding unit, 508 Third padding unit, 601 8x magnification unit, 602 4x magnification unit, 603 2x magnification unit, 604 First feature map mixing unit, 605 Second feature map mixing unit, 606 First padding removal unit, 607 Second padding removal unit, 608 Third padding removal unit, 701 Packing unit, 702 Feature map quantization unit, 703 Feature map coded information coding unit, 801 Feature map coded information decoding unit, 802 Feature map inverse quantization unit, 803 Unpacking unit, 901 Switch, 902 VVC encoding unit, 903 HEVC encoding unit, 904 AV1 encoding unit, 905 Multiplexing unit, 1005 Multiplexing / decoupling unit, 1001 Switch, 1002 VVC decoding unit, 1003 HEVC decoding unit, 1004 AV1 decoding unit, 704 Feature map analysis / conversion unit, 804 Feature map refinement section.

Claims

1. A feature map encoding device for encoding multi-scale feature maps, comprising: a packing unit that packs multiple channel feature maps to generate a single packing feature frame; a feature map encoding information encoding unit that encodes feature map encoding information to obtain a bitstream of feature map encoding information; and a feature map internal encoding unit that encodes the packing feature frame to generate a bitstream, wherein the packing unit determines the active channel to be encoded from the multiple channel feature maps, and packs only the feature map of the active channel to generate a single packing feature frame.

2. A feature map encoding method for encoding a multi-scale feature map, comprising: a packing step of packing multiple channel feature maps to generate a single packing feature frame; a feature map encoding information encoding step of encoding feature map encoding information to obtain a bitstream of feature map encoding information; and a feature map internal encoding step of encoding the packing feature frame to generate a bitstream, wherein the packing step determines the active channel to be encoded from the multiple channel feature maps, and packs only the feature map of the active channel to generate a single packing feature frame.

3. A feature map encoding program for encoding a multi-scale feature map, comprising: a packing step of packing multiple channel feature maps to generate a single packing feature frame; a feature map encoding information encoding step of encoding feature map encoding information to obtain a bitstream of feature map encoding information; and a feature map internal encoding step of encoding the packing feature frame to generate a bitstream, wherein the packing step determines the active channel to be encoded from the multiple channel feature maps, and packs only the feature map of the active channel to generate a single packing feature frame.

4. A feature map encoding device for encoding a multi-scale feature map, comprising: a packing unit that packs multiple channel feature maps to generate a single packing feature frame; a feature map analysis and conversion unit that calculates distribution characteristic information of the packing feature frame and converts the data type of the packing feature frame; a feature map encoding information encoding unit that encodes feature map encoding information to obtain a bitstream of feature map encoding information; and a feature map internal encoding unit that encodes the packing feature frame to generate a bitstream, wherein the packing unit determines the active channel to be encoded from multiple channel feature maps, packs only the feature map of the active channel to generate a single packing feature frame, and the feature map encoding information includes the distribution characteristic information.

5. A feature map encoding method for encoding a multi-scale feature map, comprising: a packing step of packing multiple channel feature maps to generate a single packing feature frame; a feature map analysis and conversion step of calculating distribution characteristic information of the packing feature frame and converting the data type of the packing feature frame; a feature map encoding information encoding step of encoding feature map encoding information to obtain a bitstream of feature map encoding information; and a feature map internal encoding step of encoding the packing feature frame to generate a bitstream, wherein the packing step determines the active channel to be encoded from multiple channel feature maps, packs only the feature map of the active channel to generate a single packing feature frame, and the feature map encoding information includes the distribution characteristic information.

6. A feature map encoding program for encoding a multi-scale feature map, comprising: a packing step of packing multiple channel feature maps to generate a single packing feature frame; a feature map analysis and conversion step of calculating distribution characteristic information of the packing feature frame and converting the data type of the packing feature frame; a feature map encoding information encoding step of encoding feature map encoding information to obtain a bitstream of feature map encoding information; and a feature map internal encoding step of encoding the packing feature frame to generate a bitstream, wherein the packing step determines the active channel to be encoded from multiple channel feature maps, packs only the feature map of the active channel to generate a single packing feature frame, and the feature map encoding information includes the distribution characteristic information.

7. A feature map decoding device for decoding a bitstream on which a multi-scale feature map is encoded, comprising: a feature map internal decoding unit that decodes the bitstream and generates a packing feature frame; a feature map encoded information decoding unit that decodes the bitstream and obtains feature map encoded information; and an unpacking unit that unpacks the packing feature frame and generates a multi-channel fractional feature map, wherein the unpacking unit generates inactive channels that are not to be decoded.

8. A feature map decoding method for decoding a bitstream on which a multi-scale feature map is encoded, comprising: a feature map internal decoding step of decoding the bitstream and generating a packing feature frame; a feature map encoded information decoding step of decoding the bitstream and obtaining feature map encoded information; and an unpacking step of unpacking the packing feature frame and generating a multi-channel fractional feature map, wherein the unpacking step generates inactive channels that are not to be decoded.

9. A feature map decoding program for decoding a bitstream on which a multi-scale feature map is encoded, wherein the program causes a computer to perform the following steps: an internal feature map decoding step of decoding the bitstream and generating a packing feature frame; a feature map encoded information decoding step of decoding the bitstream and obtaining feature map encoded information; and an unpacking step of unpacking the packing feature frame and generating a multi-channel fractional feature map, wherein the unpacking step generates inactive channels that are not to be decoded.

10. A feature map decoding device for decoding a bitstream in which a multi-scale feature map is encoded, comprising: a feature map internal decoding unit that decodes the bitstream and generates a packing feature frame; a feature map encoded information decoding unit that decodes the bitstream and obtains feature map encoded information; a feature map refinement unit that calculates distribution characteristic information of the packing feature frame, performs a refinement process on the packing feature frame based on the distribution characteristic information contained in the decoded encoded information, and obtains a refined packing feature frame; and an unpacking unit that unpacks the refined packing feature frame and generates a multi-channel fractional feature map, wherein the unpacking unit generates inactive channels that are not to be decoded.

11. A feature map decoding method for decoding a bitstream in which a multi-scale feature map is encoded, comprising: a feature map internal decoding step of decoding the bitstream and generating a packing feature frame; a feature map encoded information decoding step of decoding the bitstream and obtaining feature map encoded information; a feature map refinement step of calculating distribution characteristic information of the packing feature frame and performing a refinement process on the packing feature frame based on the distribution characteristic information contained in the decoded encoded information to obtain a refined packing feature frame; and an unpacking step of unpacking the refined packing feature frame to generate a multi-channel fractional feature map, wherein the unpacking step generates inactive channels that are not to be decoded.

12. A feature map decoding program for decoding a bitstream in which a multi-scale feature map is encoded, characterized in that the program causes a computer to execute the following steps: an internal feature map decoding step of decoding the bitstream and generating a packing feature frame; a feature map encoded information decoding step of decoding the bitstream and obtaining feature map encoded information; a feature map refining step of calculating distribution characteristic information of the packing feature frame and performing a refining process on the packing feature frame based on the distribution characteristic information contained in the decoded encoded information to obtain a refined packing feature frame; and an unpacking step of unpacking the refined packing feature frame and generating a multi-channel fractional feature map, wherein the unpacking step generates inactive channels that are not to be decoded.

Images