Feature map encoding device, feature map encoding method, feature map decoding device, and feature map decoding method

Abstract

Provided is a feature map encoding device that encodes a multi-scale feature map, wherein channel IDs that identify channels are set in a feature map of a plurality of channels to be encoded, one packed feature frame is generated by packing the feature map of the plurality of channels, and feature map encoding information including the channel IDs that identify the channels is encoded.

Description

Feature map encoding device, feature map encoding method, feature map decoding device, feature map decoding method

[0001] This paper deals with encoding and decoding of feature maps in neural networks.

[0002] As a neural network technology used for image recognition such as detecting objects at various scales in an image, dividing regions into individual objects, or tracking objects, the FPN (Feature Pyramid Network) described in Non-Patent Document 1 is known. In the FPN, multiple feature maps at various scales are generated from the image to be processed, and various image recognitions are performed using the feature maps.

[0003] The FPN used for image recognition generates multiple feature maps from an image, and its structure utilizes a CNN (Convolutional Neural Network). A CNN can be divided into a feature extraction section (backbone) that reads an image and generates a feature map through convolution and pooling, and a classification section (head) that is composed of hierarchical fully connected layers and generates output suitable for tasks such as object detection, instance segmentation, and object tracking. The FPN utilizes the CNN backbone.

[0004] The feature extraction unit of the FPN is typically configured with a hierarchical structure in which a convolution processing unit 301, an activation processing unit 302, and a pooling processing unit 303 shown in FIG. 3 are configured as one basic unit, and this basic unit is repeated.

[0005] The structure of the FPN is shown in Figure 4. The FPN consists of a bottom-up processor 322 that uses a CNN backbone to generate a multi-scale feature map consisting of multiple hierarchical layers, and a top-down processor 324 that aggregates features from deep feature maps to shallow feature maps using the inverse structure of the CNN backbone. The bottom-up processor 322 reduces the resolution of the feature map by half by repeating the convolution processor 301, activation processor 302, and pooling processor 303 (shown in Figure 3 ), which are basic units, to generate a pyramid of feature maps with multiple layers. Meanwhile, the top-down processor 324 adds feature maps with resolutions corresponding to the bottom-up processor 322, increasing the resolution of the feature map to the same resolution as the input image, thereby generating a pyramid of feature maps. In other words, the FPN generates multiple feature maps for each layer from an image 326 to be subjected to feature extraction processing.

[0006] The convolution processing unit 301 performs convolution processing on the data to be processed (image or feature map) using multiple types of predetermined filters (kernels). In the convolution processing in the convolution processing unit 301, predetermined filtering is performed on all of the data to be processed while sliding at predetermined intervals. The sliding interval is called a stride. The convolution processing unit 301 may determine the stride based on the number of data to be processed. For example, the convolution processing unit 301 may determine the stride to be 1 if the number of data to be processed is smaller than a predetermined value, and may determine the stride to be 2 if the number of data to be processed is equal to or greater than the predetermined value. Multiple predetermined filters are prepared in each layer, and multiple feature maps are generated by generating one feature map for each filter. The unit of the feature map is the channel. If the number (types) of predetermined filters is N (N types), N feature maps (N channels) are generated.

[0007] The activation processing unit 302 performs activation processing to nonlinearly convert the feature map output from the convolution processing unit 301. Here, the function used in the activation processing is called an activation function. The activation processing unit 302 uses a ReLU (Rectified Linear Unit) function, a sigmoid function, or the like as the activation function.

[0008] The pooling processing unit 303 is a process for downsampling the feature map output from the activation processing unit 302 by replacing local values ​​of the feature map with representative values.

[0009] On the other hand, when performing classification using a neural network, it has the ability to perform tasks using feature maps of multiple channels at each layer.

[0010] In image classification, the feature maps of multiple channels at each layer are convolved for each predetermined size based on the scale of the feature map, and the probability of the object class is calculated for each pixel.

[0011] Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature Pyramid Networks for Object Detection. In CVPR, 2017.

[0012] The amount of information in a feature map is enormous, making it unsuitable for transmission and storage. In view of the above-mentioned problems, an object of this embodiment is to provide a technique for encoding and decoding a feature map.

[0013] In order to solve the above problem, a feature map coding device according to one aspect of the invention includes a channel management unit that sets channel IDs to feature maps of multiple channels to be coded; a packing unit that packs the feature maps of the multiple channels to generate a single packed feature frame; a feature map coded information coding unit that codes the feature map coded information to obtain a bit stream of feature map coded information; and a feature map internal coding unit that encodes the packed feature frame to generate a bit stream, wherein the feature map coded information includes information indicating the number of channels and channel IDs that identify the channels.

[0014] A feature map decoding device according to yet another aspect of the present invention is a feature map decoding device that decodes a bitstream in which a multi-scale feature map is encoded, and includes: a feature map internal decoding unit that decodes the bitstream and generates a packed feature frame; and an unpacking unit that unpacks the packed feature frame to generate a multi-channel decimal feature map, wherein the feature map encoding information includes information indicating the number of channels and channel IDs that identify the channels.

[0015] According to this embodiment, feature maps can be efficiently encoded and decoded with a small amount of processing.

[0016] 1 is a block diagram illustrating the configuration of a feature map encoding device 100. FIG. 2 is a block diagram illustrating the configuration of a feature map decoding device 200. FIG. 3 is a block diagram illustrating the processing that is the basic unit in each layer of an FPN. FIG. 4 is a diagram illustrating the structure of an FPN. FIG. 5 is a block diagram illustrating the detailed configuration of a feature map reduction unit 102. FIG. 6 is a block diagram illustrating the detailed configuration of a feature map restoration unit 203. FIG. 7 is a block diagram illustrating the detailed configuration of a feature map transformation unit 103. FIG. 8 is a block diagram illustrating the detailed configuration of a feature map inverse transformation unit 202. FIG. 9 is a block diagram illustrating the detailed configuration of a feature map inner encoding unit 104. FIG. 10 is a block diagram illustrating the detailed configuration of a feature map inner decoding unit 201. FIG. 11 is a diagram illustrating the number of channels of feature maps x1, x2, and x3, the width and height of the feature maps. FIG. 12 is a diagram illustrating the state in which feature maps of multiple channels are packed into a packing feature frame. FIG. 13 is a diagram illustrating flipping when packing feature maps of multiple channels into a packing feature frame. FIG. 14 is a diagram illustrating layers and units handled in the feature map encoding device and feature map decoding device of this embodiment. 1 is a flowchart illustrating a feature map encoding process procedure for a channel ID according to the first embodiment. FIG. 2 is a flowchart illustrating a feature map decoding process procedure for a channel ID according to the first embodiment. FIG. 3 is an example of syntax rules for the channel ID of each channel according to the first embodiment. FIG. 4 is another example of syntax rules for the channel ID of each channel according to the first embodiment. FIG. 5 is a flowchart illustrating a channel ID encoding process procedure according to the first embodiment. FIG. 6 is a flowchart illustrating a channel ID decoding process procedure according to the first embodiment. FIG. 7 is a diagram illustrating another example of a state in which feature maps of multiple channels are packed into a packed feature frame. FIG. 8 is an example of syntax rules for the channel ID of each channel according to the second embodiment. FIG. 9 is another example of syntax rules for the channel ID of each channel according to the second embodiment. FIG. 10 is a flowchart illustrating a channel ID encoding process procedure according to the second embodiment.1 is a flowchart illustrating a channel ID decoding process according to a second embodiment. FIG. 2 is a diagram illustrating another example of a state in which feature maps of multiple channels are packed into a packed feature frame. FIG. 3 is a flowchart illustrating a feature map encoding process according to a channel ID according to a third embodiment. FIG. 4 is a flowchart illustrating a feature map decoding process according to a channel ID according to the third embodiment. FIG. 5 is a diagram illustrating an example of a state in which feature maps of active channels are packed into one frame. FIG. 6 is an example of syntax rules for feature map encoded information according to the third embodiment. FIG. 7 is another example of syntax rules for feature map encoded information according to the third embodiment. FIG. 8 is a flowchart illustrating an encoding process of feature map encoded information according to the third embodiment. FIG. 9 is a flowchart illustrating a decoding process of feature map encoded information according to the third embodiment. FIG. 10 is an example of syntax rules for feature map encoded information according to a fourth embodiment. FIG. 11 is another example of syntax rules for feature map encoded information according to the fourth embodiment. FIG. 11 is a flowchart illustrating an encoding process of feature map encoded information according to the fourth embodiment. FIG. 12 is a flowchart illustrating a decoding process of feature map encoded information according to the fourth embodiment.

[0017] The technologies and technical terms used in this embodiment will be defined below.

[0018] <Features and feature maps> In a convolutional neural network (CNN), a filter is used to scan the image (input layer data) from which features are extracted while changing its position, and the results of the convolution operation between the filter coefficients are used as features or feature maps.

[0019] <Packing> Frame packing refers to combining two or more frames (pictures) into a single frame (picture) by arranging them in a tiled pattern. In this application, packing refers to combining feature maps of multiple channels into a single frame. Figure 12 shows an example of frame packing.

[0020] <Data Type> A data type that represents an integer value is called an integer type, and a data type that represents a decimal value is called a decimal type.

[0021] <Layers and Units> The layers and units handled by the feature map encoding device and feature map decoding device of this embodiment will be described with reference to Figure 14. Reference numeral 181 in Figure 14 denotes a sequence of packed feature frames (feature frames packed by combining feature maps from multiple channels into one frame) or feature maps for all channels that continue over time, and is referred to as a sequence layer or sequence unit. Reference numeral 182 in Figure 14 denotes a sequence of feature maps for one channel that continue over time, and is referred to as a sequence layer for each channel or a sequence unit for each channel. Reference numeral 183 in Figure 14 denotes a packed feature frame or feature maps for all channels at the same time, and is referred to as a frame layer or frame unit. Reference numeral 184 in Figure 14 denotes a feature map for one channel at a certain time, and is referred to as a feature map layer for each channel or a feature map unit for each channel.

[0022] First Embodiment A feature map encoding device 100 and a feature map decoding device 200 according to a first embodiment of the present invention will be described.

[0023] 1 is a block diagram of a feature map coding device 100 according to a first embodiment. The feature map coding device 100 according to the embodiment includes a feature map reduction unit 102, a feature map conversion unit 103, and a feature map internal coding unit 104. The feature map coding device 100 is a device that encodes a feature map generated by a neural network feature extraction unit 101 to generate and output a bitstream.

[0024] The neural network feature extraction unit 101 reads an image from which features are to be extracted, generates a feature map by FPN convolution processing, activation processing, and pooling processing, and supplies the generated feature map to the feature map reduction unit 102. In this embodiment, a three-layer multi-scale feature map of x1, x2, and x3 is generated.

[0025] The feature map reduction unit 102 converts the three-layer multi-scale feature maps x1, x2, and x3 acquired from the neural network feature extraction unit 101 into a single-layer single-scale feature map xf and supplies it to the feature map conversion unit 103. The feature map reduction unit 102 will be described in detail with reference to FIG. 5 .

[0026] The feature map conversion unit 103 performs packing and quantization on the decimal-type single-scale feature map xf supplied from the feature map reduction unit 102, converting it into an integer-type packed feature frame, and supplies this to the feature map internal coding unit 104 together with the feature map coding information used in the process.

[0027] The feature map conversion unit 103 will be described in detail with reference to FIG.

[0028] The feature map internal coding unit 104 encodes the integer-type packing feature frames supplied from the feature map conversion unit 103 in accordance with an image coding standard such as VVC, HEVC, or AV1 to generate a bitstream in which the packing feature frames are encoded, and multiplexes the feature map encoding information supplied from the feature map conversion unit 103 with the encoded bitstream to output a multiplexed bitstream. The output bitstream is supplied to the feature map decoding device 200 or the like via a network or the like.

[0029] The feature map internal coding unit 104 will be described in detail with reference to FIG.

[0030] 2 is a block diagram showing the configuration of a feature map decoding device 200 according to an embodiment of the present invention, which corresponds to the feature map encoding device 100 in FIG. 1. The feature map decoding device 200 according to this embodiment includes a feature map internal decoding unit 201, a feature map inverse conversion unit 202, and a feature map restoration unit 203. The feature map decoding device 200 receives as input a bit stream encoded by the feature map encoding device 100, decodes the bit stream, and generates three-layer multi-scale feature maps x1up, x2up, and x3up, which it supplies to a neural network identification unit 204.

[0031] The feature map internal decoding unit 201 separates the multiplexed bit stream output by the feature map internal coding unit 104 of the feature map coding device 100 into a bit stream in which feature map coding information is coded and a bit stream in which packing feature frames are coded, decodes the bit stream in which packing feature frames are coded in accordance with an image coding standard such as VVC, HEVC, or AV1, generates integer-type packing feature frames, and supplies these to the feature map inverse conversion unit 202.

[0032] The feature map internal decoding unit 201 will be described in detail with reference to FIG.

[0033] The feature map inverse conversion unit 202 performs inverse quantization and unpacking on the integer-type packed feature frame supplied from the feature map internal decoding unit 201, converting it into a decimal-type single-scale feature map xr, and supplies it to the feature map restoration unit 203.

[0034] The feature map inverse conversion unit 202 will be described in detail with reference to FIG.

[0035] The feature map restoration unit 203 converts the single-scale feature map xr supplied from the feature map inverse conversion unit 202 into three-layer multi-scale feature maps x1up, x2up, and x3up, and supplies them to the neural network identification unit 204 as outputs of the feature map decoding device 200.

[0036] The feature map restoration unit 203 will be described in detail with reference to FIG.

[0037] The neural network identification unit 204 performs identification processing such as identifying objects, places, landscapes, people, and living things in the image to be identified, based on the three-layer multi-scale feature maps x1up, x2up, and x3up supplied by the feature map restoration unit 203.

[0038] <Feature Map Reduction and Feature Map Restoration> The feature map reduction unit 102 has a function of converting a multi-layered multi-scale feature map acquired from the neural network feature extraction unit 101 into a single-layered single-scale feature map.

[0039] The feature map reduction unit 102 will be described in detail using Fig. 5. The feature map reduction unit 102 is composed of a first feature map reduction unit 501, a first channel combination unit 502, a second feature map reduction unit 503, a second channel combination unit 504, a third feature map reduction unit 505, a first padding unit 506, a second padding unit 507, a third padding unit 508, and a channel management unit 509. The feature map reduction unit 102 in Fig. 5 is an example of a configuration that converts a three-layer multi-scale feature map into a single-scale feature map.

[0040] The feature map reduction unit 102 receives three-layer multi-scale feature maps, namely, a first feature map x1, a second feature map x2, and a third feature map x3, as input, converts them into a single-layer single-scale feature map xf, and supplies the resulting map to the feature map conversion unit 103. Here, n denotes an index indicating a layer, Cn denotes the number of channels in the nth layer, Wn denotes the width of the feature map, and Hn denotes the height of the feature map. In this embodiment, the values ​​of Cn, Wn, and Hn for each layer are as shown in FIG. 11 , where H and W denote the width and height, respectively, of the image from which feature extraction is performed.

[0041] The first padding unit 506 has a function of padding the first feature map x1 to generate a first padded feature map x1pad. The first padding unit 506 determines the padding size so that the width and height of x1pad are multiples of 64. The number of channels of x1pad is 256, the same as that of x1.

[0042] The second padding unit 507 performs padding on the second feature map x2 by folding to generate a second padded feature map x2pad. The second padding unit 507 determines the padding size so that the width and height of x2pad are multiples of 32. The number of channels of x2pad is 256, the same as that of x2.

[0043] The third padding unit 508 performs padding on the third feature map x3 by folding to generate a third padded feature map x3pad. The third padding unit 508 determines the padding size so that the width and height of x3pad are multiples of 16. The number of channels of x3pad is 256, the same as that of x3.

[0044] In the first padding unit 506, the second padding unit 507, and the third padding unit 508, the padding size on the left side is the same as the padding size on the right side, and the padding size on the top side is the same as the padding size on the bottom side. That is, the feature maps x1, x2, and x3 are arranged at the centers of x1pad, x2pad, and x3pad, respectively.

[0045] The first feature map reduction unit 501 performs convolution in the spatial and channel directions on the first padded feature map x1pad acquired from the first padding unit 506 to generate a first intermediate feature map y1. The number of channels of y1 is 192, its width is Wx1pad / 2, and its height is Hx1pad / 2, where Wx1pad and Hx1pad are the width and height of the first padded feature map x1pad, respectively.

[0046] The first channel combining unit 502 has a function of combining the first intermediate feature map y1 obtained from the first feature map reduction unit 501 and the second padded feature map x2pad obtained from the second padding unit 507 in the channel direction to generate an intermediate feature map y1Cx2pad. Since the number of channels of y1 is 192 and the number of channels of x2pad is 256, the number of channels of the intermediate feature map y1Cx2pad is 448 (192 + 256).

[0047] The second feature map reduction unit 503 performs convolution in the spatial and channel directions on the intermediate feature map y1Cx2pad obtained from the first channel combination unit 502 to generate a second intermediate feature map y2. The number of channels of y2 is 192, its width is Wy1Cx2pad / 2, and its height is Hy1Cx2pad / 2, where Wy1Cx2pad and Hy1Cx2pad are the width and height of the intermediate feature map y1Cx2pad, respectively.

[0048] The second channel combining unit 504 has a function of combining the second intermediate feature map y2 obtained from the second feature map reduction unit 503 and the third padded feature map x3pad obtained from the third padding unit 508 in the channel direction to generate an intermediate feature map y2Cx3pad. Since the number of channels in the intermediate feature map y2 is 192 and the number of channels in x3pad is 256, the number of channels in y1Cx2pad is 448 (192 + 256).

[0049] The third feature map reduction unit 505 performs convolution in the spatial and channel directions on the intermediate feature map y2Cx3pad obtained from the second channel combination unit 504 to generate a third intermediate feature map y3. The number of channels of y3 is 192, its width is Wy2Cx3pad / 2, and its height is Hy2Cx3pad / 2, where Wy2Cx3pad and Hy2Cx3pad are the width and height of the intermediate feature map y2Cx3pad, respectively.

[0050] The channel management unit 509 assigns a unique ID (hereinafter referred to as a channel ID) for identifying each channel to be encoded, and also assigns a predetermined order to each channel of the third intermediate feature map y3, outputs the result as a single-scale feature map xf, and supplies the single-scale feature map xf corresponding to the channel ID of each channel to the feature map conversion unit 103. At this time, if the number of channels is large and the amount of information needs to be reduced, the number of channels in the third intermediate feature map y3 may be reduced as necessary to generate a single-scale feature map xf. In this embodiment, convolution or thinning is performed on the third intermediate feature map y3, which has 192 channels, to assign a predetermined order to each channel, thereby generating a single-scale feature map xf, which has 64 channels. Furthermore, in this embodiment, integer values ​​starting from 0 and increasing by 1 are assigned to the channel ID of each channel of the single-scale feature map xf in the predetermined order. Note that, although the channel IDs are integer values ​​starting from 0 and increasing by 1 in this embodiment, the channel IDs are not limited to this and may be any unique integer values. For example, when integer values ​​starting from 0 and increasing by 1 are assigned to the channel ID of each channel of the third intermediate feature map y3, unimportant channels are eliminated, and only important channels remain to form the single-scale feature map xf, the channel IDs of the remaining channels of the third intermediate feature map y3 can be directly used as the channel IDs of the single-scale feature map xf.

[0051] The feature map restoration unit 203 has a function of converting the single-scale feature map xr acquired from the feature map inverse conversion unit 202 into three-layer multi-scale feature maps x1up, x2up, and x3up.

[0052] The feature map restoration unit 203 will be described in detail with reference to Fig. 6. The feature map restoration unit 203 is configured with an 8x enlargement unit 601, a 4x enlargement unit 602, a 2x enlargement unit 603, a first feature map blending unit 604, a second feature map blending unit 605, a first depadding unit 606, a second depadding unit 607, and a third depadding unit 608.

[0053] The feature map restoration unit 203 identifies the channels of the single-scale feature map xr based on the channel ID of each channel supplied from the feature map inverse conversion unit 202.

[0054] The 8x enlargement unit 601 performs transpose convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr obtained from the feature map inverse transformation unit 202 to enlarge the feature map and reduce the channels, thereby generating an intermediate feature map z1. The number of channels in z1 is 196. If the width and height of the single-scale feature map xr are xrwidth and xrheight, respectively, the width and height of z1 are xrwidth × 8 and xrheight × 8, respectively. Here, rwidth × 8 and xrheight × 8 are the same as the width and height, respectively, of the first padded feature map x1pad, which is the output of the first padding unit 506 of the feature map reduction unit 102.

[0055] The 4x enlargement unit 602 performs transpose convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr obtained from the feature map inverse transformation unit 202 to enlarge the features and reduce the channels, thereby generating an intermediate feature map z2. The number of channels in z2 is 196. The width and height of z2 are xrwidth×4 and xrheight×4, respectively. Here, rwidth×4 and xrheight×4 are the same as the width and height, respectively, of the second padded feature map x2pad, which is the output of the second padding unit 507 of the feature map reduction unit 102.

[0056] The 2x enlargement unit 603 performs transpose convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr obtained from the feature map inverse transformation unit 202, thereby enlarging the feature map and reducing the channels, and generating an intermediate feature map z3. The number of channels in z3 is 196. The width and height of z3 are xrwidth×2 and xrheight×2, respectively. Here, rwidth×2 and xrheight×2 are the same as the width and height, respectively, of the third padded feature map x3pad, which is the output of the third padding unit 508 of the feature map reduction unit 102.

[0057] The first feature map mixing unit 604 has a function of generating an intermediate feature map z2up by improving the quality of the intermediate feature map z2 obtained from the 4x enlargement unit 602, using the intermediate feature map z1 obtained from the 8x enlargement unit 601.

[0058] The second feature map mixing unit 605 has a function of generating an intermediate feature map z3up by using the intermediate feature map z2up obtained from the first feature map mixing unit 604, with the quality of the intermediate feature map z3 obtained from the 2x enlargement unit 603 being improved.

[0059] The padding removal unit 606 removes padding from the intermediate feature map z1 acquired from the 8x enlargement unit 601 to generate a first output feature map x1up. The width and height of x1up are the same as the width and height of the first feature map x1 input to the feature map reduction unit 102.

[0060] The padding remover 607 removes padding from the intermediate feature map z2up acquired from the first feature map mixer 604 to generate a second output feature map x2up. The width and height of x2up are the same as the width and height of the second feature map x2 input to the feature map reducer 102.

[0061] The padding remover 608 removes padding from the intermediate feature map z3up obtained from the second feature map mixer 605 to generate a third output feature map x3up. The width and height of x3up are the same as the width and height of the third feature map x3 input to the feature map reducer 102.

[0062] In the first padding remover 606, the second padding remover 607, and the third padding remover 608, the left-side padding size is the same as the right-side padding size, and the upper-side padding size is the same as the lower-side padding size, as in the first padding remover 506, the second padding remover 507, and the third padding remover 508 of the feature map reduction unit 102. In other words, the output feature maps x1up, x2up, and x3up are positioned at the center of the intermediate feature maps z1, z2up, and z3up, respectively, and padding is removed from the top, bottom, left, and right.

[0063] <Feature Map Conversion and Feature Map Inverse Conversion> The feature map conversion unit 103 has a function of performing packing processing and quantization processing on the multi-channel decimal-type single-scale feature map xf supplied from the feature map reduction unit 102, and converting it into an integer-type packed feature frame to be supplied to the feature map internal coding unit 104.

[0064] The feature map conversion unit 103 on the encoding side will be described in detail with reference to Fig. 7. The feature map conversion unit 103 is made up of a packing unit 701, a feature map quantization unit 702, and a feature map encoding information encoding unit 703.

[0065] The packing unit 701 generates a packed feature frame by packing input feature maps of multiple channels into one frame.

[0066] FIG. 12 is a diagram illustrating the channel IDs of the feature maps of each channel when feature maps of multiple channels are packed into one packing feature frame. The numbers in FIG. 12 indicate the channel IDs of each channel. Starting from the upper left of the packing feature frame, the feature maps of each channel are sequentially arranged in one packing feature frame in raster scan order, from left to right and top to bottom. This raster scan order is common to the packing unit 701 and the corresponding unpacking unit 801. Hereinafter, the order in which the feature maps of each channel are sequentially arranged from the upper left of the packing feature frame, from left to right and top to bottom, is referred to as a predetermined raster scan order.

[0067] In this embodiment, integer values ​​starting from 0 and increasing by 1 are assigned to the channel ID of each channel in a predetermined order, so as shown in Fig. 12, integer values ​​starting from 0 and increasing by 1 in a predetermined raster scan order are set as the channel ID values ​​of each channel packed in a packing feature frame. The numbers in Fig. 12 indicate the channel ID of each channel and also indicate the predetermined raster scan order.

[0068] Note that feature maps of multiple channels can also be packed while being arranged at arbitrary positions in a packing feature frame. FIG. 21 is a diagram illustrating the channel IDs of each channel when feature maps of multiple channels are arranged at arbitrary positions and packed into one packing feature frame. In the example of FIG. 21 , integer values ​​starting from 0 and increasing by 1 are assigned to the channel IDs of each channel in a predetermined order, and the packing feature maps of each channel are arranged at arbitrary positions in the packing feature frame. For example, coding efficiency can be improved by arranging channels with high correlations adjacent to each other within a frame based on the average values ​​of the elements of the feature maps of each channel or the values ​​of the elements near the boundaries of the feature maps.

[0069] Next, an example will be described in which feature maps of multiple channels, to which channel IDs with unique integer values ​​are set rather than integer values ​​starting from 0 and increasing by 1, are packed into one packing feature frame. Fig. 26 is a diagram illustrating the channel IDs of each channel when feature maps of multiple channels, to which channel IDs with unique integer values ​​are set, are packed into one packing feature frame. In the example of Fig. 26 , unique integer values ​​are assigned to the channel IDs of each channel in a predetermined order rather than integer values ​​starting from 0 and increasing by 1, and the feature maps of each channel are arranged in one packing feature frame. Even in this case, by transmitting the channel ID of each channel to the decoding side, the channel can also be identified on the decoding side based on the channel ID.

[0070] The encoded information such as the channel ID of each channel used in the packing unit 701 is supplied to a feature map encoded information encoding unit 703 as feature map encoded information.

[0071] Furthermore, the packing unit 701 performs flipping based on the position in the frame where the feature map of each channel is arranged. Whether or not to perform flipping can be selected, and the selection may be transmitted from the encoding side to the decoding side via a bitstream.

[0072] Flipping when packing feature maps of multiple channels into a packing feature frame will be described using FIG. 13 . In flipping, when packing the feature maps of each channel, the positions of the elements (pixels) of the feature maps of each channel are flipped horizontally (left and right), vertically (up and down), or horizontally and vertically (up and down, left and right). The feature maps of four channels, A (upper left), B (upper right), C (lower left), and D (lower right) in FIG. 13 , are grouped together, and no flipping is performed at position A in FIG. 13 . At position B, the feature map is flipped horizontally (left and right). At position C, the feature map is flipped vertically (up and down). At position D, the feature map is flipped horizontally and vertically (up and down, left and right). When the distribution of elements in the feature maps of each channel is similar, flipping based on the position in the frame where the channels are placed reduces the boundaries between the feature maps of each channel, improving coding efficiency.

[0073] The feature map quantization unit 702 has a function of converting elements of a decimal-type packing feature frame (feature map of all channels) into N-bit integers (N=an integer from about 8 to 16) within a predetermined range and outputting an integer-type packing feature frame. In this embodiment, the elements are converted into 10-bit integers ranging from 0 to 1023. The feature map quantization unit 702 determines the minimum and maximum values ​​of the elements of the decimal-type packing feature frame and supplies the determined minimum and maximum values ​​of the elements of the decimal-type packing feature frame (feature map of all channels) to the feature map coding information encoding unit 703 as part of the feature map coding information. To convert from a decimal-type packing feature frame (feature map) to an integer-type packing feature frame (feature map), a linear conversion is performed in which the minimum value of the decimal type is equivalent to the minimum value of the integer type and the maximum value of the decimal type is equivalent to the maximum value of the integer type. For example, when the integer type range is expressed in 10 bits, the minimum value of the elements of the integer-type packing feature frame (feature map) is 0 and the maximum value is 1023 (2 １０ -1). Values ​​between the minimum and maximum values ​​are linearly quantized.

[0074] The feature map coded information coding unit 703 codes the coded feature map information of each channel supplied from the packing unit 701 and the feature map quantization unit 702, and transmits the coded feature map information to the decoding side via the feature map internal coding unit 104. The coding of the coded feature map information of each channel will be described later.

[0075] Next, the feature map inverse transformation unit 202 has a function of performing inverse quantization processing and unpacking processing on integer-type packing feature frames decoded in VVC, HEVC, AV1, etc., supplied from the feature map internal decoding unit 201, and performing inverse transformation into a decimal-type single-scale feature map xr to be supplied to the feature map restoration unit 203.

[0076] The feature map inverse transformation unit 202 on the decoding side will be described in detail with reference to Fig. 8. The feature map inverse transformation unit 202 performs the inverse process of the feature map transformation unit 103, and is composed of a feature map coded information decoding unit 801, a feature map inverse quantization unit 802, and an unpacking unit 803.

[0077] The feature map coded information decoding unit 801 decodes the bit stream in which the feature map coded information supplied via the feature map internal decoding unit 201 is coded, and supplies the feature map coded information for each channel obtained as a result of the decoding to a feature map inverse quantization unit 802 and an unpacking unit 803. The decoding of the feature map coded information for each channel will be described later.

[0078] The feature map inverse quantization unit 802 performs the inverse process of the feature map quantization unit 702 on the encoding side, and has a function of converting elements of integer-type packing feature frames from integer type to decimal type. The feature map inverse quantization unit 802 converts the integer-type packing feature frame decoded by the feature map internal decoding unit 201 into a decimal-type packing feature frame using the minimum and maximum decimal values ​​transmitted as metadata. A linear transformation is performed to convert the integer minimum value to the decimal minimum value and the integer maximum value to the decimal maximum value. Linear inverse quantization is performed on values ​​between the minimum and maximum values.

[0079] The unpacking unit 803 extracts feature maps for each channel from packing feature frames arranged in one frame in a predetermined raster scan order and supplies them to the feature map restoration unit 203 as single-scale feature maps xr. At this time, the feature maps for each channel are extracted in a predetermined raster scan order, starting from the top left of the packing feature frame, then from left to right, and then from top to bottom. Furthermore, feature map coding information, such as a channel ID corresponding to the feature map for each channel supplied from the feature map coding information decoding unit 801, is associated with the extracted feature map for each channel. Associating the feature map for each channel with the channel ID makes it possible to identify the channel based on the channel ID. Furthermore, rearrangement is performed as necessary, for example, when feature maps for multiple channels are arranged at arbitrary positions in the packing feature frame on the encoding side.

[0080] <Feature Map Intra-Encoding / Decoding> The feature map intra-encoding unit 104 will be described in detail with reference to FIG. 9 . The feature map intra-encoding unit 104 includes a switch 901, a VVC encoding unit 902, an HEVC encoding unit 903, an AV1 encoding unit 904, and a multiplexing unit 905. The switch 901 selects an encoding standard for intra-encoding the feature map converted by the feature map conversion unit 103. The VVC encoding unit 902 encodes packing feature frames according to the VVC standard and outputs a bitstream compliant with the VVC standard. The HEVC encoding unit 903 encodes packing feature frames according to the HEVC standard and outputs a bitstream compliant with the HEVC standard. The AV1 encoding unit 904 encodes packing feature frames according to the AV1 standard and outputs a bitstream compliant with the AV1 standard.

[0081] In the VVC, HEVC, and AV1 standards, an image is divided into blocks of a predetermined size and then encoded.

[0082] It is also possible to implement only one of VVC, HEVC, and AV1, and it is also possible to use an image coding method other than VVC, HEVC, and AV1.

[0083] Furthermore, the multiplexing unit 905 multiplexes the bit stream in which the feature map coding information is coded and the bit stream in which the packing feature frame is coded, and outputs the multiplexed bit stream.

[0084] Next, the feature map internal decoding unit 201 will be described in detail with reference to Fig. 10. The feature map internal decoding unit 201 is made up of a demultiplexing unit 1005, a switch 1001, a VVC decoding unit 1002, an HEVC decoding unit 1003, and an AV1 decoding unit 1004. The switch 1001 selects the coding standard to be internally decoded based on information for selecting internal decoding within the input bitstream.

[0085] The demultiplexing unit 1005 demultiplexes the multiplexed bit stream supplied from the encoding side, and separates it into a bit stream in which feature map encoding information is encoded and a bit stream compliant with the VVC / HEVC / AV1 standard in which packing feature frames are encoded.

[0086] The VVC decoding unit 1002 decodes a bitstream in which packing feature frames are coded in accordance with the VVC standard to generate packing feature frames. The HEVC decoding unit 1003 decodes a bitstream in which packing feature frames are coded in accordance with the HEVC standard to generate packing feature frames. The AV1 decoding unit 1004 decodes a bitstream in which packing feature frames are coded in accordance with the AV1 standard to generate packing feature frames.

[0087] In the VVC, HEVC, and AV1 standards, decoding is performed for each predetermined block size.

[0088] It is also possible to implement only one of VVC, HEVC, and AV1, and it is also possible to use an image coding method other than VVC, HEVC, and AV1.

[0089] <Regarding Feature Map Encoding / Decoding Processing Procedure Related to Channel ID> Next, a description will be given of the feature map encoding processing procedure related to channel ID, which is performed by the feature map reduction unit 102, the feature map conversion unit 103, and the feature map internal encoding unit 104 of the encoding map encoding unit 100. Fig. 15 is a flowchart illustrating the feature map encoding processing procedure related to channel ID.

[0090] The channel management unit 509 of the feature map reduction unit 102 sets a channel ID in the feature map of each channel (step S101 in FIG. 15).

[0091] Next, the packing unit 701 of the feature map conversion unit 103 packs the feature map of each channel to generate a packed feature frame (step S102 in FIG. 15).

[0092] Next, the feature map quantization unit 702 of the feature map conversion unit 103 quantizes the packing feature frame, converting the packing feature frame from decimal type to integer type (step S103 in FIG. 15).

[0093] Next, the feature map coded information coding unit 703 of the feature map conversion unit 103 codes the feature map coded information, such as the number of channels, channel IDs, and information about quantization, to generate a bitstream in which the feature map coded information is coded (step S104 in FIG. 15 ).

[0094] Subsequently, the VVC encoding unit 902, the HEVC encoding unit 903, or the AV1 encoding unit 904 of the feature map intra encoding unit 104 encodes the packing feature frame to generate a bitstream in which the packing feature frame is encoded (step S105 in FIG. 15 ).

[0095] Next, the multiplexing unit 905 of the feature map internal coding unit 104 multiplexes the bit stream in which the feature map coding information is coded and the bit stream in which the packing feature frame is coded to generate a multiplexed bit stream (step S106 in FIG. 15 ).

[0096] Next, a description will be given of the feature map decoding process procedure relating to the channel ID, which is performed by the feature map internal decoding unit 201 and the feature map inverse conversion unit 202 of the coding map decoding unit 200. Fig. 16 is a flowchart illustrating the feature map decoding process procedure relating to the channel ID.

[0097] The demultiplexing unit 1005 of the feature map internal decoding unit 201 demultiplexes the multiplexed bit stream output from the encoding side, and separates it into a bit stream in which feature map encoding information is encoded and a bit stream in which packing feature frames are encoded (step S201 in FIG. 16 ).

[0098] Subsequently, the VVC decoding unit 1002, the HEVC decoding unit 1003, or the AV1 decoding unit 1004 of the feature map internal decoding unit 201 decodes the bit stream in which the packing feature frame is coded, and generates the packing feature frame (step S202 in FIG. 16 ).

[0099] Next, the feature map coded information decoding unit 801 of the feature map inverse transform unit 202 decodes the bit stream in which the feature map coded information has been coded, and generates feature map coded information such as the number of channels, channel IDs, and information about quantization (step S203 in FIG. 16 ).

[0100] Next, the feature map inverse quantization unit 802 of the feature map inverse transformation unit 202 converts the packing feature frame from integer type to decimal type (step S204 in FIG. 16).

[0101] Next, the unpacking unit 803 of the feature map inverse conversion unit 202 unpacks the packed feature frame and generates a feature map for each channel (step S205 in FIG. 16).

[0102] <Channel ID Encoding / Decoding Processing> Next, the channel ID encoding / decoding performed by the feature map encoded information encoding unit 703 / feature map encoded information decoding unit 801 according to the first embodiment will be described in detail.

[0103] In the first embodiment, there are a first mode in which information related to the channel ID is explicitly coded / decoded, and a second mode in which the channel ID is derived without coding / decoding information related to the channel ID. The first mode is called the explicit channel ID mode, and the second mode is called the implicit channel ID mode.

[0104] The explicit channel ID mode is used when the feature map of each channel is placed at an arbitrary position in the feature frame, or when an arbitrary, unique value is assigned to the channel ID of each channel.

[0105] The implicit channel ID mode is used when the channel ID of each channel in a predetermined order is assigned an integer value starting from 0 and incrementing by 1, and the feature maps of each channel are placed in the packing feature frame in a predetermined raster scan order without changing the order of the predetermined channels.

[0106] Fig. 17 shows an example of syntax rules for the channel ID of each channel in frame units (183 in Fig. 14) according to the first embodiment. Also, Fig. 18 shows an example of syntax rules when a header is prepared in feature map units (184 in Fig. 14) for each channel according to the first embodiment and the channel ID is encoded / decoded using the header.

[0107] The syntax element num_feature_channel indicates the number of channels.

[0108] The syntax element explicit_feature_channel_id_flag indicates whether the channel ID is explicitly encoded / decoded. A value of 1 (true) for the syntax element explicit_feature_channel_id_flag indicates that the syntax element feature_channel_id[n] indicating the channel ID of each channel is explicitly encoded / decoded, corresponding to the explicit channel ID mode. A value of 0 (false) for the syntax element explicit_feature_channel_id_flag indicates that the channel ID is not explicitly encoded / decoded but is derived implicitly, corresponding to the implicit channel ID mode.

[0109] The variable n is an integer variable indicating the channel index, and increases by 1 from 0 to (the number of channels - 1). The value of the variable n indicates a predetermined raster scan order.

[0110] The syntax element feature_channel_id[n] indicates the channel ID of the nth channel in a predetermined raster scan order. For example, in the example shown in Figure 12, the values ​​of feature_channel_id[n] are 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, ... in raster scan order.

[0111] In addition, in the example shown in Figure 21, the values ​​of feature_channel_id[n] are 0, 1, 4, 5, 16, 17, 20, 21, 2, 3, 6, 7, 18, 19, 22, 23, 8, 9, 12, ... in raster scan order, and in the example shown in Figure 26, the values ​​of feature_channel_id[n] are 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, ..., 64, 68, 72, ... in raster scan order.

[0112] When encoding / decoding each syntax element, arithmetic coding is used in which shorter codes are assigned to symbols with higher occurrence probabilities, and longer codes are assigned to symbols with lower occurrence probabilities.

[0113] Next, a description will be given of the channel ID encoding process performed by the feature map encoded information encoding unit 703 according to the first embodiment. Fig. 19 is a flowchart illustrating the channel ID encoding process according to the first embodiment.

[0114] First, the syntax element num_feature_channel, which indicates the number of channels to be coded, is coded (step S1101 in FIG. 19). In this embodiment, the value of the syntax element num_feature_channel is 64.

[0115] Next, the syntax element explicit_feature_channel_id_flag, which indicates whether or not to encode / decode the channel ID, is encoded (step S1102 in FIG. 19).

[0116] Next, if the value of the syntax element explicit_feature_channel_id_flag is 1 (true) (YES in step S1103 of FIG. 19), i.e., in explicit channel ID mode, the index of each channel is set to n, and the encoding process of the syntax element feature_channel_id[n] (step S1105 of FIG. 19) is repeatedly performed in a predetermined raster scan order from 0 to (number of channels - 1), i.e., until the encoding process of the channel IDs of all channels is completed, and the process of incrementing the variable n by 1 is performed (steps S1104 to S1106 of FIG. 19).

[0117] On the other hand, if the value of the syntax element explicit_feature_channel_id_flag is 0 (false) (NO in step S1103 of FIG. 19), i.e., in the implicit channel ID mode, the encoding process of the syntax element feature_channel_id[n] indicating the channel ID of each channel (steps S1104 to S1106 of FIG. 19) is omitted, and the encoding process of this channel ID is terminated.

[0118] Next, a description will be given of a channel ID decoding process performed by the feature map encoded information decoding unit 801 according to the first embodiment. Fig. 20 is a flowchart illustrating a channel ID decoding process according to the first embodiment.

[0119] First, the syntax element num_feature_channel indicating the number of channels to be coded is decoded (step S1201 in FIG. 20), followed by the syntax element explicit_feature_channel_id_flag indicating whether to code / decode a channel ID (step S1202 in FIG. 20).

[0120] Next, if the value of the syntax element explicit_feature_channel_id_flag is 1 (true) (YES in step S1203 of FIG. 20), i.e., in explicit channel ID mode, the index of each channel is set to n, and the syntax element feature_channel_id[n] is repeatedly decoded in a predetermined raster scan order from 0 to (number of channels - 1), i.e., until the decoding process of the channel IDs of all channels is completed (step S1205 of FIG. 20), and the variable n is incremented by 1 (steps S1204 to S1206 of FIG. 20).

[0121] On the other hand, if the value of the syntax element explicit_feature_channel_id_flag is 0 (false) (NO in step S1203 of FIG. 20), i.e., in the implicit channel ID mode, the explicit decoding process of the syntax element feature_channel_id[n] indicating the channel ID of each channel is omitted, and the implicit derivation process of the next feature_channel_id[n] is performed. Letting n be the index of each channel, the process repeatedly sets the value of n to feature_channel_id[n] in a predetermined raster scan order, from 0 to (the number of channels minus 1), i.e., until the derivation process of the channel IDs of all channels is completed, and increments the variable n by 1 (steps S1207 to S1209 of FIG. 20).

[0122] In the explicit channel ID mode of the first embodiment, the encoding side sets the value of explicit_feature_channel_id_flag to 1 (true) and performs encoding, and encodes the syntax element feature_channel_id[n] indicating the channel ID of each channel in a predetermined raster scan order, and the decoding side decodes the bit stream with the value of explicit_feature_channel_id_flag set to 1 (true) and decodes the syntax element feature_channel_id[n] indicating the channel ID of each channel in a predetermined raster scan order, thereby making it possible to identify the channel on the decoding side based on the channel ID of each channel.

[0123] In the implicit channel ID mode of the first embodiment, the amount of code can be reduced by omitting explicit encoding / decoding processing of the syntax element feature_channel_id[n] indicating the channel ID of each channel.

[0124] In the first embodiment, the syntax element explicit_feature_channel_id_flag can be set to 1 (true) or 0 (false), but the encoding / decoding process of the syntax element explicit_feature_channel_id_flag may be omitted by fixing explicit_feature_channel_id_flag to 1 (true) or fixing explicit_feature_channel_id_flag to 0 (false). When explicit_feature_channel_id_flag is fixed to 1 (true), the condition determinations of step S1103 in Fig. 19 and step S1203 in Fig. 20 are omitted, and the channel ID encoding / decoding process (steps S1104 to S1106 in Fig. 19 and steps S1204 to S1206 in Fig. 20) is always performed. On the other hand, if explicit_feature_channel_id_flag is fixed to 0 (false), the condition judgment of step S1103 in Figure 19 and step S1203 in Figure 20 and the channel ID encoding / decoding process (steps S1104 to S1106 in Figure 19 and steps S1204 to S1206 in Figure 20) are omitted, and the channel ID setting process (steps S1207 to S1209 in Figure 20) is performed on the decoding side.

[0125] Second Embodiment Next, a feature map encoding device 100 and a feature map decoding device 200 according to a second embodiment implemented by the feature map encoded information encoding unit 703 and the feature map encoded information decoding unit 801 will be described in detail.

[0126] The feature map coded information coding unit 703 of the feature map coding device 100 of the first embodiment codes the syntax element feature_channel_id[n] indicating the channel ID of each channel in the explicit channel ID mode, and the feature map coded information decoding unit 801 of the feature map decoding device 200 of the first embodiment decodes the syntax element feature_channel_id[n] indicating the channel ID of each channel in the explicit channel ID mode. The explicit channel ID mode of the first embodiment is referred to as the first explicit channel ID mode.

[0127] On the other hand, the feature map coded information coding unit 703 of the feature map coding device 100 of the second embodiment differs in that a second explicit channel ID mode is added to the first explicit channel ID mode. In the second explicit channel ID mode, a syntax element feature_channel_id_delta[n] indicating the difference between the channel IDs of each channel is coded, and in the feature map coded information decoding unit 801 of the feature map decoding device 200 of the second embodiment, in the second explicit channel ID mode, the syntax element feature_channel_id_delta[n] indicating the difference between the channel IDs of each channel is decoded, and the channel ID of each channel is derived from the difference between the channel ID of the immediately preceding channel in a predetermined raster scan order.

[0128] Fig. 22 shows an example of syntax rules for channel IDs of each channel in units of frames according to the second embodiment. Fig. 23 shows an example of syntax rules when a header is prepared for each feature map of each channel according to the second embodiment and the channel ID is encoded / decoded using the header.

[0129] The syntax element num_feature_channel indicates the number of channels.

[0130] In the second embodiment, the syntax element explicit_feature_channel_id_flag indicates whether the channel ID or the channel ID difference is explicitly encoded / decoded. A value of 1 for the syntax element explicit_feature_channel_id_flag indicates that the syntax element feature_channel_id[n] indicating the channel ID of each channel is explicitly encoded / decoded, corresponding to the first explicit channel ID mode. A value of 2 for the syntax element explicit_feature_channel_id_flag indicates that the syntax element feature_channel_id_delta[n] indicating the channel ID difference of each channel is explicitly encoded / decoded, corresponding to the second explicit channel ID mode. A value of 0 for the syntax element explicit_feature_channel_id_flag indicates that the channel ID difference is not explicitly encoded / decoded but is implicitly derived, corresponding to the implicit channel ID mode.

[0131] The variable n indicates the channel index, and increases by 1 from 0 to (the number of channels - 1). The value of the variable n indicates a predetermined raster scan order.

[0132] The syntax element feature_channel_id_delta[n] indicates the difference in channel ID. Specifically, it indicates the difference from the (n-1)th channel in a predetermined raster scan order, i.e., the channel ID feature_channel_id[n-1] of the immediately preceding channel. The syntax element feature_channel_id_delta[n] is derived using the following formula:

[0133] feature_channel_id_delta[n]=feature_channel_id[n]−feature_channel_id[n−1] Note that since there is no channel immediately before the first channel, the channel ID of the channel immediately before the first channel is initialized to 0.

[0134] For example, in the example shown in Figure 21, the values ​​of feature_channel_id_delta[n] are 0, 1, 3, 1, 11, 1, 3, 1, -19, 1, 3, 1, 11, 1, 3, 1, -15, 1, 3, ... in raster scan order, and in the example shown in Figure 26, the values ​​of feature_channel_id_delta[n] are 0, 2, 2, 2, ..., 4, 4, 4, ... in raster scan order.

[0135] Next, a description will be given of the channel ID encoding process performed by the feature map encoded information encoding unit 703 according to the second embodiment. Fig. 24 is a flowchart illustrating the channel ID encoding process according to the second embodiment.

[0136] First, the syntax element num_feature_channel indicating the number of channels to be coded is coded (step S1101 in FIG. 24).

[0137] Next, the syntax element explicit_feature_channel_id_flag, which indicates whether the channel ID or the difference between channel IDs is to be coded / decoded, is coded (step S1102 in FIG. 24).

[0138] Next, if the value of the syntax element explicit_feature_channel_id_flag is 1 (YES in step S1103 of FIG. 24), i.e., in the first explicit channel ID mode, the index of each channel is set to n, and the encoding process of the syntax element feature_channel_id[n] (step S1105 of FIG. 24) is repeatedly performed in a predetermined raster scan order from 0 to (number of channels - 1), i.e., until the encoding process of the channel IDs of all channels is completed, and the process of incrementing the variable n by 1 is performed (steps S1104 to S1106 of FIG. 24).

[0139] Next, if the value of the syntax element explicit_feature_channel_id_flag is 2 (NO in step S1103 and YES in step S1107 in Figure 24), i.e., in the second explicit channel ID mode, the variable feature_channel_id_prev, which indicates the channel ID of the previous channel in a specified raster scan order, is set (initialized) to 0 (step S1108 in Figure 24).

[0140] Next, with n as the index of each channel, the process of deriving the syntax element feature_channel_id_delta[n] (step S1110 in Figure 24) and the process of encoding the syntax element feature_channel_id_delta[n] (step S1111 in Figure 24) are repeatedly performed in a predetermined raster scan order, from 0 to (number of channels - 1), that is, until the encoding process of the channel ID differences of all channels is completed, and the value of the syntax element feature_channel_id_delta[n] is set to the variable feature_channel_id_prev (step S1112 in Figure 24), and the process of incrementing the variable n by 1 is performed (steps S1109 to S1113 in Figure 24).

[0141] The syntax element feature_channel_id_delta[n] is derived using the following formula:

[0142] feature_channel_id_delta[n]=feature_channel_id[n]−feature_channel_id_prev where feature_channel_id[n] indicates the channel ID of the nth channel in a predetermined raster scan order.

[0143] On the other hand, if the value of the syntax element explicit_feature_channel_id_flag is 0 (NO in step S1103 and NO in step S1107 in Figure 24), i.e., in the implicit channel ID mode, the encoding process of the syntax element feature_channel_id[n] indicating the channel ID of each channel (steps S1104 to S1106 in Figure 24) and the encoding process of the syntax element feature_channel_id_delta[n] indicating the difference between the channel IDs of each channel (steps S1108 to S1113 in Figure 24) are omitted, and the encoding process of this channel ID is terminated.

[0144] Next, a description will be given of a channel ID decoding process performed by the feature map encoded information decoding unit 801 according to the second embodiment. Fig. 25 is a flowchart illustrating a channel ID decoding process according to the second embodiment.

[0145] First, the syntax element num_feature_channel indicating the number of channels to be coded is decoded (step S1201 in FIG. 25). Next, the syntax element explicit_feature_channel_id_flag indicating whether to code / decode a channel ID or a difference between channel IDs is decoded (step S1202 in FIG. 25).

[0146] Next, if the value of the syntax element explicit_feature_channel_id_flag is 1 (YES in step S1203 of FIG. 25), i.e., in the first explicit channel ID mode, the index of each channel is set to n, and the syntax element feature_channel_id[n] is repeatedly decoded in a predetermined raster scan order from 0 to (number of channels - 1), i.e., until the decoding process of the channel IDs of all channels is completed (step S1205 of FIG. 25), and the variable n is incremented by 1 (steps S1204 to S1206 of FIG. 25).

[0147] Next, if the value of the syntax element explicit_feature_channel_id_flag is 2 (NO in step S1203 and YES in step S1210 in Figure 25), i.e., in the second explicit channel ID mode, the variable feature_channel_id_prev indicating the channel ID of the most recently encoded channel is set (initialized) to 0 (step S1211 in Figure 25).

[0148] Next, with n as the index of each channel, the decoding process of the syntax element feature_channel_id_delta[n] (step S1213 in Figure 25) and the derivation process of feature_channel_id[n] (step S1214 in Figure 24) are repeatedly performed in a predetermined raster scan order, from 0 to (number of channels - 1), that is, until the decoding process of the channel ID differences of all channels is completed, and the value of the syntax element feature_channel_id[n] is set to the variable feature_channel_id_prev (step S1215 in Figure 25), and the process of incrementing the variable n by 1 is performed (steps S1212 to S1216 in Figure 25).

[0149] On the other hand, if the value of the syntax element explicit_feature_channel_id_flag is 0 (NO in step S1203 and NO in step S1210 in Figure 25), i.e., in the implicit channel ID mode, the decoding process of feature_channel_id[n] (steps S1204 to S1206 in Figure 25), the decoding process of feature_channel_id_delta[n], and the derivation process of feature_channel_id[n] (steps S1211 to S1216 in Figure 25) are omitted, and the implicit derivation process of the next feature_channel_id[n] is performed, as in the first embodiment. Let n be the index of each channel, and in a predetermined raster scan order, the value of n is repeatedly set to feature_channel_id[n] and the variable n is incremented by 1 until n ranges from 0 to (number of channels - 1), i.e., until the process of deriving the channel IDs of all channels is completed (steps S1207 to S1209 in Figure 25).

[0150] In the second explicit channel ID mode, the encoding side sets the value of explicit_feature_channel_id_flag to 2 and performs encoding, and encodes the syntax element feature_channel_id_delta[n] indicating the difference in channel IDs between each channel in a predetermined raster scan order (step S1111 in Figure 24).The decoding side decodes the bit stream with the value of explicit_feature_channel_id_flag set to 2 and decodes the syntax element feature_channel_id_delta[n] indicating the difference in channel IDs between each channel in a predetermined raster scan order (step S1213 in Figure 25), and derives feature_channel_id_delta[n] (step S1214 in Figure 24), allowing the decoding side to identify the channel based on the channel ID of each channel.

[0151] Furthermore, in the second explicit channel ID mode, instead of the syntax element feature_channel_id[n] indicating the channel ID of each channel, the encoding side encodes the syntax element feature_channel_id_delta[n] indicating the difference between the channel IDs of each channel (step S1111 in FIG. 24 ), and the decoding side decodes the syntax element feature_channel_id_delta[n] indicating the difference between the channel IDs of each channel (step S1213 in FIG. 25 ). Therefore, when the code amount of the channel ID difference is smaller than the code amount of the channel ID, it is possible to reduce the code amount compared to the first explicit channel ID mode. For example, in the example shown in FIG. 21 , the values ​​of feature_channel_id_delta[n] indicating the channel ID difference are 0, 1, 3, 1, 11, 1, 3, 1, −19, 1, 3, 1, 11, 1, 3, 1, −15, 1, 3, ... in raster scan order, with the occurrence probability of 1 being the highest, followed by the occurrence probability of 3. In the example shown in Figure 26, the values ​​of feature_channel_id_delta[n] indicating the difference between channel IDs are 0, 2, 2, 2, ..., 4, 4, 4, ... in raster scan order, with 2 and 4 having a high probability of appearing. In the arithmetic coding used in this embodiment, shorter codes are assigned to symbols with a higher probability of appearing, and longer codes are assigned to symbols with a lower probability of appearing, so in the examples shown in Figures 21 and 26, the amount of code can be reduced by selecting the second explicit channel ID mode.

[0152] If the overall amount of code is smaller when encoding the channel ID difference than when encoding the channel ID value, for example, because there is a bias in the probability of occurrence of the channel ID difference value, the second explicit channel ID mode is selected, and if the overall amount of code is smaller when encoding the channel ID value than when encoding the channel ID difference, the overall amount of code is reduced by selecting the first explicit channel ID mode.

[0153] In addition, in the implicit channel ID mode of the second embodiment, the amount of code can be reduced by omitting explicit encoding / decoding processing of the syntax element feature_channel_id[n] indicating the channel ID of each channel and the syntax element feature_channel_id_delta[n] indicating the difference between the channel IDs of each channel.

[0154] By configuring the system to allow selection between explicit channel ID mode and implicit channel ID mode, it is possible to use it for different purposes, depending on whether you want to freely set the channel ID or whether you want to reduce the amount of code while keeping the default channel ID.

[0155] In the first embodiment described above, switching is performed between the first explicit channel ID mode and the implicit channel ID mode, and in the second embodiment, switching is performed between the first explicit channel ID mode, the second explicit channel ID mode, and the implicit channel ID mode. However, the present invention is not limited to this, and switching between the second explicit channel ID mode and the implicit channel ID mode may be performed, or only the first explicit channel ID mode, only the second explicit channel ID mode, or only the implicit channel ID mode may be used.

[0156] Third Embodiment In the third embodiment, a multi-channel decimal type single-scale feature map xf supplied from the feature map reduction unit 102 is packed and encoded into a packing feature frame. In contrast, in the third embodiment, for the multi-channel decimal type single-scale feature map xf supplied from the feature map reduction unit 102, active channels to be encoded and inactive channels to be omitted from encoding are determined, and only the active channels are packed into a packing feature frame and encoded.

[0157] An active channel is a channel that encodes a feature map in the feature map encoding device 100 on the encoding side and decodes a feature map in the feature map decoding device 200 on the decoding side. On the other hand, an inactive channel is a channel that does not encode a feature map on the encoding side and generates a feature map on the decoding side. An inactive channel is treated as a feature map having predetermined elements on the decoding side.

[0158] In the third embodiment, a flag indicating whether a channel is active or inactive is coded / decoded for each channel to determine whether it is an active channel or an inactive channel.

[0159] <Feature Map Encoding / Decoding Processing Procedure of Third Embodiment> Next, a description will be given of the feature map encoding processing procedure performed by the feature map reduction unit 102, feature map conversion unit 103, and feature map internal coding unit 104 of the coding map encoding unit 100 of the third embodiment. Fig. 27 is a flowchart illustrating the feature map encoding processing procedure according to the third embodiment.

[0160] First, the packing unit 701 of the feature map conversion unit 103 evaluates the importance of the feature map of each channel and determines active and inactive channels (step S1001 in FIG. 27). By omitting the coding of inactive channels, which are unimportant channels, coding efficiency is improved and coding degradation of highly important channels, which have a significant impact on image recognition results, is suppressed.

[0161] Next, the packing unit 701 of the feature map conversion unit 103 determines the order of the feature maps of the active channels (step S1002 in FIG. 27 ). The order of the feature maps of the active channels can be determined arbitrarily based on the importance of each channel, the correlation between channels, and other factors. FIG. 29 shows an example of the state in which the feature maps of the active channels are packed into one packed feature frame. The total number of channels, the number of active channels, and the number of inactive channels can be set arbitrarily depending on the importance of the feature maps, and other factors. For example, channels whose importance exceeds a predetermined threshold can be set as active channels, and the remaining channels can be set as inactive channels. Alternatively, a representative importance level can be calculated for all channels, and channels whose importance level exceeds the representative importance level can be set as active channels, and the remaining channels can be set as inactive channels. Statistical information such as the mean or median can be used as the representative importance level. In the third embodiment, the total number of channels is 64, the number of active channels is 48, and the number of inactive channels is 16. The encoding / decoding of channel IDs for active and inactive channels will be described later.

[0162] Next, the packing unit 701 packs the feature maps of each channel according to the determined arrangement order of the feature maps of the active channels to generate a packed feature frame (step S1003 in FIG. 27 ). Inactive channels that are not to be coded are discarded. If gaps remain even after the feature maps of all active channels are arranged in the packed feature frame, a predetermined value (such as 0) is set to the gap element.

[0163] The packing unit 701 supplies the packing feature frame to a feature map quantization unit 702, and also supplies information such as the total number of channels, the number of active channels, the number of inactive channels, the channel ID of each channel, and whether each channel is an active channel or an inactive channel to a feature map coding information encoding unit 703 as feature map coding information.

[0164] Next, the feature map quantization unit 702 of the feature map conversion unit 103 quantizes the packing feature frame, converting the packing feature frame from decimal type to integer type (step S1004 in FIG. 27).

[0165] Next, the feature map encoded information encoding unit 703 of the feature map conversion unit 103 encodes the feature map encoded information, such as the total number of channels, the channel ID of each channel, whether each channel is an active channel or an inactive channel, and information about quantization, to generate a bitstream in which the feature map encoded information is encoded (step S1005 in FIG. 27 ). The encoding of the feature map encoded information will be described later.

[0166] Subsequently, the VVC encoding unit 902, the HEVC encoding unit 903, or the AV1 encoding unit 904 of the feature map intra encoding unit 104 encodes the packing feature frame and generates a bitstream in which the packing feature frame is encoded (step S1006 in FIG. 27 ).

[0167] Next, the multiplexing unit 905 of the feature map internal coding unit 104 multiplexes the bit stream in which the feature map coding information is coded and the bit stream in which the packing feature frame is coded to generate a multiplexed bit stream (step S1007 in FIG. 27 ).

[0168] Next, a description will be given of the feature map decoding process performed by the feature map internal decoding unit 201 and the feature map inverse conversion unit 202 of the coding map decoding unit 200 according to the third embodiment. Fig. 28 is a flowchart illustrating the feature map decoding process according to the third embodiment.

[0169] The demultiplexing unit 1005 of the feature map internal decoding unit 201 demultiplexes the multiplexed bit stream output from the encoding side, and separates it into a bit stream in which feature map encoding information is encoded and a bit stream in which packing feature frames are encoded (step S2001 in FIG. 28 ).

[0170] Subsequently, the VVC decoding unit 1002, the HEVC decoding unit 1003, or the AV1 decoding unit 1004 of the feature map internal decoding unit 201 decodes the bit stream in which the packing feature frame is coded, and generates the packing feature frame (step S2002 in FIG. 28 ).

[0171] Next, the feature map coded information decoding unit 801 of the feature map inverse transform unit 202 decodes the bit stream in which the feature map coded information has been coded, and generates feature map coded information such as the total number of channels, the channel ID of each channel, whether each channel is an active channel or an inactive channel, information about quantization, etc. (step S2003 in FIG. 28 ). The decoding of the feature map coded information will be described later.

[0172] Next, the feature map inverse quantization unit 802 of the feature map inverse transformation unit 202 converts the packing feature frame from integer type to decimal type (step S2004 in FIG. 28).

[0173] Next, the unpacking unit 803 of the feature map inverse transformation unit 202 unpacks the packed feature frame and generates a feature map for each active channel (step S2005 in FIG. 28 ). Furthermore, the unpacking unit 803 of the feature map inverse transformation unit 202 generates a feature map for an inactive channel (step S2006 in FIG. 28 ). A predetermined value such as 0 is set to each element (each pixel) of the feature map for the inactive channel to generate the feature map for the inactive channel.

[0174] <Regarding the encoding / decoding process of feature map encoding information in the third embodiment> In the third embodiment, active channels to be encoded and inactive channels to be omitted from encoding are determined, and only the feature maps of the active channels are packed into a packing feature frame and encoded / decoded. Therefore, in addition to information about the total number of channels, a flag indicating whether each channel is active or inactive and the channel ID of the channel are encoded / decoded. On the decoding side, the channel ID of each channel is decoded or generated, thereby setting it explicitly or implicitly.

[0175] The encoding / decoding of feature map coded information according to the third embodiment, which is performed by the feature map coded information encoder 703 and the feature map coded information decoder 801, will now be described in detail.

[0176] Fig. 30 shows an example of syntax rules for feature map coded information in frame units (183 in Fig. 14 ) according to the third embodiment. Fig. 31 shows an example of syntax rules for encoding / decoding feature map coded information using a header feature_channel_header prepared in feature map units for each channel (184 in Fig. 14 ) according to the third embodiment.

[0177] The syntax element num_feature_channel indicates the total number of channels (the sum of the number of active and inactive channels). The explicit_non_active_channel_info_flag is a flag indicating whether to explicitly encode / decode the feature map coding information of inactive channels. A value of 1 (true) for the syntax element explicit_non_active_channel_info_flag indicates that the feature map coding information including the syntax element feature_channel_id[n] indicating the channel ID of each inactive channel is to be explicitly encoded / decoded. A value of 0 (false) for the syntax element explicit_non_active_channel_info_flag indicates that the feature map coding information including the channel ID of the inactive channel is not to be explicitly encoded / decoded, but is to be implicitly set on the decoding side.

[0178] The variable n is an integer variable indicating the channel index, and increases by 1 from 0 to (the number of channels - 1). The value of the variable n indicates a predetermined raster scan order.

[0179] The syntax element active_channel_flag[n] indicates whether the nth channel is an active or inactive channel in a given raster scan order. A value of 1 (true) for the syntax element active_channel_flag[n] indicates that the nth channel is an active channel in a given raster scan order, and a value of 0 (false) for the syntax element active_channel_flag[n] indicates that the nth channel is an inactive channel in a given raster scan order.

[0180] The syntax element feature_channel_id[n] indicates the channel ID of the nth channel in a predetermined raster scan order.

[0181] When encoding / decoding each syntax element, arithmetic coding is used in which shorter codes are assigned to symbols with higher occurrence probabilities, and longer codes are assigned to symbols with lower occurrence probabilities.

[0182] Next, a description will be given of a procedure for encoding feature map encoded information according to the third embodiment. Fig. 32 is a flowchart illustrating a procedure for encoding feature map encoded information performed by the feature map encoded information encoding unit 703 according to the third embodiment.

[0183] First, the syntax element num_feature_channel, which indicates the total number of channels to be coded, is coded (step S1121 in FIG. 32). In this embodiment, the value of the syntax element num_feature_channel is 64.

[0184] Next, the syntax element explicit_non_active_channel_info_flag, which indicates whether or not to encode / decode the feature map encoded information of the inactive channel, is encoded (step S1123 in FIG. 32).

[0185] Next, in a predetermined raster scan order, the encoding process of the syntax element active_channel_flag[n] of the nth channel (step S1125 in FIG. 32) and the encoding process of the feature map encoding information of the nth channel including the syntax element feature_channel_id[n] (steps S1126 to S1127 in FIG. 32) are repeatedly performed, with n ranging from 0 to num_feature_channel-1 (the number of channels minus 1), that is, until the encoding process of the feature map encoding information of all channels is completed, and the process of incrementing the variable n by 1 is performed (steps S1124 to S1129 in FIG. 32).

[0186] In the example of Figure 29, the channels with index n shown in Figure 29 of 48, 56, 59, 58, 23, 7, 6, 35, 20, 1, 42, 27, 0, 30, 16, 63, 9, 54, 14, 43, 61, 11, 60, 55, 46, 51, 50, 22, 13, 3, 10, 18, 40, 52, 28, 31, 37, 5, 34, 33, 47, 12, 53, 4, 29, 19, 57, and 25 are active channels, and a syntax element active_channel_flag[n] set to a value of 1 (true) is coded. 29 are inactive channels, and the syntax element active_channel_flag[n] set to a value of 0 (false) is coded. Note that in the third embodiment, only the feature maps of the active channels are packed into the packing feature frames in raster scan order, and the feature maps of the inactive channels are not packed into the packing feature frames. However, in the encoding / decoding process of the feature map coding information, the feature maps of the inactive channels are virtually arranged in raster scan order following the feature maps of the active channels. If the syntax element active_channel_flag[n] is 1 (true) or the value of the syntax element explicit_non_active_channel_info_flag is 1 (true) (YES in step S1126 of FIG. 32 ), the feature map coded information of the n-th channel including the syntax element feature_channel_id[n] is coded (step S1127 of FIG. 32 ). On the other hand, if the syntax element active_channel_flag[n] is 0 (false) and the value of the syntax element explicit_non_active_channel_info_flag is 0 (false), the process of explicitly coding the feature map coded information including the channel ID of the inactive channel is omitted (NO in step S1126 of FIG. 32 ).

[0187] Next, a description will be given of a procedure for decoding feature map coded information, which is performed by the feature map coded information decoding unit 801 according to the third embodiment. Fig. 33 is a flowchart illustrating the procedure for decoding feature map coded information according to the third embodiment.

[0188] First, the syntax element num_feature_channel indicating the total number of channels is decoded (step S1221 in FIG. 33).

[0189] Next, the syntax element explicit_non_active_channel_info_flag, which indicates whether or not to encode / decode feature map encoded information including the channel ID of an inactive channel, is decoded (step S1223 in FIG. 33).

[0190] Next, in a predetermined raster scan order, the decoding process of the syntax element active_channel_flag[n] of the nth channel (step S1225 in Figure 33) and the decoding or setting process of the feature map coding information of the nth channel including the syntax element feature_channel_id[n] (steps S1226 to S1228 in Figure 33) are repeatedly performed, with n ranging from 0 to num_feature_channel-1 (the number of all channels minus 1), that is, until the decoding process of the feature map coding information of all channels is completed, and the process of incrementing the variable n by 1 (steps S1224 to S1229 in Figure 33) is performed. If the syntax element active_channel_flag[n] is 1 (true) or the value of the syntax element explicit_non_active_channel_info_flag is 1 (true) (YES in step S1226 of FIG. 33 ), the feature map coding information of the nth channel including the syntax element feature_channel_id[n] is decoded (step S1227 of FIG. 33 ). On the other hand, if the syntax element active_channel_flag[n] is 0 (false) and the value of the syntax element explicit_non_active_channel_info_flag is 0 (false) (NO in step S1226 of FIG. 33 ), the nth channel is designated as an inactive channel, and feature map coding information including the channel ID feature_channel_id[n] is set (step S1228 of FIG. 33 ). Here, the channel ID feature_channel_id[n] of the inactive channel is set to a unique value other than the channel ID of the decoded active channel. In the example of Figure 29, 2, 8, 15, 17, 21, 24, 26, 32, 36, 38, 39, 41, 44, 45, 49, and 62, which are not shown in Figure 29, are set as the channel IDs feature_channel_id[n] of the inactive channels.

[0191] It should be noted that a predetermined value such as 0 is set to each element (each pixel) of the feature map of an inactive channel.

[0192] In the third embodiment, the syntax element explicit_non_active_channel_info_flag can be set to 1 (true) or 0 (false), but the encoding / decoding process of the syntax element explicit_non_active_channel_info_flag may be omitted by fixing explicit_non_active_channel_info_flag to 1 (true) or fixing explicit_non_active_channel_info_flag to 0 (false). When explicit_non_active_channel_info_flag is fixed to 1 (true), the condition determinations of step S1126 in Fig. 32 and step S1226 in Fig. 33 are omitted, and the encoding / decoding process of the feature map encoding information (step S1127 in Fig. 32 and step S1227 in Fig. 33) is performed even for inactive channels.

[0193] Furthermore, in the third embodiment, a syntax element feature_channel_id[n] indicating the channel ID of each channel is encoded in a predetermined raster scan order, and the decoding side decodes the syntax element feature_channel_id[n] indicating the channel ID of each channel in a predetermined raster scan order, thereby enabling the decoding side to identify the channel based on the channel ID of each channel. Furthermore, in the third embodiment, a syntax element active_channel_flag[n] indicating whether a channel is active or inactive is encoded, and the decoding side decodes the syntax element active_channel_flag[n], enabling the decoding side to distinguish between active and inactive channels. In the procedure for encoding / decoding feature map coded information in the third embodiment, the coding / decoding of feature map coded information for an active channel is followed by the coding / decoding of feature map coded information for an inactive channel. Therefore, after coding / decoding a syntax element feature_channel_id[n] with a value of 0, the values ​​of subsequent syntax elements feature_channel_id[n] may be set to 0, and the coding / decoding of the syntax elements feature_channel_id[n] may be omitted.

[0194] Furthermore, in the third embodiment, when the channel ID of each channel is set to a value that starts from 0 and increases by 1, the value of the syntax element explicit_non_active_channel_info_flag is set to 0 (false) and the value of the syntax element active_channel_flag[n] of the inactive channel is set to 0 (false). This makes it possible to omit encoding of the channel IDs of the inactive channels on the encoding side and generate the channel IDs of the inactive channels on the decoding side, thereby reducing the amount of coding of feature map encoding information including the syntax element feature_channel_id[n].

[0195] Fourth Embodiment In the fourth embodiment, similarly to the third embodiment, a multi-channel decimal type single-scale feature map xf supplied from the feature map reduction unit 102 is packed and encoded into a packing feature frame, whereas in the third embodiment, for the multi-channel decimal type single-scale feature map xf supplied from the feature map reduction unit 102, active channels to be encoded and inactive channels to be omitted from encoding are determined, and only the active channels are packed into a packing feature frame and encoded.

[0196] In the third embodiment, a flag indicating whether a channel is active or inactive is coded / decoded for each channel, whereas in the fourth embodiment, the number of active channels is coded / decoded in addition to the total number of channels, allowing the decoding side to determine whether each channel is an active channel or an inactive channel.

[0197] <Feature Map Encoding / Decoding Processing Procedure of Fourth Embodiment> Next, the feature map encoding processing procedure performed by the feature map reduction unit 102, feature map conversion unit 103, and feature map internal encoding unit 104 of the encoding map encoding unit 100 of the fourth embodiment will be described, focusing only on the differences from the third embodiment.

[0198] The feature map encoded information encoder 703 of the feature map converter 103 of the fourth embodiment encodes feature map encoded information, such as the total number of channels, the number of active channels, the channel ID of each channel, and information about quantization, to generate a bitstream in which the feature map encoded information is encoded (step S1005 in FIG. 27 ). The encoding of the feature map encoded information of the fourth embodiment will be described later. The remaining processing is the same as that of the third embodiment.

[0199] Next, the feature map decoding process performed by the feature map internal decoding unit 201 and the feature map inverse conversion unit 202 of the coding map decoding unit 200 of the fourth embodiment will be described, focusing only on the differences from the third embodiment.

[0200] The feature map coded information decoding unit 801 in the feature map inverse transform unit 202 decodes the bit stream in which the feature map coded information has been coded, and generates feature map coded information such as the total number of channels, the number of active channels, the channel ID of each channel, and information about quantization (step S2003 in FIG. 28 ). The decoding of the feature map coded information in the fourth embodiment will be described later. The rest of the processing is the same as in the third embodiment.

[0201] <Regarding the encoding / decoding process of feature map encoded information in the fourth embodiment> In the third embodiment, a flag indicating whether each channel is an active channel or an inactive channel is encoded / decoded and determined. However, in the fourth embodiment, in addition to information regarding the total number of channels, information regarding the number of active channels is encoded / decoded, and on the decoding side, it is determined whether each channel is an active channel or an inactive channel based on the information regarding the total number of channels and the information regarding the number of active channels.

[0202] The encoding / decoding of feature map coded information according to the fourth embodiment, which is performed by the feature map coded information encoder 703 and the feature map coded information decoder 801, will now be described in detail.

[0203] Fig. 34 shows an example of syntax rules for feature map coded information in frame units (183 in Fig. 14) according to the fourth embodiment. Fig. 35 shows an example of syntax rules for encoding / decoding feature map coded information using a header feature_channel_header prepared in feature map units for each channel (184 in Fig. 14) according to the fourth embodiment.

[0204] The syntax element num_feature_channel indicates the total number of channels (the sum of the number of active and inactive channels), and the syntax element num_active_feature_channel indicates the number of active channels. The explicit_non_active_channel_info_flag is a flag that indicates whether to explicitly encode / decode feature map coding information including the channel IDs of inactive channels. A value of 1 (true) for the syntax element explicit_non_active_channel_info_flag indicates that feature map coding information including the syntax element feature_channel_id[n] indicating the channel IDs of each inactive channel is to be explicitly encoded / decoded. A value of 0 (false) for the syntax element explicit_non_active_channel_info_flag indicates that feature map coding information including the channel IDs of inactive channels is not to be explicitly encoded / decoded, but is to be implicitly derived. The variable n is an integer variable that increases by 1 from 0 to num_active_feature_channel-1 (the number of active channels minus 1).

[0205] The syntax element feature_channel_id[n], where n is from 0 to num_active_feature_channel-1 (the number of active channels minus 1), indicates the channel ID of the active channel to be placed nth in a specified raster scan order. The syntax element feature_channel_id[n], where n is from num_active_feature_channel (the number of active channels) to num_feature_channel-1 (the number of all channels minus 1), indicates the channel IDs of the remaining inactive channels excluding the active channels.

[0206] When encoding / decoding each syntax element, arithmetic coding is used in which shorter codes are assigned to symbols with higher occurrence probabilities, and longer codes are assigned to symbols with lower occurrence probabilities.

[0207] Next, a description will be given of the procedure for encoding feature map encoded information performed by the feature map encoded information encoding unit 703 according to the fourth embodiment. Fig. 36 is a flowchart illustrating the procedure for encoding feature map encoded information according to the fourth embodiment.

[0208] First, the syntax element num_feature_channel, which indicates the total number of channels, is coded (step S1131 in FIG. 36). In this embodiment, the value of the syntax element num_feature_channel is 64.

[0209] Next, the syntax element num_active_feature_channel, which indicates the number of active channels, is coded (step S1132 in FIG. 36). In this embodiment, the value of the syntax element num_active_feature_channel is 48.

[0210] Next, the syntax element explicit_non_active_channel_info_flag, which indicates whether or not to encode / decode the feature map encoded information of the inactive channel, is encoded (step S1133 in FIG. 36).

[0211] Next, in a predetermined raster scan order, the encoding process of the syntax element feature_channel_id[n] is repeatedly performed (step S1135 in FIG. 36 ) until n is from 0 to num_active_feature_channel−1 (the number of active channels−1), that is, until the encoding process of the feature map encoding information of all active channels is completed, and the process of incrementing the variable n by 1 is performed (steps S1134 to S1136 in FIG. 36 ).

[0212] Next, if the value of the syntax element explicit_non_active_channel_info_flag is 1 (true) (YES in step S1137 of Figure 36), the encoding process of the syntax element feature_channel_id[n] (step S1139 of Figure 36) is repeatedly performed in a specified raster scan order from num_active_feature_channel (number of active channels) to num_feature_channel-1 (number of all channels - 1), that is, until the encoding process of the channel IDs of all remaining inactive channels excluding the active channels is completed, and the process of incrementing variable n by 1 is performed (steps S1138 to S1140 of Figure 36).

[0213] On the other hand, if the value of the syntax element explicit_non_active_channel_info_flag is 0 (false), the process of explicitly encoding feature map encoding information including the channel ID of the inactive channel is omitted (NO in step S1137 in FIG. 36).

[0214] Next, a description will be given of a procedure for decoding feature map coded information, which is performed by the feature map coded information decoding unit 801 according to the fourth embodiment. Fig. 37 is a flowchart illustrating a procedure for decoding feature map coded information according to the fourth embodiment.

[0215] First, the syntax element num_feature_channel indicating the total number of channels is decoded (step S1231 in FIG. 37), followed by the syntax element num_active_feature_channel indicating the number of active channels (step S1232 in FIG. 37).

[0216] Next, the syntax element explicit_non_active_channel_info_flag, which indicates whether to encode / decode the feature map encoded information of the inactive channel, is decoded (step S1233 in FIG. 37).

[0217] Next, in a predetermined raster scan order, the decoding process of the syntax element feature_channel_id[n] is repeatedly performed (step S1235 in FIG. 37) until n is from 0 to num_active_feature_channel-1 (the number of active channels minus 1), that is, until the decoding process of the feature map coding information related to all active channels is completed, and the process of incrementing the variable n by 1 is performed (steps S1234 to S1236 in FIG. 37).

[0218] Next, if the value of the syntax element explicit_non_active_channel_info_flag is 1 (true) (YES in step S1237 of FIG. 37 ), the decoding process of the syntax element feature_channel_id[n] (step S1239 of FIG. 37 ) is repeatedly performed in a predetermined raster scan order from num_active_feature_channel (the number of active channels) to num_feature_channel−1 (the number of all channels−1), that is, until the decoding process of the feature map encoding information including the channel IDs of all remaining inactive channels excluding the active channels is completed, and the process of incrementing the variable n by 1 is performed (steps S1238 to S1240 of FIG. 37 ).

[0219] On the other hand, if the value of the syntax element explicit_non_active_channel_info_flag is 0 (false) (NO in step S1237 of FIG. 37 ), feature map encoding information including the channel IDs feature_channel_id[n] of the inactive channels is set for all remaining inactive channels excluding the active channels, where n is from num_active_feature_channel to num_feature_channel−1 (the total number of channels minus 1), in a predetermined raster scan order (step S1242 of FIG. 37 ), and the variable n is incremented by 1 (steps S1241 to S1243 of FIG. 37 ). Here, the channel IDs feature_channel_id[n] of the inactive channels (where n is from num_active_feature_channel to num_feature_channel−1) are set to unique values ​​other than the channel IDs of the decoded active channels. In the example of Figure 29, 2, 8, 15, 17, 21, 24, 26, 32, 36, 38, 39, 41, 44, 45, 49, and 62 not shown in Figure 29 are set to the channel IDs feature_channel_id[n] of the inactive channels (n is num_active_feature_channel to num_feature_channel-1).

[0220] It should be noted that a predetermined value such as 0 is set to each element (each pixel) of the feature map of an inactive channel.

[0221] As in the third embodiment, in the fourth embodiment, the syntax element explicit_non_active_channel_info_flag can be set to 1 (true) or 0 (false), but the encoding / decoding process of the syntax element explicit_non_active_channel_info_flag may be omitted by fixing explicit_non_active_channel_info_flag to 1 (true) or fixing explicit_non_active_channel_info_flag to 0 (false). When explicit_non_active_channel_info_flag is fixed to 1 (true), the condition determination in step S1137 of Fig. 36 and step S1237 of Fig. 37 is unnecessary, and the encoding / decoding process of the feature map encoding information (steps S1138 to S1140 of Fig. 36 and steps S1238 to S1240 of Fig. 37) is performed even for inactive channels. Furthermore, in the fourth embodiment, as in the third embodiment, the syntax element feature_channel_id[n] indicating the channel ID of each channel is encoded in a predetermined raster scan order, and on the decoding side, the syntax element feature_channel_id[n] indicating the channel ID of each channel is decoded in a predetermined raster scan order, so that the decoding side can identify the channel based on the channel ID of each channel.

[0222] Furthermore, in the fourth embodiment, the syntax element num_active_feature_channel indicating the number of active channels is coded, and the syntax element num_active_feature_channel is decoded on the decoding side, thereby enabling the decoding side to distinguish between active and inactive channels. In the third embodiment, the syntax element active_channel_flag[n] indicating whether each channel is active or inactive is coded / decoded, whereas in the fourth embodiment, only one syntax element num_active_feature_channel indicating the number of active channels is coded / decoded on a frame basis (183 in FIG. 14 ) or on a feature map basis (184 in FIG. 14 ) of each channel, thereby enabling a reduction in the amount of code compared to the third embodiment.

[0223] Furthermore, in the fourth embodiment, when the channel ID of each channel is set to a value that starts from 0 and increases by 1, by setting the value of the syntax element explicit_non_active_channel_info_flag to 0 (false), it is possible to omit encoding of the channel IDs of inactive channels on the encoding side and generate channel IDs of inactive channels on the decoding side, thereby reducing the amount of code for feature map encoding information including the syntax element feature_channel_id[n].

[0224] In the first to fourth embodiments, the syntax rules for channel IDs are set on a frame-by-frame basis (183 in FIG. 14 ) or on a feature map-by-feature map basis (184 in FIG. 14 ) for each channel, and encoding / decoding is performed. However, if the channel IDs of each channel and the order of packings do not change over time, the syntax rules for channel IDs may be set on a sequence-by-sequence basis (181 in FIG. 14 ) or on a sequence-by-sequence basis for each channel (182 in FIG. 14 ) for each channel, and encoding / decoding may also be performed.

[0225] In the first to fourth embodiments, arithmetic coding is used to encode / decode each syntax element, but this is not limited to this, and other coding methods such as entropy coding, variable length coding, fixed length coding, etc. may also be used.

[0226] In all of the above-described embodiments, the bitstream output by the feature map encoder has a specific data format that can be decoded according to the encoding method used in the embodiment, and the feature map decoder corresponding to the feature map encoder can decode the bitstream in this specific data format.

[0227] When a wired or wireless network is used to exchange bitstreams between the feature map encoder and the feature map decoder, the bitstreams may be converted into a data format suitable for the transmission mode of the communication channel before transmission. In this case, a transmitter is provided that converts the bitstream output by the feature map encoder into coded data in a data format suitable for the transmission mode of the communication channel and transmits the coded data to the network, and a receiver is provided that receives the coded data from the network, restores the coded data to a bitstream, and supplies the bitstream to the feature map decoder. The transmitter includes a memory that buffers the bitstream output by the feature map encoder, a packet processor that packetizes the bitstream, and a transmitter that transmits the packetized coded data via the network. The receiver includes a receiver that receives the packetized coded data via the network, a memory that buffers the received coded data, and a packet processor that packetizes the coded data to generate a bitstream and provides it to the feature map decoder.

[0228] The above encoding and decoding processes may be realized not only as a transmission, storage, and receiving device using hardware, but also as firmware stored in a ROM (read-only memory), flash memory, etc., or as software for a computer, etc. The firmware program or software program may be provided by recording it on a computer-readable recording medium, or may be provided from a server via a wired or wireless network, or may be provided as data broadcasting on terrestrial or satellite digital broadcasting.

[0229] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the components and treatment processes, and that such modifications are also within the scope of the present invention.

[0230] The present invention can be used in encoding and decoding techniques for feature maps.

[0231] 100 Feature map encoding device, 101 Neural network feature extraction unit, 102 Feature map reduction unit, 103 Feature map conversion unit, 104 Feature map internal encoding unit, 200 Feature map decoding device, 201 Feature map internal decoding unit, 202 Feature map inverse conversion unit, 203 Feature map restoration unit, 204 Neural network identification unit, 301 Convolution processing unit, 302 Activation processing unit, 303 Pooling processing unit, 322 Bottom-up processing unit, 324 Top-down processing unit, 326 Image to be subjected to feature extraction processing, 501 First feature map reduction unit, 502 First channel combination unit, 503 Second feature map reduction unit, 504 Second channel combination unit, 505 Third feature map reduction unit, 506 First padding unit, 507 Second padding unit, 508 Third padding unit, 509 Channel management unit, 601 8x enlargement unit, 602 4x enlargement unit, 603 2x enlargement unit, 604 First feature map mixing unit, 605 Second feature map mixing unit, 606 First depadding unit, 607 Second depadding unit, 608 Third depadding unit, 701 Packing unit, 702 Feature map quantization unit, 703 Feature map coded information coding unit, 801 Feature map coded information decoding unit, 802 Feature map dequantization unit, 803 Unpacking unit, 901 Switch, 902 VVC encoding unit, 903 HEVC encoding unit, 904 AV1 encoding unit, 905 Multiplexing unit, 1005 Demultiplexing unit, 1001 Switch, 1002 VVC decoding unit, 1003 HEVC decoding unit, 1004 AV1 decoding unit.

Claims

1. A feature map encoding device for encoding multi-scale feature maps, comprising: a channel management unit that sets channel IDs that identify channels in feature maps of multiple channels to be encoded; a packing unit that packs the feature maps of the multiple channels to generate a single packed feature frame; a feature map encoded information encoding unit that encodes the feature map encoded information to obtain a bitstream of feature map encoded information; and a feature map internal encoding unit that encodes the packed feature frame to generate a bitstream, wherein the feature map encoded information includes information indicating the number of channels and the channel IDs that identify the channels.

2. A feature map encoding method for encoding multi-scale feature maps, comprising: a channel management step of setting channel IDs that identify channels in feature maps of multiple channels to be encoded; a packing step of packing the feature maps of the multiple channels to generate one packed feature frame; a feature map encoding information encoding step of encoding the feature map encoding information to obtain a bitstream of feature map encoding information; and a feature map inner encoding step of encoding the packed feature frame to generate a bitstream, wherein the feature map encoding information includes information indicating the number of channels and channel IDs that identify the channels.

3. A feature map encoding program for encoding multi-scale feature maps, the program causing a computer to execute the following steps: a channel management step of setting channel IDs that identify channels in feature maps of multiple channels to be encoded; a packing step of packing the feature maps of the multiple channels to generate one packed feature frame; a feature map encoded information encoding step of encoding the feature map encoded information to obtain a bit stream of feature map encoded information; and a feature map internal encoding step of encoding the packed feature frame to generate a bit stream, wherein the feature map encoded information includes information indicating the number of channels and channel IDs that identify the channels.

4. A feature map decoding device that decodes a bitstream in which a multi-scale feature map is encoded, comprising: a feature map internal decoding unit that decodes the bitstream and generates a packed feature frame; and an unpacking unit that unpacks the packed feature frame and generates a multi-channel decimal feature map, wherein the feature map encoding information includes information indicating the number of channels and a channel ID that identifies the channel.

5. A feature map decoding method for decoding a bitstream in which a multi-scale feature map is encoded, comprising: a feature map inner decoding step for decoding the bitstream and generating a packed feature frame; and an unpacking step for unpacking the packed feature frame to generate a multi-channel fractional feature map, wherein the feature map encoding information includes information indicating the number of channels and a channel ID for identifying the channel.

6. A feature map decoding method for decoding a bitstream in which a multi-scale feature map is encoded, comprising: a feature map internal decoding step of decoding the bitstream and generating a packed feature frame; and an unpacking step of unpacking the packed feature frame to generate a multi-channel decimal feature map, wherein the feature map encoding information includes information indicating the number of channels and a channel ID that identifies the channel.

Images