Feature map encoding device, feature map encoding method, feature map decoding device, and feature map decoding method

Abstract

To provide a method and apparatus for efficiently encoding and decoding a feature map with a small amount of processing.SOLUTION: A feature map encoding apparatus 100 for encoding a feature map generated by a neural network feature extraction unit to generate and output a bitstream includes a feature map reduction unit for transforming a multi-scale feature map to generate a single-scale feature map, a packing unit for packing the single-scale feature map into a frame to generate a decimal-type packed feature frame, a feature map reduction update unit for determining whether to update a filter weight for transforming the multi-scale feature map, a feature map restoration parameter encoding unit for transmitting information indicating whether to update the filter weight in units of layers and further transmitting information indicating the filter weight when the filter weight of the layer is updated, and a feature map internal encoding unit for encoding an integer-type packed feature frame.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] Coding and decoding of feature maps in neural networks. [Background technology]

[0002] A well-known neural network technology used for image recognition, such as detecting objects at various scales in an image, dividing regions into individual objects, or tracking objects, is the FPN (Feature Pyramid Network) described in Non-Patent Document 1. In FPN, multiple feature maps at various scales are generated from the image to be processed, and various image recognitions are performed using the feature maps.

[0003] FPNs used for image recognition generate multiple feature maps from images, and their structure utilizes a CNN (Convolutional Neural Network). CNNs can be divided into a feature extraction section (backbone) that reads an image and generates a feature map using convolution and pooling, and a classification section (head) that is composed of hierarchical fully connected layers and generates output suitable for tasks such as object detection, instance segmentation, and object tracking. FPNs utilize the CNN backbone.

[0004] The feature extraction unit of the FPN is typically configured with a hierarchical structure in which a convolution process 301, an activation process 302, and a pooling process 303 shown in FIG. 3 are configured as one basic unit, and this basic unit is repeated.

[0005] Figure 4 shows the structure of the FPN. The FPN consists of a bottom-up processor 322 that uses a CNN backbone to generate a multi-scale feature map consisting of multiple hierarchical layers, and a top-down processor 324 that aggregates features from deep-layer feature maps to shallow-layer feature maps using the inverse structure of the CNN backbone. The bottom-up processor 322 reduces the resolution of the feature map by half by repeating the basic units of convolution 301, activation 302, and pooling 303 shown in Figure 3, generating a pyramid of feature maps with multiple layers. Meanwhile, the top-down processor 324 adds feature maps with resolutions corresponding to those of the bottom-up processor 322, increasing the resolution of the feature map to the same resolution as the input image, generating a pyramid of feature maps. In other words, the FPN generates multiple feature maps for each layer from the image 326 to be subjected to feature extraction processing.

[0006] The convolution processing unit 301 performs convolution processing on the data to be processed (image or feature map) using multiple types of predetermined filters (kernels). In the convolution processing in the convolution processing unit 301, predetermined filtering is performed on all of the data to be processed while sliding at predetermined intervals. The sliding interval is called the stride. The convolution processing unit 301 may determine the stride based on the number of data to be processed. For example, the convolution processing unit 301 may determine the stride to be 1 if the number of data to be processed is smaller than a predetermined value, and may determine the stride to be 2 if the number is equal to or greater than the predetermined value. Multiple feature maps are generated by preparing multiple predetermined filters at each layer and generating one feature map for each filter. The unit of the feature map is the channel. If the number (types) of predetermined filters is N (N types), N feature maps (N channels) are generated.

[0007] The activation processing unit 302 performs activation processing to nonlinearly convert the feature map output from the convolution processing unit 301. Here, the function used in the activation processing is called an activation function. The activation processing unit 302 uses a ReLU (Rectified Linear Unit) function, a sigmoid function, or the like as the activation function.

[0008] The pooling processing unit 303 is a process for downsampling the feature map output from the activation processing unit 302 by replacing local values ​​of the feature map with representative values.

[0009] On the other hand, when performing classification using a neural network, it has the ability to perform tasks using feature maps of multiple channels at each layer.

[0010] In image classification, the feature maps of multiple channels at each layer are convolved for each predetermined size based on the scale of the feature map, and the probability of the object class is calculated for each pixel. [Prior art documents] [Non-patent literature]

[0011] [Non-Patent Document 1] Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature Pyramid Networks for Object Detection. In CVPR, 2017. Summary of the Invention [Problem to be solved by the invention]

[0012] The amount of information in a feature map is enormous, making it unsuitable for transmission and storage. In view of the above problems, the present invention has an object to provide a technology for encoding and decoding feature maps. [Means for solving the problem]

[0013] In order to solve the above problems, the feature map encoding device of the present application includes a feature map reduction unit that transforms a multi-scale feature map to generate a single-scale feature map; a packing unit that packs the single-scale feature map into a frame to generate a decimal-packed feature frame; a feature map reduction update unit that decides whether to update filter weights for transforming the multi-scale feature map; a feature map restoration parameter encoding unit that transmits information indicating whether to update the filter weights on a layer-by-layer basis and, if the filter weights of the layer are to be updated, further transmits information indicating the filter weights; and a feature map inner encoding unit that encodes the integer-packed feature frame.

[0014] The feature map decoding device of the present application includes a feature map restoration parameter decoding unit 205 that transforms a single-scale feature map, decodes information indicating whether to update filter weights for generating a multi-scale feature map, on a layer-by-layer basis, and, if the filter weight of the layer is to be updated, further decodes information indicating the filter weight; a feature map internal decoding unit that decodes the single-scale feature map that has been packed into a frame and encoded, and generates an integer-type packing feature frame; a dequantization unit that converts elements of the integer-type packing feature frame into decimal values, and generates a fractional-type packing feature frame; an unpacking unit that unpacks the fractional-type packing feature frame, and generates the single-scale feature map; and a feature map restoration unit that transforms the single-scale feature map, and generates the multi-scale feature map. [Effects of the Invention]

[0015] According to the present invention, feature maps can be efficiently coded and decoded with a small amount of processing. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a block diagram illustrating the configuration of a feature map encoding device 100. [Figure 2] FIG. 2 is a block diagram illustrating the configuration of a feature map decoding device 200. [Figure 3] FIG. 1 is a block diagram for explaining the basic unit of processing in each layer of the FPN. [Figure 4] FIG. 1 is a block diagram illustrating the structure of an FPN. [Figure 5] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map reduction unit 102. [Figure 6] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map restoration unit 203. [Figure 7] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map conversion unit 103. [Figure 8] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map inverse conversion unit 202. [Figure 9] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map internal coding unit 104. [Figure 10] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map internal decoding unit 201. [Figure 11] FIG. 10 is a diagram for explaining the number of channels of feature maps x1, x2, and x3, the width of the feature maps, and the height of the feature maps. [Figure 12] FIG. 10 is a diagram illustrating a state in which feature maps of multiple channels are packed into one frame. [Figure 13] FIG. 10 is a diagram illustrating flipping when packing feature maps of multiple channels into one frame. [Figure 14] 1 is a diagram illustrating layers and units handled by a feature map encoding device and a feature map decoding device according to an embodiment of the present invention. FIG. [Figure 15] 10 is a flowchart illustrating the details of the operations of a feature map reduction update unit 105 and a feature map restoration parameter encoding unit 106 according to the first embodiment. [Figure 16]10 is a flowchart for explaining details of the operations of a feature map restoration update unit 206 and a feature map restoration parameter decoding unit 205 according to the first embodiment. [Figure 17] FIG. 3 is a diagram illustrating a filter template according to the first embodiment. [Figure 18] FIG. 3 is a diagram for explaining the syntax structure of feature map restoration parameters in the first embodiment. [Figure 19] 10 is a flowchart illustrating the details of the operations of a feature map reduction update unit 105 and a feature map restoration parameter encoding unit 106 according to the second embodiment. [Figure 20] 10 is a flowchart illustrating the details of the operations of a feature map restoration update unit 206 and a feature map restoration parameter decoding unit 205 according to the second embodiment. [Figure 21] FIG. 10 is a diagram for explaining the syntax structure of feature map restoration parameters in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] The technologies and technical terms used in this embodiment will be defined below.

[0018] <Features and feature maps> In a convolutional neural network (CNN), a filter is used to scan the image (input layer data) from which features are extracted, and the results of the convolution operation between the data and the filter coefficients are used as features or a feature map.

[0019] <Packing> Frame packing refers to combining two or more frames (pictures) into a single frame (picture) by arranging them in a tiled pattern. In this application, packing refers to combining feature maps from multiple channels into a single frame. Figure 12 shows an example of frame packing.

[0020] <Data Type> A data type that represents an integer value is called an integer type, and a data type that represents a decimal value is called a decimal type.

[0021] <Layer, Unit> The layers and units handled by the feature map encoding device and feature map decoding device of this embodiment will be described using Figure 14. Reference numeral 181 in Figure 14 denotes a sequence of consecutive feature frames or feature maps of all channels over time, which is referred to as a sequence layer or sequence unit. Reference numeral 182 in Figure 14 denotes a sequence of consecutive feature maps of one channel over time, which is referred to as a sequence layer for each channel or a sequence unit for each channel. Reference numeral 183 in Figure 14 denotes a feature frame or feature maps of all channels at the same time, which is referred to as a frame layer or frame unit. Reference numeral 184 in Figure 14 denotes a feature map of one channel at a certain time, which is referred to as a feature map layer for each channel (a feature map unit for each channel).

[0022] (First embodiment) A feature map encoding device 100 and a feature map decoding device 200 according to a first embodiment of the present invention will be described.

[0023] 1 is a block diagram of a feature map encoding device 100 according to a first embodiment. The feature map encoding device 100 of this embodiment includes a feature map reduction unit 102, a feature map conversion unit 103, a feature map internal encoding unit 104, a feature map reduction / update unit 105, and a feature map restoration parameter encoding unit 106. The feature map encoding device 100 encodes a feature map generated by a neural network feature extraction unit 101 to generate and output a bitstream.

[0024] The neural network feature extraction unit 101 reads an image from which features are to be extracted, generates a feature map by FPN convolution processing, activation processing, and pooling processing, and supplies the generated feature map to the feature map reduction unit 102. In this embodiment, a three-layer multi-scale feature map of x1, x2, and x3 is generated.

[0025] The feature map reduction unit 102 converts the three-layer multi-scale feature maps x1, x2, and x3 obtained from the neural network feature extraction unit 101 into a single-layer single-scale feature map xf using feature map reduction parameters obtained from the feature map reduction update unit 105 (described later), and supplies the map to the feature map conversion unit 103. The feature map reduction unit 102 will be described in detail with reference to FIG. 5.

[0026] The feature map conversion unit 103 performs packing and quantization on the decimal-type single-scale feature map xf supplied from the feature map reduction unit 102, converting it into an integer-type packed feature frame, and supplies it to the feature map internal coding unit 104.

[0027] The feature map conversion unit 103 will be described in detail with reference to FIG.

[0028] The feature map intra-encoding unit 104 encodes the integer-type packing feature frames supplied from the feature map conversion unit 103 in accordance with an image encoding standard such as VVC, HEVC, or AV1 to generate and output a bitstream. The output bitstream is supplied to the feature map decoding device 200 or the like via a network or the like.

[0029] The feature map internal coding unit 104 will be described in detail with reference to FIG.

[0030] The feature map reduction update unit 105 has a function to update the feature map reduction parameters used to convert the input feature map into a feature map in the layer one hierarchically lower. The feature map reduction parameters are updated according to any setting made by the user.

[0031] The feature map restoration parameter encoding unit 106 determines feature map restoration parameters corresponding to the feature map reduction parameters updated by the feature map reduction update unit 105, encodes the determined feature map restoration parameters, and generates and outputs a feature map restoration parameter bit stream. The output bit stream is supplied to the feature map decoding device 200 and the like via a network or the like.

[0032] The details of the operations of the feature map reduction update unit 105 and the feature map restoration parameter encoding unit 106 will be described later with reference to FIG.

[0033] 2 is a block diagram showing the configuration of a feature map decoding device 200 according to an embodiment of the present invention, which corresponds to the feature map encoding device 100 in FIG. 1. The feature map decoding device 200 according to this embodiment includes a feature map internal decoding unit 201, a feature map inverse conversion unit 202, a feature map restoration unit 203, a feature map restoration parameter decoding unit 205, and a feature map restoration update unit 206. The feature map decoding device 200 receives as input a bitstream encoded by the feature map encoding device 100, decodes the bitstream, and generates three-layer multi-scale feature maps x1up, x2up, and x3up, which it supplies to a neural network identification unit 204.

[0034] The feature map internal decoding unit 201 decodes the bitstream coded by the feature map internal coding unit 104 of the feature map coding device 100 in accordance with an image coding standard such as VVC, HEVC, or AV1, generates integer-type packing feature frames, and supplies them to the feature map inverse conversion unit 202.

[0035] The feature internal decoding unit 201 will be described in detail with reference to FIG.

[0036] The feature map inverse conversion unit 202 performs inverse quantization and unpacking on the integer-type packed feature frame supplied from the feature internal decoding unit 201, converting it into a decimal-type single-scale feature map xr, and supplies it to the feature map restoration unit 203.

[0037] The feature map inverse conversion unit 202 will be described in detail with reference to FIG.

[0038] The feature map restoration unit 203 converts the single-scale feature map xr supplied from the feature map inverse conversion unit 202 into three-layer multi-scale feature maps x1up, x2up, and x3up, and supplies them to the neural network identification unit 204 as the output of the feature map decoding device 200.

[0039] The feature map restoration unit 203 will be described in detail with reference to FIG.

[0040] The neural network classification unit 204 performs classification processing such as classification of objects, places, landscapes, people, and living things in the classification target image based on the three-layer multi-scale feature maps x1up, x2up, and x3up supplied by the feature map restoration unit 203.

[0041] The feature map restoration parameter decoding unit 205 decodes the feature map restoration parameter bit stream coded by the feature map restoration parameter coding unit 106 of the feature map coding device 100, generates feature map restoration parameters, and supplies them to the feature map restoration update unit 206.

[0042] The feature map restoration update unit 206 supplies the feature map restoration parameters generated by the feature map restoration parameter decoding unit 205 to the feature map restoration unit 203 .

[0043] The operations of the feature map restoration parameter decoding unit 205 and the feature map restoration update unit 206 will be described in detail later with reference to FIG.

[0044] <Feature map reduction and restoration> The feature map reduction unit 102 has a function of converting the multi-layered multi-scale feature map acquired from the neural network feature extraction unit 101 into a single-layered single-scale feature map.

[0045] The feature map reduction unit 102 will be described in detail using FIG. 5. The feature map reduction unit 102 is composed of a first feature map reduction unit 501, a first channel combination unit 502, a second feature map reduction unit 503, a second channel combination unit 504, a third feature map reduction unit 505, a first padding unit 506, a second padding unit 507, and a third padding unit 508. The feature map reduction unit 102 also acquires feature map reduction parameters from the feature map reduction update unit 105. The feature map reduction parameters consist of three elements: a feature map reduction parameter [0] input to the first feature map reduction unit 501, a feature map reduction parameter [1] input to the second feature map reduction unit 503, and a feature map reduction parameter [2] input to the third feature map reduction unit 505. The feature map reduction unit 102 in FIG. 5 is an example of a configuration that converts a three-layer multi-scale feature map into a single-scale feature map.

[0046] The feature map reduction unit 102 receives three-layer multi-scale feature maps, namely, a first feature map x1, a second feature map x2, and a third feature map x3, as input, converts them into a single-layer single-scale feature map xf, and supplies the result to the feature map conversion unit 103. Here, n denotes an index indicating a layer, Cn denotes the number of channels in the nth layer, Wn denotes the width of the feature map, and Hn denotes the height of the feature map. In this embodiment, the values ​​of Cn, Wn, and Hn for each layer are as shown in FIG. 11, where H and W denote the width and height, respectively, of the image from which features are extracted.

[0047] The first padding unit 506 has a function of padding the first feature map x1 to generate a first padded feature map x1pad. The first padding unit 506 determines the padding size so that the width and height of x1pad are multiples of 64. The number of channels of x1pad is 256, the same as that of x1.

[0048] The second padding unit 507 performs padding on the second feature map x2 by folding to generate a second padded feature map x2pad. The second padding unit 507 determines the padding size so that the width and height of x2pad are multiples of 32. The number of channels of x2pad is 256, the same as that of x2.

[0049] The third padding unit 508 performs padding on the third feature map x3 by folding to generate a third padded feature map x3pad. The third padding unit 508 determines the padding size so that the width and height of x3pad are multiples of 16. The number of channels in x3pad is 256, the same as that of x3.

[0050] In the first padding unit 506, the second padding unit 507, and the third padding unit 508, the left padding size is the same as the right padding size, and the top padding size is the same as the bottom padding size. That is, the feature maps x1, x2, and x3 are positioned at the centers of x1pad, x2pad, and x3pad, respectively.

[0051] The first feature map reducing unit 501 determines a filter to apply to spatial convolution based on the feature map reduction parameter [0] acquired from the feature map reduction update unit 105. Based on the determined filter, the first padded feature map x1pad acquired from the first padding unit 506 is convolved in the spatial and channel directions to generate a first intermediate feature map y1. The number of channels of y1 is 192, its width is Wx1pad / 2, and its height is Hx1pad / 2, where Wx1pad and Hx1pad are the width and height of the first padded feature map x1pad, respectively. If the feature map reduction parameter [0] is not acquired from the feature map reduction update unit 105, that is, if the feature map reduction update unit 105 determines not to update the feature map reduction parameter, a predetermined filter is applied.

[0052] The first channel combining unit 502 has a function of combining the first intermediate feature map y1 obtained from the first feature map reduction unit 501 and the second padded feature map x2pad obtained from the second padding unit 507 in the channel direction to generate an intermediate feature map y1Cx2pad. Since y1 has 192 channels and x2pad has 256 channels, the intermediate feature map y1Cx2pad has 448 channels (192 + 256).

[0053] The second feature map reduction unit 503 determines a filter to be applied to spatial convolution based on the feature map reduction parameter [1] acquired from the feature map reduction update unit 105. Based on the determined filter, the second feature map reduction unit 503 performs spatial and channel convolution on the intermediate feature map y1Cx2pad acquired from the first channel combination unit 502 to generate a second intermediate feature map y2. The number of channels of y2 is 192, its width is Wy1Cx2pad / 2, and its height is Hy1Cx2pad / 2, where Wy1Cx2pad and Hy1Cx2pad are the width and height of the intermediate feature map y1Cx2pad, respectively. If the feature map reduction parameter [1] is not acquired from the feature map reduction update unit 105, that is, if the feature map reduction update unit 105 determines not to update the feature map reduction parameter, a predetermined filter is applied.

[0054] The second channel combining unit 504 has a function of combining the second intermediate feature map y2 obtained from the second feature map reduction unit 503 and the third padded feature map x3pad obtained from the third padding unit 508 in the channel direction to generate an intermediate feature map y2Cx3pad. Since the number of channels in the intermediate feature map y2 is 192 and the number of channels in x3pad is 256, the number of channels in y1Cx2pad is 448 (192 + 256).

[0055] The third feature map reduction unit 505 determines a filter to be applied to spatial convolution based on the feature map reduction parameter [1] acquired from the feature map reduction update unit 105. Based on the determined filter, the third feature map reduction unit 505 performs spatial and channel convolution on the intermediate feature map y2Cx3pad acquired from the second channel combination unit 504 to generate a third intermediate feature map y3. The number of channels of y3 is 192, its width is Wy2Cx3pad / 2, and its height is Hy2Cx3pad / 2, where Wy2Cx3pad and Hy2Cx3pad are the width and height of the intermediate feature map y2Cx3pad, respectively. If the feature map reduction parameter [2] is not acquired from the feature map reduction update unit 105, i.e., if the feature map reduction update unit 105 determines not to update the feature map reduction parameter, a predetermined filter is applied.

[0056] The feature map reduction unit 102 outputs the third intermediate feature map y3 as a single-scale feature map xf and supplies it to the feature map conversion unit 103.

[0057] The feature map restoration unit 203 has a function of converting the single-scale feature map xr acquired from the feature map inverse conversion unit 202 into three-layer multi-scale feature maps x1up, x2up, and x3up.

[0058] The feature map restoration unit 203 will be described in detail using Figure 6. The feature map restoration unit 203 is composed of an 8x enlargement unit 601, a 4x enlargement unit 602, a 2x enlargement unit 603, a first feature map mixer 604, a second feature map mixer 605, a first de-padding unit 606, a second de-padding unit 607, and a third de-padding unit 608. The feature map restoration unit 203 also acquires feature map restoration parameters from the feature map restoration update unit 206. The feature map restoration parameters are composed of three elements: a feature map restoration parameter [0] input to the 8x enlargement unit 601, a feature map restoration parameter [1] input to the 4x enlargement unit 602, and a feature map restoration parameter [2] input to the 2x enlargement unit 603.

[0059] The 8x enlargement unit 601 determines a filter to apply to the spatial transpose convolution based on the feature map restoration parameter [0] obtained from the feature map restoration update unit 206. Based on the determined filter, it performs spatial transpose convolution and channel convolution on the single-scale feature map xr obtained from the feature map inverse transformation unit 202 to enlarge the feature map and reduce the channels, thereby generating an intermediate feature map z1. The number of channels in z1 is 196. If the width and height of the single-scale feature map xr are xrwidth and xrheight, respectively, the width and height of z1 are xrwidth × 8 and xrheight × 8, respectively. Here, rwidth × 8 and xrheight × 8 are the same as the width and height of the first padded feature map x1pad, which is the output of the first padding unit 506 of the feature extraction unit 102. If the feature map restoration parameter [0] is not acquired from the feature map restoration update unit 206, that is, if the feature map restoration parameter decoding unit 205 decodes information indicating that the feature map restoration parameter will not be updated, a predetermined filter is applied.

[0060] The 4x enlargement unit 602 determines a filter to apply to spatial transposition convolution based on the feature map restoration parameter [0] acquired from the feature map restoration update unit 206. Based on the determined filter, it performs spatial transposition convolution and channel convolution on the single-scale feature map xr acquired from the feature map inverse transformation unit 202 to enlarge the features and reduce the channels, thereby generating an intermediate feature map z2. The number of channels in z2 is 196. The width and height of z2 are xrwidth×4 and xrheight×4, respectively. Here, rwidth×4 and xrheight×4 are the same as the width and height of the second padded feature map x2pad, which is the output of the second padding unit 507 of the feature extraction unit 102. If the feature map restoration parameter [1] is not acquired from the feature map restoration update unit 206, i.e., if the feature map restoration parameter decoding unit 205 decodes information indicating that the feature map restoration parameters will not be updated, a predetermined filter is applied.

[0061] The 2x enlargement unit 603 determines a filter to apply to spatial transposition convolution based on the feature map restoration parameter [0] acquired from the feature map restoration update unit 206. Based on the determined filter, it performs spatial transposition convolution and channel convolution on the single-scale feature map xr acquired from the feature map inverse transformation unit 202, thereby enlarging the feature map and reducing the channels, thereby generating an intermediate feature map z3. The number of channels in z3 is 196. The width and height of z3 are xrwidth×2 and xrheight×2, respectively. Here, rwidth×2 and xrheight×2 are the same as the width and height of the third padded feature map x3pad, which is the output of the third padding unit 508 of the feature extraction unit 102. If the feature map restoration parameter [2] is not acquired from the feature map restoration update unit 206, i.e., if the feature map restoration parameter decoding unit 205 decodes information indicating that the feature map restoration parameters will not be updated, a predetermined filter is applied.

[0062] The first feature map mixing unit 604 has a function of generating an intermediate feature map z2up by improving the quality of the intermediate feature map z2 obtained from the 4x enlargement unit 602, using the intermediate feature map z1 obtained from the 8x enlargement unit 601.

[0063] The second feature map mixing unit 605 has a function of generating an intermediate feature map z3up that improves the quality of the intermediate feature map z3 obtained from the 2x enlargement unit 603, by using the intermediate feature map z2up obtained from the first feature map mixing unit 604.

[0064] The padding removal unit 606 removes padding from the intermediate feature map z1 acquired from the 8x enlargement unit 601 to generate a first output feature map x1up. The width and height of x1up are the same as the width and height of the first feature map x1 input to the feature extraction unit 102.

[0065] The padding removal unit 607 has a function of removing padding from the intermediate feature map z2up acquired from the first feature map blending unit 604 to generate a second output feature map x2up. The width and height of x2up are the same as the width and height of the second feature map x2 input to the feature extraction unit 102.

[0066] The padding removal unit 608 has a function of removing padding from the intermediate feature map z3up acquired from the second feature map blending unit 605 to generate a third output feature map x3up. The width and height of x3up are the same as the width and height of the third feature map x3 input to the feature extraction unit 102.

[0067] In the first padding removal unit 606, the second padding removal unit 607, and the third padding removal unit 608, the left padding size is the same as the right padding size, and the top padding size is the same as the bottom padding size, similar to the first padding unit 506, the second padding unit 507, and the third padding unit 508 of the feature extraction unit 102. In other words, the padding on the top, bottom, left, and right sides is removed by assuming that each output feature map x1up, x2up, and x3up is placed at the center of each intermediate feature map z1, z2up, and z3up, respectively.

[0068] <Updating feature map reduction and restoration parameters> FIG. 18 shows an example of the syntax structure of feature map restoration parameters in this embodiment. num_of_layers represents the number of layers involved in feature map restoration. In this embodiment, feature map restoration is performed by the 8x enlargement unit 601, 4x enlargement unit 602, and 2x enlargement unit 603, so num_of_layers=3 is set. overwrites_weights is a flag specifying whether to update the filter weights of the target layer i (=1, 2, 3). overwrites_weights[i]=0 indicates that the filter weights of the layer are not updated. overwrites_weights[i]=1 indicates that the filter weights of the layer are updated, followed by weight_type. weight_type is an index specifying which template from multiple predetermined filter weight templates is to be applied to the layer. FIG. 17 shows an example in which four types of templates are specified. 1100 corresponds to weight_type=0, 1101 corresponds to weight_type=1, 1102 corresponds to weight_type=2, and 1103 corresponds to weight_type=3. 1104, 1106, 1108, and 1110 are filter weights applied in convolution by the first feature map reduction unit 501, the second feature map reduction unit 503, and the third feature map reduction unit 505 of the feature map reduction unit 102. 1105, 1107, 1109, and 1111 are filter weights applied in transposed convolution by the 8x enlargement unit 601, the 4x enlargement unit 602, and the 2x enlargement unit 603 of the feature map restoration unit 203.

[0069] 18 is transmitted in frame units in the feature map internal decoding unit 201, making it possible to determine whether or not to update the filter for each frame. Also, this syntax structure is transmitted every several frames, for example, every intra-frame, in the feature map internal decoding unit 201, making it possible to determine whether or not to update the filter once every several frames.

[0070] The operations of the feature map reduction / update unit 105 and the feature map restoration parameter encoding unit 106 will be described in detail below with reference to the flowchart of FIG.

[0071] The feature map reduction / update unit 105 executes S102 and subsequent steps for all layers in order, and when all processing is completed, proceeds to step S106 (step S101).

[0072] The feature map reduction / update unit 105 determines whether to update the filter weights of the current layer based on the user's input (step S102).

[0073] If the filter weights are not to be updated, the feature map reduction / update unit 105 sets overwrites_weights[i]=0 (i is an index indicating the current layer) and returns to step S101. If the filter weights are to be updated, the feature map reduction / update unit 105 sets overwrites_weights[i]=1 (i is an index indicating the current layer) and proceeds to step S104 (step S103).

[0074] The feature map reduction / update unit 105 determines weights_type[i][j] (i is an index indicating the current layer, and j is an index specifying the convolution filter) for all convolution filters in the current layer (steps S104 and S105).

[0075] The feature map restoration parameter encoding unit 106 transmits the feature map restoration parameters in accordance with the syntax structure of FIG. 18 (step S106).

[0076] The operations of the feature map restoration parameter decoding unit 205 and the feature map restoration update unit 206 will be described with reference to the flowchart in FIG.

[0077] The feature map restoration / update unit 206 executes S202 and subsequent steps for all layers in order, and ends the process when all the processes are completed (S201).

[0078] The feature map restoration parameter decoding unit 205 decodes overwrites_weights from the bitstream and sends it to the feature map restoration update unit 206 (S202).

[0079] The feature map restoration / update unit 206 determines overwrites_weights. If overwrites_weights=1 (Yes in step S203), the process proceeds to step S204. If overwrites_weights=0 (No in step S203), the process ends for the layer and returns to step S201 (step S203).

[0080] The feature map restoration parameter decoding unit 205 sequentially decodes weights_type[i][j] (i is an index indicating the current layer, and j is an index specifying a convolution filter) for all convolution filters in the current layer from the bitstream and sends them to the feature map restoration updating unit 206. The feature map restoration updating unit 206 sends the filter weights specified by weights_type[i][j] to the 8x enlargement unit 601, 4x enlargement unit 602, and 2x enlargement unit 603 of the feature map restoration unit 203, and rewrites the filter weights previously stored in the 8x enlargement unit 601, 4x enlargement unit 602, and 2x enlargement unit 603 (steps S204, S205, S206).

[0081] Incremental learning is known, in which a task is added to a trained CNN model. In general, in a CNN, the earlier layers store general knowledge for the task, while the later layers store knowledge specific to a specific class. Therefore, in incremental learning, the weights of the earlier layers tend to remain fixed, while only the weights of the later layers tend to change. By adopting the configuration of this embodiment, by updating only the filter weights of a specific layer, the filter weights of each layer can be optimized with a minimum amount of coding, thereby enabling efficient reduction and restoration of the feature map. As a result, the amount of coding required for internal coding of the feature map can be reduced.

[0082] Furthermore, since the filter weight can be updated by selecting from a predetermined template, the filter can be updated with a minimum amount of code and processing load.

[0083] <Feature map transformation and feature map inverse transformation> The feature map conversion unit 103 has a function of performing packing and quantization on the multi-channel decimal-type single-scale feature map xf supplied from the feature map reduction unit 102, and converting it into an integer-type packed feature frame to be supplied to the feature map internal coding unit 104.

[0084] The feature map conversion unit 103 on the encoding side will be described in detail with reference to Fig. 7. The feature map conversion unit 103 is made up of a packing unit 701 and a feature map quantization unit 702.

[0085] The packing unit 701 has a function of generating a packed feature frame by combining input feature maps of multiple channels into one frame. Fig. 12 is a diagram explaining the state in which feature maps of multiple channels are packed into one frame. The feature maps of each channel are sequentially arranged in one frame in raster scan order, from left to right and top to bottom.

[0086] Furthermore, the packing unit 701 performs flipping based on the position in the frame where the channel feature map is placed. Whether or not to perform flipping can be selected, and the selection may be transmitted from the encoding side to the decoding side via a bitstream.

[0087] Flipping is performed when packing feature maps of multiple channels into one frame, as shown in FIG. 13. In flipping, the positions of the elements (pixels) of each channel's feature map are flipped horizontally (left and right), vertically (up and down), or horizontally and vertically (up and down, left and right) when packing the feature maps of each channel. The feature maps of four channels, A (upper left), B (upper right), C (lower left), and D (lower right) in FIG. 13, are grouped together as one set. No flipping is performed at position A in FIG. 13. At position B, the feature map is flipped horizontally (left and right). At position C, the feature map is flipped vertically (up and down). At position D, the feature map is flipped horizontally and vertically (up and down, left and right). When the distribution of elements in the feature maps of each channel is similar, flipping based on the position within the frame where the channels are placed reduces the boundaries between the feature maps of each channel, improving coding efficiency.

[0088] The feature map quantization unit 702 has a function of converting elements of a decimal-type packing feature frame (feature map of all channels) into N-bit integer type elements (N=an integer from about 8 to 16) within a predetermined range, and outputting an integer-type packing feature frame. In this embodiment, the elements are converted into 10-bit integer type elements ranging from 0 to 1023. The feature map quantization unit 702 detects the minimum and maximum values ​​of the elements of the decimal-type packing feature frame, and transmits the detected minimum and maximum values ​​of the elements of the packing feature frame (feature map of all channels) to the decoding side as metadata. To convert from a decimal-type packing feature frame (feature map) to an integer-type packing feature frame (feature map), a linear conversion is performed in which the minimum value of the decimal type is equivalent to the minimum value of the integer type, and the maximum value of the decimal type is equivalent to the maximum value of the integer type. For example, when the integer type range is expressed in 10 bits, the minimum value of the elements of the integer-type packing feature frame (feature map) is 0 and the maximum value is 1023 (2 10 -1). Values ​​between the minimum and maximum values ​​are linearly quantized.

[0089] Next, the feature map inverse transform unit 202 has a function of performing inverse quantization and unpacking on the integer-type packing feature frame decoded by VVC, HEVC, AV1, etc., supplied from the feature map internal decoding unit 201, and performing inverse transform into a decimal-type single-scale feature map xr to be supplied to the feature map restoration unit 203.

[0090] The feature map inverse transformation unit 202 on the decoding side will be described in detail with reference to Fig. 8. The feature map inverse transformation unit 202 performs the inverse process of the feature map transformation unit 103, and is made up of a feature map inverse quantization unit 801 and an unpacking unit 802.

[0091] The feature map inverse quantization unit 801 performs the inverse process of the feature map quantization unit 702 on the encoding side, and has a function of converting elements of integer-type packing feature frames from integer type to decimal type. The feature map inverse quantization unit 801 converts the integer-type packing feature frame decoded by the feature map internal decoding unit 201 into a decimal-type packing feature frame using the minimum and maximum decimal values ​​transmitted as metadata. A linear conversion is performed to convert the integer minimum value to the decimal minimum value and the integer maximum value to the decimal maximum value. Linear inverse quantization is performed on values ​​between the minimum and maximum values.

[0092] The unpacking unit 802 extracts a feature map for each channel from the packing feature frames arranged in one frame in raster scan order and supplies it to the feature map restoration unit 203 as a single-scale feature map xr.

[0093] <Feature map inner encoding and decoding> The feature map internal coding unit 104 will be described in detail using Figure 9. The feature map internal coding unit 104 is made up of a switch 901, a VVC encoding unit 902, an HEVC encoding unit 903, and an AV1 encoding unit 904. The switch 901 selects the encoding standard for internally coding the feature map converted by the feature map conversion unit 103. The VVC encoding unit 902 encodes the feature map in the VVC standard and outputs a bitstream that complies with the VVC standard. The HEVC encoding unit 903 encodes the feature map in the HEVC standard and outputs a bitstream that complies with the HEVC standard. The AV1 encoding unit 904 encodes the feature map in the AV1 standard and outputs a bitstream that complies with the AV1 standard.

[0094] In the VVC, HEVC, and AV1 standards, images are divided into blocks of a specified size and then encoded.

[0095] It is also possible to implement only one of VVC, HEVC, and AV1, and it is also possible to use image coding methods other than VVC, HEVC, and AV1.

[0096] Next, the feature map internal decoding unit 201 will be described in detail with reference to Fig. 10. The feature map internal decoding unit 201 is made up of a switch 1001, a VVC decoding unit 1002, an HEVC decoding unit 1003, and an AV1 decoding unit 1004. The switch 1001 selects the encoding standard for internal decoding based on information for selecting internal decoding in the input bitstream. The VVC decoding unit 1002 decodes the feature map in accordance with the VVC standard. The HEVC decoding unit 1003 decodes the feature map in accordance with the HEVC standard. The AV1 encoding unit 1004 decodes the feature map in accordance with the AV1 standard.

[0097] In the VVC, HEVC, and AV1 standards, decoding is performed for each predetermined block size.

[0098] It is also possible to implement only one of VVC, HEVC, and AV1, and it is also possible to use image coding methods other than VVC, HEVC, and AV1.

[0099] (Second embodiment) The first embodiment is configured to select filter weights to be updated from a predetermined template using weights_type.

[0100] The second embodiment differs from the first embodiment in that the filter weights to be updated are transmitted.

[0101] 21 shows an example of the syntax structure of feature map restoration parameters in this embodiment. num_of_layers, overwrites_weights, and num_of_weights are the same as in the first embodiment. weights_width[i] and weights_height[i] are elements that specify the horizontal and vertical sizes of the filter in layer i (=1, 2, 3), respectively. weight[i][j] is an element that specifies the weight of the j-th filter in layer i.

[0102] 18, the syntax structure in Fig. 21 is transmitted in frame units in the feature map internal decoding unit 201, making it possible to determine whether or not to update the filter for each frame. Also, this syntax structure is transmitted every several frames, for example, every intra-frame, in the feature map internal decoding unit 201, making it possible to determine whether or not to update the filter once every several frames.

[0103] The operations of the feature map reduction / update unit 105 and the feature map restoration parameter encoding unit 106 will be described in detail below with reference to the flowchart of FIG.

[0104] The feature map reduction / update unit 105 executes S302 and subsequent steps for all layers in order, and when all processing is completed, proceeds to step S306 (step S301).

[0105] The feature map reduction / update unit 105 determines whether to update the filter weights of the current layer based on the user's input (step S302).

[0106] If the filter weights are not to be updated, the feature map reduction / update unit 105 sets overwrites_weights[i]=0 (i is an index indicating the current layer) and returns to step S301. If the filter weights are to be updated, the feature map reduction / update unit 105 sets overwrites_weights[i]=1 (i is an index indicating the current layer) and proceeds to step S304 (step S303).

[0107] The feature map reduction / update unit 105 determines filter weights weight[i][j] (i is an index indicating the current layer, and j is an index specifying a convolution filter) for all convolution filters in the current layer (steps S304 and S305).

[0108] The feature map restoration parameter encoding unit 106 transmits the feature map restoration parameters in accordance with the syntax structure of FIG. 21 (step S306).

[0109] The operations of the feature map restoration parameter decoding unit 205 and the feature map restoration update unit 206 will be described using the flowchart in FIG.

[0110] The feature map restoration / update unit 206 executes S402 and subsequent steps for all layers in order, and ends the process when all the processes are completed (S401).

[0111] The feature map restoration parameter decoding unit 205 decodes overwrites_weights from the bitstream and sends it to the feature map restoration update unit 206 (S402).

[0112] The feature map restoration / update unit 206 determines overwrites_weights. If overwrites_weights=1 (Yes in step S403), the process proceeds to step S404. If overwrites_weights=0 (No in step S403), the process ends for the layer and returns to step S401 (step S403).

[0113] The feature map restoration parameter decoding unit 205 sequentially decodes filter weights weights[i][j] (i is an index indicating the current layer, and j is an index specifying a convolution filter) for all convolution filters in the current layer from the bitstream and sends them to the feature map restoration updating unit 206. The feature map restoration updating unit 206 sends the weights to the 8x enlargement unit 601, 4x enlargement unit 602, and 2x enlargement unit 603 of the feature map restoration unit 203, and rewrites the filter weights previously stored in the 8x enlargement unit 601, 4x enlargement unit 602, and 2x enlargement unit 603 (steps S404, S405, S406).

[0114] This embodiment can update only the filter weights of a specific layer, and therefore has the same effect as the first embodiment in that the amount of coding can be reduced.

[0115] Furthermore, this embodiment can directly transmit the values ​​of filter weights to be updated. Since it is possible to transmit an update filter that is optimal for the feature map to be coded, it is possible to reduce and restore the feature map more efficiently than in the first embodiment, reduce the amount of code in the feature map internal coding, and improve coding efficiency.

[0116] In all of the above-described embodiments, the bitstream output by the feature map encoder has a specific data format that can be decoded according to the encoding method used in the embodiment, and the feature map decoder corresponding to the feature map encoder can decode the bitstream in this specific data format.

[0117] When a wired or wireless network is used to exchange bitstreams between the feature map encoder and the feature map decoder, the bitstreams may be converted into a data format suitable for the transmission mode of the communication channel before transmission. In this case, a transmitter is provided that converts the bitstream output by the feature map encoder into coded data in a data format suitable for the transmission mode of the communication channel and transmits the coded data to the network, and a receiver is provided that receives the coded data from the network, restores the coded data to a bitstream, and supplies the bitstream to the feature map decoder. The transmitter includes a memory that buffers the bitstream output by the feature map encoder, a packet processor that packetizes the bitstream, and a transmitter that transmits the packetized coded data via the network. The receiver includes a receiver that receives the packetized coded data via the network, a memory that buffers the received coded data, and a packet processor that packetizes the coded data to generate a bitstream and provides it to the feature map decoder.

[0118] The above encoding and decoding processes may be realized not only as a transmission, storage, and receiving device using hardware, but also as firmware stored in a ROM (read-only memory) or flash memory, or as software for a computer, etc. The firmware program or software program may be provided by recording it on a computer-readable recording medium, or may be provided from a server via a wired or wireless network, or may be provided as data broadcasting on terrestrial or satellite digital broadcasting.

[0119] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the components and processing steps, and that such modifications are also within the scope of the present invention. [Explanation of symbols]

[0120] 100 Feature map encoding device, 101 Neural network feature extraction unit, 102 Feature map reduction unit, 103 Feature map conversion unit, 104 Feature map internal encoding unit, 105 Feature map reduction and update unit, 106 Feature map restoration parameter encoding unit, 200 Feature map decoding device, 201 Feature map internal decoding unit, 202 Feature map inverse conversion unit, 203 Feature map restoration unit, 204 Neural network identification unit, 205 Feature map restoration parameter decoding unit, 206 Feature map restoration and update unit, 301 Convolution processing unit, 302 Activation processing unit, 303 Pooling processing unit, 322 Bottom-up processing unit, 324 Top-down processing unit, 326 Image to be subjected to feature extraction processing, 501 First feature map reduction unit, 502 First channel combination unit, 503 Second feature map reduction unit, 504 Second channel combination unit, 505 Third feature map reduction unit, 506, first padding unit, 507, second padding unit, 508, third padding unit, 601, 8x enlargement unit, 602, 4x enlargement unit, 603, 2x enlargement unit, 604, first feature map mixing unit, 605, second feature map mixing unit, 606, first unpadding unit, 607, second unpadding unit, 608, third unpadding unit, 701, packing unit, 702, feature map quantization unit, 801, feature map inverse quantization unit, 802, unpacking unit, 901, switch, 902, VVC encoding unit, 903, HEVC encoding unit, 904, AV1 encoding unit, 1001, switch, 1002, VVC decoding unit, 1003, HEVC decoding unit, 1004, AV1 decoding unit.

Claims

1. a feature map reducer that converts the multi-scale feature maps to generate single-scale feature maps; a packing unit that packs the single-scale feature map into a frame to generate a fractional packed feature frame; a feature map reduction update unit that determines whether to update filter weights for transforming the multi-scale feature maps; a feature map restoration parameter encoding unit that transmits, on a layer-by-layer basis, information indicating whether to update the filter weights, and, if the filter weights of the layer are to be updated, further transmits information indicating the filter weights; a feature map inner coding unit that codes the integer-type packing feature frame; Equipped with A feature map encoding device characterized by:

2. a feature map restoration parameter decoding unit 205 that converts the single-scale feature map, decodes information indicating whether to update filter weights for generating a multi-scale feature map, on a layer-by-layer basis, and further decodes information indicating the filter weights if the filter weights of the layer are to be updated; a feature map internal decoder that decodes the single-scale feature map packed and encoded into a frame to generate an integer-type packed feature frame; an inverse quantization unit that converts elements of the integer-type packing feature frame into decimal values ​​to generate a decimal-type packing feature frame; an unpacking unit that unpacks the fractional packed feature frame to generate the single-scale feature map; a feature map restoration unit that transforms the single-scale feature map to generate the multi-scale feature map; Equipped with A feature map decoding device characterized by:

Images