Feature map encoding device, feature map encoding method, feature map decoding device, and feature map decoding method

Abstract

To provide a device for encoding / decoding a feature map in a neural network.SOLUTION: A feature map encoding device 100 comprises: a feature map reduction section 102 which converts a multi-scale feature map and generates a single-scale feature map; a feature map conversion section 103 including a packing section which packs the single-scale feature map into a frame and generates a decimal type packing feature frame and a quantization section which converts elements of the decimal type packing feature frame into an integral value and generates an integral type packing feature frame; and a feature map internal encoding section 104 which encodes the integral type packing feature frame. In a case where the number of channels of the single-scale feature map is smaller than a predetermined minimum channel number, the packing section adds a dummy feature map to the single-scale feature map and packs it into a frame.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] Coding and decoding of feature maps in neural networks. [Background technology]

[0002] A well-known neural network technology used for image recognition, such as detecting objects at various scales in an image, dividing regions into individual objects, or tracking objects, is the FPN (Feature Pyramid Network) described in Non-Patent Document 1. In FPN, multiple feature maps at various scales are generated from the image to be processed, and various image recognitions are performed using the feature maps.

[0003] FPNs used for image recognition generate multiple feature maps from images, and their structure utilizes a CNN (Convolutional Neural Network). CNNs can be divided into a feature extraction section (backbone) that reads an image and generates a feature map using convolution and pooling, and a classification section (head) that is composed of hierarchical fully connected layers and generates output suitable for tasks such as object detection, instance segmentation, and object tracking. FPNs utilize the CNN backbone.

[0004] The feature extraction unit of the FPN is typically configured with a hierarchical structure in which a convolution process 301, an activation process 302, and a pooling process 303 shown in FIG. 3 are configured as one basic unit, and this basic unit is repeated.

[0005] Figure 4 shows the structure of the FPN. The FPN consists of a bottom-up processor 322 that uses a CNN backbone to generate a multi-scale feature map consisting of multiple hierarchical layers, and a top-down processor 324 that aggregates features from deep-layer feature maps to shallow-layer feature maps using the inverse structure of the CNN backbone. The bottom-up processor 322 reduces the resolution of the feature map by half by repeating the basic units of convolution 301, activation 302, and pooling 303 shown in Figure 3, generating a pyramid of feature maps with multiple layers. Meanwhile, the top-down processor 324 adds feature maps with resolutions corresponding to those of the bottom-up processor 322, increasing the resolution of the feature map to the same resolution as the input image, generating a pyramid of feature maps. In other words, the FPN generates multiple feature maps for each layer from the image 326 to be subjected to feature extraction processing.

[0006] The convolution processing unit 301 performs convolution processing on the data to be processed (image or feature map) using multiple types of predetermined filters (kernels). In the convolution processing in the convolution processing unit 301, predetermined filtering is performed on all of the data to be processed while sliding at predetermined intervals. The sliding interval is called the stride. The convolution processing unit 301 may determine the stride based on the number of data to be processed. For example, the convolution processing unit 301 may determine the stride to be 1 if the number of data to be processed is smaller than a predetermined value, and may determine the stride to be 2 if the number is equal to or greater than the predetermined value. Multiple feature maps are generated by preparing multiple predetermined filters at each layer and generating one feature map for each filter. The unit of the feature map is the channel. If the number (types) of predetermined filters is N (N types), N feature maps (N channels) are generated.

[0007] The activation processing unit 302 performs activation processing to nonlinearly convert the feature map output from the convolution processing unit 301. Here, the function used in the activation processing is called an activation function. The activation processing unit 302 uses a ReLU (Rectified Linear Unit) function, a sigmoid function, or the like as the activation function.

[0008] The pooling processing unit 303 is a process for downsampling the feature map output from the activation processing unit 302 by replacing local values ​​of the feature map with representative values.

[0009] On the other hand, when performing classification using a neural network, it has the ability to perform tasks using feature maps of multiple channels at each layer.

[0010] In image classification, the feature maps of multiple channels at each layer are convolved for each predetermined size based on the scale of the feature map, and the probability of the object class is calculated for each pixel. [Prior art documents] [Non-patent literature]

[0011] [Non-Patent Document 1] Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature Pyramid Networks for Object Detection. In CVPR, 2017. Summary of the Invention [Problem to be solved by the invention]

[0012] The amount of information in a feature map is enormous, making it unsuitable for transmission and storage. In view of the above problems, the present invention has an object to provide a technology for encoding and decoding feature maps. [Means for solving the problem]

[0013] In order to solve the above problems, the feature map encoding device of the present application includes a feature map reduction unit that converts a multi-scale feature map to generate a single-scale feature map; a packing unit that packs the single-scale feature map into a frame to generate a decimal-type packed feature frame; a quantization unit that converts elements of the decimal-type packed feature frame into integer values ​​to generate an integer-type packed feature frame; and a feature map inner encoding unit that encodes the integer-type packed feature frame, wherein the packing unit adds a dummy feature map to the single-scale feature map and packs it into the frame if the number of channels of the single-scale feature map is smaller than a predetermined minimum number of channels. [Effects of the Invention]

[0014] According to the present invention, feature maps can be efficiently coded and decoded with a small amount of processing. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a block diagram illustrating the configuration of a feature map encoding device 100. [Figure 2] FIG. 2 is a block diagram illustrating the configuration of a feature map decoding device 200. [Figure 3] FIG. 1 is a block diagram for explaining the basic unit of processing in each layer of the FPN. [Figure 4] FIG. 1 is a block diagram illustrating the structure of an FPN. [Figure 5] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map reduction unit 102. [Figure 6] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map restoration unit 203. [Figure 7] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map conversion unit 103. [Figure 8]FIG. 2 is a block diagram for explaining the detailed configuration of a feature map inverse conversion unit 202. [Figure 9] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map internal coding unit 104. [Figure 10] FIG. 2 is a block diagram for explaining the detailed configuration of a feature map internal decoding unit 201. [Figure 11] FIG. 10 is a diagram for explaining the number of channels of feature maps x1, x2, and x3, the width of the feature maps, and the height of the feature maps. [Figure 12] FIG. 10 is a diagram illustrating a state in which feature maps of multiple channels are packed into one frame. [Figure 13] FIG. 10 is a diagram illustrating flipping when packing feature maps of multiple channels into one frame. [Figure 14] 1 is a diagram illustrating layers and units handled by a feature map encoding device and a feature map decoding device according to an embodiment of the present invention. FIG. [Figure 15] 10 is a flowchart illustrating the operation of a packing unit 701. [Figure 16] 10 is a flowchart illustrating the processing for setting the number of encoding channels in a packing unit 701. [Figure 17] 10 is a flowchart for explaining packing processing in a packing unit 701. [Figure 18] 10 is a flowchart illustrating the operation of an unpacking unit 802. [Figure 19] FIG. 10 is a diagram illustrating metadata that sets the minimum number of channels. [Figure 20] FIG. 10 is a diagram illustrating packed feature frames according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0016] The technologies and technical terms used in this embodiment will be defined below.

[0017] <Features and feature maps> In a convolutional neural network (CNN), a filter is used to scan the image (input layer data) from which features are extracted, and the results of the convolution operation between the data and the filter coefficients are used as features or a feature map.

[0018] <Packing> Frame packing refers to combining two or more frames (pictures) into a single frame (picture) by arranging them in a tiled pattern. In this application, packing refers to combining feature maps from multiple channels into a single frame. Figure 12 shows an example of frame packing.

[0019] <Data Type> A data type that represents an integer value is called an integer type, and a data type that represents a decimal value is called a decimal type.

[0020] <Layer, Unit> The layers and units handled by the feature map encoding device and feature map decoding device of this embodiment will be described using Figure 14. Reference numeral 181 in Figure 14 denotes a sequence of consecutive feature frames or feature maps of all channels over time, which is referred to as a sequence layer or sequence unit. Reference numeral 182 in Figure 14 denotes a sequence of consecutive feature maps of one channel over time, which is referred to as a sequence layer for each channel or a sequence unit for each channel. Reference numeral 183 in Figure 14 denotes a feature frame or feature maps of all channels at the same time, which is referred to as a frame layer or frame unit. Reference numeral 184 in Figure 14 denotes a feature map of one channel at a certain time, which is referred to as a feature map layer for each channel (a feature map unit for each channel).

[0021] (First embodiment) A feature map encoding device 100 and a feature map decoding device 200 according to a first embodiment of the present invention will be described.

[0022] 1 is a block diagram of a feature map encoding device 100 according to a first embodiment. The feature map encoding device 100 according to the embodiment includes a feature map reduction unit 102, a feature map conversion unit 103, and a feature map internal encoding unit 104. The feature map encoding device 100 encodes a feature map generated by a neural network feature extraction unit 101 to generate and output a bitstream.

[0023] The neural network feature extraction unit 101 reads an image from which features are to be extracted, generates a feature map by FPN convolution processing, activation processing, and pooling processing, and supplies the generated feature map to the feature map reduction unit 102. In this embodiment, a three-layer multi-scale feature map of x1, x2, and x3 is generated.

[0024] The feature map reduction unit 102 converts the three-layer multi-scale feature maps x1, x2, and x3 acquired from the neural network feature extraction unit 101 into a single-layer single-scale feature map xf, and supplies it to the feature map conversion unit 103. The feature map reduction unit 102 will be described in detail with reference to FIG. 5.

[0025] The feature map conversion unit 103 performs packing and quantization on the decimal-type single-scale feature map xf supplied from the feature map reduction unit 102, converting it into an integer-type packed feature frame, and supplies it to the feature map internal coding unit 104.

[0026] The feature map conversion unit 103 will be described in detail with reference to FIG.

[0027] The feature map intra-encoding unit 104 encodes the integer-type packing feature frames supplied from the feature map conversion unit 103 in accordance with an image encoding standard such as VVC, HEVC, or AV1 to generate and output a bitstream. The output bitstream is supplied to the feature map decoding device 200 or the like via a network or the like.

[0028] The feature map internal coding unit 104 will be described in detail with reference to FIG.

[0029] 2 is a block diagram showing the configuration of a feature map decoding device 200 according to an embodiment of the present invention, which corresponds to the feature map encoding device 100 in FIG. 1. The feature map decoding device 200 according to this embodiment includes a feature map internal decoding unit 201, a feature map inverse conversion unit 202, and a feature map restoration unit 203. The feature map decoding device 200 receives as input a bit stream encoded by the feature map encoding device 100, decodes the bit stream, and generates three-layer multi-scale feature maps x1up, x2up, and x3up, which it supplies to a neural network classification unit 204.

[0030] The feature map internal decoding unit 201 decodes the bitstream coded by the feature map internal coding unit 104 of the feature map coding device 100 in accordance with an image coding standard such as VVC, HEVC, or AV1, generates integer-type packing feature frames, and supplies them to the feature map inverse conversion unit 202.

[0031] The feature internal decoding unit 201 will be described in detail with reference to FIG.

[0032] The feature map inverse conversion unit 202 performs inverse quantization and unpacking on the integer-type packed feature frame supplied from the feature internal decoding unit 201, converting it into a decimal-type single-scale feature map xr, and supplies it to the feature map restoration unit 203.

[0033] The feature map inverse conversion unit 202 will be described in detail with reference to FIG.

[0034] The feature map restoration unit 203 converts the single-scale feature map xr supplied from the feature map inverse conversion unit 202 into three-layer multi-scale feature maps x1up, x2up, and x3up, and supplies them to the neural network identification unit 204 as the output of the feature map decoding device 200.

[0035] The feature map restoration unit 203 will be described in detail with reference to FIG.

[0036] The neural network classification unit 204 performs classification processing such as classification of objects, places, landscapes, people, and living things in the classification target image based on the three-layer multi-scale feature maps x1up, x2up, and x3up supplied by the feature map restoration unit 203.

[0037] <Feature map reduction and restoration> The feature map reduction unit 102 has a function of converting the multi-layered multi-scale feature map acquired from the neural network feature extraction unit 101 into a single-layered single-scale feature map.

[0038] The feature map reduction unit 102 will be described in detail using Figure 5. The feature map reduction unit 102 is composed of a first feature map reduction unit 501, a first channel combination unit 502, a second feature map reduction unit 503, a second channel combination unit 504, a third feature map reduction unit 505, a first padding unit 506, a second padding unit 507, and a third padding unit 508. The feature map reduction unit 102 in Figure 5 is an example of a configuration that converts a three-layer multi-scale feature map into a single-scale feature map.

[0039] The feature map reduction unit 102 receives three-layer multi-scale feature maps, namely, a first feature map x1, a second feature map x2, and a third feature map x3, as input, converts them into a single-layer single-scale feature map xf, and supplies the result to the feature map conversion unit 103. Here, n denotes an index indicating a layer, Cn denotes the number of channels in the nth layer, Wn denotes the width of the feature map, and Hn denotes the height of the feature map. In this embodiment, the values ​​of Cn, Wn, and Hn for each layer are as shown in FIG. 11, where H and W denote the width and height, respectively, of the image from which features are extracted.

[0040] The first padding unit 506 has a function of padding the first feature map x1 to generate a first padded feature map x1pad. The first padding unit 506 determines the padding size so that the width and height of x1pad are multiples of 64. The number of channels of x1pad is 256, the same as that of x1.

[0041] The second padding unit 507 performs padding on the second feature map x2 by folding to generate a second padded feature map x2pad. The second padding unit 507 determines the padding size so that the width and height of x2pad are multiples of 32. The number of channels of x2pad is 256, the same as that of x2.

[0042] The third padding unit 508 performs padding on the third feature map x3 by folding to generate a third padded feature map x3pad. The third padding unit 508 determines the padding size so that the width and height of x3pad are multiples of 16. The number of channels in x3pad is 256, the same as that of x3.

[0043] In the first padding unit 506, the second padding unit 507, and the third padding unit 508, the left padding size is the same as the right padding size, and the top padding size is the same as the bottom padding size. That is, the feature maps x1, x2, and x3 are positioned at the centers of x1pad, x2pad, and x3pad, respectively.

[0044] The first feature map reduction unit 501 performs convolution in the spatial and channel directions on the first padded feature map x1pad obtained from the first padding unit 506 to generate a first intermediate feature map y1. The number of channels of y1 is 192, its width is Wx1pad / 2, and its height is Hx1pad / 2, where Wx1pad and Hx1pad are the width and height of the first padded feature map x1pad, respectively.

[0045] The first channel combining unit 502 has a function of combining the first intermediate feature map y1 obtained from the first feature map reduction unit 501 and the second padded feature map x2pad obtained from the second padding unit 507 in the channel direction to generate an intermediate feature map y1Cx2pad. Since y1 has 192 channels and x2pad has 256 channels, the intermediate feature map y1Cx2pad has 448 channels (192 + 256).

[0046] The second feature map reduction unit 503 generates a second intermediate feature map y2 by performing convolution in the spatial and channel directions on the intermediate feature map y1Cx2pad obtained from the first channel combination unit 502. The number of channels of y2 is 192, its width is Wy1Cx2pad / 2, and its height is Hy1Cx2pad / 2, where Wy1Cx2pad and Hy1Cx2pad are the width and height of the intermediate feature map y1Cx2pad, respectively.

[0047] The second channel combining unit 504 has a function of combining the second intermediate feature map y2 obtained from the second feature map reduction unit 503 and the third padded feature map x3pad obtained from the third padding unit 508 in the channel direction to generate an intermediate feature map y2Cx3pad. Since the number of channels in the intermediate feature map y2 is 192 and the number of channels in x3pad is 256, the number of channels in y1Cx2pad is 448 (192 + 256).

[0048] The third feature map reduction unit 505 generates a third intermediate feature map y3 by performing convolution in the spatial and channel directions on the intermediate feature map y2Cx3pad obtained from the second channel combination unit 504. The number of channels of y3 is 192, its width is Wy2Cx3pad / 2, and its height is Hy2Cx3pad / 2, where Wy2Cx3pad and Hy2Cx3pad are the width and height of the intermediate feature map y2Cx3pad, respectively.

[0049] The feature map reduction unit 102 outputs the third intermediate feature map y3 as a single-scale feature map xf and supplies it to the feature map conversion unit 103.

[0050] The feature map restoration unit 203 has a function of converting the single-scale feature map xr acquired from the feature map inverse conversion unit 202 into three-layer multi-scale feature maps x1up, x2up, and x3up.

[0051] The feature map restoration unit 203 will be described in detail using Fig. 6. The feature map restoration unit 203 is composed of an 8x enlargement unit 601, a 4x enlargement unit 602, a 2x enlargement unit 603, a first feature map blending unit 604, a second feature map blending unit 605, a first de-padding unit 606, a second de-padding unit 607, and a third de-padding unit 608.

[0052] The 8x enlargement unit 601 performs transposition convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr obtained from the feature map inverse transformation unit 202 to enlarge the feature map and reduce the channels, thereby generating an intermediate feature map z1. The number of channels in z1 is 196. If the width and height of the single-scale feature map xr are xrwidth and xrheight, respectively, the width and height of z1 are xrwidth × 8 and xrheight × 8, respectively. Here, rwidth × 8 and xrheight × 8 are the same as the width and height, respectively, of the first padded feature map x1pad, which is the output of the first padding unit 506 of the feature extraction unit 102.

[0053] The 4x enlargement unit 602 performs transposition convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr obtained from the feature map inverse transformation unit 202 to enlarge the features and reduce the channels, generating an intermediate feature map z2. The number of channels in z2 is 196. The width and height of z2 are xrwidth×4 and xrheight×4, respectively. Here, rwidth×4 and xrheight×4 are the same as the width and height, respectively, of the second padded feature map x2pad, which is the output of the second padding unit 507 of the feature extraction unit 102.

[0054] The 2x enlargement unit 603 performs transposition convolution in the spatial direction and convolution in the channel direction on the single-scale feature map xr obtained from the feature map inverse transformation unit 202 to enlarge the feature map and reduce the channels, thereby generating an intermediate feature map z3. The number of channels in z3 is 196. The width and height of z3 are xrwidth×2 and xrheight×2, respectively. Here, rwidth×2 and xrheight×2 are the same as the width and height, respectively, of the third padded feature map x3pad, which is the output of the third padding unit 508 in the feature extraction unit 102.

[0055] The first feature map mixing unit 604 has a function of generating an intermediate feature map z2up by improving the quality of the intermediate feature map z2 obtained from the 4x enlargement unit 602, using the intermediate feature map z1 obtained from the 8x enlargement unit 601.

[0056] The second feature map mixing unit 605 has a function of generating an intermediate feature map z3up that improves the quality of the intermediate feature map z3 obtained from the 2x enlargement unit 603, by using the intermediate feature map z2up obtained from the first feature map mixing unit 604.

[0057] The padding removal unit 606 removes padding from the intermediate feature map z1 acquired from the 8x enlargement unit 601 to generate a first output feature map x1up. The width and height of x1up are the same as the width and height of the first feature map x1 input to the feature extraction unit 102.

[0058] The padding removal unit 607 has a function of removing padding from the intermediate feature map z2up acquired from the first feature map blending unit 604 to generate a second output feature map x2up. The width and height of x2up are the same as the width and height of the second feature map x2 input to the feature extraction unit 102.

[0059] The padding removal unit 608 has a function of removing padding from the intermediate feature map z3up acquired from the second feature map blending unit 605 to generate a third output feature map x3up. The width and height of x3up are the same as the width and height of the third feature map x3 input to the feature extraction unit 102.

[0060] In the first padding removal unit 606, the second padding removal unit 607, and the third padding removal unit 608, the left padding size is the same as the right padding size, and the top padding size is the same as the bottom padding size, similar to the first padding unit 506, the second padding unit 507, and the third padding unit 508 of the feature extraction unit 102. In other words, the padding on the top, bottom, left, and right sides is removed by assuming that each output feature map x1up, x2up, and x3up is placed at the center of each intermediate feature map z1, z2up, and z3up, respectively.

[0061] <Feature map transformation and feature map inverse transformation> The feature map conversion unit 103 has a function of performing packing and quantization on the multi-channel decimal-type single-scale feature map xf supplied from the feature map reduction unit 102, and converting it into an integer-type packed feature frame to be supplied to the feature map internal coding unit 104.

[0062] The feature map conversion unit 103 on the encoding side will be described in detail with reference to Fig. 7. The feature map conversion unit 103 is made up of a packing unit 701 and a feature map quantization unit 702.

[0063] The packing unit 701 has a function of generating a packed feature frame by combining input feature maps of multiple channels into one frame. FIG. 12 is a diagram illustrating the state in which feature maps of multiple channels are packed into one frame. The feature maps of each channel are sequentially arranged in one frame in raster scan order, from left to right and top to bottom. Details of the packing unit 701 will be described later.

[0064] Furthermore, the packing unit 701 performs flipping based on the position in the frame where the channel feature map is placed. Whether or not to perform flipping can be selected, and the selection may be transmitted from the encoding side to the decoding side via a bitstream.

[0065] Flipping is performed when packing feature maps of multiple channels into one frame, as shown in FIG. 13. In flipping, the positions of the elements (pixels) of each channel's feature map are flipped horizontally (left and right), vertically (up and down), or horizontally and vertically (up and down, left and right) when packing the feature maps of each channel. The feature maps of four channels, A (upper left), B (upper right), C (lower left), and D (lower right) in FIG. 13, are grouped together as one set. No flipping is performed at position A in FIG. 13. At position B, the feature map is flipped horizontally (left and right). At position C, the feature map is flipped vertically (up and down). At position D, the feature map is flipped horizontally and vertically (up and down, left and right). When the distribution of elements in the feature maps of each channel is similar, flipping based on the position within the frame where the channels are placed reduces the boundaries between the feature maps of each channel, improving coding efficiency.

[0066] The feature map quantization unit 702 has a function of converting elements of a decimal-type packing feature frame (feature map of all channels) into N-bit integer type elements (N=an integer from about 8 to 16) within a predetermined range, and outputting an integer-type packing feature frame. In this embodiment, the elements are converted into 10-bit integer type elements ranging from 0 to 1023. The feature map quantization unit 702 detects the minimum and maximum values ​​of the elements of the decimal-type packing feature frame, and transmits the detected minimum and maximum values ​​of the elements of the packing feature frame (feature map of all channels) to the decoding side as metadata. To convert from a decimal-type packing feature frame (feature map) to an integer-type packing feature frame (feature map), a linear conversion is performed in which the minimum value of the decimal type is equivalent to the minimum value of the integer type, and the maximum value of the decimal type is equivalent to the maximum value of the integer type. For example, when the integer type range is expressed in 10 bits, the minimum value of the elements of the integer-type packing feature frame (feature map) is 0 and the maximum value is 1023 (2 10 -1). Values ​​between the minimum and maximum values ​​are linearly quantized.

[0067] Next, the feature map inverse transform unit 202 has a function of performing inverse quantization and unpacking on the integer-type packing feature frame decoded by VVC, HEVC, AV1, etc., supplied from the feature map internal decoding unit 201, and performing inverse transform into a decimal-type single-scale feature map xr to be supplied to the feature map restoration unit 203.

[0068] The feature map inverse transformation unit 202 on the decoding side will be described in detail with reference to Fig. 8. The feature map inverse transformation unit 202 performs the inverse process of the feature map transformation unit 103, and is made up of a feature map inverse quantization unit 801 and an unpacking unit 802.

[0069] The feature map inverse quantization unit 801 performs the inverse process of the feature map quantization unit 702 on the encoding side, and has a function of converting elements of integer-type packing feature frames from integer type to decimal type. The feature map inverse quantization unit 801 converts the integer-type packing feature frame decoded by the feature map internal decoding unit 201 into a decimal-type packing feature frame using the minimum and maximum decimal values ​​transmitted as metadata. A linear conversion is performed to convert the integer minimum value to the decimal minimum value and the integer maximum value to the decimal maximum value. Linear inverse quantization is performed on values ​​between the minimum and maximum values.

[0070] The unpacking unit 802 extracts a feature map for each channel in raster scan order from the packing feature frames arranged in one frame, and supplies the extracted feature map as a single-scale feature map xr to the feature map restoration unit 203. Details of the unpacking unit 802 will be described later.

[0071] <Feature map inner encoding and decoding> The feature map internal coding unit 104 will be described in detail using Figure 9. The feature map internal coding unit 104 is made up of a switch 901, a VVC encoding unit 902, an HEVC encoding unit 903, and an AV1 encoding unit 904. The switch 901 selects the encoding standard for internally coding the feature map converted by the feature map conversion unit 103. The VVC encoding unit 902 encodes the feature map in the VVC standard and outputs a bitstream that complies with the VVC standard. The HEVC encoding unit 903 encodes the feature map in the HEVC standard and outputs a bitstream that complies with the HEVC standard. The AV1 encoding unit 904 encodes the feature map in the AV1 standard and outputs a bitstream that complies with the AV1 standard.

[0072] In the VVC, HEVC, and AV1 standards, images are divided into blocks of a specified size and then encoded.

[0073] It is also possible to implement only one of VVC, HEVC, and AV1, and it is also possible to use image coding methods other than VVC, HEVC, and AV1.

[0074] Next, the feature map internal decoding unit 201 will be described in detail with reference to Fig. 10. The feature map internal decoding unit 201 is made up of a switch 1001, a VVC decoding unit 1002, an HEVC decoding unit 1003, and an AV1 decoding unit 1004. The switch 1001 selects the encoding standard for internal decoding based on information for selecting internal decoding in the input bitstream. The VVC decoding unit 1002 decodes the feature map in accordance with the VVC standard. The HEVC decoding unit 1003 decodes the feature map in accordance with the HEVC standard. The AV1 encoding unit 1004 decodes the feature map in accordance with the AV1 standard.

[0075] In the VVC, HEVC, and AV1 standards, decoding is performed for each predetermined block size.

[0076] It is also possible to implement only one of VVC, HEVC, and AV1, and it is also possible to use image coding methods other than VVC, HEVC, and AV1.

[0077] <Feature map internal encoding / decoding division unit> In general, in image coding standards, one frame is divided into multiple blocks and processed sequentially. For example, in the VVC standard, a block called a CTU (Coding Tree Unit) is used as the basic unit of division. The size of a CTU can be set from a minimum of 32×32 pixels to a maximum of 128×128 pixels, and the set value is transmitted as metadata. Furthermore, a CTU can be recursively divided into CBs (Coding Blocks), which are units of prediction. The size of a CB can be set from a minimum of 4×4 pixels to 1 / 4 of the CTU size (when the CTU size is 128×128 pixels, the CB size is 32×32 pixels), and the set value of the minimum CB size is transmitted as metadata. <Explanation of packing section 701> Using the flowchart of FIG. 15, the processing of the packing unit 701 will be described. In the packing unit 701, an encoding channel is set (step S101), and feature maps of multiple channels are packed into one frame (step S102).

[0078] FIG. 16 is a flowchart for explaining the details of step S101. First, a comparison is made between the number of channels num_of_ch of the feature map to be packed and the minimum number of channels min_num_ch (step S201). Here, the minimum number of channels min_num_ch represents the minimum value of the number of channels of the feature map to be encoded. min_num_of_ch is set to be greater than or equal to (the minimum frame size defined in the internal encoding / decoding of the feature map / the size of the feature map).

[0079] When num_of_ch < min_num_ch, the number of channels num_of_ch_to_be_coded of the feature map to be encoded is set to min_num_ch (step S202). Otherwise, that is, when num_of_ch >= min_num_ch, the number of channels num_of_ch_to_be_coded of the feature map to be encoded is set to num_of_ch (step S203). The number of channels num_of_ch_to_be_coded of the feature map to be encoded is a variable calculated in the internal encoding / decoding of the feature map.

[0080] FIG. 17 is a flowchart for explaining the details of step S102. First, an index ch_idx for indicating the feature map to be packed is initialized to 0 (step S301). While ch_idx is smaller than num_of_ch (step S302), the ch_idx-th feature map is packed into the packed feature frame (step S303), and then proceed to the next ch_idx (step S304). In step S302, when ch_idx becomes greater than or equal to num_of_ch, proceed to step S305.

[0081] While ch_idx is smaller than num_of_ch_to_be_coded (step S305), a dummy feature map is packed into the packed feature frame (step S306), and the process proceeds to the next ch_idx (step S307). If ch_idx becomes equal to or larger than num_of_ch_to_be_coded in step S305, the process ends. Here, the dummy feature map is a feature map with the same size and predetermined element values ​​as the feature map of the channel to be packed. <Details of the unpacking unit 802> The processing of the unpacking unit 802 will be described using the flowchart in Fig. 18. First, an index ch_idx for indicating a feature map to be unpacked is initialized to 0 (step S401). While ch_idx is smaller than num_of_ch (step S402), the ch_idx-th feature map is unpacked from the packed feature frame (step S403), and processing proceeds to the next ch_idx (step S404). If ch_idx becomes equal to or greater than num_of_ch in step S402, this procedure ends.

[0082] If ch_idx is greater than or equal to num_of_ch and less than num_of_ch_to_be_coded, the packed feature frames represent dummy feature maps, so ignore them without unpacking them as feature maps.

[0083] FIG. 19 shows an example of metadata related to packing. min_number_of_channels specifies the minimum number of channels, min_num_ch. number_of_channels specifies the number of channels, num_of_ch, of the feature map to be packed. packing_frame_width is the horizontal size of the packed feature frame, and packing_frame_height is the horizontal size of the packed feature frame. feature_width is the horizontal size of the packed feature map, and feature_height is the vertical size of the packed feature map. min_number_of_channels can be agreed upon in advance between the packing unit 701 and the unpacking unit 802. In this case, min_number_of_channels does not need to be transmitted as metadata.

[0084] Figure 20 shows an example of a packed feature frame when min_num_ch = 32 and num_of_ch = 16. In this example, CH1 to CH16 are packed feature maps, and DCH17 to DCH32 are dummy feature maps.

[0085] According to this embodiment, it is possible to guarantee that a packed feature frame is always configured with a feature map having at least the minimum number of channels min_num_ch, regardless of the number of channels in the feature map. It is possible to guarantee that a packed feature frame is configured with a feature map having at least the minimum number of channels min_num_ch, in other words, that the packed feature frame has a predetermined size or larger. This avoids violation of the frame size limit in the intra-encoding and decoding of the feature map, and enables reliable transmission of packed feature frames.

[0086] Furthermore, the minimum number of channels (min_num_ch) and the number of channels (num_of_ch) of the packed feature map can be set independently of the size of the CTU, which is metadata related to the internal encoding / decoding part of the feature map, and the minimum size of the CB. This allows the internal encoding / decoding part to be changed without changing the minimum number of channels (min_num_ch) or the number of channels (num_of_ch) of the packed feature map, providing excellent scalability.

[0087] In all of the above-described embodiments, the bitstream output by the feature map encoder has a specific data format that can be decoded according to the encoding method used in the embodiment, and the feature map decoder corresponding to the feature map encoder can decode the bitstream in this specific data format.

[0088] When a wired or wireless network is used to exchange bitstreams between the feature map encoder and the feature map decoder, the bitstreams may be converted into a data format suitable for the transmission mode of the communication channel before transmission. In this case, a transmitter is provided that converts the bitstream output by the feature map encoder into coded data in a data format suitable for the transmission mode of the communication channel and transmits the coded data to the network, and a receiver is provided that receives the coded data from the network, restores the coded data to a bitstream, and supplies the bitstream to the feature map decoder. The transmitter includes a memory that buffers the bitstream output by the feature map encoder, a packet processor that packetizes the bitstream, and a transmitter that transmits the packetized coded data via the network. The receiver includes a receiver that receives the packetized coded data via the network, a memory that buffers the received coded data, and a packet processor that packetizes the coded data to generate a bitstream and provides it to the feature map decoder.

[0089] The above encoding and decoding processes may be realized not only as a transmission, storage, and receiving device using hardware, but also as firmware stored in a ROM (read-only memory) or flash memory, or as software for a computer, etc. The firmware program or software program may be provided by recording it on a computer-readable recording medium, or may be provided from a server via a wired or wireless network, or may be provided as data broadcasting on terrestrial or satellite digital broadcasting.

[0090] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the components and treatment processes, and that such modifications are also within the scope of the present invention. [Explanation of symbols]

[0091] 100 Feature map encoding device, 101 Neural network feature extraction unit, 102 Feature map reduction unit, 103 Feature map conversion unit, 104 Feature map internal encoding unit, 200 Feature map decoding device, 201 Feature map internal decoding unit, 202 Feature map inverse conversion unit, 203 Feature map restoration unit, 204 Neural network identification unit, 301 Convolution processing unit, 302 Activation processing unit, 303 Pooling processing unit, 322 Bottom-up processing unit, 324 Top-down processing unit, 326 Image to be subjected to feature extraction processing, 501 First feature map reduction unit, 502 First channel combination unit, 503 Second feature map reduction unit, 504 Second channel combination unit, 505 Third feature map reduction unit, 506 First padding unit, 507 Second padding unit, 508 Third padding unit, 601 8x magnification unit, 602 4x magnification unit, 603 2x magnification unit, 604 first feature map mixing unit, 605 second feature map mixing unit, 606 first unpadding unit, 607 second unpadding unit, 608 third unpadding unit, 701 packing unit, 702 feature map quantization unit, 801 feature map inverse quantization unit, 802 unpacking unit, 901 switch, 902 VVC encoding unit, 903 HEVC encoding unit, 904 AV1 encoding unit, 1001 switch, 1002 VVC decoding unit, 1003 HEVC decoding unit, 1004 AV1 decoding unit.

Claims

1. a feature map reducer that converts the multi-scale feature maps to generate single-scale feature maps; a packing unit that packs the single-scale feature map into a frame to generate a fractional packed feature frame; a quantization unit that converts elements of the minority-type packing feature frame into integer values ​​to generate an integer-type packing feature frame; a feature map inner coding unit that codes the integer-type packing feature frame; Equipped with When the number of channels of the single-scale feature map is smaller than a predetermined minimum number of channels, the packing unit adds a dummy feature map to the single-scale feature map and packs the single-scale feature map into a frame. A feature map encoding device characterized by:

2. a feature map internal decoder that decodes the frame-packed and encoded single-scale feature map to generate an integer-packed feature frame; an inverse quantization unit that converts elements of the integer-type packing feature frame into decimal values ​​to generate a decimal-type packing feature frame; an unpacking unit that divides the fractional packing feature frame into one or more packing groups based on information about the packing groups and unpacks the fractional packing feature frame to generate a single-scale feature map; a feature map restoration unit that transforms the single-scale feature map to generate a multi-scale feature map; Equipped with When the number of channels of the single-scale feature map is smaller than a predetermined minimum number of channels, the unpacking unit performs unpacking while excluding a dummy feature map. A feature map decoding device characterized by:

Images