Video encoding device and video decoding device

Abstract

To efficiently encode and decode feature maps using existing video encoding methods.SOLUTION: A video encoding device for encoding a feature map includes a quantization unit that quantizes the feature map, a channel packing unit that packs the quantized feature map into a plurality of sub-channels each consisting of three components, and a first video encoding unit that encodes the sub-channels, and the first video encoding unit encodes a quantization offset value and / or a quantization scale value used in the quantization unit.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] An embodiment of the present invention relates to a video encoding device and a video decoding device. [Background technology]

[0002] In order to efficiently transmit or record moving images, a moving image encoding device is used that generates encoded data by encoding moving images, and a moving image decoding device is used that generates a decoded image by decoding the encoded data.

[0003] Specific examples of video encoding methods include H.266 / VVC (Versatile Video Coding) and H.265 / HEVC (High Efficiency Video Coding) (Non-Patent Document 1).

[0004] Meanwhile, in recent years, coding methods suitable for machine analysis processing such as object detection, object segmentation, and object tracking have also been studied. Non-Patent Document 2 discloses a method for coding a feature map derived from a video image by deep learning or the like for machine recognition. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] ITU-T Rec. H.266 [Non-Patent Document 2] ISO / IEC JTC 1 / SC 29 / WG 2 N104 Summary of the Invention [Problem to be solved by the invention]

[0006] When encoding and decoding a feature map consisting of multiple channels extracted from a video using an existing video encoding method such as Non-Patent Document 1, there is a problem that the method of arranging them within the screen cannot utilize correlation such as the same movement of each channel. In addition, there is a problem that the correspondence between the multiple channels (e.g. 64 channels) and the pictures within the screen is unclear, and even if the video is decoded, the feature map is not determined and cannot be used for machine recognition.

[0007] The object of the present invention is to efficiently encode and decode feature maps using an existing video encoding method such as that described in Non-Patent Document 1. [Means for solving the problem]

[0008] In order to solve the above problems, a video coding device according to one embodiment of the present invention is a video coding device that codes a feature map, the video coding device including a quantization unit that quantizes the feature map, a channel packing unit that packs the quantized feature map into a plurality of sub-channels each consisting of three components, and a first video coding unit that codes the sub-channels, the first video coding unit being characterized in that it codes a quantization offset value and / or a quantization scale value used in the quantization unit.

[0009] In order to solve the above problem, a video decoding device according to one embodiment of the present invention is a video decoding device that decodes a feature map from an encoding stream of the feature map, the video decoding device including: a first video decoding unit that decodes a plurality of sub-channels, each of which is composed of three components, from the encoding stream; an inverse channel packing unit that reconstructs a feature map from the sub-channels; and an inverse quantization unit that inverse quantizes the reconstructed feature map, wherein the first video decoding unit decodes a quantization offset value and / or a quantization scale value used in the inverse quantization unit. Effect of the Invention

[0010] According to one aspect of the present invention, it is possible to efficiently encode and decode a feature map by taking into account the correlation between each channel. [Brief description of the drawings]

[0011] [Figure 1] 1 is a schematic diagram showing a configuration of an image transmission system according to an embodiment of the present invention. [Diagram 2] 1 is a diagram showing the configuration of a transmitting device equipped with a video encoding device according to the present embodiment, and a receiving device equipped with a video decoding device, in which PROD_A indicates the transmitting device equipped with the video encoding device, and PROD_B indicates the receiving device equipped with the video decoding device. [Diagram 3] 1 is a diagram showing the configuration of a recording device equipped with a video encoding device according to the present embodiment, and a playback device equipped with a video decoding device, in which PROD_C indicates a recording device equipped with a video encoding device, and PROD_D indicates a playback device equipped with a video decoding device. [Figure 4] FIG. 2 is a diagram showing a hierarchical structure of data in an encoded stream. [Diagram 5] FIG. 1 is a functional block diagram showing a schematic configuration of a video encoding device 11 according to a first embodiment. [Figure 6] 2 is a diagram for explaining input and output of feature map extraction unit 101. FIG. [Figure 7] FIG. 2 is a diagram for explaining an example of a channel pack. [Figure 8] FIG. 2 is a diagram for explaining an example of a channel pack. [Figure 9] This is an example in which each subchannel is assigned in the time direction and coded. [Figure 10] This is an example of hierarchical coding in which each subchannel is assigned in the layer direction. [Figure 11] 1 is a functional block diagram showing a schematic configuration of a video decoding device 31 according to a first embodiment. [Figure 12] FIG. 11 is a functional block diagram showing a schematic configuration of a video encoding device 11 according to a second embodiment. [Figure 13]FIG. 11 is a functional block diagram showing a schematic configuration of a video decoding device 31 according to a second embodiment. [Figure 14] FIG. 11 is a functional block diagram showing a schematic configuration of a video encoding device 11 according to a third embodiment. [Figure 15] FIG. 11 is a functional block diagram showing a schematic configuration of a video decoding device 31 according to a third embodiment. [Figure 16] FIG. 13 is a diagram for explaining the operation of a conversion unit 1091. [Figure 17] FIG. 2 is a diagram for explaining an example of a channel pack. [Figure 18] FIG. 11 is a diagram illustrating an example of the syntax of feature map information. [Figure 19] FIG. 11 is a diagram showing an example of syntax when feature map information is notified by a sequence parameter set and a picture parameter set. [Figure 20] FIG. 13 is a diagram for explaining an example of a channel pack using a sub-picture. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0013] FIG. 1 is a schematic diagram showing the configuration of an image transmission system 1 according to this embodiment.

[0014] The image transmission system 1 is a system that transmits an encoded stream obtained by encoding an image to be encoded, decodes the transmitted encoded stream, and displays and / or analyzes the image. The image transmission system 1 includes a video encoding device (image encoding device) 11, a network 21, a video decoding device (image decoding device) 31, a video display device (image display device) 41, and a video analysis device (image analysis device) 51.

[0015] An image T is input to the video encoding device 11 .

[0016] The network 21 transmits the encoded stream Te and the encoded stream Fe generated by the video encoding device 11 to the video decoding device 31. The network 21 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination of these. The network 21 is not necessarily limited to a bidirectional communication network, and may be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting and satellite broadcasting. The network 21 may also be replaced by a storage medium on which the encoded stream Te is recorded, such as a DVD (Digital Versatile Disc: registered trademark) or a BD (Blu-ray Disc: registered trademark).

[0017] The video decoding device 31 decodes each of the coded streams Te and Fe transmitted by the network 21, and generates one or more decoded images Td and a decoded feature map Fd.

[0018] The video display device 41 displays all or part of one or more decoded images Td generated by the video decoding device 31. The video display device 41 includes a display device such as a liquid crystal display or an organic EL (Electro-luminescence) display. The display may be in the form of a stationary display, a mobile display, an HMD, or the like. When the video decoding device 31 has high processing power, it displays high quality images, and when it has only low processing power, it displays images that do not require high processing power or display power.

[0019] The video analysis device 51 performs analysis processes such as object detection, object segmentation, and object tracking using one or more decoded feature maps Fd generated by the video decoding device 31, and displays all or a part of the analysis results on the video display device 41. For example, the video analysis device 51 may use the feature map to output a list consisting of object IDs indicating the position, size, and type of an object, and certainty factors.

[0020] <Structure of the coding stream Te / Fe> Before going into a detailed description of the video encoding device 11 and video decoding device 31 according to this embodiment, the data structure of the encoded stream Te / Fe generated by the video encoding device 11 and decoded by the video decoding device 31 will be described.

[0021] FIG. 4 is a diagram showing a hierarchical structure of data in the coded stream Te / Fe. The coded stream Te / Fe illustratively includes a sequence and a plurality of pictures that constitute the sequence. 4 includes a coded video sequence that defines a sequence SEQ, a coded picture that defines a picture PICT, a coded slice that defines a slice S, and slice data A diagram showing the coded slice data defining the above, the coding tree units contained in the coded slice data, and the coding units contained in the coding tree units is shown.

[0022] (Coded Video Sequence) In the case of a coded video sequence, a video decoder is used to decode the sequence SEQ to be processed. A set of data to be referred to by the device 31 is defined. The sequence SEQ includes a Video Parameter Set, a Sequence The parameter set includes a sequence parameter set SPS (Sequence Parameter Set), a picture parameter set PPS (Picture Parameter Set), a picture PICT, and supplemental enhancement information SEI (Supplemental Enhancement Information).

[0023] The video parameter set VPS is used to determine the number of layers of a video image. A set of coding parameters common to multiple video images and a set of coding parameters related to multiple layers and individual layers included in the video images are defined.

[0024] The sequence parameter set SPS is used to decode the target sequence. A set of coding parameters to be referenced by the PPS is specified. For example, the width and height of a picture are specified. Note that there may be multiple SPSs. In that case, one of the multiple SPSs can be selected from the PPS. Select .

[0025] The picture parameter set PPS specifies the number of pictures to be decoded for each picture in the target sequence. A set of coding parameters to be referred to by the video decoding device 31 is specified. For example, the set includes a reference value of the quantization width used in decoding a picture (pic_init_qp_minus26) and a flag indicating application of weighted prediction (weighted_pred_flag). Note that there may be multiple PPSs. In that case, one of the multiple PPSs is selected for each picture in the target sequence.

[0026] (Encoded Picture) The coded picture defines a set of data to be referenced by the video decoding device 31 in order to decode the picture PICT to be processed. The picture PICT is, as shown in the coded picture of FIG. As shown in the figure, slices 0 to NS-1 are included (NS is the total number of slices included in the picture PICT). .

[0027] In the following, when there is no need to distinguish between slices 0 to NS-1, the symbols are The subscripts may be omitted. Also, in the coding stream Te / Fe described below, The same applies to other included data that are subscripted.

[0028] (Coded Slice) In the case of the coded slice, the video decoding device 31 refers to the slice S to be processed in order to decode the slice S. A slice is a set of data that is encoded as shown in the coding slice of Figure 4. It includes a slice header and slice data.

[0029] The slice header includes a group of coding parameters to be referred to by the video decoding device 31 in order to determine a decoding method for the current slice. Slice type designation information (slice_type) that designates the slice type is an example of a coding parameter included in the slice header.

[0030] Slice types that can be specified by the slice type specification information include (1) an I slice that uses only intra prediction during encoding, (2) a P slice that uses unidirectional prediction or intra prediction during encoding, and (3) a B slice that uses unidirectional prediction, bidirectional prediction, or intra prediction during encoding. Note that inter prediction is not limited to uni-prediction or bi-prediction, and a predicted image may be generated using more reference pictures. Hereinafter, P When called a B slice, it is a slice that includes blocks that can use inter prediction. This refers to the s.

[0031] In addition, the slice header may include a reference to a picture parameter set PPS (pic_parameter_set_id).

[0032] (Encoded slice data) The coded slice data defines a set of data to be referenced by the video decoding device 31 in order to decode the slice data to be processed. As shown in the slice header, the slice includes a CTU, which is a block of a fixed size (for example, 64x64) that constitutes a slice, and is also called a Largest Coding Unit (LCU).

[0033] (Subpicture) A picture may be further divided into rectangular sub-pictures, for example, four horizontal and four vertical sub-pictures. The size of a sub-picture is CTU The number of tiles may be a multiple of . A subpicture is defined as a set of tiles that are consecutive in both the vertical and horizontal directions. The slice header may include sh_subpic_id, which indicates the ID of the subpicture.

[0034] (coding tree unit) 4 specifies a set of data to be referenced by the video decoding device 31 in order to decode the CTU to be processed. The CTU is divided into a recursive quad tree (QT (Quad Tree) division), a binary tree (BT (Binary Tree) division), or a ternary tree (TT (Ternary Tree) division). The BT and TT divisions are collectively called multi-tree division (MT (Multi Tree) division). The nodes of the tree structure obtained by recursive quad-tree division are called coding nodes. The intermediate nodes of the 1-ary, 2-ary, and 3-ary trees are coding nodes, and the CTU itself is the highest coding It is defined as an encoding node.

[0035] (Encoding Unit) As shown in the coding unit of FIG. 4, a video is generated to decode the coding unit to be processed. The CU defines a set of data to be referenced by the image decoding device 31. Specifically, a CU is composed of a CU header CUH, prediction parameters, transformation parameters, quantization transformation coefficients, etc. The CU header defines a prediction mode, etc.

[0036] The prediction process may be performed in units of CUs, or in units of sub-CUs obtained by further dividing a CU. When the size of a CU and a sub-CU are equal, there is one sub-CU in the CU. When the size of a CU is larger than that of a sub-CU, the CU is divided into sub-CUs. For example, when a CU is 8x8 and a sub-CU is 4x4, the CU is divided into 2 parts horizontally and 2 parts vertically, into 4 sub-CUs.

[0037] There are two types of prediction (prediction modes): intra prediction and inter prediction. Intra prediction is a prediction within the same picture, while inter prediction refers to a prediction process performed between different pictures (for example, between display times or between layer images).

[0038] Transformation and quantization are performed in units of CUs, but quantization coefficients are stored in units of subblocks such as 4x4. It may be entropy coded.

[0039] (Prediction parameters) The predicted image is derived from prediction parameters associated with the block, which include intra-prediction and inter-prediction parameters.

[0040] (Configuration of a video encoding device according to the first embodiment) FIG. 5 is a functional block diagram showing a schematic configuration of a video encoding device 11 according to the first embodiment.

[0041] The video encoding device 11 includes a feature map extraction unit 101, a feature map conversion unit 102, and a video encoding unit 103. The video encoding device 11 may also include a video encoding unit 104.

[0042] The feature map extraction unit 101 includes a convolutional neural network and has C1=3 channels (e.g. For example, it inputs an image T consisting of three RGB channels and outputs a feature map F consisting of two C channels.

[0043] For example, the output of the first convolutional layer of Faster R-CNN (Region based Convolutional Neural Network) X101-FPN (Feature Pyramid Network), which is one of the neural networks used for object detection, may be used as the feature map. In this case, the number of channels of the feature map F is C2 = 64.

[0044] 6 is a diagram for explaining input and output of the feature map extraction unit 101. The feature map extraction unit 101 inputs an image T of W1 (width)×H1 (height)×C1 (number of channels), and outputs a feature map F of W2 (width)×H2 (height)×C2 (number of channels) via a convolution layer, an activation function, a pooling layer, etc. Here, each value of the image T may be an 8-bit integer, and each value of the feature map F may be a 16-bit fixed-point number. Alternatively, each value may be a 32-bit floating-point number. The feature map F is an image of width W2, height H2, and number of channels C2.

[0045] The feature map conversion unit 102 includes a quantization unit 1021 and a channel pack unit 1022. The feature map conversion unit 102 converts the image data of the feature map F output from the feature map extraction unit 101 into a The image is quantized. It is also divided and rearranged (hereafter "packed") into sets of multiple videos / images (hereafter "subchannels") and output. Note that "subchannel" is an abbreviation of subset channel (subset of channel).

[0046] The quantization unit 1021 quantizes the feature map F to an image of integer values ​​(for example, 10-bit integers, bitDepth=10) for the video encoding unit 103, and outputs the quantized image.

[0047] The quantized feature map qF is expressed by the following equation.

[0048] qF = Offset + Round(F ÷ Scale) Here, Round(a) is a function that returns the integer value of a and is defined as follows:

[0049] F: feature map (32-bit floating point number) qF: quantized feature map (e.g. 10-bit integer) Offset: Quantization offset value (e.g. 10-bit integer) Scale: Quantization scale value The quantization unit 1021 notifies the video encoding unit 103 of the offset and scale. / is an integer division that truncates towards zero. ÷ is an integer division that truncates or rounds. No division.

[0050] (Configuration assigned to layer) The channel pack unit 1022 assigns (packs) qF to images (subchannels) subSamples of multiple images each composed of three components (e.g., luminance Y and color differences U and V), and outputs the images. For example, the channel pack unit 1022 assigns (packs) qF to channels with IDs designated by c=0..C2-1. The following process is performed on the image qF of the feature vector, and the i-th (i=0..(C2+2) / 3) image / video (subchannel) identified by subChannelID is derived. Here, x=y..z is an integer number ranging from y to This indicates that the integer value x between the values ​​z is derived in order and the processing is performed.

[0051] c = 0 do { subChannelID = c / 3 subSamples[y][x][0] = qF[y][x][c]; c = c + 1 subSamples[y][x][1] = qF[y][x][c]; c = c + 1 subSamples[y][x][2] = qF[y][x][c]; c = c + 1 } while (c < C2) Also the following is acceptable: subChannelID = c / numComps subSamples[y][x][c%numComps] = qF[y][x][c] c=0..C2-1. numComps=3 is also acceptable. Here, % indicates the modulus (MOD) operation. Here, subSamples is an array of images specified by a subchannel ID (subChannelID). The channel pack unit 1022 determines whether a channel of a feature map exists, and if a channel of a feature map does not exist, a predetermined value FillVal, for example, 1<<(bitDepth-1), may be assigned according to the bit depth bitDepth of the image. In the case of 10 bits, FillVal may be 512.

[0052] subChannelID = c / numComps subSamples[y][x][0] = c >= C2 ? FillVal : qF[y][x][c]; c = c + 1 subSamples[y][x][1] = c >= C2 ? FillVal : qF[y][x][c]; c = c + 1 subSamples[y][x][2] = c >= C2 ? FillVal : qF[y][x][c]; c = c + 1 where y=0..H2-1, x=0..W2-1. numComps=3. It may also be derived as follows.

[0053] subSamples[y][x][0] = qF[y][x][c / 3]; subSamples[y][x][1] = qF[y][x][c / 3+1]; subSamples[y][x][2] = qF[y][x][c / 3+2]; c = c + 3 Alternatively, it may be derived as follows.

[0054] subSamples[y][x][0] = qF[y][x][3*subChannelID; subSamples[y][x][1] = qF[y][x][3*subChannelID+1]; subSamples[y][x][2] = qF[y][x][3*subChannelID+2]; where subChannelID=0..numSubChannels-1.

[0055] When the number of channels of the feature map is numChannels, the channel pack unit 1022 The number of channels, numSubChannels, is derived as follows depending on the number of components, numComps, of the image:

[0056] numSubChannels = Ceil(numChannels ÷ numComps) numComps can be 3 (for 4:2:0, 4:4:4). For 4:0:0, numComps=1. Here, Ceil(a) is a function that returns the smallest integer greater than or equal to a. It may also be derived using division with truncation as follows:

[0057] numSubChannels = (numChannels + numComps-1) / numComps In addition, when the feature map is assigned not only to the image components but also to the subpictures, it is derived as follows using the number of subpictures, numSubpics.

[0058] numSubChannels = (numChannels + numComps*numSubpics-1) / (numComps*numSubpics) FIG. 7 is a diagram for explaining an example of a channel pack.

[0059] FIG. 7(a) shows an example in which each channel of the feature map is assigned to multiple 4:4:4 format videos (e.g., a format corresponding to sps_chroma_format_idc = 3 in H.266 / VVC or H.265 / HEVC). numChannels is 64. The number of subchannels, numSubChannels, is numSubChannels = Ceil(64 ÷ 3) = 22.

[0060] The channel pack unit 1022 may assign each channel (ch0, ch1, ..., ch63) of the feature map by scanning the channels in the order of the channels, the outer loop in the order of the subchannels, and the inner loop in the order of the components. Component U of channel 0, ch2 is used as subchannel 0 component V, ch3 is used as subchannel Component Y, ch4 of channel 1 to sub-channel 1 Component U, ch5 to sub-channel 1 For the final sub-channel 21, since there is no feature map to be assigned to the components U and V, the channel ch63 of the feature map assigned to the component Y is copied. Alternatively, it may be filled with the above-mentioned predetermined pixel value FillVal. .

[0061] 7(b) shows an example in which each channel of a feature map is assigned to multiple 4:2:0 format videos (e.g., a format corresponding to sps_chroma_format_idc=1 in H.266 / VVC or H.265 / HEVC). The channel pack unit 1022 downsamples the feature map by 1 / 2 in both the horizontal and vertical directions and assigns it to the U and V components.

[0062] FIG. 8 is a diagram for explaining an example of a channel pack.

[0063] Figure 8(a) shows an example of assigning each channel of the feature map to multiple 4:4:4 format images. The channel pack unit 1022 packs each channel (ch63, ch62, ..., ch0) of the feature map into a The channels may be scanned and assigned in the reverse order of subchannels in the outer loop and components in the reverse order in the inner loop. In other words, ch63 is assigned to component V of subchannel 21, ch62 to component U of subchannel 21, ch61 to component Y of subchannel 21, ch60 to component V of subchannel 20, ch59 to component U of subchannel 20, ch58 to component Y of subchannel 20, etc. For the first subchannel 0, Since there are no feature maps to be assigned to the components U and Y, the channel ch0 of the feature map assigned to the component V is copied. Alternatively, the channel may be filled with the above-mentioned predetermined pixel value FillVal.

[0064] Figure 8(b) shows an example of assigning each channel of the feature map to multiple 4:2:0 format images. The channel pack unit 1022 down-converts the feature map by half in both the horizontal and vertical directions. For example, the image of the feature map of the channel with channelID % 3 == 0 is left as is, and the image of the feature maps of the channels with channelID % 3 == 1 and 2 is Downsample the image to 1 / 2.

[0065] subSamples[y][x][0] = qF[y][x][c]; c = c + 1 subSamples[y>>1][x>>1][1] = qF[y][x][c]; c = c + 1 subSamples[y>>1][x>>1][2] = qF[y][x][c]; c = c + 1 (Configuration to assign to subpicture) The channel pack unit 1022 is a unit of a number of channels, numChannels, and a number of subpictures, numSubpics. The number of subchannels, numSubChannels, is derived from

[0066] numSubChannels = Ceil(numChannels ÷ (numSubpics * numComps)) The channel pack unit 1022 derives an image consisting of multiple sub-pictures from the image of the feature map qF as follows.

[0067] For example, the channel pack unit 1022 assigns a feature map value qF to pixel values ​​subSamples[y][x][c] of a video composed of subpictures, where the number of subpictures in the horizontal direction is numSubpicsX and the number of subpictures in the vertical direction is numSubpicsY, as follows: do.

[0068] s = subpic_id = (c / numComps) % (numSubpicX*numSubPicsY) subSamples[s][y][x][0] = qF[y][x][c]; c = c + 1 subSamples[s][y][x][1] = qF[y][x][c]; c = c + 1 subSamples[s][y][x][2] = qF[y][x][c]; c = c + 1 where y=0..H2-1, x=0..W2-1, c=0..C2-1, numComps=3. Subpic_id=0..numSubChannels -1 is scanned. Another example of allocation to sub-pictures will be described below.

[0069] (Configuration 1 for assigning subpictures and layers) Also, a channel (subchannel) of a feature map may be assigned to an image composed of multiple subpictures. The channel pack unit 1022 derives an image in which a specific channel of a feature map is assigned to a specific channel of a subpicture. For example, if the number of subpictures arranged in one picture is numSubpics, the channel pack unit 1022 generates an image of y=0..H2-1, x=0..W2-1, c=0..C2-1.

[0070] s = subpic_id = (c / 3) % numSubpics subChannelID = (c / 3) / numSubpics subSamples[s][y][x][0] = qF[y][x][c]; c = c + 1 subSamples[s][y][x][1] = qF[y][x][c]; c = c + 1 subSamples[s][y][x][2] = qF[y][x][c]; c = c + 1 Here, subSamples is an array of images specified by a specified subchannel ID (subChannelID). The channel pack unit 1022 may determine whether a channel of a feature map exists, and may assign a predetermined value FillVal if the channel of the feature map does not exist. The header encoding unit 1031 of the video encoding unit 103 described later may assign the subChannelID to a layer ID (layer_id) and encode it as encoded data.

[0071] s = subpic_id = (c / 3) % numSubpics subChannelID = (c / 3) / numSubpics subSamples[s][y][x][0] = c >= C2 ? FillVal : qF[y][x][c]; c = c + 1 subSamples[s][y][x][1] = c >= C2 ? FillVal : qF[y][x][c]; c = c + 1 subSamples[s][y][x][2] = c >= C2 ? FillVal : qF[y][x][c]; c = c + 1 (Configuration 2: Assigning to subpicture) The channel pack unit 1022 may also assign each channel of the feature map to a subpicture. For example, a 64-channel feature map may be used to generate a 4:0:0 video using subpictures. It can be allocated to 4x16 sub-pictures, 4 horizontally and 16 vertically.

[0072] For example, the channel pack unit 1022 assigns a feature map value qF to pixel values ​​subSamples[y][x][c] of a video composed of subpictures, where the number of subpictures in the horizontal direction is numSubpicsX and the number of subpictures in the vertical direction is numSubpicsY, as follows: do.

[0073] sx = c % numSubpicsX sy = c / numSubpicsX subSamples[y+sy*subPicH][x+sx*subPicW][0] = qF[y][x][c]; c = c + 1 Here, subPicW, subPicH are the width and height of the subpicture, and subPicH=H2 / numSubpicsY, subPicW=W2 / numSubpicsX, where y=0..H2-1, x=0..W2-1, c=0..C2-1. It scans between sx=0..numSubpicsX-1, sy=0..numSubpicsY-1.

[0074] (Configuration 2 for assigning layers and subpictures) Furthermore, the channel pack unit 1022 may assign the image of each channel of the feature map to a subpicture of multiple images. For example, the number of subpictures in the horizontal direction is numSubpicsX, the number of subpictures in the vertical direction is numSubpicsX, and the number of subpictures in the vertical direction is numSubpicsX. If the number of subpictures in the vertical direction is numSubpicsY, the channel pack unit 1022 performs the following processing on y=0..H2-1, x=0..W2-1, c=0..C2-1, sx=0..numSubpicsX-1, and sy=0..numSubpicsY-1 to generate the i-th image (i=0..((C2+(3*(numSubpicsX*numSubpicsY)-1)) / (3*(numSubpicsX*numSubpicsY))).

[0075] subChannelID = (c / 3) / (numSubpicsX*numSubpicsY) sx = (c / 3) % numSubpicsX sy = (c / 3) / numSubpicsX subSamples[y+sy*subPicH][x+sx*subPicW][0] = qF[y][x][c]; c = c + 1 subSamples[y+sy*subPicH][x+sx*subPicW][1] = qF[y][x][c]; c = c + 1 subSamples[y+sy*subPicH][x+sx*subPicW][2] = qF[y][x][c]; c = c + 1 Here, subSamples is an array of images specified by a specified subchannel ID (subChannelID). Note that the channel pack unit 1022 may determine whether a channel of a feature map exists, and assign a predetermined value, for example, FillVal, if the channel of the feature map does not exist.

[0076] subChannelID = (c / 3) / (numSubpicsX*numSubpicsY) subSamples[y+sy*subPicH][x+sx*subPicW][0] = c >= C2 ? FillVal : qF[y][x][c]; c = c + 1 subSamples[y+sy*subPicH][x+sx*subPicW][1] = c >= C2 ? FillVal : qF[y][x][c]; c = c + 1 subSamples[y+sy*subPicH][x+sx*subPicW][2] = c >= C2 ? FillVal : qF[y][x][c]; c = c + 1 FIG. 20 is a diagram for explaining an example of a channel pack using a sub-picture.

[0077] Figure 20(a) shows an example of allocating each channel of a feature map to multiple 4:4:4 format images. numChannels is 64. numSubpics is 4. numSubChannels is numSubChannels = Ceil(NumChannels ÷ (NumSubPictures * 3)) = 6.

[0078] The channel pack unit 1022 may scan and assign each channel (ch0, ch1, ..., ch63) of the feature map in the order of the channels, such that the first loop is the subchannel order, the second loop is the component order, and the third loop is the subpicture order. Subpicture 0, ch1 of component Y of subchannel 0 is subpicture 1, ch2 of component Y of subchannel 0 is subpicture 2, ch3 of component Y of subchannel 0 is subpicture 1, ch2 of component Y of subchannel 0 is subpicture 2, ch3 of component Y of subchannel 0 is Subpicture 3, ch4 of component Y of subchannel 0 is subpicture 0, ch5 of component U of subchannel 0 is subpicture 1, ch6 of component U of subchannel 0 is subpicture 1, ch7 of component U of subchannel 0 is subpicture 2, ch9 of component U of subchannel 0 is subpicture 3, ch4 of component Y of subchannel 0 is subpicture 0, ch5 of component U of subchannel 0 is subpicture 1, ch6 of component U of subchannel 0 is subpicture 2, ch1 of component U of subchannel 0 is subpicture 3, ch4 of component Y of subchannel 0 is subpicture 0, ch5 of component U of subchannel 0 is subpicture 1, ch6 ...2, ch1 of component U of subchannel 0 is subpicture Sub-pictures 2 and 7 of component U of sub-channel 0 are assigned to sub-picture 3 of component U of sub-channel 0, and so on. For the final sub-channel 5, since there are no feature maps to assign to components U and V, channels ch60, ch61, ch62, and ch63 of the feature map assigned to component Y are copied. Alternatively, it may be filled with the above-mentioned predetermined pixel value FillVal.

[0079] When assigning each channel of a feature map to multiple 4:2:0 format images, the channel The packing unit 1022 downsamples the feature map by half in both the horizontal and vertical directions. Assign it to components U and V.

[0080] The channel pack unit 1022 notifies the video encoding unit 103 of the number of channels of the feature map, numChannels, the number of subpictures, numSubpics, and the like.

[0081] In the above example, the number of components of the image is 3, but any number of components, numComps, can be used (same below). The channel pack unit 1022 converts qF into a plurality of images composed of numComps components. For example, the channel packing unit 1022 assigns (packs) the feature quantity of an image qF of a channel having an ID specified by c=0..C2-1 to the following The above process is performed to derive the i-th (i=0..(C2+numComps-1) / numComps) image / video (sub-channel) identified by subChannelID.

[0082] c = 0 do { subChannelID = c / numComps subSamples[y][x][c%numComps] of the image specified by subChannelID = qF[y][x][c]; c = c + 1 } while (c < C2) The video encoding unit 103 encodes the sub-channels output from the channel pack unit 1022. The encoded stream is output as Fe. The encoding method is VVC / H.266 or HEVC / H.265. The video encoding unit 103 encodes the layer ID and The image encoding unit 1031 includes a header encoding unit 1031 that encodes information and a predicted image generating unit 1032 that generates an inter-predicted image.

[0083] The header encoding unit 1031 may encode the number of subpictures minus 1 as sps_num_subpics_minus1, and a flag sps_independent_subpics_flag indicating whether the subpicture is independent. The top left position (sps_subpic_ctu_top_left_x[i], sps_subpic_ctu_top_left_y[i]), width sps_subpic_width_minus1[i], and height sps_subpic_height_minus1[i] of the th subpicture may be coded. Furthermore, sps_subpic_treated_as_pic_flag[i] indicating independence on a subpicture basis may be coded. When assigning a feature map to a subpicture and coding it, the header coding unit 1031 codes it as sps_independent_subpics_flag=1 or a specific subpicture i has sps_independent_subpics_flag[i]=1.

[0084] When the prediction image generation unit 1032 refers to a picture different from the target sub-picture, if sps_independent_subpics_flag = 1 or when a specific sub-picture i has sps_independent_subpics_flag[i] == 1, it pads the pixels at the sub-picture boundary and does not refer to the pixels of another sub-picture. Specifically, the prediction image generation unit 1031 clips and limits the X coordinate xInt of the reference pixel as follows using the left boundary position SubpicLeftBoundaryPos and the right boundary position SubpicRightBoundaryPos of the sub-picture. It clips and limits the Y coordinate yInt of the reference pixel as follows using the upper boundary position SubpicTopBoundaryPos and the lower boundary position SubpicBottomBoundaryPos of the sub-picture. The prediction image generation unit 1032 generates a prediction image by filter processing or the like using the reference pixels at the clipped positions. xInt = Clip3( SubpicLeftBoundaryPos, SubpicRightBoundaryPos, xIntL ) yInt = Clip3( SubpicTopBoundaryPos, SubpicBotBoundaryPos, yIntL ) Here, Clip3(a, b, c) is a function that clips c to a value between a and b. When c < a it returns a, when c > b it returns b, and in other cases it returns c (however, a <= b).

[0085] Figure 9 shows an example of encoding by allocating each sub-channel in the time direction. Sub-channel 0 is allocated to frame frame0, sub-channel 1 to frame1,..., sub-channel 20 to frame20, and sub-channel 21 to frame21 for encoding. The number of frames numFrames is numFrames = numSubChannels = 22. The number of layers numLayers is numLayers = 1.

[0086] 10 shows an example of hierarchical coding in which each subchannel is assigned in the layer direction. Subchannel 0 is assigned to layer 0 of frame 0, subchannel 1 to layer 1 of frame 0, ..., subchannel 20 to layer 20 of frame 0, and subchannel 21 to layer 21 of frame 0 for hierarchical coding. For numFrames, numFrames = 1. For numLayers, numLayers = numSubChannels = 22.

[0087] The video encoding unit 103 converts the feature map information including the Offset and Scale notified by the quantization unit 1021 and the numChannels notified by the channel packing unit 1022 into notification data (e.g., additional The notification data is encoded as extended information (SEI) and output as the coded stream Fe. The data is not limited to SEI, which is data accompanying an image, and may be, for example, the syntax of a transmission format such as ISO base media file format (ISOBMFF), DASH, MMT, or RTP.

[0088] FIG. 18(a) shows an example of the syntax of feature map information. The mantics are as follows: fm_quantization_offset: Quantization offset value Offset fm_quantization_scale: Quantization scale value Scale fm_num_channels_minus1:fm_num_channels_minus1 + 1 indicates the number of channels, numChannels, in the feature map.

[0089] Alternatively, feature map information may be signaled by the sequence parameter set SPS. Figure 19(a) shows an example of the syntax for notifying feature map information in the sequence parameter set SPS. The semantics of each field are as follows: sps_fm_info_present_flag: 1 if feature_map_info() exists, 0 if it does not. sps_fm_info_payload_size_minus1: sps_fm_info_payload_size_minus1 + 1 indicates the size of feature_map_info(). sps_fm_alignment_zero_bit: 1 bit with value 1.

[0090] Alternatively, the feature map information may be signaled by the picture parameter set PPS. 19B shows an example of the syntax when notifying by a picture parameter set PPS. The semantics of each field are as follows: pps_fm_info_present_flag: 1 if feature_map_info() exists, 0 if it does not exist. pps_fm_info_payload_size_minus1: pps_fm_info_payload_size_minus1 + 1 indicates the size of feature_map_info().

[0091] In addition, the video encoding unit 103 receives the feature map image and a channel ID (e.g., a channel number). Information indicating the correspondence between the two may be coded as supplemental enhancement information SEI.

[0092] FIG. 18(b) shows an example of the syntax of the feature map information. The mantics are as follows: fm_param_flag: If 1, the relationship between each component of each subpicture of each layer and the feature map is coded. For example, the channel ID of the feature map corresponding to each component of each subpicture of each layer is coded. If 0, the channel ID is not coded and is set to a predetermined value. The above relationship is derived based on the corresponding method. fm_num_layers_minus1:fm_num_layers_minus1 + 1 indicates the number of image / video layers used to transmit the feature maps. fm_num_subpics_minus1: fm_num_subpics_minus1 + 1 indicates the number of image / video subpictures used to transmit the feature map. fm_channel_id[i][j][k]: The jth component of the kth subpicture of the ith layer. The channel ID of the feature map stored in the database.

[0093] The video encoder 104 encodes the image T and outputs it as an encoded stream Te. As with the video encoder 103, the encoding method may be VVC / H.266, HEVC / H.265, or the like.

[0094] (Configuration of an image decoding device according to the first embodiment) FIG. 11 is a functional block diagram showing a schematic configuration of a video decoding device 31 according to the first embodiment. do.

[0095] The video decoding device 31 includes a video decoding unit 301 , a feature map inverse conversion unit 302 , and a video decoding unit 303 .

[0096] The video decoding unit 301 has a function of decoding a coded stream coded by VVC / H.266, HEVC / H.265, or the like, and decodes the coded stream Fe, and outputs an image / video (packed sub-channel) on which a feature map is arranged to the feature map inverse conversion unit 302. For example, the image / video shown in FIG. 7 or FIG. 8. It is advisable to assign the subChannelID to the layer ID and decode it from the encoded data.

[0097] The video decoding unit 301 decodes feature map additional enhancement information SEI included in the encoded stream Fe, derives feature map information including the quantization offset value Offset, the quantization scale value Scale, the number of channels of the feature map numChannels, etc., and notifies the feature map inverse conversion unit 302.

[0098] Offset = fm_quantization_offset Scale = fm_quantization_scale numChannels = fm_num_channels_minus1 + 1 numLayers = fm_num_layers_minus1 + 1 numSubpics = fm_num_subpics_minus1 + 1 Alternatively, these values ​​may be derived by decoding the sequence parameter set SPS shown in FIG. 19(a).

[0099] Alternatively, these values ​​may be derived by decoding the picture parameter set PPS shown in FIG. 19(b).

[0100] The video decoding unit 301 includes a header decoding unit 3011 that decodes a layer ID (Layler_id) of a video from the NAL unit header of the encoded data and decodes information of subpictures that constitute the video. NAL is an abbreviation of Network Abstraction Layer. NAL is composed of a NAL unit header and NAL unit data, and the NAL unit header is composed of a parameter set, slice data, etc. The NAL unit header may include a NAL unit type and a temporal ID, and indicates an abstract type of encoded data.

[0101] The feature map inverse transform unit 302 includes an inverse channel packing unit 3021 and an inverse quantization unit 3022. The feature map inverse conversion unit 302 converts the feature map FdBase (or the differential feature map FdResi ) to output.

[0102] The inverse channel packing unit 3021 reconstructs a feature map FdBase (or a differential feature map FdResi) from multiple sub-channels each consisting of three components (for example, luminance Y and chrominance U, V). For example, the reverse channel pack unit 3021 receives a subChannelID decoded by the video decoder 301. The process shown in the pseudo code below is performed on the image subSamples specified. The following process is performed on y=0..H2-1, x=0..W2-1, c=0..C2-1 to generate qF from the i-th subSample (i=0..(C2+2) / 3).

[0103] for (c = 0; c < C2; c++) { for (y = 0; y < H2; y++) { for (x = 0; x < W2; x++) { qF[y][x][c] = subSamples[y][x][c%numComps] } } } numComps=3 Note that subChannelID is the layer_id (subChannelID=layer_id) obtained from the encoded data. Alternatively, an ID assigned to a video / image of a subchannel obtained from a transmission format may be used. When a subpicture is used, the reverse channel packing unit 3021 performs a process of reconstructing qF from subSamples as shown in the following pseudo code for the image subSamples of subChannelID decoded by the video decoding unit 301. The following process is performed on y=0..H2-1, x=0..W2-1, c=0..C2-1, s=0..numSubpics-1 to generate the i-th (i=0..(C2+2) / 3) image.

[0104] for (c = 0; c < C2; c++) { for (y = 0; y < H2; y++) { for (x = 0; x < W2; x++) { for (s = 0; s < numSubpics; s++) { qF[y][x][c] = subSamples[s][y][x][c%numComps] } } } } numComps=3 When using sub-pictures, the reverse channel pack unit 3021 receives the decoded sub-pictures from the video decoder 301. For the image subSamples of the subChannelID, the process of reconstructing qF from the subSamples is performed as shown in the following pseudo code. The following process is performed for y=0..H2-1, x=0..W2-1, c=0..C2-1, sy=0..numSubpicsY-1, sx=0..numSubpicsX-1, and the i-th (i=0..(C2+2) / 3) image is Generate.

[0105] for (c = 0; c < C2; c++) { for (y = 0; y < H2; y++) { for (x = 0; x < W2; x++) { for (sy = 0; sy < numSubpicsY; sy++) { for (sx = 0; sx < numSubpicsX; sx++) { qF[y][x][c] = subSamples[y+sy*subPicH][x+sx*subPicW][c%numComps] } } } } } numComps=3 (Feature map derivation using correspondence information fm_channel_id and layers) The inverse channel packing unit 3021 may map data stored in the component comp_id(0,1,2) of the layer layer_id(0..numLayers-1) to feature data of the channel indicated by fm_channel_id[layer_id][comp_id][0]. The inverse channel packing unit 3021 may derive a feature map qF using fm_channel_id.

[0106] for (c = 0; c < C2; c++) { for (y = 0; y < H2; y++) { for (x = 0; x < W2; x++) { subChannelID = layer_id channel_id = fm_channel_id[subChannelID][c%numComps][0] qF[y][x][channel_id] = subSamples[y][x][c%numComps] of subChannelID } } } numComps=3 The reverse channel pack unit 3021 may perform the above process when fm_param_flag is 1.

[0107] When fm_param_flag=0 and fm_channel_id does not appear, the header decoding unit 3031 may derive the correspondence information fm_channel_id from the layer subChannelID.

[0108] subChannelID = c / 3 fm_channel_id[subChannelID][c%3][0] = c c=0.. C2-1 (Feature map derivation using correspondence information fm_channel_id and sublayers) The reverse channel packing unit 3021 may map the data stored in the subpicture to the data of channel fm_channel_id[0][comp_id][subpic_id] of the feature map.

[0109] for (c = 0; c < C2; c++) { for (y = 0; y < H2; y++) { for (x = 0; x < W2; x++) { sx = (c / numComps) % numSubpicsX sy = (c / numComps) / numSubpicsX subpic_id = sy*numSubpicsX+sx channel_id = fm_channel_id[0][c%3][subpic_id] qF[y][x][channel_id] = subSamples[y+sy*subPicH][x+sx*subPicW][c%numComps] } } } numComps=3 The reverse channel pack unit 3021 may perform the above process when fm_param_flag is 1.

[0110] When fm_param_flag=0 and fm_channel_id does not appear, the header decoding unit 3031 may derive the correspondence information fm_channel_id from the subpicture ID subpic_id.

[0111] sx = (c / numComps) % numSubpicsX sy = (c / numComps) / numSubpicsX subpic_id = sy*numSubpicsX+sx fm_channel_id[0][c%numComps][subpic_id] = c c=0.. C2-1 numComps=3 (Feature map derivation using correspondence information fm_channel_id, layers and sublayers) For example, when fm_param_flag is 1, the reverse channel pack unit 3021 stores the subpicture. The data stored in the feature map may be mapped to data in channel fm_channel_id[layer_id][comp_id][subpic_id] of the feature map.

[0112] for (c = 0; c < C2; c++) { for (y = 0; y < H2; y++) { for (x = 0; x < W2; x++) { subChannelID = layer_id sx = (c / numComps / numLayers) % numSubpicsX sy = (c / numComps / numLayers) / numSubpicsX subpic_id = sy*numSubpicsX+sx channel_id = fm_channel_id[layer_id][c%numComps][subpic_id] qF[y][x][channel_id] = subSamples[y+sy*subPicH][x+sx*subPicW][c%3] of subChannelID } } } numComps=3 The reverse channel pack unit 3021 may perform the above process when fm_param_flag is 1.

[0113] When fm_param_flag=0 and fm_channel_id does not appear, the header decoding unit 3031 may derive the correspondence information fm_channel_id from the layer subChannelID and the subpic_id.

[0114] subChannelID = c / (numComps*numSubpics) sx = (c / numComps) % numSubpicsX sy = (c / numComp) / numSubpicsX subpic_id = sy*numSubpicsX+sx fm_channel_id[subChannelID][c%numComps][subpic_id] = c c=0..C2-1, numComps=3 When multiple 4:4:4 format images are assigned as shown in FIG. 7(a), the reverse channel The packing unit 3021 reconstructs a feature map composed of 64 channels as shown in FIG. 6B. For example, a component Y of subchannel 0, a component U of subchannel 0, a component V of subchannel 0, a component Y of subchannel 1, a component V of subchannel 2, a component V of subchannel 3, a component V of subchannel 4, a component V of subchannel 5, a component V of subchannel 6, a component V of subchannel 7, a component V of subchannel 8, a component V of subchannel 9, a component V of subchannel 10, a component V of subchannel 11, a component V of subchannel 12, a component V of subchannel 13, a component V of subchannel 14, a component V of subchannel 15, a component V of subchannel 16, a component V of subchannel 17, a component V of subchannel 18, a component V of subchannel 19, a component V of subchannel 20, a component V of component U, component V of subchannel 1, …, component V of subchannel 21 From Y, we reconstruct the feature map.

[0115] When the subchannels are assigned to multiple 4:2:0 format images as shown in FIG. 7(b), In this case, the inverse channel packing unit 3021 upsamples the feature maps assigned to the components U and V by a factor of two in both the horizontal and vertical directions and reconstructs them.

[0116] When the subchannels are assigned to multiple 4:4:4 format images as shown in FIG. 6B. For example, the feature map is reconstructed from a component V of subchannel 0, a component Y of subchannel 1, a component U of subchannel 1, a component V of subchannel 1, ..., a component Y of subchannel 21, a component U of subchannel 21, and a component V of subchannel 21.

[0117] When the subchannels are assigned to multiple 4:2:0 format images as shown in FIG. 8(b), In this case, the inverse channel packing unit 3021 upsamples the feature maps assigned to the components U and V by a factor of two in both the horizontal and vertical directions and reconstructs them.

[0118] The inverse quantization unit 3022 inverse quantizes the quantized feature map qF, and outputs a feature map represented as a 32-bit floating-point number as a decoded feature map Fd.

[0119] The decoded feature map Fd can be expressed as follows:

[0120] Fd = (qF - Offset) * Scale Here, each parameter is defined as follows:

[0121] Fd: Decoded feature map (32-bit floating point number) qF: quantized feature map (10-bit integer) Offset: Quantization offset value (10-bit integer) Scale: Quantization scale value The video decoding unit 303 has a function of decoding a coded stream coded in VVC / H.266, HEVC / H.265, or the like, and decodes the coded stream Te and outputs it as a decoded image Td.

[0122] The image analysis device 51 performs analysis processes such as object detection, object segmentation, and object tracking using a decoded feature map Fd obtained by decoding the coded stream Fe. When assigning a feature map to a subpicture for encoding, the header decoding unit 3031 decodes encoded data in which sps_independent_subpics_flag=1 or a specific subpicture i to which a feature map is assigned is encoded with sps_independent_subpics_flag[i]=1. That is, the header decoding unit 3031 decodes a feature map image that can be independently decoded without reference between subpictures in predictive image generation, loop filter, and the like.

[0123] As described above, because the feature map is packed and coded into multiple subchannels consisting of three components, the feature map can be efficiently coded and decoded by taking into account the correlation of each channel using coding tools such as intra prediction that uses the correlation between color components in the coding method and inter prediction that uses the same motion vector between color components. In addition, by making it possible to independently decode subpictures and performing out-of-screen padding at the subpicture boundaries, unnecessary errors between subpictures are prevented, improving prediction efficiency. In addition, the channels of each feature map assigned to a subpicture can be decoded in parallel.

[0124] (Configuration of an image encoding device according to the second embodiment) FIG. 12 is a functional block diagram showing a schematic configuration of a video encoding device 11 according to the second embodiment. The video encoding device 11 of this configuration derives encoded data of an image (first encoded data, base layer) and encoded data of a feature map (second encoded data, enhancement layer) and outputs them as two encoded data. This configuration utilizes so-called hierarchical encoding to derive a difference value of the encoded data of a feature map using the encoded data of an image, but is characterized in that downsampling is used in deriving the encoded data of an image.

[0125] The video encoding device 11 includes a feature map extraction unit 101, a feature map conversion unit 102, a video encoding unit 103, a video encoding unit 104, a feature map extraction unit 105, a subtraction unit 106, a downsampling unit 107, and and an upsampling unit 108.

[0126] The same functional blocks as those in the first embodiment are given the same reference numerals and the description thereof will be omitted. do.

[0127] The video encoding device 11 according to the first embodiment includes a feature map extraction unit 105, a subtraction unit 106, 1 in that it includes a downsampling unit 107 and an upsampling unit 108.

[0128] The downsampling unit 107 downsamples the image T and outputs it.

[0129] The upsampling unit 108 upsamples the locally decoded image output from the video encoding unit 104 and outputs the result.

[0130] The feature map extraction unit 105 inputs the locally decoded image output from the upsampling unit 108, and outputs the base feature map FdBase, similarly to the feature map extraction unit 101.

[0131] The subtraction unit 106 calculates a difference feature map FdResi which is a difference between the feature map of the original image input from the feature map extraction unit 101 and the feature map of the locally decoded image input from the feature map extraction unit 105. Output.

[0132] As described above, the amount of information in the coding stream Te required for coding the feature map can be reduced by coding the first coded data by downsampling the original image, and coding the difference between the feature map of the original image and the feature map obtained from the image obtained by upsampling the locally decoded image of the coded image, as the second coded data. (Configuration of an image decoding device according to the second embodiment) FIG. 13 is a functional block diagram showing a schematic configuration of a video decoding device 31 according to the second embodiment. The video decoding device 31 of this configuration decodes the coded data of the image (first coded data) and the coded data of the feature map coded as a difference value (second coded data) to derive the feature map. Here, the video decoding device 31 is characterized in that an upsampled image of the decoded image is used.

[0133] The video decoding device 31 includes a video decoding unit 301, a feature map inverse conversion unit 302, a video decoding unit 303, It includes a feature map extraction unit 304 , an addition unit 305 , and an upsampling unit 306 .

[0134] The same functional blocks as those in the first embodiment are given the same reference numerals and the description thereof will be omitted. do.

[0135] The video decoding device 31 according to the first embodiment includes a feature map extraction unit 304, an addition unit 305, and The difference is that an upsampling unit 306 is provided.

[0136] The upsampling unit 306 upsamples the decoded image output from the video decoding unit 303 and outputs it as a decoded image Td.

[0137] The feature map extraction unit 304 receives the decoded image Td output from the upsampling unit 306, and outputs the base feature map FdBase in the same manner as the feature map extraction unit 101.

[0138] The adder 305 adds the differential feature map FdResi input from the feature map inverse transformer 302 and the feature map FdBase of the decoded image input from the feature map extractor 304, and outputs a decoded feature map Fd.

[0139] The image analysis device 51 performs analysis processes such as object detection, object segmentation, and object tracking using a decoded feature map Fd obtained by decoding the coded streams Te and Fe.

[0140] As described above, by decoding the first encoded data to obtain a downsampled image, and then upsampling the decoded image to obtain a feature map, the difference between the feature map obtained from this image and the feature map obtained by decoding the second encoded data is added to derive a feature map, thereby making it possible to reduce the amount of information in the encoded stream Te required to generate the decoded feature map Fd.

[0141] (Configuration of an image encoding device according to a third embodiment) FIG. 14 is a functional block diagram showing a schematic configuration of a video encoding device 11 according to the third embodiment. be.

[0142] The video encoding device 11 includes a feature map extraction unit 101, a feature map conversion unit 109, a video encoding unit 103, a video encoding unit 104, a feature map extraction unit 105, a subtraction unit 106, a downsampling unit 107, and and an upsampling unit 108.

[0143] The feature map conversion unit 109 includes a conversion unit 1091, a quantization unit 1021, and a channel pack unit 1022. It is composed of:

[0144] The same functional blocks as those in the first and second embodiments are given the same reference numerals and the description thereof will be omitted.

[0145] The video encoding device 11 according to the second embodiment differs in that the feature map conversion unit 109 includes a conversion unit 1091. The conversion unit 1091 performs Principal Component Analysis (PCA) Perform conversion to reduce the dimension of the feature map. FIG. 16 is a diagram for explaining the operation of the conversion unit 1091. The conversion unit 1091 derives the average feature map F_mean (FIG. 16(a)), C2 basis vectors BV (FIG. 16(b)), and a conversion coefficient TCoeff consisting of C2 x C2 in PCA. Next Output the average feature map F_mean and C3 (<C2) basis vectors BV to the quantization unit 1021 as the feature map F_red after dimension reduction. Also, notify the moving image encoding unit 103 of the number of channels C3 after dimension reduction and the conversion coefficient of C3 x C2 corresponding to the output basis vectors.

[0146] Generally, the conversion of PCA is represented by the product of the matrix A and the input vector, and the inverse conversion of PCA is shown by the product of the transposed matrix A and the input vector. The following processing may be used.

[0147] The conversion unit 1091 performs conversion on the one-dimensional array u[] with length C2 using the conversion matrix transMatrix[][], and derives the coefficient v[] of the one-dimensional array with length C3 (C2 < C3) as the output.

[0148] v[i] = Clip3(CoeffMin, CoeffMax, Σ(transMatrix[i][j]*u[j]+64)>>7) Here, Σ is the sum from j = 0..C2 - 1. Also, the process is performed for i from 0..C3 - 1. CoeffMin and CoeffMax indicate the value range of the conversion coefficient.

[0149] The quantization unit 1021 quantizes the feature map F_red (average feature map F_mean and basis vector BV) after dimension reduction output from the conversion unit 1091, and outputs the quantized feature map qF_red (quantized average feature map qF_mean and quantized basis vector qBV).

[0150] qF_red = Offset + Round(F_red / Scale) qF_mean = Offset + Round(F_mean / Scale) qBV = Offset + Round(BV / Scale) Here, each parameter is defined as follows:

[0151] F_red: feature map after dimensionality reduction F_mean: Mean feature map BV: basis vectors qF_red: Quantized dimensionality reduced feature map qF_mean: quantized mean feature map qF_BV: quantized basis vectors Offset: Quantization offset value (10-bit integer) Scale: Quantization scale value In the above example, the mean feature map F_mean and the basis vector BV are set to the same quantization offset value, and quantization scale value, it may be quantized using a different quantization offset value and quantization scale value.

[0152] The channel packing unit 1022 packs qF_red into multiple sub-channels each consisting of three components (for example, luminance Y and chrominance U and V), and outputs the packed sub-channels.

[0153] When the number of channels of the basis vectors BV after dimension reduction is numChannelsRed, the number of subchannels output by the channel pack unit 1022, numSubChannels, can be expressed as follows.

[0154] numSubChannels = Ceil((1 + numChannelsRed) ÷ 3) FIG. 17 is a diagram for explaining an example of a channel pack.

[0155] FIG. 17(a) shows an example of assigning each channel of a feature map to multiple 4:4:4 format videos. numChannelsRed is 32. numSubChannels is numSubChannels = Ceil((1 + 32) ÷ 3) = 11.

[0156] Each channel of the feature map (F_mean, BV0, BV1, ..., BV31) may be scanned and assigned so that the order of channels is subchannel order in the outer loop and component order in the inner loop. That is, F_mean is the component Y of subchannel 0, and BV0 is the component Y of subchannel 0. Component U, BV1 is assigned to component V of sub-channel 0, BV2 is assigned to component Y of sub-channel 1, BV3 is assigned to component U of sub-channel 1, BV4 is assigned to component V of sub-channel 1, and so on.

[0157] FIG. 17(b) shows an example in which each channel of a feature map is assigned to multiple 4:2:0 format images. The channel pack unit 1022 down-converts the feature map by half in both the horizontal and vertical directions. The resulting image is sampled and assigned to the components U and V.

[0158] The video encoding unit 103 converts the characteristics including NumChannelsRed and TCoeff output from the conversion unit 1091 into The map information is coded as supplemental enhancement information SEI and output as a coded stream Fe.

[0159] FIG. 18(c) shows an example of the syntax of the feature map information. The mantics are as follows: fm_quantization_offset: Quantization offset value Offset fm_quantization_scale: Quantization scale value Scale fm_num_channels_minus1:fm_num_channels_minus1 + 1 indicates the number of channels, numChannels, in the feature map. fm_transform_flag: Flag indicating whether transformation is required fm_num_channels_red_minus1: fm_num_channels_red_minus1 + 1 indicates the number of channels, numChannelsRed, of the basis vectors BV after dimensionality reduction. fm_transform_coefficient[i][j]: (i,j) component of the transform coefficient TCoeff (32-bit floating-point number) ) Alternatively, the feature map information may be notified by the sequence parameter set SPS or the picture parameter set PPS, as in the first to third embodiments.

[0160] (Configuration of an image decoding device according to the third embodiment) FIG. 15 is a functional block diagram showing a schematic configuration of a video decoding device 31 according to the third embodiment. do.

[0161] The video decoding device 31 includes a video decoding unit 301, a feature map inverse conversion unit 307, a video decoding unit 303, The image processing unit includes a feature map extractor 304, an adder 305, an upsampling unit 306, and an upsampling unit 307. It is composed of:

[0162] The feature map inverse transform unit 307 includes an inverse channel packing unit 3021 , an inverse quantization unit 3022 , and an inverse transform unit 3071 .

[0163] The same functional blocks as those in the first and second embodiments are given the same reference numerals and the description thereof will be omitted.

[0164] The video decoding device 31 according to the second embodiment differs in that the feature map inverse transform unit 307 includes an inverse transform unit 3071 .

[0165] The video decoding unit 301 decodes the feature map additional enhancement information SEI included in the coded stream Fe, derives Offset, Scale, numChannels, a flag indicating whether conversion is required, numChannelsRed, and conversion coefficients, and notifies the feature map inverse conversion unit 302.

[0166] Offset = fm_quantization_offset Scale = fm_quantization_scale numChannels = fm_num_channels_minus1 + 1 numChannelsRed = fm_num_channels_red_minus1 + 1 TCoeff[][] = fm_transform_coefficient[][] Alternatively, similarly to the first to third embodiments, a sequence parameter set SPS, or These values ​​are obtained by decoding the above syntax in the picture parameter set PPS. It may be derived.

[0167] The inverse channel packing unit 3021 reconstructs a feature map from multiple sub-channels consisting of three components (eg, luma Y and chroma U, V).

[0168] When the quantized feature maps are assigned to multiple 4:4:4 format images shown in Fig. 17(a), the average feature map shown in Fig. 16(a) and the 32 channel feature map are obtained from each component. For example, the component Y of subchannel 0 is , Subchannel 0 component U, Subchannel 0 component V, Subchannel 1 component Y, Subchannel 1 component U, Subchannel 1 component Y From the components V of subchannels V, ..., 10, we reconstruct the average feature map and basis vectors.

[0169] When quantized feature maps are assigned to multiple 4:2:0 format images as shown in FIG. 17(b), the feature maps assigned to the components U and V are upsampled by a factor of 2 in both the horizontal and vertical directions and reconstructed.

[0170] The inverse quantization unit 3022 inverse quantizes qF_red, and outputs Fd_mean and BVd.

[0171] Fd_red = (qF_red - Offset) * Scale Fd_mean = (qF_mean - Offset) * Scale BVd = (qBV - Offset) * Scale Here, each parameter is defined as follows:

[0172] Fd_red: Decoded feature map before dimension recovery (32-bit floating point number) Fd_mean: the decoded mean feature map BVd: Decoded basis vectors qF_red: quantized feature map before dimension recovery (10-bit integer) qF_mean: quantized mean feature map qBV: quantized basis vectors Offset: Quantization offset value (10-bit integer) Scale: Quantization scale value The inverse transform unit 3071 uses Fd_red and TCoeff to perform an inverse transform of the principal component analysis, restores the dimensions of the feature map, and outputs a decoded feature map Fd.

[0173] The image analysis device 51 performs analysis processes such as object detection, object segmentation, and object tracking using a decoded feature map Fd obtained by decoding the coded streams Te and Fe.

[0174] As described above, the dimensions of the feature map are reduced by using principal component analysis for encoding and decoding, so the amount of information required for encoding and decoding the feature map can be reduced without degrading the performance of the image analysis process.

[0175] In addition, a part of the video encoding device 11 and the video decoding device 31 in the above-mentioned embodiment, for example, the feature map extraction unit 101, the feature map conversion unit 102, the video encoding unit 103, the video encoding unit 104, the feature map extraction unit 105, the subtraction unit 106, the downsampling unit 107, the upsampling unit 108, the video decoding unit 301, the feature map inverse conversion unit 302, the video decoding unit 303, the feature map extraction unit 304, the addition unit 305, and the upsampling unit 306 may be realized by a computer. In this case, a program for realizing this control function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into and executed by a computer system. In addition, the "computer system" referred to here is a computer system built into either the video encoding device 11 or the video decoding device 31, and includes hardware such as an OS and peripheral devices. In addition, "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" may also include devices that dynamically hold a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and devices that hold a program for a certain period of time, such as volatile memory inside a computer system that serves as a server or client in such cases. Furthermore, the above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0176] In addition, a part or the whole of the video encoding device 11 and the video decoding device 31 in the above-mentioned embodiment may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the video encoding device 11 and the video decoding device 31 may be individually made into a processor, or a part or the whole may be integrated into a processor. The integrated circuit method is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. Furthermore, when an integrated circuit technology that replaces LSI appears due to the progress of semiconductor technology, an integrated circuit based on that technology may be used.

[0177] Although one embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention.

[0178] [Application example] The above-mentioned video encoding device 11 and video decoding device 31 can be mounted on various devices that transmit, receive, record, and play back video. The moving images may be natural moving images captured by a camera or artificial moving images (including CG and GUI) generated by a computer or the like.

[0179] First, it will be described with reference to FIG. 2 that the above-mentioned video encoding device 11 and video decoding device 31 can be used for transmitting and receiving videos.

[0180] FIG. 2 is a block diagram showing the configuration of a transmission device PROD_A equipped with a video encoding device 11. As shown in the figure, the transmitting device PROD_A includes an encoding unit PROD_A1 that obtains encoded data by encoding a moving image, a modulation unit PROD_A2 that obtains a modulated signal by modulating a carrier wave with the encoded data obtained by the encoding unit PROD_A1, and a transmitting unit PROD_A3 that transmits the modulated signal obtained by the modulation unit PROD_A2. The above-mentioned moving image encoding device 11 is used as this encoding unit PROD_A1.

[0181] The transmission device PROD_A captures moving images as a supply source of moving images to be input to the encoding unit PROD_A1. The apparatus further includes a camera PROD_A4 for recording moving images, a recording medium PROD_A5 for recording moving images, an input terminal PROD_A6 for inputting moving images from the outside, and an image processing unit A7 for generating or processing images. In the figure, a configuration in which the transmitting device PROD_A includes all of these components is illustrated, but some of them may be omitted.

[0182] The recording medium PROD_A5 may be one in which unencoded moving images are recorded. Alternatively, the recording medium PROD_A5 may be a recording medium that has recorded thereon a moving image encoded by a recording encoding method different from the transmission encoding method. In the latter case, a decoding unit ( It is advisable to use a conductor (not shown) between the two.

[0183] FIG. 2 is a block diagram showing the configuration of a receiving device PROD_B equipped with a video decoding device 31. FIG. 1 shows a receiving device PROD_B. As shown in the figure, the receiving device PROD_B includes a receiving unit PROD_B1 that receives a modulated signal, a demodulation unit PROD_B2 that obtains coded data by demodulating the modulated signal received by the receiving unit PROD_B1, and a decoding unit PROD_B3 that obtains a moving image by decoding the coded data obtained by the demodulation unit PROD_B2. The above-mentioned moving image decoding device 31 is used as this decoding unit PROD_B3.

[0184] The receiving device PROD_B is a supply destination of the video output by the decoding unit PROD_B3, and displays the video. The receiving device PROD_B may further include a display PROD_B4 for recording a moving image, a recording medium PROD_B5 for recording the moving image, an output terminal PROD_B6 for outputting the moving image to the outside, and an analysis unit PROD_B7 for analyzing the moving image. In the figure, the receiving device PROD_B is illustrated as having all of these components, but some of them may be omitted.

[0185] The recording medium PROD_B5 is for recording unencoded moving images. Alternatively, the image may be encoded by a coding method for recording that is different from the coding method for transmission. In the latter case, a signal from the decoding unit PROD_B3 to the recording medium PROD_B5 may be provided between the decoding unit PROD_B3 and the recording medium PROD_B5. It is preferable to interpose an encoding unit (not shown) that encodes the acquired video image according to an encoding method for recording.

[0186] The transmission medium for transmitting the modulated signal may be wireless or wired. The transmission mode for transmitting the modulated signal may be broadcast (here, this refers to a transmission mode in which the destination is not specified in advance) or communication (here, this refers to a transmission mode in which the destination is specified in advance). In other words, the transmission of the modulated signal may be realized by any of wireless broadcast, wired broadcast, wireless communication, and wired communication.

[0187] For example, a broadcasting station (such as a broadcasting facility) / receiving station (such as a television receiver) for terrestrial digital broadcasting is an example of a transmitting device PROD_A / receiving device PROD_B that transmits and receives modulated signals by wireless broadcasting. Also, a broadcasting station (such as a broadcasting facility) / receiving station (such as a television receiver) for cable television broadcasting is an example of a transmitting device PROD_A / receiving device PROD_B that transmits and receives modulated signals by cable broadcasting.

[0188] Also, a server (such as a workstation) / client (such as a television receiver, a personal computer, a smartphone, etc.) of a VOD (Video On Demand) service or a video sharing service using the Internet is an example of a transmitting device PROD_A / receiving device PROD_B that transmits and receives modulated signals by communication (usually, in a LAN, either wireless or wired is used as a transmission medium, and in a WAN, wired is used as a transmission medium). Here, personal computers include desktop PCs, laptop PCs, and tablet PCs. Also, smartphones include multi-function mobile phone terminals.

[0189] The client of the video hosting service has a function to decode the encoded data downloaded from the server and display it on a display, as well as a function to encode the video captured by the camera and upload it to the server. That is, the client of the video hosting service functions as both the transmitting device PROD_A and the receiving device PROD_B.

[0190] Next, it will be described with reference to FIG. 3 that the above-mentioned video encoding device 11 and video decoding device 31 can be used for recording and reproducing videos.

[0191] FIG. 3 is a block diagram showing the configuration of a recording device PROD_C equipped with the above-mentioned video encoding device 11. As shown in the figure, the recording device PROD_C includes an encoding unit PROD_C1 that encodes a moving image to obtain encoded data, and a writing unit PROD_C2 that writes the encoded data obtained by the encoding unit PROD_C1 onto a recording medium PROD_M. is used as this encoding unit PROD_C1.

[0192] The recording medium PROD_M may be (1) a type built into the recording device PROD_C, such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), (2) a type connected to the recording device PROD_C, such as an SD memory card or USB (Universal Serial Bus) flash memory, or (3) a type loaded into a drive device (not shown) built into the recording device PROD_C, such as a DVD (Digital Versatile Disc: registered trademark) or BD (Blu-ray Disc: registered trademark).

[0193] The recording device PROD_C supplies a video image to the encoding unit PROD_C1. The recording device PROD_C may further include a camera PROD_C3 for capturing an image, an input terminal PROD_C4 for inputting a moving image from the outside, a receiving unit PROD_C5 for receiving the moving image, and an image processing unit PROD_C6 for generating or processing an image. In the figure, the recording device PROD_C is illustrated as having all of these components, but some of them may be omitted.

[0194] The receiving unit PROD_C5 may receive uncoded video. Alternatively, the receiving unit PROD_C5 may receive encoded data that has been encoded by a transmission encoding method different from the recording encoding method. In the latter case, a transmission decoding unit (not shown) that decodes the encoded data that has been encoded by the transmission encoding method may be provided between the receiving unit PROD_C5 and the encoding unit PROD_C1.

[0195] Examples of such a recording device PROD_C include a DVD recorder, a BD recorder, and an HDD (Hard Disk Drive) recorder (in this case, the input terminal PROD_C4 or the receiving unit PROD_C5 is the main source of the moving image). Other examples of such a recording device PROD_C include a camcorder (in this case, the camera PROD_C3 is the main source of the moving image), a personal computer (in this case, the receiving unit PROD_C5 or the image processing unit C6 is the main source of the moving image), and a smartphone (in this case, the camera PROD_C3 or the receiving unit PROD_C5 is the main source of the moving image).

[0196] FIG. 3 shows the configuration of a playback device PROD_D equipped with the above-mentioned video decoding device 31. A block diagram is shown. As shown in the figure, the playback device PROD_D includes a reading unit PROD_D1 that reads out coded data written on a recording medium PROD_M, and a decoding unit PROD_D2 that obtains a video by decoding the coded data read by the reading unit PROD_D1. The above-mentioned video decoding device 31 is used as this decoding unit PROD_D2.

[0197] The recording medium PROD_M may be (1) a type built into the playback device PROD_D, such as an HDD or SSD, or (2) a type of storage device, such as an SD memory card or a USB flash memory. (3) DVD, BD, etc. The DVD-ROM 10 may be loaded into a drive device (not shown) built into the playback device PROD_D, as shown in FIG.

[0198] In addition, the playback device PROD_D receives the video output from the decoding unit PROD_D2 and outputs the video to the playback device PROD_D. The playback device PROD_D may further include a display PROD_D3 for displaying the video, an output terminal PROD_D4 for outputting the video to the outside, a transmission unit PROD_D5 for transmitting the video, and an analysis unit PROD_D6 for analyzing the video. In the figure, the playback device PROD_D is illustrated as having all of these components, but some of them may be omitted.

[0199] The transmission unit PROD_D5 may transmit uncoded video. Alternatively, the decoder unit PROD_D2 may transmit encoded data encoded by a transmission encoding method different from the recording encoding method. In the latter case, it is preferable to interpose an encoding unit (not shown) between the decoder unit PROD_D2 and the transmitter unit PROD_D5, which encodes the video image by the transmission encoding method.

[0200] Examples of such a playback device PROD_D include a DVD player, a BD player, and a HDD player (in this case, the output terminal PROD_D4 to which a television receiver or the like is connected operates). In this case, the display PROD_D3 or the transmitting unit PROD_D5 is the main destination of the moving images. Other examples of the playback device PROD_D include a television receiver (in this case, the display PROD_D3 is the main destination of the moving images), a digital signage (also called an electronic billboard or an electronic bulletin board, and the display PROD_D3 or the transmitting unit PROD_D5 is the main destination of the moving images), a desktop PC (in this case, the output terminal PROD_D4 or the transmitting unit PROD_D5 is the main destination of the moving images), a laptop or tablet PC (in this case, the display PROD_D3 or the transmitting unit PROD_D5 is the main destination of the moving images), and a smartphone (in this case, the display PROD_D3 or the transmitting unit PROD_D5 is the main destination of the moving images).

[0201] (Hardware and Software Realizations) Furthermore, each block of the above-mentioned video decoding device 31 and video encoding device 11 may be realized in hardware by a logic circuit formed on an integrated circuit (IC chip), or may be realized by a CPU. This may be realized in software using a Central Processing Unit (Central Processing Unit).

[0202] In the latter case, each of the above devices includes a CPU that executes the instructions of a program that realizes each function, ROM (Read Only Memory) stores the program, and RAM (Random Access Memory) expands the program. The object of the embodiment of the present invention is to provide a computer-readable storage device (recording medium) for storing the program code (executable program, intermediate code program, source program) of the control program of each of the above devices, which is software for realizing the above-mentioned functions. This can also be achieved by supplying the recording medium to each of the above-mentioned devices and having the computer (or CPU or MPU) read and execute the program code recorded on the recording medium.

[0203] Examples of the recording medium that can be used include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy disks (registered trademark) and hard disks, disks including optical disks such as CD-ROMs (Compact Disc Read-Only Memory), MO disks (Magneto-Optical discs), MDs (Mini Discs), DVDs (Digital Versatile Discs: registered trademark), CD-Rs (CD Recordable), and Blu-ray Discs (registered trademark), cards such as IC cards (including memory cards) and optical cards, semiconductor memories such as mask ROMs, EPROMs (Erasable Programmable Read-Only Memory), EEPROMs (Electrically Erasable and Programmable Read-Only Memory: registered trademark), and flash ROMs, and logic circuits such as PLDs (Programmable logic devices) and FPGAs (Field Programmable Gate Arrays).

[0204] Moreover, each of the above devices may be configured to be connectable to a communication network, and the above program code may be supplied via the communication network. This communication network is not particularly limited as long as it is capable of transmitting the program code. For example, the Internet, an intranet, an extranet, a LAN (Local Area Network), an ISDN (Integrated Services Digital Network), a VAN (Value-Added Network), a CATV (Community Antenna television / Cable Television) communication network, a virtual private network, a telephone line network, a mobile communication network, a satellite communication network, etc. may be used. Moreover, the transmission medium constituting this communication network is not limited to a specific configuration or type as long as it is a medium capable of transmitting the program code. For example, it can be used in wired communication such as IEEE (Institute of Electrical and Electronic Engineers) 1394, USB, power line carrier, cable TV line, telephone line, ADSL (Asymmetric Digital Subscriber Line) line, etc., or in wireless communication such as infrared such as IrDA (Infrared Data Association) or remote control, Bluetooth (registered trademark), IEEE802.11 wireless, HDR (High Data Rate), NFC (Near Field Communication), DLNA (Digital Living Network Alliance: registered trademark), mobile phone network, satellite line, terrestrial digital broadcasting network, etc. Note that the embodiment of the present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

[0205] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the claims. In other words, the technical scope of the present invention also includes embodiments obtained by combining technical means that are appropriately modified within the scope of the claims. [Industrial Applicability]

[0206] The embodiments of the present invention can be suitably applied to a video decoding device that decodes coded data in which image data is coded, and a video coding device that generates coded data in which image data is coded, and can also be suitably applied to the data structure of coded data that is generated by a video coding device and referenced by the video decoding device. [Explanation of symbols]

[0207] 11 Image encoding device 101 Feature map extraction unit 102 Feature map conversion unit 1021 Quantization section 1022 Channel Pack Part 103 Video Encoding Unit 104 Video Encoding Unit 105 Feature map extraction unit 106 Subtraction section 107 Downsampling section 108 Upsampling section 109 Feature map conversion unit 1091 Conversion Unit 21 Network 31 Image Decoding Device 301 Video Decoding Unit 302 Feature map inverse transformation unit 3021 Reverse channel pack part 3022 Inverse quantization section 303 Video Decoding Unit 304 Feature map extraction unit 305 Addition section 306 Upsampling section 307 Feature map inverse transformation unit 3071 Reverse conversion unit 41 Image display device 51 Image analysis device

Claims

1. 1. A video encoding device for encoding a feature map, comprising: a quantization unit for quantizing the feature map; and a channel packing unit for packing the quantized feature map into a plurality of sub-channels each consisting of three components; a first video encoding unit that encodes the sub-channel; The video encoding device, wherein the first video encoding section encodes a quantization offset value and / or a quantization scale value used in the quantization section.

2. Further, a second video encoding unit is provided to encode the image, 2. The video encoding device according to claim 1, wherein the feature map is differential data between the feature map of the image and the feature map of the locally decoded image of the second video encoding unit.

3. 3. The video encoding device according to claim 2, characterized in that the feature map is differential data between a feature map of the image and a feature map obtained by upsampling a feature map of a locally decoded image obtained by downsampling the image and encoding it by the second video encoding unit.

4. 4. The video encoding device according to claim 1, further comprising a conversion unit that reduces a dimension of the feature map.

5. 1. A video decoding device that decodes a feature map from an encoded stream of the feature map, a first video decoding unit that decodes a plurality of sub-channels, each of which is composed of three components, from the encoded stream; an inverse channel packing unit for reconstructing a feature map from the sub-channels; an inverse quantization unit for inverse quantizing the reconstructed feature map; The video decoding device, wherein the first video decoding unit decodes a quantization offset value and / or a quantization scale value used in the inverse quantization unit.

6. The video decoding device according to claim 5, further comprising a second video decoding unit that decodes an image from an encoded stream of images, wherein the feature map is an additive data of the inverse quantized feature map and a feature map of an image decoded by the second video decoding unit.

7. The video decoding device according to claim 6, characterized in that the feature map is an addition data of the inverse quantized feature map and a feature map obtained by upsampling a feature map of an image decoded by the second video decoding unit.

8. 8. The video decoding device according to claim 5, further comprising an inverse transform unit that restores the dimensions of the feature map.

Images