Video encoding device, video decoding device, video encoding method, and video decoding method

Abstract

This video encoding device includes a neural network, a quantization means, and an entropy encoding means. The video encoding device also includes a grouping means for grouping quantized values output by the quantization means into sub-blocks and makes it possible to perform, with low latency, parallel processing of entropy encoding of quantized feature vectors using a multi-resolution representation.

Description

Video encoding device, video decoding device, video encoding method, and video decoding method

[0001] The present disclosure relates to a video encoding device, a video decoding device, a video encoding method, and a video decoding method.

[0002] In order to transmit or record video efficiently, a video encoding device is used to generate a coded representation (hereinafter referred to as a bitstream) of input video, and a video decoding device is used to decode the bitstream to generate decoded video.

[0003] [Video Coding Based on Predictive Coding in Coding Units] Video coding standards include H.264 / AVC (Advanced Video Coding), H.265 / HEVC (High-Efficiency Video Coding), and H.266 / VVC (Versatile Video Coding), which are standardized by ITU-T SG16 and ISO / IEC / SC29. Another recent video coding technology is the technology described in Non-Patent Document 1.

[0004] In these video coding methods, video data is coded and decoded while being managed in a hierarchical structure. The hierarchical structure is made up of, for example, pictures that make up the video data, slices (or tiles) obtained by dividing pictures, coding tree units (CTUs) obtained by dividing slices, and coding units (CUs) obtained by dividing coding tree units.

[0005] The input image of the target CU is usually encoded earlier than the target CU and is predictively coded based on a predicted image generated based on the decoded image that has been decoded. That is, a prediction error image obtained by subtracting the predicted image from the input image is coded and decoded. Predictive coding includes intra-picture prediction (intra-prediction) that uses a decoded image included in a picture with the same display time as the target CU, and inter-picture prediction (inter-prediction) that uses a decoded image included in a picture with a different display time from the target CU.

[0006] The prediction error image is encoded based on frequency transform, quantization, and entropy coding. The prediction error image is decoded based on entropy decoding, inverse quantization, and inverse frequency transform. The frequency transform value of the quantized prediction error image is called a quantized value.

[0007] [Video Coding Based on Neural Networks] Non-Patent Document 2 describes a new video coding technique that combines an auto-encoder, which is a type of neural network, quantization, and entropy coding.

[0008] An autoencoder compresses input data into a low-dimensional feature vector that contains only important features. The autoencoder then generates reconstructed data by reconstructing the low-dimensional feature vector back to its original dimensions. Figure 1 is an explanatory diagram showing the autoencoder algorithm. In Figure 1, the circular parts are called nodes and the arrows are called edges. The process of reducing the data into a low-dimensional feature vector (the first half) is called encoding. The process of generating reconstructed data (the second half) is called decoding.

[0009] The autoencoder is trained to minimize the reconstruction error (the difference between the input data and the reconstructed data). To obtain meaningful features, the autoencoder is designed to impose constraints on the encoding structure and to add regularization terms to the network's loss function.

[0010] Non-Patent Document 3 discloses a method (MSPSM (Multi-Scale Progressive Statistical Model) image compression) for entropy encoding hidden layers (quantized feature vectors, i.e., quantized feature vectors) using a suitable probability distribution. In MSPSM image compression, as shown in FIG. 2, a video encoding device represents quantized feature vectors at multiple resolutions and performs entropy encoding starting from a coarser resolution. If the target resolution is i (i = 0, 1, 2, 3, or 4 in the example shown in FIG. 2), when performing entropy encoding processing for resolution i, the entropy encoding result for the immediately preceding resolution i+1 can be used. A video decoding device can also use the entropy encoding result for the immediately preceding resolution i+1. The video decoding device can then obtain the probability distribution of the quantized feature vectors for the target resolution i. Note that when i = 4, the video encoding device and video decoding device cannot use the result for resolution i+1.

[0011] 2 corresponds to an element (feature) in the quantized feature vector. Markings inside the small rectangles are provided to make it easier to understand the correspondence between elements at different resolutions.

[0012] Specifically, the video encoding device performs entropy encoding processing on elements of coarse resolutions for each of a plurality of resolutions. For example, the quantized feature vectors of resolution r (r=0 to 4) shown in FIG. 2 are expressed as y ｒ In this case, the video encoding device ４ , y ３ , y ２ , y １ , y ０ The process is performed in the order of y. The quantized feature vector of resolution r is obtained by downsampling the quantized feature vector of resolution (r-1). ｒ A probability model is estimated from the quantized feature vectors and context information of p. For example, ｒ is the quantized feature vector and context information z ｒ＋１The video encoding device converts the quantized feature vector of the target resolution i into a probability model p estimated from the quantized feature vector of the resolution (i+1). ｉ＋１ Entropy encode using

[0013] "Algorithm description of Enhanced Compression Model 9(ECM 9)", JVET-AD2025, JVET of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 30th Meeting, Antalya, TR, 21-28 April 2023J. Ball'e, V. Laparra, and EP Simoncelli, "End-to-end Optimized Image Compression", published as a conference paper at ICLR 2017H. Zhang et al, "Lossless image compression using a multi-scale progressive statistical mode", published ACCV 2020 LNCS version

[0014] Non-Patent Document 3 proposes that, in order to parallelize processing, each resolution block is divided into multiple sub-blocks and then entropy-encoded. FIG. 3 is an explanatory diagram showing an example of processing in units of sub-blocks. FIG. 3 shows an example in which an 8×16 (8×16 samples (elements)) quantized feature vector is divided into sixteen 2×4 (2×4 samples) sub-blocks. Note that in FIG. 3, hatched rectangles indicate that entropy encoding has already been performed on the next smaller resolution (the processing result for the resolution i+1 immediately preceding the resolution i to be processed).

[0015] However, as shown in Figure 4, video encoding devices and video decoding devices generally process quantized feature vectors of a target resolution i simply in a sample (element) order from top left to bottom right. This poses a problem in parallel processing in 2x4 subblock units, which involves waiting for processing. Note that in Figure 4, W_tensor(i) and H_tensor(i) represent the horizontal width (size in the row direction) and vertical width (size in the column direction) of the quantized values ​​of a target resolution i. Processing in a top left to bottom right order means that each sample is processed in the row direction, and once processing of the samples in that row is completed, each sample in the next row is processed in the same manner as the previous row.

[0016] FIG. 5 is an explanatory diagram for explaining the problem. FIG. 5 shows an example in which a 4×8 quantized feature vector to be processed (W_tensor(i)=8, H_tensor(i)=4) is divided into four 2×4 sub-blocks. The numbers in FIG. 5 indicate the processing order of the entropy encoding process. Note that in FIG. 5, hatched rectangles indicate that entropy encoding has already been performed for a resolution one level smaller (the processing result for the resolution i+1 immediately preceding the resolution i to be processed: see FIG. 2).

[0017] As described above, since processing is generally performed on a quantized feature vector sample by sample in order from the top left to the bottom right, for example, if we focus on the top left sub-block, processing of the top left sub-block is not completed until processing of the top four samples in the top right sub-block is completed. Specifically, processing of the sample numbered 8 cannot begin until processing of the samples numbered 4, 5, 6, and 7 is completed. In other words, despite the aim of dividing the quantized feature vector into sub-blocks to enable parallel processing for high speed, a wait time occurs until processing of the sub-blocks is completed.

[0018] The present invention aims to provide a video encoding device, a video decoding device, a video encoding method, and a video decoding method that enable entropy coding of quantized feature vectors using multi-resolution representation to be performed in parallel with low delay.

[0019] A video encoding device according to the present disclosure is a video encoding device including a neural network, a quantization means, and an entropy encoding means, and further including a grouping means for grouping quantized values ​​output by the quantization means into sub-blocks.

[0020] A video decoding device based on the present disclosure is a video decoding device comprising a neural network, an inverse quantization means, and an entropy decoding means, and further including an inverse grouping means for inverse grouping the quantized values ​​of sub-blocks output by the entropy decoding means.

[0021] A video coding method according to the present disclosure performs a neural network-based encoding process, a quantization process, and an entropy coding process, and groups quantized values ​​produced by the quantization process into sub-blocks.

[0022] A video decoding method according to the present disclosure is a video decoding method that performs a neural network-based decoding process, an inverse quantization process, and an entropy decoding process, and inversely groups the quantized values ​​of sub-blocks obtained by the entropy decoding process.

[0023] A video encoding program according to the present disclosure causes a computer to perform a neural network-based encoding process, a quantization process, and entropy coding, and to group quantized values ​​produced in the quantization process into sub-blocks.

[0024] A video decoding program according to the present disclosure causes a computer to perform a neural network-based decoding process, an inverse quantization process, and an entropy decoding process, and to inversely group the quantized values ​​of sub-blocks obtained by the entropy decoding process.

[0025] According to the present invention, entropy coding of quantized feature vectors using multi-resolution representation can be performed in parallel with low delay.

[0026] FIG. 1 is an explanatory diagram showing an algorithm of an autoencoder. FIG. 2 is an explanatory diagram showing examples of quantized feature vectors for each resolution. FIG. 3 is an explanatory diagram showing an example of processing in units of sub-blocks. FIG. 4 is an explanatory diagram for explaining the processing order for quantized feature vectors. FIG. 5 is an explanatory diagram for explaining a problem. FIG. 6 is a block diagram showing a video encoder and a video decoder that encode and decode each picture constituting video data based on a neural network. FIG. 7 is an explanatory diagram showing an example of rearrangement in an NN video encoder and an example of reverse rearrangement in an NN video decoder. FIG. 8 is an explanatory diagram showing an example of processing order in a sub-block. FIG. 9 is a flowchart showing an example of operation of an NN video encoder. FIG. 10 is a flowchart showing an example of operation of an NN video decoder. FIG. 11 is a block diagram showing an example of the configuration of a video encoder that is a block diagram showing example configurations of an encoder and a decoder. FIG. 12 is a block diagram showing an example of the configuration of a video decoder. FIG. 13 is an explanatory diagram showing another example of the processing order in a sub-block. FIG. 14 is a block diagram showing an example of the configuration of an information processing system. FIG. 15 is a block diagram showing main parts of a video encoding device. FIG. 16 is a block diagram showing main parts of a video decoding device.

[0027] Hereinafter, an embodiment will be described with reference to the drawings.

[0028] Embodiment 1. Figure 6 is a block diagram showing a video encoder 101 and a video decoder 201 that encode and decode each picture constituting video data based on a neural network. Hereinafter, the video encoder 101 will be referred to as the NN video encoder 101, and the video decoder 201 will be referred to as the NN video decoder 201.

[0029] The NN video encoder 101 includes an encoder 1001 , a quantizer 1002 , a reordering unit 1003 , an entropy encoder 1004 , and a multiplexer 1005 .

[0030] The NN video decoder 201 includes a decoder 2001 , an inverse quantizer 2002 , an inverse rearrangement unit 2003 , an entropy decoder 2004 , and a demultiplexer 2005 .

[0031] 6 simply indicates the direction of signal (data) flow, but does not exclude bidirectionality. This also applies to other block diagrams.

[0032] [Description on the Encoding Side] In the NN video encoder 101, the encoder 1001 extracts features from the image of the input picture. Specifically, the encoder 1001 obtains a feature vector from the image of the input picture.

[0033] The quantizer 1002 quantizes each element of the feature vector supplied from the encoder 1001 to obtain a quantized value (each element of the quantized feature vector).

[0034] The rearrangement unit 1003 rearranges the quantized values ​​supplied from the quantizer 1002 .

[0035] 7 is an explanatory diagram showing an example of rearrangement in the NN video encoder 101 and an example of reverse rearrangement in the NN video decoder 201. In the example shown in Fig. 7, the rearrangement unit 1003 rearranges elements (samples) of a quantized feature vector having a width W_tensor(i) and a height H_tensor(i) for each 2x4 sub-block.

[0036] In this way, each sample in the quantized feature vector is allocated in sub-block units from the top left to the bottom right. That is, each sample in the quantized feature vector is rearranged in sub-block units from the top left to the bottom right. The meaning of the expression "top left to bottom right" is as described above.

[0037] FIG. 8 is an explanatory diagram showing an example of the processing order for sub-blocks. FIG. 8 shows an example in which a 4×8 quantized feature vector (W_tensor(i)=8, H_tensor(i)=4) is divided into four 2×4 sub-blocks. The numbers in FIG. 8 indicate the processing order of the entropy coding process. Note that in FIG. 8, hatched rectangles indicate that entropy coding has already been performed for the next smaller resolution (the processing result for the resolution i+1 immediately preceding the resolution i to be processed: see FIG. 2).

[0038] The entropy encoder 1004 first performs entropy encoding on the top left sub-block of the four sub-blocks, then on the top right sub-block, then on the bottom left sub-block, and finally on the bottom right sub-block.

[0039] The entropy encoder 1004 entropy encodes the samples for each block in order from top left to bottom right.

[0040] An example of a specific operation of the rearrangement unit 1003 will be described with reference to the example of FIG.

[0041] The reordering unit 1003 temporarily stores the quantized feature vectors, i.e., a series of quantized values ​​(see the left side of FIG. 7 ), input from the quantizer 1002, in, for example, a buffer memory (not shown), and then supplies the quantized values ​​to the entropy encoder 1004 in the processing order illustrated in FIG.

[0042] By performing the above-described processing, the rearrangement unit 1003 essentially rearranges each sample in the feature vector in units of sub-blocks. Note that rearrangement unit 1003 rearranging each sample in units of sub-blocks can also be said to mean that rearrangement unit 1003 groups each sample in units of sub-blocks, or groups each sample into sub-blocks (divides each sample into a plurality of sub-blocks).

[0043] In other words, the rearrangement unit 1003 groups the feature vectors into sub-blocks and controls the entropy coding to be performed for each sub-block, because the quantized values ​​are supplied to the entropy encoder 1004 according to the processing order illustrated in Fig. 8, and thus the quantized values ​​are supplied to the entropy encoder 1004 for each sub-block.

[0044] Furthermore, by performing the above-described processing by the rearrangement unit 1003, the samples are rearranged within the sub-block from the upper left to the lower right. This is because the quantized values ​​are supplied to the entropy encoder 1004 in accordance with the processing order illustrated in Fig. 8, and thereby the entropy encoding process is performed in each sub-block from the upper left to the lower right. Note that, as will be described later, the processing order within a sub-block is not limited to the processing order illustrated in Fig. 8.

[0045] The entropy encoder 1004 entropy encodes the sorted quantized values ​​supplied from the sorting unit 1003 for each subblock, thereby obtaining entropy-coded data.

[0046] For simplicity of explanation, the entropy encoder 1004 has been described as entropy encoding each block in a cascade manner, but the entropy encoder 1004 can also perform entropy encoding on multiple blocks in parallel.

[0047] The multiplexer 1005 outputs the entropy-encoded data supplied from the entropy encoder 1004 as a bit stream (called an NN bit stream).

[0048] Next, a description will be given of the operation of the NN video encoder 101. Fig. 9 is a flowchart showing an example of the operation of the NN video encoder 101.

[0049] In the NN video encoder 101, the encoder 1001 extracts features from an input picture and generates a feature vector (step S101).

[0050] The quantizer 1002 quantizes the feature vector to generate a quantized value (step S102).

[0051] The rearrangement unit 1003 rearranges the quantized values ​​supplied from the quantizer 1002 in units of subblocks by the above-described process (step S103).

[0052] The entropy encoder 1004 entropy encodes the rearranged quantized values ​​to generate entropy-coded data (step S104).

[0053] The multiplexer 1005 outputs the entropy-encoded data as an NN bit stream (step S105).

[0054] [Description on the Decoding Side] In the NN video decoder 201, the demultiplexer 2005 demultiplexes the NN bitstream to obtain entropy coded data.

[0055] The entropy decoder 2004 entropy decodes the entropy-encoded data supplied from the demultiplexer 2005 to obtain quantized values.

[0056] The inverse rearrangement unit 2003 rearranges the quantized values ​​in sub-block units supplied from the entropy decoder 2004 in sample units. That is, the inverse rearrangement unit 2003 inversely groups the quantized values ​​of the sub-blocks output by the entropy decoder 2004. Specifically, as shown in FIG. 7 , the inverse rearrangement unit 2003 restores the order of the quantized values ​​rearranged by the rearrangement unit 1003 in the NN video encoder 101 (see the left side of FIG. 7 ). That is, the inverse rearrangement unit 2003 can restore the order of the quantized values ​​in the sub-blocks by performing, as inverse rearrangement, a process that is the reverse of the rearrangement performed by the rearrangement unit 1003.

[0057] Explaining this in more detail with reference to the examples of Figures 5 and 8, the inverse rearrangement unit 2003 rearranges the quantized values ​​input in the order illustrated in Figure 8 into the order illustrated in Figure 5. In other words, the inverse rearrangement unit 2003 outputs the quantized values ​​input from the entropy decoder 2004 to the inverse quantizer 2002 in the order illustrated in Figure 5. In other words, the inverse rearrangement unit 2003 outputs the quantized values ​​input from the entropy decoder 2004 to the inverse quantizer 2002 so that the inverse quantizer 2002 can process them in order from the top left to the bottom right (see Figure 4, etc.). Note that, if necessary, the inverse rearrangement unit 2003 temporarily stores the quantized values ​​input from the entropy decoder 2004 in, for example, a buffer memory (not shown).

[0058] The inverse quantizer 2002 inverse quantizes the sorted quantized values ​​supplied from the inverse sorting unit 2003 to obtain a reconstructed feature vector.

[0059] The decoder 2001 obtains a reconstructed image of a decoded picture (also called an NN decoded picture) from the reconstructed feature vector.

[0060] Next, a description will be given of the operation of the NN video decoder 201. Fig. 10 is a flowchart showing an example of the operation of the NN video decoder 201.

[0061] In the NN video decoder 201, the demultiplexer 2005 demultiplexes the bitstream (step S201). The demultiplexer 2005 obtains entropy-encoded data through demultiplexing.

[0062] The entropy decoder 2004 entropy decodes the entropy-encoded data to obtain quantized values ​​(step S202).

[0063] The inverse rearrangement unit 2003 rearranges the quantized values ​​supplied from the entropy decoder 2004 in subblock units (for each subblock) (step S203). That is, the inverse rearrangement unit 2003 restores the order of the quantized values ​​to their original order (see the left side of FIG. 7).

[0064] The inverse quantizer 2002 inversely quantizes the quantized value (step S204). The inverse quantizer 2002 obtains a reconstructed feature vector through the inverse quantization.

[0065] The decoder 2001 obtains a reconstructed image of the decoded picture from the reconstructed feature vector (step S205).

[0066] In this embodiment, the rearrangement unit 1003 rearranges the quantized feature vectors (specifically, the elements (samples) of the quantized feature vectors) so that entropy coding in neural network-based video coding is performed in subblock units, i.e., for each subblock. In this case, the rearrangement unit 1003 rearranges the quantized feature vectors so that processing is performed in order from top left to bottom right within each subblock. As a result, entropy coding of the quantized feature vectors can be performed in parallel with low latency.

[0067] Furthermore, the NN video decoder 201 receives a bitstream from the NN video encoder 101 having a reordering unit 1003, and the inverse reordering unit 2003 performs the reverse of the reordering performed by the reordering unit 1003. Therefore, the NN video decoder 201 can smoothly play back video based on the bitstream from the NN video encoder 101.

[0068] [Encoder and Decoder Configuration] Fig. 11 is a block diagram showing an example configuration of the encoder 1001 and decoder 2001 according to the first embodiment. In Fig. 11, a "downward arrow (↓) 2" indicates subsampling by 1 / 2 (also called pooling). An "upward arrow (↑) 2" indicates upsampling by 2.

[0069] In the example shown in Fig. 11, the encoder 1001 is composed of four residual blocks and one convolution block. Each residual block is composed of two convolutional blocks and one shortcut link. Each convolutional block is composed of one convolution layer and one activation function.

[0070] The decoder 2001 consists of four residual blocks and one PixelShuffle convolution layer. The Pixel Shuffler is a mechanism proposed as Sub-Pixel Convolution. The Pixel Shuffler rearranges input feature vectors and outputs high-resolution feature vectors.

[0071] As an activation function, a Parametric ReLU (Parametric Rectified Linear Unit) can be used, in which the output value is α times the input value when the input value is below 0 (where α is a parameter determined by learning), and the output value is the same as the input value when the input value is 0 or greater.

[0072] Furthermore, the configuration shown in FIG. 11 is just an example, and the configurations of the encoder 1001 and the decoder 2001 are not limited to the configuration shown in FIG.

[0073] Second Embodiment Fig. 12 is a block diagram showing an example of the configuration of a video encoder. Fig. 13 is a block diagram showing an example of the configuration of a video decoder.

[0074] The video encoder 100 shown in FIG. 12 includes a switch 3000, an NN video encoder 3001, an NN video decoder 3003, a CP video encoder 3002, a CP video decoder 3004, a decoded picture buffer 3005, and a multiplexer 4000 that performs multiplexing processing of entropy-encoded data and other information.

[0075] Note that "NN" stands for neural network, and "CP" stands for coding unit-based prediction.

[0076] The CP video encoder 3002 performs video encoding processing using a video encoding method based on predictive coding for each coding unit. The CP video decoder 3004 performs decoding processing using a video encoding method based on predictive coding for each coding unit. The decoded picture buffer 3005 is a storage unit that stores decoded pictures (reconstructed pictures). As described above, video encoding methods that comply with H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc. can be used.

[0077] The NN video encoder 101 in each of the above embodiments can be used as the NN video encoder 3001. The NN video decoder 201 in each of the above embodiments can be used as the NN video decoder 3003.

[0078] The video decoder 200 shown in FIG. 13 includes a demultiplexer 5000 that demultiplexes a bitstream, an NN video decoder 3003, a CP video decoder 3004, and a decoded picture buffer 3005.

[0079] In other words, the video encoder 100 shown in FIG. 12 and the video decoder 200 shown in FIG. 13 are a video encoder and a video decoder that combine the above-mentioned NN video encoder 101 as an NN video encoder 3001 and the above-mentioned NN video decoder 201 as an NN video decoder 3003 with a CP video encoder 3002 and a CP video decoder 3004 that are based on a video encoding method based on predictive encoding in the encoding unit.

[0080] [Description of the Encoding Side] In the video encoder 100 shown in FIG. 12, a switch 3000 supplies an input picture to either an NN video encoder 3001 or a CP video encoder 3002 .

[0081] The NN video encoder 3001 operates in the same manner as the NN video encoder 101 of the first embodiment to generate an NN bitstream.

[0082] The NN video decoder 3003 receives the NN bitstream supplied from the NN video encoder 3001, and operates in the same manner as the NN video decoder 201 of the first embodiment to obtain decoded pictures (NN decoded pictures). The NN video decoder 3003 stores the NN decoded pictures in the decoded picture buffer 3005.

[0083] The CP video encoder 3002 uses the input picture and the decoded picture stored in the decoded picture buffer 3005 to perform video encoding based on predictive encoding in the encoding unit, and generates a bitstream (also called a CP bitstream).

[0084] The CP video decoder 3004 receives the CP bitstream supplied from the CP video encoder 3002, performs entropy decoding, and then performs decoding based on predictive coding in the coding unit to obtain a decoded picture (also referred to as a CP decoded picture). The CP video decoder 3004 stores the CP decoded picture in the decoded picture buffer 3005.

[0085] The decoded pictures stored in the decoded picture buffer 3005 are used as reference pictures.

[0086] In the configuration shown in Fig. 12, the NN video decoder 3003 receives the NN bitstream from the NN video encoder 3001 and performs entropy decoding. The NN video decoder 3003 then obtains NN-decoded pictures from the encoded data obtained by entropy decoding. However, the NN video decoder 3003 may also be configured to receive intermediate data (e.g., quantized values) before entropy encoding from the NN video encoder 3001 and obtain NN-decoded pictures from the intermediate data. In this case, the NN video decoder 3003 does not need to perform entropy decoding.

[0087] 12 , the CP video decoder 3004 receives the CP bitstream from the CP video encoder 3002 and performs entropy decoding. The CP video decoder 3004 then obtains CP-decoded pictures from the encoded data obtained by entropy decoding. However, the CP video decoder 3004 may also be configured to receive intermediate data (e.g., quantized values) before entropy encoding from the CP video encoder 3002 and obtain CP-decoded pictures from the intermediate data. In this case, the CP video decoder 3004 does not need to perform entropy decoding.

[0088] 13, the NN video decoder 3003 operates in the same manner as the NN video decoder 201 of the first embodiment to obtain NN-decoded pictures based on the coded data obtained by demultiplexing the NN bitstream. The NN video decoder 3003 stores the NN-decoded pictures in the decoded picture buffer 3005.

[0089] The CP video decoder 3004 demultiplexes the CP bitstream, performs entropy decoding, and obtains CP decoded pictures based on the resulting coded data. The CP video decoder 3004 stores the CP decoded pictures in the decoded picture buffer 3005.

[0090] The video decoder 200 outputs the NN decoded picture or the CP decoded picture stored in the decoded picture buffer 3005 as a decoded picture.

[0091] [Variation 1] In the above embodiment, the entropy encoder 1004, under the control of the sorting unit 1003, performed entropy encoding processing on the quantized feature vectors (W_tensor(i)=8, H_tensor(i)=4) at the resolution to be processed in units of 2×4 sub-blocks in the order from top left to bottom right, and within each sub-block in the order from top left to bottom right.

[0092] In order to obtain a more sophisticated probability distribution within a sub-block, rearrangement unit 1003 may further change the processing order within the sub-block from the processing order exemplified in Fig. 8. For example, rearrangement unit 1003 may rearrange within the sub-block the processing order shown in Fig. 14 so that a probability distribution can be obtained by suitably interpolating the quantized values ​​to be subsequently processed within the sub-block from the surrounding values.

[0093] Each of the above embodiments can be configured by hardware, but can also be realized by a computer program.

[0094] The information processing system shown in Fig. 15 includes a processor 701 such as a CPU (Central Processing Unit), a program memory 702, a storage medium 703 for storing video data, and a storage medium 704 for storing a bitstream. The storage medium 703 and the storage medium 704 may be separate storage media or may be storage areas formed by the same storage medium. A magnetic storage medium such as a hard disk can be used as the storage medium. The information processing system may also include multiple processors 701.

[0095] In the information processing system, a program memory 702 stores a program (video encoding program or video decoding program) for implementing the functions of each block shown in each of the above embodiments.

[0096] The processor 701 then executes processing in accordance with the program stored in the program memory 702, thereby realizing the functions of the video encoder 100, NN video encoder 101, video decoder 200, and NN video decoder 201 shown in each embodiment.

[0097] For example, the processor 701 executes processing in accordance with a video encoding program for realizing the functions of each block (except the decoded picture buffer 3005) in the NN video encoder 101 and the video encoder 100 shown in Figures 6 and 12, thereby realizing the functions of the NN video encoder 101 and the video encoder 100.

[0098] Also, for example, the processor 701 executes processing in accordance with a video decoding program for realizing the functions of each block (except the decoded picture buffer 3005) in the NN video decoder 201 and the video decoder 200 shown in Figures 6 and 13, thereby realizing the functions of the NN video decoder 201 and the video decoder 200.

[0099] At least the program memory 702 is a non-transitory computer-readable medium. However, the program may be stored in various types of transitory computer-readable medium. The program is supplied to the transitory computer-readable medium, for example, via a wired or wireless communication channel, i.e., via an electrical signal, an optical signal, or an electromagnetic wave.

[0100] 16 is a block diagram showing the main components of a video encoding device. The video encoding device 10 shown in Fig. 16 (implemented by a video encoder 100 and an NN video encoder 101 in the embodiment) includes a neural network 11 (implemented by an encoder 1001 in the embodiment), a quantization unit 12 (implemented by a quantizer 1002 in the embodiment), an entropy encoding unit 13 (implemented by an entropy encoder 1004 in the embodiment), and a grouping unit 14 (implemented by a rearrangement unit 1003 in the embodiment) that groups the quantized values ​​output by the quantization unit 12 into sub-blocks.

[0101] 17 is a block diagram showing the main components of a video decoding device. The video decoding device 20 shown in Fig. 17 (implemented by a video decoder 200 and an NN video decoder 201 in the embodiment) includes a neural network 21 (implemented by a decoder 2001 in the embodiment), an inverse quantization unit 22 (implemented by an inverse quantizer 2002 in the embodiment), and an entropy decoding unit 23 (implemented by an entropy decoder 2004 in the embodiment), as well as a degrouping unit 24 (implemented by an inverse sorting unit 2003 in the embodiment) that degroups the quantized values ​​of the sub-blocks output by the entropy decoding unit.

[0102] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes.

[0103] (Supplementary Note 1) A video encoding device comprising a neural network, a quantization means, and an entropy encoding means, further comprising: a grouping means for grouping quantized values ​​output by the quantization means into sub-blocks.

[0104] (Supplementary Note 2) The video encoding device according to Supplementary Note 1, wherein the grouping means rearranges the quantization values ​​within the sub-block.

[0105] (Supplementary Note 3) A video decoding device comprising a neural network, an inverse quantization means, and an entropy decoding means, further comprising a degrouping means for degrouping quantized values ​​of sub-blocks output by the entropy decoding means.

[0106] (Supplementary Note 4) The video decoding device according to Supplementary Note 3, wherein the degrouping means rearranges the quantization values ​​within the sub-block.

[0107] (Supplementary Note 5) A video coding method that performs neural network-based encoding, quantization, and entropy coding, wherein quantized values ​​produced by the quantization are grouped into sub-blocks.

[0108] (Supplementary Note 6) The video encoding method according to Supplementary Note 5, wherein the quantization values ​​within the sub-block are rearranged.

[0109] (Supplementary Note 7) A video decoding method that executes a neural network-based decoding process, an inverse quantization process, and an entropy decoding process, and further comprises inverse grouping quantized values ​​of sub-blocks obtained by the entropy decoding process.

[0110] (Supplementary Note 8) The video decoding method according to Supplementary Note 7, wherein quantization values ​​within a sub-block are rearranged.

[0111] (Supplementary Note 9) A video encoding program for causing a computer to perform neural network-based encoding, quantization, and entropy coding, and for grouping quantized values ​​generated by the quantization into sub-blocks.

[0112] (Supplementary Note 10) The video encoding program according to Supplementary Note 9, which causes a computer to rearrange quantization values ​​within a sub-block.

[0113] (Supplementary Note 11) A video decoding program for causing a computer to execute a neural network-based decoding process, an inverse quantization process, and an entropy decoding process, and for inverse grouping the quantized values ​​of sub-blocks obtained by the entropy decoding process.

[0114] (Supplementary Note 12) The video decoding program according to Supplementary Note 11, which causes a computer to rearrange quantization values ​​within a sub-block.

[0115] (Supplementary Note 13) A recording medium having recorded thereon a video encoding program for causing a computer to execute encoding processing based on a neural network, quantization processing, and entropy coding, and for grouping quantized values ​​generated by the quantization processing into sub-blocks.

[0116] (Supplementary Note 14) A recording medium having recorded thereon a video decoding program for causing a computer to execute a neural network-based decoding process, an inverse quantization process, and an entropy decoding process, and for inverse grouping the quantized values ​​of sub-blocks obtained by the entropy decoding process.

[0117] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0118] This application claims priority based on Japanese Patent Application No. 2024-061862, filed on April 8, 2024, the disclosure of which is incorporated herein in its entirety.

[0119] REFERENCE SIGNS LIST 10 Video encoding device 11 Neural network 12 Quantization means 13 Entropy encoding means 14 Grouping means 20 Video decoding device 21 Neural network 22 Inverse quantization means 23 Entropy decoding means 24 Inverse grouping means 100 Video encoder 101 NN video encoder 200 Video decoder 201 NN video decoder 701 Processor 702 Program memory 703, 704 Storage medium 1001 Encoder 1002 Quantizer 1003 Rearrangement unit 1004 Entropy encoder 1005 Multiplexer 2001 Decoder 2002 Inverse quantizer 2003 Inverse rearrangement unit 2004 Entropy decoder 2005 Demultiplexer 3000 Switch 3001 NN video encoder 3002 CP video encoder 3003 NN video decoder 3004 CP video decoder 3005 Decoded picture buffer 4000 Multiplexer 5000 Demultiplexer

Claims

1. A video encoding device comprising a neural network, a quantization means, and an entropy encoding means, the video encoding device further comprising a grouping means for grouping quantized values ​​output by the quantization means into sub-blocks.

2. The video encoding device according to claim 1, wherein the grouping means rearranges the quantization values ​​within the sub-blocks.

3. A video decoding device comprising a neural network, an inverse quantization means, and an entropy decoding means, further comprising a degrouping means for degrouping the quantized values ​​of the sub-blocks output by the entropy decoding means.

4. The video decoding device according to claim 3, wherein the degrouping means rearranges the quantization values ​​within the subblock.

5. A video coding method that performs neural network based encoding, quantization, and entropy coding, wherein quantized values ​​produced by said quantization are grouped into sub-blocks.

6. The video coding method according to claim 5, wherein the quantization values ​​within the sub-block are rearranged.

7. A video decoding method that performs a neural network-based decoding process, an inverse quantization process, and an entropy decoding process, and inversely groups the quantized values ​​of sub-blocks obtained by the entropy decoding process.

8. The video decoding method according to claim 7, wherein quantization values ​​within a sub-block are rearranged.

9. A video coding program for causing a computer to perform neural network-based encoding, quantization, and entropy coding, and to group quantized values ​​produced by said quantization into sub-blocks.

10. A video decoding program for causing a computer to execute a neural network-based decoding process, an inverse quantization process, and an entropy decoding process, and for inverse grouping the quantized values ​​of sub-blocks obtained by the entropy decoding process.

Images