Video encoding apparatus, video decoding apparatus, video encoding method, and video decoding method

Abstract

The present invention relates to a video encoding apparatus, a video decoding apparatus, a video encoding method, and a video decoding method. The video encoding apparatus is a video encoding apparatus including a neural network, a quantization device, and an entropy encoding device, and including a grouping device for grouping quantization values output from the quantization device into sub-blocks, whereby low-delay parallel processing of entropy encoding of a quantization feature vector using a multi-resolution representation is enabled.

Description

Technical Field

[0001] This disclosure relates to video encoding equipment, video decoding equipment, video encoding methods, and video decoding methods. Background Technology

[0002] In order to efficiently transmit or record video, video encoding devices are used to generate an encoded representation (hereinafter referred to as a bitstream) obtained by encoding the input video, and video decoding devices are used to decode the bitstream to generate a decoded video.

[0003] Video coding based on predictive coding in coding units

[0004] Examples of video coding system standards include H.264 / AVC (Advanced Video Coding), H.265 / HEVC (High Efficiency Video Coding), and H.266 / VVC (Multi-Functional Video Coding), which are standardized by ITU-T SG16 and ISO / IEC / SC29. Furthermore, the technology described in Non-Patent Document 1 (NPL 1) is an example of recent video coding technologies.

[0005] In these video coding systems, video data is managed in a hierarchical structure and is encoded and decoded. The hierarchical structure includes, for example, the images that make up the video data, slices (or tiles) obtained by segmenting (dividing) the images, coding tree units (CTUs) obtained by segmenting the slices, and coding units (CUs) obtained by segmenting the coding tree units.

[0006] The input image to the CU being processed is typically encoded relative to the past and is predictively encoded based on a prediction image generated from the decoded image. That is, a prediction error image obtained by subtracting the prediction image from the input image is encoded and decoded. Predictive encoding includes: intra-frame prediction, which uses a decoded image included in a picture displayed at the same time as the CU being processed; and inter-frame prediction, which uses a decoded image included in a picture displayed at a different time than the CU being processed.

[0007] The prediction error image is encoded using frequency transform, quantization, and entropy coding. The prediction error image is decoded using entropy decoding, inverse quantization, and inverse frequency transform. The frequency transform value of the quantized prediction error image is called the quantized value.

[0008] [Neural Network-Based Video Coding]

[0009] NPL 2 discloses a new video coding technique that combines autoencoders, quantization, and entropy coding, which is a type of neural network.

[0010] An autoencoder compresses the input data into a low-dimensional feature vector so that only the important features are included. The autoencoder then generates reconstructed data by reconstructing the low-dimensional feature vector back to its original dimensions. Figure 1 This is an explanatory diagram illustrating the algorithm of an autoencoder. Figure 1 In this diagram, the circular parts are called nodes, and the arrows are called edges. The process of reducing the dimensionality to a low-dimensional feature vector (the first half) is called encoding. The process of generating the reconstructed data (the second half) is called decoding.

[0011] Autoencoders learn in a way that minimizes the reconstruction error (the difference between the input data and the reconstructed data). To obtain meaningful feature quantities, autoencoders are designed to impose constraints on the structure of the encoding, or to add regularization terms to the network's loss function.

[0012] NPL 3 discloses a method for entropy encoding of the hidden layer (quantized feature vector, i.e., quantized feature vector) using an appropriate probability distribution (Multi-Scale Asymptotic Statistical Model (MSPSM) image compression). In MSPSM image compression, such as... Figure 2 As shown, the video encoding device represents the quantized feature vector at multiple resolutions and performs entropy coding starting from the lowest resolution. Assume the resolution being processed is i (in... Figure 2 In the example shown, i = 0, 1, 2, 3, or 4. When performing entropy coding on resolution i, the entropy coding result of the immediately preceding resolution i+1 can be utilized. The video decoder can also utilize the entropy coding result of the immediately preceding resolution i+1. Thus, the video decoder can obtain the probability distribution of the quantized feature vector of the resolution i being processed. It should be noted that when i = 4, neither the video encoder nor the video decoder can utilize the result of resolution i+1.

[0013] Figure 2 The small rectangles in the diagram correspond to the elements (features) in the quantized feature vector. The markings inside the small rectangles are used to facilitate understanding the correspondence between elements at different resolutions.

[0014] To explain more specifically, video encoding devices perform entropy encoding on elements at lower resolutions for each of multiple resolutions. For example, when... Figure 2 The quantized feature vector of the resolution r (r=0 to 4) shown is represented as y. r At that time, the video encoding device follows the y 4 , y 3 , y 2 , y 1 and y 0The processing is performed sequentially. It should be noted that the quantized feature vector at resolution r is obtained by downsampling the quantized feature vector at resolution (r-1). The video coding device estimates the probability model based on the quantized feature vector and contextual information. For example, by quantizing the feature vector y... r and context information z r+1 Inputting into a convolutional neural network to obtain a probabilistic model p r The video encoding device uses a probabilistic model p estimated based on the quantized feature vectors at resolution (i+1). i+1 Entropy encoding is performed on the quantized feature vector of resolution i.

[0015] Reference List

[0016] Non-patent literature

[0017] NPL 1: "Algorithmic Description of Enhanced Compression Model 9 (ECM 9)", JVET-AD2025, 30th Meeting of JVET, ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29, Antalya, Turkey, April 21-28, 2023

[0018] NPL 2: J. Ball'e, V. Laparra, and EP Simoncelli, “End-to-End Optimization of Image Compression,” published as a conference paper at ICLR 2017.

[0019] NPL 3: H. Zhang et al., “Lossless Image Compression Using Multiscale Asymptotic Statistical Models”, published in ACCV 2020 LNCS version. Summary of the Invention

[0020] Technical issues

[0021] NPL 3 proposes dividing each resolution block into multiple sub-blocks and performing entropy coding on these sub-blocks to achieve parallel processing. Figure 3 This is an illustrative diagram showing an example of processing based on sub-blocks. Figure 3 An example is shown where an 8×16 (8×16 samples (elements)) quantized feature vector is divided into 16 2×4 (2×4 samples) sub-blocks. It should be noted that in... Figure 3 In the diagram, the rectangle marked with a slash indicates that entropy encoding has been performed for the next lower resolution (i.e., the processing result of resolution i+1 immediately preceding the currently being processed resolution i).

[0022] Usually, such as Figure 4As shown, the video encoder and decoder process the quantized feature vectors of resolution i on a sample-by-sample (element-by-element) basis, sequentially from top left to bottom right. Therefore, parallel processing based on 2×4 sub-blocks introduces processing latency. Figure 4 In this context, W_tensor(i) and H_tensor(i) represent the width (size in the row direction) and height (size in the column direction) of the quantized value at resolution i being processed. Furthermore, processing from top left to bottom right means that samples are processed sequentially in the row direction, and after the samples in a row are processed, the samples in the next row are processed in the same way as the previous row.

[0023] Figure 5 It is an explanatory diagram used to illustrate a problem. Figure 5 An example is shown where a 4×8 quantized feature vector (W_tensor(i)=8, H_tensor(i)=4) to be processed is divided into four 2×4 sub-blocks. Figure 5 The numbers in the text indicate the processing order of the entropy encoding process. It should be noted that in... Figure 5 In the diagram, the rectangle marked with a slash indicates that entropy encoding has been performed for the next lower resolution (i.e., the processing result of resolution i+1 immediately preceding the currently being processed resolution i: see [link]). Figure 2 ).

[0024] As mentioned above, generally speaking, the processing of quantized feature vectors is performed sequentially from top left to bottom right on a sample-by-sample basis. Therefore, for example, when focusing on the top left sub-block, the processing of the top left sub-block cannot be completed until the processing of the top four samples of the top right sub-block is finished. Specifically, the processing of sample labeled 8 cannot begin until the processing of samples labeled 4, 5, 6, and 7 is completed. In other words, although the purpose of dividing the quantized feature vector into sub-blocks is to improve speed through parallel processing, a waiting time is incurred before the processing of each sub-block is completed.

[0025] The purpose of this invention is to provide a video encoding device, a video decoding device, a video encoding method, and a video decoding method, which can perform low-latency parallel processing on entropy encoding of quantized feature vectors using multi-resolution representation.

[0026] Technical solutions

[0027] The video encoding device based on this disclosure includes a neural network, a quantization device, an entropy encoding device, and a grouping device, wherein the grouping device is used to group the quantized values ​​output from the quantization device into sub-blocks.

[0028] The video decoding device based on this disclosure includes a neural network, an inverse quantization device, an entropy decoding device, and a degrouping device, wherein the degrouping device is used to degroup the quantized values ​​of the sub-blocks output from the entropy decoding device.

[0029] The video coding method based on this disclosure performs a neural network-based coding process, a quantization process, and an entropy coding process, and includes grouping the quantized values ​​generated by the quantization process into sub-blocks.

[0030] The video decoding method based on this disclosure performs a neural network-based decoding process, an inverse quantization process, and an entropy decoding process, and includes degrouping the quantized values ​​of sub-blocks obtained through the entropy decoding process.

[0031] The video coding program based on this disclosure enables a computer to perform a neural network-based coding process, a quantization process, an entropy coding process, and to group the quantized values ​​generated by the quantization process into sub-blocks.

[0032] The video decoding program based on this disclosure enables a computer to perform a neural network-based decoding process, an inverse quantization process, an entropy decoding process, and to degroup the quantized values ​​of sub-blocks obtained through the entropy decoding process.

[0033] Beneficial effects

[0034] According to the present invention, low-latency parallel processing is possible for entropy encoding of quantized feature vectors using multi-resolution representation. Attached Figure Description

[0035] Figure 1 An illustrative diagram illustrating the algorithm of an autoencoder is shown.

[0036] Figure 2 An illustrative diagram depicts examples of quantized feature vectors at various resolutions.

[0037] Figure 3 An illustrative diagram is depicted to show an example of processing based on sub-blocks.

[0038] Figure 4 An explanatory diagram illustrating the processing order for quantizing eigenvectors is provided.

[0039] Figure 5 An explanatory diagram was drawn to illustrate the problem.

[0040] Figure 6 A block diagram illustrating a video encoder and a video decoder that encode and decode the individual images constituting video data based on a neural network is depicted.

[0041] Figure 7 An illustrative diagram is depicted illustrating an example of reordering in an NN video encoder and inverse reordering in an NN video decoder.

[0042] Figure 8 An illustrative diagram depicting an example of the processing order in a sub-block is provided.

[0043] Figure 9 A flowchart illustrating an operational example of an NN video encoder is provided.

[0044] Figure 10 A flowchart illustrating an operational example of the NN video decoder is provided.

[0045] Figure 11 A block diagram illustrating an example configuration of the encoder and decoder is provided.

[0046] Figure 12 A block diagram illustrating an example configuration of a video encoder is provided.

[0047] Figure 13 A block diagram illustrating an example configuration of a video decoder is depicted.

[0048] Figure 14 An illustrative diagram is depicted to show another example of the processing order in a sub-block.

[0049] Figure 15 A block diagram illustrating a configuration example of an information processing system is provided.

[0050] Figure 16 A block diagram illustrating the main components of a video encoding device is shown.

[0051] Figure 17 A block diagram illustrating the main components of a video decoding device is shown. Detailed Implementation

[0052] Example embodiments are described below with reference to the accompanying drawings.

[0053] First example embodiment.

[0054] Figure 6 This is a block diagram illustrating a video encoder 101 and a video decoder 201 that encode and decode the individual images constituting the video data based on a neural network. Hereinafter, the video encoder 101 will be referred to as the NN video encoder 101, and the video decoder 201 will be referred to as the NN video decoder 201.

[0055] The NN video encoder 101 includes an encoder 1001, a quantizer 1002, a reordering unit 1003, an entropy encoder 1004, and a multiplexer 1005.

[0056] The NN video decoder 201 includes a decoder 2001, an inverse quantizer 2002, an inverse reordering unit 2003, an entropy decoder 2004, and a demultiplexer 2005.

[0057] It should be noted that Figure 6 The arrows in the diagram only indicate the direction of signal (data) flow and do not preclude bidirectional flow. The same applies to other block diagrams.

[0058] [Description of the encoding side]

[0059] In the NN video encoder 101, encoder 1001 extracts features from the input image. Specifically, encoder 1001 obtains a feature vector from the input image.

[0060] The quantizer 1002 quantizes each element of the feature vector supplied from the encoder 1001 to obtain quantized values ​​(each element of the quantized feature vector).

[0061] The reordering unit 1003 reorders the quantized values ​​supplied from the quantizer 1002.

[0062] Figure 7 This is an illustrative diagram showing an example of reordering in the NN video encoder 101 and inverse reordering in the NN video decoder 201. Figure 7 In the example shown, the reordering unit 1003 reorders the elements (samples) of the quantized feature vector with a width of W_tensor(i) and a height of H_tensor(i) in 2×4 sub-blocks.

[0063] As described above, samples in the quantized feature vector are allocated sequentially from top left to bottom right, on a sub-block basis. Specifically, the samples in the quantized feature vector are reordered in order from top left to bottom right, on a sub-block basis. Note that the meaning of "in order from top left to bottom right" is as described above.

[0064] Figure 8 This is an illustrative diagram showing an example of the processing order in a sub-block. Figure 8 An example is shown where a 4×8 quantized feature vector (W_tensor(i)=8, H_tensor(i)=4) is divided into four 2×4 sub-blocks. Figure 8 The numbers in the text indicate the processing order of the entropy encoding process. It should be noted that in... Figure 8 In the diagram, the rectangle marked with a slash indicates that entropy encoding has been performed for the next lower resolution (i.e., the processing result of resolution i+1 immediately preceding the currently being processed resolution i: see [link]). Figure 2 ).

[0065] The entropy encoder 1004 first performs entropy encoding on the top-left sub-block of the four sub-blocks. Next, the entropy encoder 1004 performs entropy encoding on the top-right sub-block. Next, the entropy encoder 1004 performs entropy encoding on the bottom-left sub-block. Then, the entropy encoder 1004 performs entropy encoding on the bottom-right sub-block.

[0066] The entropy encoder 1004 performs entropy encoding on the samples for each block, in order from top left to bottom right.

[0067] Reference Figure 8 The example shown illustrates a specific operational example of the reordering unit 1003.

[0068] For example, the reordering unit 1003 will take the quantized feature vector input from the quantizer 1002, i.e., the sequence of quantized values ​​(see...). Figure 7 (on the left side), temporarily stored in a buffer memory (not shown). Then, according to... Figure 8 As shown in the processing sequence, the reordering unit 1003 supplies the quantized value to the entropy encoder 1004.

[0069] By performing the above processing by the reordering unit 1003, the samples in the feature vector are effectively reordered in units of sub-blocks. It should be noted that the fact that the reordering unit 1003 reorders the samples in units of sub-blocks can also be described as the reordering unit 1003 grouping the samples in units of sub-blocks, or grouping the samples into sub-blocks (dividing a sample into multiple sub-blocks).

[0070] In other words, the reordering unit 1003 groups the feature vectors into sub-blocks and controls the process to perform entropy encoding for each sub-block. This is because, by following... Figure 8 The processing sequence shown supplies quantized values ​​to the entropy encoder 1004, with the quantized values ​​supplied to the entropy encoder 1004 in units of sub-blocks.

[0071] Furthermore, due to the processing performed by the reordering unit 1003, the samples within each sub-block are also reordered from top left to bottom right. This is because, by following... Figure 8 The processing sequence shown supplies the quantized values ​​to the entropy encoder 1004, and entropy encoding is performed in each sub-block in order from top left to bottom right. It should be noted that, as described later, the processing order within a sub-block is not limited to... Figure 8 The order shown.

[0072] The entropy encoder 1004 is capable of performing entropy encoding on the reordered quantized values ​​supplied from the reordering unit 1003 for each sub-block to obtain entropy-coded data.

[0073] For ease of explanation, the entropy encoder 1004 has been described as performing entropy encoding sequentially for each block; however, the entropy encoder 1004 can also perform entropy encoding in parallel for multiple blocks.

[0074] The multiplexer 1005 outputs the entropy-encoded data supplied by the entropy encoder 1004 as a bit stream (called the NN bit stream).

[0075] Next, the operation of the NN video encoder 101 will be explained. Figure 9This is a flowchart illustrating an operational example of the NN video encoder 101.

[0076] In the NN video encoder 101, the encoder 1001 extracts features from the input image to generate a feature vector (step S101).

[0077] Quantizer 1002 quantizes the feature vector to generate quantized values ​​(step S102).

[0078] The reordering unit 1003 reorders the quantized values ​​supplied from the quantizer 1002 in units of sub-blocks through the above processing (step S103).

[0079] The entropy encoder 1004 performs entropy encoding on the reordered quantized values ​​to generate entropy-coded data (step S104).

[0080] The multiplexer 1005 outputs the entropy-encoded data as an NN bit stream (step S105).

[0081] [Description of the decoding side]

[0082] In the NN video decoder 201, the demultiplexer 2005 demultiplexes the NN bitstream to obtain entropy-coded data.

[0083] Entropy decoder 2004 performs entropy decoding on the entropy-encoded data supplied from demultiplexer 2005 to obtain quantized values.

[0084] The inverse reordering unit 2003 reorders the quantized values ​​supplied by the entropy decoder 2004, which are in units of sub-blocks, into units of samples. In other words, the inverse reordering unit 2003 degroups the quantized values ​​of the sub-blocks output from the entropy decoder 2004. Specifically, as... Figure 7 As shown, the inverse reordering unit 2003 restores the original order of the quantized values ​​reordered by the reordering unit 1003 in the NN video encoder 101 (see [reference]). Figure 7 (Left side). That is, the inverse reordering unit 2003 can restore the original order of quantized values ​​within a sub-block by performing the opposite operation to the reordering operation performed by the reordering unit 1003 as an inverse reordering operation.

[0085] For more details, please refer to Figure 5 and Figure 8 To illustrate with an example, the inverse reordering unit 2003 will follow... Figure 8 The quantized values ​​input in the given order are reordered as follows: Figure 5 The order shown. In other words, the reverse reordering unit 2003 follows... Figure 5The order shown indicates that the quantized values ​​received from the entropy decoder 2004 are output to the inverse quantizer 2002. In other words, the inverse reordering unit 2003 enables the inverse quantizer 2002 to follow the order from top left to bottom right (see [link to documentation]). Figure 4 The quantized value is processed by the entropy decoder 2004 and output to the inverse quantizer 2002. In addition, if necessary, for example, the inverse reordering unit 2003 temporarily stores the quantized value input from the entropy decoder 2004 in a buffer memory (not shown).

[0086] Inverse quantizer 2002 performs inverse quantization on the reordered quantized values ​​supplied by inverse reordering unit 2003 to obtain a reconstructed feature vector.

[0087] Decoder 2001 obtains the reconstructed image of the decoded image (also known as the NN decoded image) based on the reconstructed feature vector.

[0088] Next, the operation of the NN video decoder 201 will be explained. Figure 10 This is a flowchart illustrating an operational example of the NN video decoder 201.

[0089] In the NN video decoder 201, the demultiplexer 2005 demultiplexes the bitstream (step S201). Through demultiplexing, the demultiplexer 2005 obtains entropy-coded data.

[0090] Entropy decoder 2004 performs entropy decoding on the entropy-encoded data to obtain quantized values ​​(step S202).

[0091] The inverse reordering unit 2003 reorders the quantized values ​​supplied from the entropy decoder 2004 on a sub-block basis (for each sub-block) (step S203). In other words, the inverse reordering unit 2003 restores the original order of the quantized values ​​(see [link to relevant documentation]). Figure 7 (Left side).

[0092] Inverse quantizer 2002 performs inverse quantization on the quantized value (step S204). Inverse quantizer 2002 obtains the reconstructed feature vector through inverse quantization.

[0093] Decoder 2001 obtains the reconstructed image of the decoded image based on the reconstructed feature vector (step S205).

[0094] In this example embodiment, the reordering unit 1003 reorders the quantized feature vector (specifically, the elements (samples) of the quantized feature vector) so that entropy coding in neural network-based video coding is performed on a sub-block basis, i.e., for each sub-block. At this time, the reordering unit 1003 reorders the quantized feature vector so that processing is performed in each sub-block in an order from top left to bottom right. Therefore, low-latency parallel processing of the entropy coding of the quantized feature vector is possible.

[0095] Furthermore, the NN video decoder 201 receives a bitstream from the NN video encoder 101, which includes a reordering unit 1003, and the inverse reordering unit 2003 performs a process that is the reverse of the reordering process performed by the reordering unit 1003. Therefore, the NN video decoder 201 is able to smoothly play video based on the bitstream from the NN video encoder 101.

[0096] [Encoder and decoder configuration]

[0097] Figure 11 This is a block diagram illustrating a configuration example of the encoder 1001 and decoder 2001 in a first example embodiment. Figure 11 In the diagram, "down arrow (↓) 2" indicates 1 / 2 subsampling (also known as pooling). "up arrow (↑) 2" indicates 2x upsampling.

[0098] exist Figure 11 In the example shown, encoder 1001 includes four residual blocks and one convolutional block. Each residual block contains two convolutional blocks and a shortcut link. Each convolutional block contains a convolutional layer and an activation function.

[0099] The decoder 2001 consists of four residual blocks and a pixel-shuffle convolutional layer. The pixel-shuffler is a mechanism proposed as a sub-pixel convolution. The pixel-shuffler rearranges the input feature vector to output a high-resolution feature vector.

[0100] For the activation function, a parametric ReLU can be used, where the output value is α times the input value when the input value is less than 0 (where α is a parameter determined through training); and the output value is equal to the input value when the input value is greater than or equal to 0.

[0101] It should be noted that Figure 11 The configuration shown is only an example, and the configuration of encoder 1001 and decoder 2001 is not limited to this. Figure 11 The configuration shown.

[0102] Second example embodiment.

[0103] Figure 12 This is a block diagram illustrating an example configuration of a video encoder. Figure 13 This is a block diagram illustrating an example configuration of a video decoder.

[0104] Figure 12 The video encoder 100 shown includes a switch 3000, an NN video encoder 3001, an NN video decoder 3003, a CP video encoder 3002, a CP video decoder 3004, a decoded image buffer 3005, and a multiplexer 4000 that performs multiplexing on entropy-coded data and other information.

[0105] It should be noted that "NN" stands for Neural Network. "CP" refers to prediction based on encoding units.

[0106] The CP video encoder 3002 performs video encoding using a video encoding method based on predictive coding on a per-coding-unit basis. The CP video decoder 3004 performs decoding using a video decoding method based on predictive coding on a per-coding-unit basis. The decoded image buffer 3005 is a storage unit for storing decoded images (reconstructed images). As described above, video encoding methods conforming to H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc., can be used.

[0107] The NN video encoder 3001 can use the NN video encoder 101 described in the above example embodiments. The NN video decoder 3003 can use the NN video decoder 201 described in the above example embodiments.

[0108] Figure 13 The video decoder 200 shown includes a demultiplexer 5000 that performs demultiplexing on the bitstream, an NN video decoder 3003, a CP video decoder 3004, and a decoded image buffer 3005.

[0109] In other words, Figure 12 The video encoder 100 shown and Figure 13 The video decoder 200 shown is a video encoder and a video decoder: it combines the NN video encoder 101, which is designated as NN video encoder 3001, and the NN video decoder 201, which is designated as NN video decoder 3003, with the CP video encoder 3002 and CP video decoder 3004, which are based on a video coding method that uses predictive coding in the coding unit.

[0110] [Description of the encoding side]

[0111] exist Figure 12 In the video encoder 100 shown, switch 3000 supplies the input image to either NN video encoder 3001 or CP video encoder 3002.

[0112] The NN video encoder 3001 operates in the same manner as the NN video encoder 101 of the first example embodiment to generate an NN bitstream.

[0113] The NN video decoder 3003 takes as input the NN bitstream supplied from the NN video encoder 3001 and operates in the same manner as the NN video decoder 201 in the first example embodiment to obtain a decoded image (NN decoded image). The NN video decoder 3003 stores the NN decoded image in the decoded image buffer 3005.

[0114] The CP video encoder 3002 uses the input image and the decoded image stored in the decoded image buffer 3005 to perform video encoding based on prediction coding in the coding unit, thereby generating a bitstream (also known as the CP bitstream).

[0115] The CP video decoder 3004 takes as input the CP bitstream supplied from the CP video encoder 3002, performs entropy decoding, and then performs decoding based on the predictive coding performed in the coding unit to obtain a decoded picture (also known as a CP decoded picture). The CP video decoder 3004 stores the CP decoded picture in the decoded picture buffer 3005.

[0116] The decoded image stored in the decoded image buffer 3005 is used as a reference image.

[0117] exist Figure 12 In the configuration shown, the NN video decoder 3003 takes as input the NN bitstream from the NN video encoder 3001 and performs entropy decoding. Then, the NN video decoder 3003 obtains the NN decoded image based on the encoded data obtained through entropy decoding. However, the NN video decoder 3003 can also be configured to take as input intermediate data (e.g., quantization values) from the NN video encoder 3001 prior to entropy encoding and obtain the NN decoded image based on this intermediate data. In this case, the NN video decoder 3003 does not need to perform entropy decoding.

[0118] In addition, Figure 12 In the configuration shown, the CP video decoder 3004 takes as input the CP bitstream from the CP video encoder 3002 and performs entropy decoding. Then, the CP video decoder 3004 obtains the CP decoded image based on the encoded data obtained through entropy decoding. However, the CP video decoder 3004 can also be configured to take as input intermediate data (e.g., quantization values) from the CP video encoder 3002 prior to entropy encoding and obtain the CP decoded image based on this intermediate data. In this case, the CP video decoder 3004 does not need to perform entropy decoding.

[0119] [Description of the decoding side]

[0120] exist Figure 13 In the video decoder 200 shown, the NN video decoder 3003 operates in the same manner as the NN video decoder 201 of the first example embodiment, based on the encoded data obtained by demultiplexing the NN bitstream, to obtain an NN decoded image. The NN video decoder 3003 stores the NN decoded image in the decoded image buffer 3005.

[0121] The CP video decoder 3004 demultiplexes the CP bitstream, performs entropy decoding, and obtains the CP decoded image based on the resulting encoded data. The CP video decoder 3004 stores the CP decoded image in the decoded image buffer 3005.

[0122] The video decoder 200 outputs either the NN decoded image or the CP decoded image stored in the decoded image buffer 3005 as a decoded image.

[0123] [Variation 1]

[0124] In the above example embodiment, the entropy encoder 1004 performs entropy encoding sequentially from the top left to the bottom right in units of 2×4 sub-blocks for the quantized feature vector (W_tensor(i)=8, H_tensor(i)=4) at the resolution to be processed, and performs entropy encoding in the order from the top left to the bottom right within each sub-block.

[0125] To determine the probability distribution within a sub-block with higher accuracy, the reordering unit 1003 can also reorder the processing order within the sub-block from... Figure 8 The order shown is changed. For example, in order to determine the probability distribution by appropriately interpolating the quantized values ​​to be processed subsequently within the sub-block based on surrounding values, the reordering unit 1003 can follow... Figure 14 The order shown reorders the processing within the sub-block.

[0126] The above example embodiments can be constructed by hardware or implemented by computer programs.

[0127] Figure 15 The information processing system shown includes: a processor 701 such as one or more CPUs (Central Processing Units), a program memory 702, a storage medium 703 for storing video data, and a storage medium 704 for storing bitstreams. Storage mediums 703 and 704 can be separate storage media or a storage area composed of the same storage medium. Magnetic storage media such as hard disks can be used as storage media. Furthermore, the information processing system may include multiple processors 701.

[0128] In the information processing system, the program (video encoding program or video decoding program) used to implement the functions of the various blocks described in the above example embodiments is stored in the program memory 702.

[0129] The processor 701 performs processing according to the program stored in the program memory 702, thereby implementing the functions of the video encoder 100, NN video encoder 101, video decoder 200 and NN video decoder 201 shown in the above example embodiment.

[0130] For example, the functions of NN video encoder 101 and video encoder 100 are implemented in the following manner: processor 701 is configured to implement... Figure 6 and Figure 12 The video encoding program performs the functions of each block (except for the decoded image buffer 3005) in the NN video encoder 101 and video encoder 100 shown.

[0131] Furthermore, for example, the functions of the NN video decoder 201 and the video decoder 200 are implemented in the following manner: the processor 701 is configured to implement... Figure 6 and Figure 13 The video decoding program performs the functions of each block (except for the decoded image buffer 3005) in the NN video decoder 201 and video decoder 200 shown.

[0132] At least the program memory 702 is a non-transitory computer-readable medium. However, programs can be stored in various types of transient computer-readable media. Programs may be provided to transient computer-readable media, for example, via wired or wireless communication paths, i.e., via electrical signals, optical signals, or electromagnetic waves.

[0133] Figure 16 This is a block diagram showing the main parts of a video encoding device. Figure 16 The video encoding device 10 shown (implemented by video encoder 100 or NN video encoder 101 in an example embodiment) includes a neural network 11 (implemented by encoder 1001 in an example embodiment), a quantization device 12 (implemented by quantizer 1002 in an example embodiment), an entropy encoding device (implemented by entropy encoder 1004 in an example embodiment), and a grouping device (implemented by reordering unit 1003 in an example embodiment) for grouping the quantized values ​​output from quantization device 12 into sub-blocks.

[0134] Figure 17 This is a block diagram showing the main components of a video decoder. Figure 17The video decoding device 20 shown (implemented by video decoder 200 or NN video decoder 201 in an example embodiment) includes a neural network 21 (implemented by decoder 2001 in an example embodiment), an inverse quantization device 22 (implemented by inverse quantizer 2002 in an example embodiment), an entropy decoding device 23 (implemented by entropy decoder 2004 in an example embodiment), and a degrouping device 24 (implemented by inverse reordering unit 2003 in an example embodiment), the degrouping device being used to degroup the quantized values ​​of the sub-blocks output from the entropy decoding device.

[0135] Some or all of the above example embodiments may be described as follows, but are not limited thereto.

[0136] (Supplementary Explanation 1) A video encoding device, comprising a neural network, a quantization device, an entropy encoding device, and...

[0137] A grouping device for grouping quantized values ​​output from a quantization device into sub-blocks.

[0138] (Supplementary Explanation 2) In the video encoding device according to Supplementary Explanation 1, the grouping device reorders the quantization values ​​within the sub-blocks.

[0139] (Supplementary Explanation 3) A video decoding device includes a neural network, an inverse quantization device, an entropy decoding device, and a degrouping device, wherein the degrouping device is used to degroup the quantized values ​​of sub-blocks output from the entropy decoding device.

[0140] (Supplementary Explanation 4) In the video decoding device according to Supplementary Explanation 3, the degrouping device reorders the quantization values ​​within the sub-blocks.

[0141] (Supplementary Note 5) A video coding method that performs a neural network-based coding process, a quantization process, and an entropy coding process includes: grouping quantized values ​​generated by the quantization process into sub-blocks.

[0142] (Supplementary Note 6) The video coding method described in Supplementary Note 5 further includes: reordering the quantization values ​​within the sub-block.

[0143] (Supplementary Explanation 7) A video decoding method that performs a neural network-based decoding process, an inverse quantization process, and an entropy decoding process includes: degrouping the quantized values ​​of sub-blocks obtained through the entropy decoding process.

[0144] (Supplementary Note 8) The video decoding method described in Supplementary Note 7 further includes: reordering the quantization values ​​within the sub-block.

[0145] (Supplementary Note 9) A video encoding program that causes a computer to perform: The process of encoding, quantization, and entropy encoding based on neural networks; and the grouping of quantized values ​​generated by the quantization process into sub-blocks.

[0146] (Supplementary Note 10) The video encoding procedure described in Supplementary Note 9 causes the computer to reorder the quantization values ​​within the sub-block.

[0147] (Supplementary Note 11) A video decoding program that causes a computer to execute: The decoding process based on neural networks, the inverse quantization process, and the entropy decoding process; as well as the degrouping of the quantized values ​​of the sub-blocks obtained through the entropy decoding process.

[0148] (Supplementary Note 12) The video decoding program described in Supplementary Note 11 enables the computer to reorder the quantization values ​​within the sub-block.

[0149] (Supplementary Note 13) A storage medium for storing a video encoding program, the video encoding program causing a computer to perform: a neural network-based encoding process, a quantization process, an entropy encoding process; and grouping quantized values ​​generated by the quantization process into sub-blocks.

[0150] (Supplementary Note 14) A storage medium for storing a video decoding program, the video decoding program causing a computer to execute: a neural network-based decoding process, an inverse quantization process, and an entropy decoding process; and degrouping the quantized values ​​of sub-blocks obtained through the entropy decoding process.

[0151] While the invention of this application has been described above with reference to exemplary embodiments, the invention is not limited to the above exemplary embodiments. Within the scope of this invention, those skilled in the art will understand that various changes can be made to the configuration and details of the invention.

[0152] This application is based on and claims the priority of Japanese Patent Application No. 2024-061862, filed on April 8, 2024, the disclosure of which is incorporated herein by reference in its entirety.

[0153] List of reference numerals

[0154] 10. Video encoding equipment

[0155] 11 Neural Networks

[0156] 12 Quantization device

[0157] 13 Entropy coding device

[0158] 14. Grouping device

[0159] 20 Video decoding equipment

[0160] 21 Neural Networks

[0161] 22 Inverse Quantization Device

[0162] 23 Entropy Decoding Device

[0163] 24 Degrouping device

[0164] 100 video encoder

[0165] 101 NN video encoder

[0166] 200 video decoders

[0167] 201 NN video decoder

[0168] 701 processor

[0169] 702 Program Memory

[0170] 703 and 704 storage media

[0171] 1001 Encoder

[0172] 1002 quantizer

[0173] 1003 Reordering Units

[0174] 1004 Entropy Encoder

[0175] 1005 Multiplexer

[0176] 2001 Decoder

[0177] 2002 Inverse Quantizer

[0178] 2003 Inverse Reordering Unit

[0179] 2004 Entropy Decoder

[0180] 2005 Demultiplexer

[0181] 3000 switch

[0182] 3001 NN Video Encoder

[0183] 3002 CP Video Encoder

[0184] 3003 NN Video Decoder

[0185] 3004 CP Video Decoder

[0186] 3005 Decoding Image Buffer

[0187] 4000 multiplexer

[0188] 5000 Demultiplexer

Claims

1. A video encoding device, comprising: Neural networks, quantization devices, entropy coding devices, and grouping devices. The grouping device is used to group the quantized values ​​output from the quantization device into sub-blocks.

2. The video encoding device according to claim 1, wherein, The grouping device reorders the quantization values ​​within the sub-blocks.

3. A video decoding device, comprising: Neural network, inverse quantization device, entropy decoding device, and degrouping device, The degrouping device is used to degroup the quantized values ​​of the sub-blocks output from the entropy decoding device.

4. The video decoding device according to claim 3, wherein, The degrouping device reorders the quantization values ​​within the sub-blocks.

5. A video coding method that performs a neural network-based coding process, a quantization process, and an entropy coding process, comprising: The quantized values ​​generated by the quantization process are grouped into sub-blocks.

6. The video encoding method according to claim 5, further comprising: The quantization values ​​within the sub-block are reordered.

7. A video decoding method that performs a neural network-based decoding process, an inverse quantization process, and an entropy decoding process, comprising: The quantized values ​​of the sub-blocks obtained through the entropy decoding process are degrouped.

8. The video decoding method according to claim 7, further comprising: The quantization values ​​within the sub-block are reordered.

9. A video encoding program that causes a computer to perform: The encoding process, quantization process, and entropy encoding process based on neural networks; and The quantized values ​​generated by the quantization process are grouped into sub-blocks.

10. A video decoding program that causes a computer to perform: The decoding process based on neural networks, the inverse quantization process, and the entropy decoding process; and The quantized values ​​of the sub-blocks obtained through the entropy decoding process are degrouped.

Images