Video data processing device and video data processing method

Abstract

In order to allow the processing order of entropy encoding to be made consistent between video encoding based on predictive encoding in an encoding unit and video encoding based on a neural network when these encodings are combined, this video data processing device comprises a neural network, a quantization means, and an entropy encoding means, and further comprises a rearrangement means for rearranging quantization values output by the quantization means.

Description

Video data processing device and video data processing method

[0001] The present disclosure relates to a video data processing device and a video data processing method.

[0002] In order to transmit or record video efficiently, a video encoding device is used to generate a coded representation (hereinafter referred to as a bitstream) of input video, and a video decoding device is used to decode the bitstream to generate decoded video.

[0003] [Video Coding Based on Predictive Coding in Coding Units] Video coding standards include H.264 / AVC (Advanced Video Coding), H.265 / HEVC (High-Efficiency Video Coding), and H.266 / VVC (Versatile Video Coding), which are standardized by ITU-T SG16 and ISO / IEC / SC29. Another recent video coding technology is the technology described in Non-Patent Document 1.

[0004] In these video coding methods, video data is coded and decoded while being managed in a hierarchical structure. The hierarchical structure is made up of, for example, pictures that make up the video data, slices (or tiles) obtained by dividing pictures, coding tree units (CTUs) obtained by dividing slices, and coding units (CUs) obtained by dividing coding tree units.

[0005] The input image of the target CU is usually encoded earlier than the target CU and is predictively coded based on a predicted image generated based on the decoded image that has been decoded. That is, a prediction error image obtained by subtracting the predicted image from the input image is coded and decoded. Predictive coding includes intra-picture prediction (intra-prediction) that uses a decoded image included in a picture with the same display time as the target CU, and inter-picture prediction (inter-prediction) that uses a decoded image included in a picture with a different display time from the target CU.

[0006] The prediction error image is encoded based on frequency transform, quantization, and entropy coding. The prediction error image is decoded based on entropy decoding, inverse quantization, and inverse frequency transform. The frequency transform value of the quantized prediction error image is called a quantized value.

[0007] The functions provided by slices are independence (Independent Decoding) and divisibility (Data Partitioning). Independence is a function that allows decoding without using the decoded results of other slices in the same picture. Dividability is a function that allows dividing the bitstream into slices of any size.

[0008] [Video Coding Based on Neural Networks] Non-Patent Document 2 describes a new video coding technique that combines an auto-encoder, which is a type of neural network, quantization, and entropy coding.

[0009] An autoencoder compresses input data into a low-dimensional feature vector so that it contains only important features. The autoencoder then generates reconstructed data by reconstructing the low-dimensional feature vector back to its original dimensions. Figure 1 is an explanatory diagram showing the autoencoder algorithm. In Figure 1, the circular parts are called nodes and the arrows are called edges. The process of reducing the data into a low-dimensional feature vector (the first half) is called encoding. The process of generating reconstructed data (the second half) is called decoding.

[0010] The autoencoder is trained to minimize the reconstruction error (the difference between the input data and the reconstructed data). To obtain meaningful features, the autoencoder is designed to impose constraints on the encoding structure and to add regularization terms to the network's loss function.

[0011] "Algorithm description of Enhanced Compression Model 9(ECM 9)", JVET-AD2025, JVET of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 30th Meeting, Antalya, TR, 21-28 April 2023J. Ball'e, V. Laparra, and EP Simoncelli, "End-to-end Optimized Image "Compression", published as a conference paper at ICLR 2017

[0012] When video coding based on predictive coding in a coding unit and video coding based on a neural network are combined, there is a problem in that the processing order of entropy coding for quantized values ​​cannot be unified.

[0013] Fig. 2 is an explanatory diagram illustrating the above-mentioned problem. The left side of Fig. 2 illustrates an example of the processing order of entropy coding in video coding based on neural networks. The right side of Fig. 2 illustrates an example of the processing order of entropy coding in video coding based on predictive coding in a coding unit.

[0014] In FIG. 2, W_tensor indicates the row size of the feature vector. H_tensor indicates the column size of the feature vector. W_img indicates the width (horizontal size) of the input picture. H_img indicates the height (vertical size) of the input picture. CTU_img indicates the width of the CTU. Note that FIG. 2 illustrates a square CTU.

[0015] In neural network-based video coding, quantization values ​​are processed sample by sample in order from top left to bottom right. In coding unit-based video coding based on predictive coding, quantization values ​​are processed block by block (e.g., CTU) in order from top left to bottom right, but within each block, quantization values ​​are processed from top left to bottom right.

[0016] When attention is also paid to processing within a block, the processing order of entropy coding in neural network-based video coding and the processing order of entropy coding in predictive coding-based video coding in a coding unit are not aligned. Therefore, when the width (height) of the CTU in predictive coding-based video coding in a coding unit is equal to CTU_img, if the width (W_tensor) and height (H_tensor) of the quantized values ​​for one picture obtained by neural network-based video coding are equal to the width (W_img) and height (H_img) of the quantized values ​​for one picture obtained by predictive coding-based video coding in a coding unit, the transmission order of the quantized values ​​of the picture will not be aligned.

[0017] Furthermore, neural network-based video coding has the problem that it cannot provide the independent and divisible slicing function as in video coding based on predictive coding in the coding unit.

[0018] The reason why independence cannot be provided is that encoding on the encoding device side (encoding illustrated in FIG. 1 ) and decoding on the decoding device side (decoding illustrated in FIG. 1 ) may cross a slice boundary, and the reason why divisibility cannot be provided is that entropy encoding processing on the encoding device and entropy decoding processing on the decoding device may be continuous across a slice boundary.

[0019] Continuing the entropy coding process means not aligning the output. Furthermore, if the entropy coding is based on adaptive arithmetic coding, continuing the entropy coding process also includes not resetting the most probable symbols and their probabilities, and not resetting the arithmetic coding engine. On the other hand, continuing the corresponding entropy decoding process means not decoding the data for data alignment. Furthermore, if the entropy coding is based on adaptive arithmetic coding, continuing the entropy coding process also includes not resetting the most probable symbols and their probabilities, and not resetting the arithmetic decoding engine.

[0020] The present invention aims to provide a video data processing device and a video data processing method that make it possible to align the processing order of entropy coding at least when video coding based on predictive coding in a coding unit and video coding based on a neural network are combined.

[0021] A video data processing device according to the present disclosure includes a neural network, a quantization means, and an entropy coding means, and also includes a sorting means for sorting quantized values ​​output by the quantization means.

[0022] Another video data processing device according to the present disclosure includes a neural network, an inverse quantization means, and an entropy decoding means, and also includes an inverse sorting means for sorting quantized values ​​output by the entropy decoding means.

[0023] A video data processing method according to the present disclosure performs a neural network-based encoding process, a quantization process, and entropy coding, and rearranges the quantized values ​​produced by the quantization process.

[0024] Another video data processing method according to the present disclosure performs a neural network-based decoding process, an inverse quantization process, and an entropy decoding process, and rearranges the quantized values ​​obtained by the entropy decoding process.

[0025] A video data processing program according to the present disclosure causes a computer to perform encoding processing based on a neural network, quantization processing, and entropy coding, and to perform processing to sort quantized values ​​created by the quantization processing.

[0026] Another video data processing program based on the present disclosure causes a computer to perform a neural network-based decoding process, an inverse quantization process, and an entropy decoding process, and to perform a process of rearranging the quantized values ​​obtained by the entropy decoding process.

[0027] According to the present invention, when video coding based on predictive coding in a coding unit and video coding based on a neural network are combined, it is possible to unify the processing order of entropy coding.

[0028] 1 is an explanatory diagram showing an algorithm of an autoencoder. 2 is an explanatory diagram explaining a problem. 3 is a block diagram showing a video data processing system that encodes and decodes each picture constituting video data based on a neural network. 4 is an explanatory diagram showing an example of rearrangement in a NN video data processor on the encoding side and rearrangement in a NN video data processor on the decoding side. 5 is a flowchart showing an example of operation of a NN video data processor on the encoding side. 6 is a flowchart showing an example of operation of a NN video data processor on the decoding side. 7 is a block diagram showing a video data processing system that can guarantee independence and divisibility. 8 is an explanatory diagram showing an example of slice division in a NN video data processor on the encoding side. 9 is a flowchart showing another example of operation of a NN video data processor on the encoding side. 10 is an explanatory diagram showing an example of slice division in a NN video data processor on the decoding side. 11 is a flowchart showing another example of operation of a NN video data processor on the decoding side. 12 is a block diagram showing example configurations of an encoder and a decoder. 13 is an explanatory diagram showing another example of slice division in a NN video data processor on the encoding side. 14 is an explanatory diagram showing another example of slice division in a NN video data processor on the decoding side. 15 is an explanatory diagram explaining the operation of a rearrangement unit and a reverse rearrangement unit. 16 is a block diagram showing an example configuration of a video data processor on the encoding side. It is a block diagram showing an example of the configuration of a video data processor on the decoding side.It is a block diagram showing an example of the configuration of an information processing system.It is a block diagram showing the main parts of a video data processing device.It is a block diagram showing the main parts of a video data processing device of another aspect.

[0029] Hereinafter, an embodiment will be described with reference to the drawings.

[0030] Embodiment 1. Figure 3 is a block diagram showing an encoding-side video data processor 101 and a decoding-side video data processor 201 that encode and decode each picture constituting video data based on a neural network. Hereinafter, the video data processor 101 will be referred to as the NN video data processor 101. The video data processor 201 will be referred to as the NN video data processor 201. The NN video data processor 101 is an encoding-side video data processor that performs video processing (e.g., encoding processing) based on a neural network. The NN video data processor 201 is a decoding-side video data processor that performs video processing (e.g., decoding processing) based on a neural network. A system including the NN video data processors 101 and 201 is called a video data processing system.

[0031] The NN video data processor 101 includes an encoder 1001 , a quantizer 1002 , a rearrangement unit 1003 , an entropy encoder 1004 , and a multiplexer 1005 .

[0032] The NN video data processor 201 includes a decoder 2001 , an inverse quantizer 2002 , an inverse rearrangement unit 2003 , an entropy decoder 2004 , and a demultiplexer 2005 .

[0033] 3 simply indicates the direction of signal (data) flow, but does not exclude bidirectionality. This also applies to other block diagrams.

[0034] [Description of Encoding Side] In the NN video data processor 101, the encoder 1001 extracts features from the image of the input picture. Specifically, the encoder 1001 obtains a feature vector from the image of the input picture.

[0035] The quantizer 1002 quantizes the feature vector supplied from the encoder 1001 to obtain a quantized value.

[0036] The rearrangement unit 1003 rearranges the quantized values ​​supplied from the quantizer 1002 .

[0037] 4 is an explanatory diagram showing an example of rearrangement in the NN video encoder 101 and an example of rearrangement in the NN video decoder 201. In the example shown in FIG. 4, the rearrangement unit 1003 rearranges the quantized values ​​in units of CTUs with a width of CTU_tensor, where CTU_tensor is the width of the coding unit in the encoded feature vector. Therefore, the ratio between CTU_tensor and CTU_img is the same as the ratio between W_tensor and W_img.

[0038] Although only one CTU in the upper right corner is illustrated in FIG. 4, the quantization values ​​are rearranged for each CTU for all CTUs.

[0039] Also, for example, if the size of the CTU is 4x4, the rearrangement unit 1003 rearranges the quantization values ​​so that the four quantization values ​​in the nth (n: 1 to 3)th row in the CTU are lined up next to the four quantization values ​​in the (n+1)th row.

[0040] The rearrangement method is not limited to this example, and the blocks may be rearranged based on a specified processing order. For example, if the image is processed in a spiral from the periphery to the center, the blocks may be rearranged in accordance with this processing order.

[0041] The entropy encoder 1004 entropy encodes the sorted quantized values ​​supplied from the sorting unit 1003 to obtain entropy-coded data.

[0042] The multiplexer 1005 outputs the entropy-encoded data supplied from the entropy encoder 1004 as a bit stream (called an NN bit stream).

[0043] Next, a description will be given of the operation of the NN video data processor 101. FIG.

[0044] In the NN video data processor 101, the encoder 1001 extracts features from the input picture and generates a feature vector (step S101).

[0045] The quantizer 1002 quantizes the feature vector to generate a quantized value (step S102).

[0046] The reordering unit 1003 reorders the quantization values ​​supplied from the quantizer 1002 for each CTU in predictive coding in the coding unit (step S103). In the process of step S103, as illustrated in Fig. 4, the reordering unit 1003 reorders the quantization values ​​in each CTU so that the processing order is the same as the processing order of video coding based on predictive coding in the coding unit.

[0047] The entropy encoder 1004 entropy encodes the rearranged quantized values ​​to generate entropy-coded data (step S104).

[0048] The multiplexer 1005 outputs the entropy-encoded data as an NN bit stream (step S106).

[0049] [Description on the Decoding Side] In the NN video data processor 201, the demultiplexer 2005 demultiplexes the NN bitstream to obtain entropy-encoded data.

[0050] The entropy decoder 2004 entropy decodes the entropy-encoded data supplied from the demultiplexer 2005 to obtain quantized values.

[0051] The inverse rearrangement unit 2003 rearranges, on a sample-by-sample basis, the quantized values ​​supplied from the entropy decoder 2004. Specifically, as shown in Fig. 4 , the inverse rearrangement unit 2003 restores the order of the quantized values ​​rearranged by the rearrangement unit 1003 in the NN video encoder 101 (see the left side of Fig. 4 ). In other words, the inverse rearrangement unit 2003 performs rearrangement that is the reverse of the rearrangement performed by the rearrangement unit 1003.

[0052] The inverse quantizer 2002 inversely quantizes the sorted quantized values ​​supplied from the inverse sorting unit 2003 to obtain a reconstructed feature vector.

[0053] The decoder 2001 obtains a reconstructed image of a decoded picture (also called an NN decoded picture) from the reconstructed feature vector.

[0054] Next, a description will be given of the operation of the NN video data processor 201. Fig. 6 is a flowchart showing an example of the operation of the NN video data processor 201.

[0055] In the NN video data processor 201, the demultiplexer 2005 demultiplexes the bitstream (step S201). The demultiplexer 2005 obtains entropy-encoded data by demultiplexing.

[0056] The entropy decoder 2004 entropy decodes the entropy-encoded data to obtain quantized values ​​(step S202).

[0057] The inverse rearrangement unit 2003 rearranges the quantized values ​​supplied from the entropy decoder 2004 in units of samples (step S203). That is, the inverse rearrangement unit 2003 returns the order of the quantized values ​​to the original order (see the left side of FIG. 4).

[0058] The inverse quantizer 2002 inversely quantizes the quantized value (step S204). The inverse quantizer 2002 obtains a reconstructed feature vector through the inverse quantization.

[0059] The decoder 2001 obtains a reconstructed image of the decoded picture from the reconstructed feature vector (step S205).

[0060] In this embodiment, when video coding based on predictive coding in a coding unit and video coding based on a neural network are combined, the processing order of entropy coding in the video coding based on neural network can be aligned with the processing order of entropy coding in the video coding based on predictive coding by the rearrangement unit 1003 and the inverse rearrangement unit 2003. Therefore, when video coding based on predictive coding and video coding based on a neural network are combined in a coding unit, the processing order of entropy coding can be aligned.

[0061] Furthermore, the rearrangement unit 1003 and the inverse rearrangement unit 2003 can align the processing order of entropy coding, and also simplify the process of determining slice boundaries, because whether an image to be processed and a slice to which a feature vector belongs can be determined by simply comparing the coordinates of the upper left corner, the slice number, or the number of the first CTU between the image to be processed and the slice to which the feature vector belongs.

[0062] Embodiment 2. Figure 7 is a block diagram showing an NN video data processor 102 on the encoding side and an NN video data processor 202 on the decoding side that encode and decode each picture constituting video data based on a neural network, and that can guarantee independence and divisibility in video encoding based on a neural network. The NN video data processor 102 and NN video data processor 202 together constitute a video data processing system.

[0063] The NN video data processor 102 includes an encoder 1001 A, a quantizer 1002 , a rearrangement unit 1003 , an entropy encoder 1004 , a first control unit 1006 , a second control unit 1007 , and a multiplexer 1005 .

[0064] The NN video data processor 202 includes a decoder 2001 A, an inverse quantizer 2002 , an inverse rearrangement unit 2003 , an entropy decoder 2004 , a first control unit 2006 , a second control unit 2007 , and a demultiplexer 2005 .

[0065] [Description of the Encoding Side] In the NN video data processor 102, the quantizer 1002, rearrangement unit 1003, entropy encoder 1004, and multiplexer 1005 have the same configurations and functions as those in the first embodiment.

[0066] The encoder 1001A obtains a feature vector from an image of an input picture, similar to the encoder 1001 in the first embodiment. However, the encoder 1001A does not use input images of slice areas that are not the target of processing when obtaining a feature vector of a slice area to be processed, in accordance with control by the first control unit 1006. In other words, the feature vector of the slice area to be processed will be the same as the feature vector obtained by encoding the input image of the slice area to be processed as a single picture.

[0067] The first control unit 1006 obtains feature vector domain slice boundaries based on the input slice division information (information for identifying slice boundary information). Alternatively, the first control unit 1006 obtains information for identifying feature vector domain slice boundaries. The information for identifying feature vector domain slice boundaries may be, for example, image domain slice boundaries of each slice region. The first control unit 1006 controls the encoding process (hereinafter simply referred to as "encoding") by the encoder 1001 so that the process does not cross image domain slice boundaries. Furthermore, the first control unit 1006 supplies position information of the feature vector domain slice boundaries as slice boundary information to the multiplexer 1005 via the second control unit 1007.

[0068] The second control unit 1007 monitors the entropy encoder 1004 and causes the entropy encoder 1004 to terminate its processing after entropy encoding the last quantized value of each slice region.

[0069] The end of entropy coding means that the output of the entropy-coded data is data-aligned. That is, the entropy encoder 1004 adds alignment data for data alignment to the entropy-coded data. Furthermore, if the entropy coding is based on adaptive arithmetic coding, the entropy encoder 1004 also resets the most probable symbols and their probabilities, and the arithmetic coding engine, in order to end entropy coding.

[0070] In this embodiment, the multiplexer 1005 multiplexes the slice boundary information and the entropy-encoded data supplied from the entropy encoder 1004, and outputs the multiplexed data as an NN bit stream.

[0071] Next, the operation of the NN video data processor 102 will be described. Here, the operation of the NN video encoder 102 will be described using an example in which an input picture is divided equally into three slices in the vertical direction. Figure 8 is an explanatory diagram showing an example of slice division in the NN video data processor 102. Figure 9 is a flowchart showing an example of the operation of the NN video data processor 102.

[0072] In Fig. 8, W_img indicates the width (horizontal size) of the input picture, H_img indicates the height (vertical size) of the input picture, W_tensor indicates the row size of the feature vector, and H_tensor indicates the column size of the feature vector.

[0073] Note that the example in which the input picture is divided equally vertically into three slices is merely an example, and the division method is not limited to this. For example, the input picture may be divided unevenly. The number of divisions may be two, three, or more.

[0074] In the NN video data processor 102, the encoder 1001A divides an input picture into a plurality of slices (step S101). In this embodiment, the input picture is divided equally into three slices in the vertical direction (see FIG. 8).

[0075] The encoder 1001A extracts features from each slice and generates a feature vector (step S101A).

[0076] The quantizer 1002 quantizes the feature vector for each slice to generate a quantized value (step S102A).

[0077] The processes in steps S103 and S104 are the same as those in the first embodiment.

[0078] In step S104, the entropy encoder 1004 entropy encodes the rearranged quantized values ​​to generate entropy-encoded data, and then terminates the process (step S105). As described above, the termination is performed under the control of the second control unit 1007.

[0079] The multiplexer 1005 multiplexes the slice boundary information and the entropy-encoded data and outputs the multiplexed data as an NN bit stream (step S106A).

[0080] [Description on the Decoding Side] In the NN video data processor 202, the configurations and functions of the inverse quantizer 2002, inverse rearrangement unit 2003, entropy decoder 2004, and demultiplexer 2005 are the same as those in the first embodiment.

[0081] The first control unit 2006 controls the decoding process (hereinafter simply referred to as decoding) by the decoder 2001A based on slice boundary information obtained via the second control unit 2007 so that it does not cross a feature vector domain slice boundary.

[0082] The decoder 2001A obtains a reconstructed image of an NN-decoded picture from the reconstructed feature vector. However, under the control of the first control unit 2006, when obtaining a reconstructed image of a slice region to be processed, the decoder 2001A does not use the reconstructed feature vector of a slice region that is not the target of processing. In other words, the reconstructed image of a slice region to be processed will be the same as a reconstructed image obtained by decoding the reconstructed feature vector of the slice region to be processed as a single picture.

[0083] The second control unit 2007 monitors the entropy decoder 2004 and causes the entropy decoder 2004 to terminate its processing after entropy decoding the last quantized value of each slice region.

[0084] The end of entropy decoding means data alignment of the entropy-coded data, i.e., reading alignment data for data alignment from the entropy-coded data. Furthermore, if the entropy coding is based on adaptive arithmetic coding, the end of entropy decoding also involves resetting the most probable symbols and their probabilities, and resetting the arithmetic coding engine.

[0085] Next, the operation of the NN video data processor 202 will be described. Here, the operation of the NN video data processor 202 will be described using an example in which an input picture is divided equally into three slices in the vertical direction. Fig. 10 is an explanatory diagram showing an example of slice division in the NN video data processor 202. Fig. 11 is a flowchart showing an example of the operation of the NN video data processor 202.

[0086] 10, W_tensor indicates the row size of a feature vector, H_tensor indicates the column size of a feature vector, W_img indicates the width (horizontal size) of a decoded picture, and H_img indicates the height (vertical size) of a decoded picture.

[0087] In the NN video data processor 202, the demultiplexer 2005 demultiplexes the bitstream (step S201). In this embodiment, the demultiplexer 2005 obtains slice boundary information and entropy-encoded data through demultiplexing.

[0088] The processes in steps S202 to S204 are the same as those in the first embodiment.

[0089] The decoder 2001A obtains a reconstructed image of the decoded picture from the reconstructed feature vector (step S205A). As described above, the decoder 2001A does not use the reconstructed feature vector of a slice area that is not the target of processing when obtaining a reconstructed image of the target slice area under the control of the first control unit 2006. In other words, the decoder 2001A performs decoding based on slice boundary information so that the feature vector does not cross a slice boundary in the domain.

[0090] Then, the entropy decoder 2004 terminates the process (step S206). As described above, the termination is performed under the control of the second control unit 2007.

[0091] In this embodiment, the effects of the first embodiment can be obtained, and in addition, through the control of the first control unit 1006 and the first control unit 2006, encoding on the encoding side and decoding on the decoding side do not cross slice boundaries, so independence is guaranteed in video coding based on neural networks.

[0092] Furthermore, by controlling the termination of the second control unit 1007 and the second control unit 2007, the entropy encoding process on the encoding side and the entropy decoding process on the decoding side are no longer continuous at slice boundaries, thereby ensuring divisibility in video encoding based on neural networks.

[0093] [Encoder and Decoder Configuration] Fig. 12 is a block diagram showing an example configuration of the encoder 1001 and decoder 2001 in the first embodiment. In Fig. 12, a "downward arrow (↓) 2" represents subsampling by 1 / 2 (also called pooling). An "upward arrow (↑) 2" represents upsampling by 2. Note that the encoder 1001A and decoder 2001A in the second embodiment can also be configured as illustrated in Fig. 12.

[0094] In the example shown in Fig. 12, the encoder 1001 is composed of four residual blocks and one convolution block. Each residual block is composed of two convolutional blocks and one shortcut link. Each convolutional block is composed of one convolution layer and one activation function.

[0095] The decoder 2001 consists of four residual blocks and one PixelShuffle convolution layer. The Pixel Shuffler is a mechanism proposed as Sub-Pixel Convolution. The Pixel Shuffler rearranges input feature vectors and outputs high-resolution feature vectors.

[0096] As an activation function, a Parametric ReLU (Parametric Rectified Linear Unit) can be used, in which the output value is α times the input value when the input value is below 0 (where α is a parameter determined by learning), and the output value is the same as the input value when the input value is 0 or greater.

[0097] Furthermore, the configuration shown in FIG. 12 is just an example, and the configurations of the encoders 1001 and 1001A and the decoders 2001 and 2001A are not limited to the configuration shown in FIG.

[0098] Embodiment 3 In the second embodiment, the encoder 1001A in the NN video data processor 102 on the encoding side divides the input picture vertically into three equal slices (see FIG. 8). However, the encoder 1001A may use a division method as shown in FIG. 13 (an example of division in which slice area 2 and slice area 1 exist in the same row, and slice area 2 and slice area 3 exist in another row). In FIG. 13, dashed lines indicate slice boundaries.

[0099] The NN video data processor 202 on the decoding side uses three slices obtained by dividing the input picture equally vertically (see FIG. 10 ). However, a division method such as that shown in FIG. 14 (where slice area 2 and slice area 1 exist in the same row, and slice area 2 and slice area 3 exist in another row) may also be used. In FIG. 14 , dashed lines indicate slice boundaries.

[0100] The configurations of the NN video data processor on the encoding side and the NN video data processor on the decoding side in this embodiment may be the same as those in the second embodiment shown in FIG.

[0101] Embodiment 4. The NN video data processor on the encoding side and the NN video data processor on the decoding side in each of the above embodiments can also support highly efficient encoding that utilizes correlation between color components (also called inter-channel). For example, the above-mentioned rearrangement 1003 and reverse rearrangement 2003 may group and process color component CTUs that belong to the same position in space, rather than treating each color component independently.

[0102] For example, when the input video is in RGB 4:4:4 format, the sorting 1003 performs the sorting process by grouping CTUs of each color that belong to the same position in space, as illustrated on the right side of Fig. 15. Specifically, the sorting 1003 groups CTUs of each color that belong to the same position, and sorts the quantized values ​​within the CTU in color order from the top left to the bottom right.

[0103] Similarly, the reverse sorting 2003 sorts the quantized values ​​of the picture in groups from top left to bottom right in CTU units, and sorts them from top left to bottom right within each CTU.

[0104] It is also possible to introduce a unit finer than a CTU and rearrange the quantized values ​​of the CTUs of each color within a group using this finer unit. For example, if the finer unit is 1x1, the quantized values ​​of the CTUs of each color are rearranged from the bottom left to the bottom right in the order of Red, Green, and Blue.

[0105] In addition, the NN video data processor on the encoding side and the NN video data processor on the decoding side in each of the above embodiments can also apply the above grouping to color spaces other than RGB (such as YCoCr) and formats other than the above 4:4:4 (such as 4:2:0).

[0106] Fifth embodiment Fig. 16 is a block diagram showing an example of the configuration of a video data processor on the encoding side, and Fig. 17 is a block diagram showing an example of the configuration of a video data processor on the decoding side.

[0107] The video data processor 100 shown in FIG. 16 includes a switch 3000, an NN video encoder 3001, an NN video decoder 3003, a CP video encoder 3002, a CP video decoder 3004, a decoded picture buffer 3005, and a multiplexer 4000 that performs multiplexing processing of entropy-encoded data and other information.

[0108] Note that "NN" stands for neural network, and "CP" stands for coding unit-based prediction.

[0109] The CP video encoder 3002 performs video encoding processing using a video encoding method based on predictive coding for each coding unit. The CP video decoder 3004 performs decoding processing using a video encoding method based on predictive coding for each coding unit. The decoded picture buffer 3005 is a storage unit that stores decoded pictures (reconstructed pictures). As described above, video encoding methods that comply with H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc. can be used.

[0110] The NN video data processor 101 or 102 in each of the above embodiments can be used as the NN video encoder 3001. The NN video data processor 201 or 202 in each of the above embodiments can be used as the NN video decoder 3003.

[0111] The video data processor 200 shown in FIG. 17 includes a demultiplexer 5000 that demultiplexes a bitstream, an NN video decoder 3003, a CP video decoder 3004, and a decoded picture buffer 3005.

[0112] In other words, video data processor 100 shown in FIG. 16 and video data processor 200 shown in FIG. 17 are video data processors in which NN video encoder 3001 is the NN video data processor 101 or 102 described above, NN video data processor 201 or 202 is the NN video decoder 3003 is a combination of these, and a CP video encoder 3002 and a CP video decoder 3004 that are based on a video encoding method based on predictive encoding in the encoding unit.

[0113] [Description of the Encoding Side] In the video data processor 100 shown in FIG. 16, a switch 3000 supplies an input picture to either an NN video encoder 3001 or a CP video encoder 3002 .

[0114] The NN video encoder 3001 operates in the same manner as the NN video data processor 101 of the first embodiment or the NN video data processor 102 of the second embodiment to generate an NN bitstream.

[0115] The NN video decoder 3003 receives the NN bitstream supplied from the NN video encoder 3001 and operates in the same manner as the NN video data processor 201 of the first embodiment or the NN video data processor 202 of the second embodiment to obtain decoded pictures (NN decoded pictures). The NN video decoder 3003 stores the NN decoded pictures in the decoded picture buffer 3005.

[0116] The CP video encoder 3002 uses the input picture and the decoded picture stored in the decoded picture buffer 3005 to perform video encoding based on predictive encoding in the encoding unit, and generates a bitstream (also called a CP bitstream).

[0117] The CP video decoder 3004 receives the CP bitstream supplied from the CP video encoder 3002, performs entropy decoding, and then performs decoding based on predictive coding in the coding unit to obtain a decoded picture (also referred to as a CP decoded picture). The CP video decoder 3004 stores the CP decoded picture in the decoded picture buffer 3005.

[0118] The decoded pictures stored in the decoded picture buffer 3005 are used as reference pictures.

[0119] In the configuration shown in FIG. 16 , the NN video decoder 3003 receives an NN bitstream from the NN video encoder 3001 and performs entropy decoding. The NN video decoder 3003 then obtains NN-decoded pictures from the encoded data obtained by entropy decoding. However, the NN video decoder 3003 may also be configured to receive intermediate data (e.g., quantized values) before entropy encoding from the NN video encoder 3001 and obtain NN-decoded pictures from the intermediate data. In this case, the NN video decoder 3003 does not need to perform entropy decoding.

[0120] 16 , the CP video decoder 3004 receives the CP bitstream from the CP video encoder 3002 and performs entropy decoding. The CP video decoder 3004 then obtains CP-decoded pictures from the encoded data obtained by entropy decoding. However, the CP video decoder 3004 may also be configured to receive intermediate data (e.g., quantized values) before entropy encoding from the CP video encoder 3002 and obtain CP-decoded pictures from the intermediate data. In this case, the CP video decoder 3004 does not need to perform entropy decoding.

[0121] 17, the NN video decoder 3003 operates in the same manner as the NN video data processor 201 of the first embodiment or the NN video data processor 202 of the second embodiment, based on the coded data obtained by demultiplexing the NN bitstream, to obtain NN-decoded pictures. The NN video decoder 3003 stores the NN-decoded pictures in the decoded picture buffer 3005.

[0122] The CP video decoder 3004 demultiplexes the CP bitstream, performs entropy decoding, and obtains CP decoded pictures based on the resulting coded data. The CP video decoder 3004 stores the CP decoded pictures in the decoded picture buffer 3005.

[0123] The video data processor 200 outputs the NN decoded picture or the CP decoded picture stored in the decoded picture buffer 3005 as a decoded picture.

[0124] [Variation 1] In each of the above embodiments, the rearrangement unit 1003 and the reverse rearrangement unit 2003 use the CTU unit, which is the width of the coding unit CTU_tensor in the encoded feature vector, as the implicit predetermined unit in the rearrangement and reverse rearrangement. However, the predetermined unit may be explicitly embedded in the bitstream. That is, the predetermined unit may be multiplexed into the bitstream. Furthermore, the predetermined unit may be a value other than CTU_tensor as long as it is equal to or smaller than CTU_img.

[0125] When a predetermined unit is multiplexed into a bitstream, in a decoding-side video data processor or NN video data processor corresponding to the encoding-side video data processor or NN video data processor of variant 1, a demultiplexer 2005, 5000 demultiplexes the predetermined unit of reverse rearrangement from the bitstream.

[0126] [Variation 2] In the second embodiment and in the third to fifth embodiments using the components of the second embodiment, both slice independence and divisibility are satisfied. However, if complete slice independence is not required, the first control unit 1006 in the NN video encoder 102 may relax the slice boundaries that must not be crossed. For example, when encoding the feature vector of a slice region to be processed, a configuration is possible in which the input image of the slice region corresponding to the subsequent slice in the bitstream is not crossed. That is, the feature vector of slice region 1 (see FIG. 8 ) is generated using the input image of slice region 1. The feature vector of slice region 2 is generated using the input images of slice region 1 and slice region 2. The feature vector of slice region 3 is generated using the input images of slice regions 1, 2, and 3. This configuration (Variation 2) does not achieve complete independence, but it can mitigate the degradation of coding efficiency associated with partitioning loss.

[0127] In addition, in a decoding-side video data processor or NN video data processor corresponding to the encoding-side video data processor or NN video data processor of Modification 2, when decoding the reconstructed feature vector of a slice region to be processed, the first control unit 2006 controls so that the reconstructed feature vector does not cross over the reconstructed feature vector of a slice region corresponding to a subsequent slice in the bitstream. That is, when obtaining a reconstructed image of slice region 1, the reconstructed feature vector of slice region 1 (see FIG. 10 ) is used. When obtaining a reconstructed image of slice region 2, the reconstructed feature vectors of slice region 1 and slice region 2 are used. When obtaining a feature reconstructed image of slice region 3, the reconstructed feature vectors of slice regions 1, 2, and 3 are used.

[0128] The slice boundary information described above can be configured as follows: When a frame is divided into slices of equal size, the information includes the number of slice divisions and the coordinates of the upper left corner of each slice or the slice number. When a frame is divided into slices of non-uniform size, the information includes the coordinates of the upper left corner of each slice and the horizontal and vertical widths of each slice. In the example shown in FIG. 8 , the slice numbers are reference numbers assigned to three slices (slice regions) in order from top to bottom. Slice numbers may also be assigned in order from bottom to top.

[0129] Furthermore, when a video coding method using a neural network is combined with a video coding method using coding unit-based prediction, the slice boundary information can be configured as follows: if a frame is divided into slices of equal size, the number of slice divisions and the coordinate of the upper left corner of each slice or the number of the first CTU; if a frame is divided into slices of non-uniform size, the coordinate of the upper left corner of each slice or the number of the first CTU, and the horizontal and vertical widths of each slice.

[0130] Each of the above embodiments can be configured by hardware, but can also be realized by a computer program.

[0131] The information processing system shown in Fig. 18 includes a processor 701 such as one or more CPUs (Central Processing Units), a program memory 702, a storage medium 703 for storing video data, and a storage medium 704 for storing a bitstream. The storage medium 703 and the storage medium 704 may be separate storage media or may be storage areas formed by the same storage medium. A magnetic storage medium such as a hard disk can be used as the storage medium.

[0132] In the information processing system, a program memory 702 stores a program (video data processing program) for realizing the functions of each block shown in each of the above embodiments.

[0133] The processor 701 then executes processing in accordance with the program stored in the program memory 702, thereby realizing the functions of the video data processor 100, NN video data processors 101 and 102, video data processor 200, and NN video data processors 201 and 202 shown in each embodiment.

[0134] For example, the functions of the NN video data processors 101, 102 and the video data processor 100 are realized by the processor 701 executing processing in accordance with a video encoding program for realizing the functions of each block (excluding the decoded picture buffer 3005) in the NN video data processors 101, 102 and the video data processor 100 shown in Figures 3, 7 and 16.

[0135] Furthermore, for example, the functions of the NN video data processors 201, 202 and the video data processor 200 are realized by the processor 701 executing processing in accordance with a video decoding program for realizing the functions of each block (excluding the decoded picture buffer 3005) in the NN video data processors 201, 202 and the video data processor 200 shown in Figures 3, 7 and 17.

[0136] At least the program memory 702 is a non-transitory computer-readable medium. However, the program may be stored in various types of transitory computer-readable medium. The program is supplied to the transitory computer-readable medium, for example, via a wired or wireless communication channel, i.e., via an electrical signal, an optical signal, or an electromagnetic wave.

[0137] Fig. 19 is a block diagram showing the main parts of a video data processing device. The video data processing device 10 shown in Fig. 19 (implemented by a video data processor 100 and NN video data processors 101 and 102 in the embodiment) includes a neural network 11 (implemented by encoders 1001 and 1001A in the embodiment), a quantization means 12 (implemented by a quantizer 1002 in the embodiment), and an entropy coding means 13 (implemented by an entropy coder 1004 in the embodiment), and further includes a rearrangement means 14 (implemented by a rearrangement unit 1003 in the embodiment) that rearranges quantized values ​​output by the quantization means 12.

[0138] Fig. 20 is a block diagram showing the main components of another aspect of a video data processing device. The video data processing device 20 shown in Fig. 20 (implemented by a video data processor 200 and NN video data processors 201 and 202 in the embodiment) includes a neural network 21 (implemented by decoders 2001 and 2001A in the embodiment), inverse quantization means 22 (implemented by an inverse quantizer 2002 in the embodiment), and entropy decoding means 23 (implemented by an entropy decoder 2004 in the embodiment), as well as inverse rearrangement means 24 (implemented by an inverse rearrangement unit 2003 in the embodiment) that rearranges the quantized values ​​output by the entropy decoding means 23.

[0139] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes.

[0140] (Supplementary Note 1) A video data processing device comprising a neural network, a quantization means, and an entropy coding means, further comprising a rearrangement means for rearranging quantized values ​​output by the quantization means.

[0141] (Supplementary Note 2) The video data processing device of Supplementary Note 1, further comprising a first control means (implemented by the first control unit 1006 in the embodiment) that prevents the encoding process based on the neural network from spanning a slice boundary.

[0142] (Supplementary Note 3) The video data processing device of Supplementary Note 1 or Supplementary Note 2, further comprising predictive coding means (implemented by the CP video coder 3002 in the embodiment) that performs predictive coding on a coding unit basis.

[0143] (Supplementary Note 4) A video data processing device comprising a neural network, an inverse quantization means, and an entropy decoding means, further comprising an inverse rearrangement means for rearranging quantized values ​​output by the entropy decoding means.

[0144] (Supplementary Note 5) The video data processing device of Supplementary Note 4, further comprising a first control means (implemented by the first control unit 2006 in the embodiment) that prevents the decoding process based on the neural network from spanning a slice boundary.

[0145] (Supplementary Note 6) The video data processing device of Supplementary Note 4 or Supplementary Note 5, further comprising predictive decoding means (implemented by the CP video decoder 3004 in the embodiment) that performs predictive decoding on a coding unit basis.

[0146] (Supplementary Note 7) A video data processing method that performs neural network-based encoding, quantization, and entropy coding, and further comprises rearranging quantized values ​​generated by the quantization.

[0147] (Supplementary Note 8) The video data processing method according to Supplementary Note 7, wherein the encoding process is controlled so as not to cross a slice boundary.

[0148] (Supplementary Note 9) The video data processing method according to Supplementary Note 7 or Supplementary Note 8, wherein predictive coding processing is performed on a coding unit basis.

[0149] (Supplementary Note 10) A video data processing method that executes a neural network-based decoding process, an inverse quantization process, and an entropy decoding process, the video data processing method comprising: rearranging quantized values ​​obtained by the entropy decoding process.

[0150] (Supplementary Note 11) The video data processing method according to Supplementary Note 10, wherein the decoding process is controlled so as not to cross a slice boundary.

[0151] (Supplementary Note 12) The video data processing method according to Supplementary Note 10 or Supplementary Note 11, wherein predictive decoding is performed on a coding unit basis.

[0152] (Supplementary Note 13) A video data processing program for causing a computer to execute an encoding process based on a neural network, a quantization process, and entropy coding, and to execute a process of rearranging quantized values ​​created by the quantization process.

[0153] (Supplementary Note 14) The video data processing program of Supplementary Note 13, which causes a computer to control the encoding process so as not to cross a slice boundary.

[0154] (Supplementary Note 15) The video data processing program of Supplementary Note 13 or Supplementary Note 14, which causes a computer to execute predictive coding processing on a coding unit basis.

[0155] (Supplementary Note 16) A video data processing program for causing a computer to execute a neural network-based decoding process, an inverse quantization process, and an entropy decoding process, and to execute a process of rearranging quantized values ​​obtained by the entropy decoding process.

[0156] (Supplementary Note 17) The video data processing program of Supplementary Note 16, which causes a computer to control the decoding process so as not to cross a slice boundary.

[0157] (Supplementary Note 18) The video data processing program of Supplementary Note 16 or Supplementary Note 17, which causes a computer to execute predictive decoding processing on a coding unit basis.

[0158] (Supplementary Note 19) A storage medium for storing a bitstream generated by a video data processing device comprising a neural network, a quantization means, and an entropy coding means, the video data processing device comprising a rearrangement means for rearranging quantized values ​​output by the quantization means.

[0159] (Supplementary Note 20) A storage medium for storing a bitstream generated by a video data processing method that performs neural network-based encoding, quantization, and entropy coding, and that rearranges quantized values ​​created by the quantization.

[0160] (Supplementary Note 21) The video data processing device according to any one of Supplementary Note 1 to Supplementary Note 3, wherein a predetermined unit of rearrangement is multiplexed into a bit stream. This Supplementary Note corresponds to the above-mentioned Modification 1.

[0161] (Supplementary Note 22) The video data processing device according to any one of Supplementary Note 4 to Supplementary Note 6, wherein a predetermined unit of reverse rearrangement is demultiplexed from the bitstream. This Supplementary Note corresponds to the above-mentioned Modification 1.

[0162] (Supplementary Note 23) The video data processing device according to Supplementary Note 2, wherein the first control means prevents the neural network-based encoding process from crossing only a slice boundary of a slice area that follows in bitstream order. This supplementary note corresponds to the above-mentioned Modification 2.

[0163] (Supplementary Note 24) The video data processing device according to Supplementary Note 5, wherein the first control means prevents the decoding process based on the neural network from crossing only a slice boundary of a slice area that follows in bitstream order. This supplementary note corresponds to the above-mentioned Modification 2.

[0164] This application claims priority based on Japanese Patent Application No. 2023-155650 filed on September 21, 2023, and Japanese Patent Application No. 2023-179926 filed on October 19, 2023, the disclosures of which are incorporated herein in their entireties.

[0165] REFERENCE SIGNS LIST 10 Video data processing device 11 Neural network 12 Quantization means 13 Entropy coding means 14 Rearrangement means 20 Video data processing device 21 Neural network 22 Inverse quantization means 23 Entropy decoding means 24 Inverse rearrangement means 100 Video encoder 101, 102 NN video data processor 200 Video data processor 201, 202 NN video data processor 701 Processor 702 Program memory 703, 704 Storage medium 1001, 1001A Encoder 1002 Quantizer 1003 Rearrangement unit 1004 Entropy encoder 1005 Multiplexer 1006 First control unit 1007 Second control unit 2001, 2001A Decoder 2002 Inverse quantizer 2003 Inverse rearrangement unit 2004 Entropy decoder 2005 Demultiplexer 2006 First control unit 2007 Second control unit 3000 Switch 3001 NN video encoder 3002 CP video encoder 3003 NN video decoder 3004 CP video decoder 3005 Decoded picture buffer 4000 Multiplexer 5000 Demultiplexer

Claims

1. A video data processing device comprising a neural network, a quantization means, and an entropy coding means, further comprising a sorting means for sorting quantized values ​​output by the quantization means.

2. The video data processing device according to claim 1, further comprising a first control means for preventing said neural network-based encoding process from spanning a slice boundary.

3. The video data processing device according to claim 1 or 2, further comprising predictive coding means for performing predictive coding on a coding unit basis.

4. A video data processing device comprising a neural network, an inverse quantization means, and an entropy decoding means, further comprising an inverse sorting means for sorting quantized values ​​output by the entropy decoding means.

5. A video data processing device according to claim 4, further comprising first control means for preventing the decoding process based on the neural network from spanning a slice boundary.

6. The video data processing device according to claim 4 or 5, further comprising predictive decoding means for performing predictive decoding on a coding unit basis.

7. A method of processing video data comprising performing a neural network based encoding process, a quantization process, and an entropy coding process, the method comprising: reordering quantized values ​​produced by the quantization process.

8. The video data processing method according to claim 7, further comprising controlling the encoding process so as not to straddle a slice boundary.

9. The video data processing method according to claim 7 or 8, wherein predictive coding processing is executed on a coding unit basis.

10. A video data processing method which executes a neural network-based decoding process, an inverse quantization process, and an entropy decoding process, and which rearranges quantization values ​​obtained by the entropy decoding process.

11. The video data processing method according to claim 10, further comprising controlling the decoding process so as not to cross a slice boundary.

12. A video data processing method according to claim 10 or 11, wherein predictive decoding processing is executed on a coding unit basis.

13. A video data processing program for causing a computer to execute encoding processing based on a neural network, quantization processing, and entropy coding, and to execute a processing of rearranging the quantized values ​​created by the quantization processing.

14. The video data processing program according to claim 13, which causes a computer to control the encoding process so as not to straddle a slice boundary.

15. A video data processing program as claimed in claim 13 or 14, which causes a computer to execute predictive coding processing on a coding unit basis.

16. A video data processing program for causing a computer to execute a neural network-based decoding process, an inverse quantization process, and an entropy decoding process, and to execute a process of rearranging the quantization values ​​obtained by the entropy decoding process.

17. The video data processing program according to claim 16, which causes a computer to control the decoding process so as not to cross a slice boundary.

18. A video data processing program as claimed in claim 16 or 17, which causes a computer to execute predictive decoding processing on an encoding unit basis.

19. A storage medium for storing a bitstream generated by a video data processing device comprising a neural network, a quantization means, and an entropy coding means, the video data processing device comprising a rearrangement means for rearrangement of quantized values ​​output by the quantization means.

20. A storage medium for storing a bitstream generated by a video data processing method that performs a neural network-based encoding process, a quantization process, and entropy coding, and that rearranges the quantized values ​​created by the quantization process.

Images