Video data processing device and video data processing method

Abstract

In order to guarantee independence and divisibility in video encoding based on a neural network, this video data processing device includes a neural network, a quantization means, and an entropy encoding means, and includes a first control means that causes slice boundaries not to be crossed in encoding processing based on the neural network.

Description

Video data processing device and video data processing method

[0001] The present disclosure relates to a video data processing device and a video data processing method.

[0002] In order to transmit or record video efficiently, a video encoding device is used to generate a coded representation (hereinafter referred to as a bitstream) of input video, and a video decoding device is used to decode the bitstream to generate decoded video.

[0003] [Video Coding Based on Predictive Coding in Coding Units] Video coding standards include H.264 / AVC (Advanced Video Coding), H.265 / HEVC (High-Efficiency Video Coding), and H.266 / VVC (Versatile Video Coding), which are standardized by ITU-T SG16 and ISO / IEC / SC29. Another recent video coding technology is the technology described in Non-Patent Document 1.

[0004] In these video coding methods, video data is coded and decoded while being managed in a hierarchical structure. The hierarchical structure is made up of, for example, pictures that make up the video data, slices (or tiles) obtained by dividing pictures, coding tree units (CTUs) obtained by dividing slices, and coding units (CUs) obtained by dividing coding tree units.

[0005] The input image of the target CU is usually encoded earlier than the target CU and is predictively coded based on a predicted image generated based on the decoded image that has been decoded. That is, a prediction error image obtained by subtracting the predicted image from the input image is coded and decoded. Predictive coding includes intra-picture prediction (intra-prediction) that uses a decoded image included in a picture with the same display time as the target CU, and inter-picture prediction (inter-prediction) that uses a decoded image included in a picture with a different display time from the target CU.

[0006] The prediction error image is encoded based on frequency transform, quantization, and entropy coding. The prediction error image is decoded based on entropy decoding, inverse quantization, and inverse frequency transform. The frequency transform value of the quantized prediction error image is called a quantized value.

[0007] The functions provided by slices are independence (Independent Decoding) and divisibility (Data Partitioning). Independence is a function that allows decoding without using the decoding results of other slices in the same picture. Dividability is a function that allows dividing the bitstream into slices of any size.

[0008] [Video Coding Based on Neural Networks] Non-Patent Document 2 describes a new video coding technique that combines an auto-encoder, which is a type of neural network, quantization, and entropy coding.

[0009] An autoencoder compresses input data into a low-dimensional feature vector so that it contains only important features. The autoencoder then generates reconstructed data by reconstructing the low-dimensional feature vector back to its original dimensions. Figure 1 is an explanatory diagram showing the autoencoder algorithm. In Figure 1, the circular parts are called nodes and the arrows are called edges. The process of reducing the data into a low-dimensional feature vector (the first half) is called encoding. The process of generating reconstructed data (the second half) is called decoding.

[0010] The autoencoder is trained to minimize the reconstruction error (the difference between the input data and the reconstructed data). To obtain meaningful features, the autoencoder is designed to impose constraints on the encoding structure and to add regularization terms to the network's loss function.

[0011] "Algorithm description of Enhanced Compression Model 9(ECM 9)", JVET-AD2025, JVET of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 30th Meeting, Antalya, TR, 21-28 April 2023J. Ball'e, V. Laparra, and EP Simoncelli, "End-to-end Optimized Image "Compression", published as a conference paper at ICLR 2017

[0012] Neural network-based video coding has the drawback of not being able to provide the same independent and divisible slicing functionality as video coding based on predictive coding in the coding unit.

[0013] The reason why independence cannot be provided is that encoding on the encoding device side (encoding illustrated in FIG. 1 ) and decoding on the decoding device side (decoding illustrated in FIG. 1 ) may cross a slice boundary, and the reason why divisibility cannot be provided is that entropy encoding processing on the encoding device and entropy decoding processing on the decoding device may be continuous across a slice boundary.

[0014] Continuing the entropy coding process means not aligning the output. Furthermore, if the entropy coding is based on adaptive arithmetic coding, continuing the entropy coding process also includes not resetting the most probable symbols and their probabilities, and not resetting the arithmetic coding engine. On the other hand, continuing the corresponding entropy decoding process means not decoding the data for data alignment. Furthermore, if the entropy coding is based on adaptive arithmetic coding, continuing the entropy coding process also includes not resetting the most probable symbols and their probabilities, and not resetting the arithmetic decoding engine.

[0015] Furthermore, neural network-based video coding has the problem that dividing a picture into multiple slices increases the amount of processing, reduces coding efficiency, or both.

[0016] An object of the present invention is to provide a video data processing device and a video data processing method that provide, in neural network-based video coding, a slicing function similar to that of video coding based on predictive coding in a coding unit.

[0017] A video data processing device according to the present disclosure includes a neural network, a quantization means, and an entropy coding means, and also includes a first control means for preventing the encoding process based on the neural network from spanning a slice boundary.

[0018] Another video data processing device according to the present disclosure includes a neural network, an inverse quantization means, and an entropy decoding means, and includes first control means for preventing the decoding process based on the neural network from crossing a slice boundary.

[0019] A video data processing method according to the present disclosure performs encoding based on a neural network, quantization, and entropy coding, and controls the encoding so that it does not cross a slice boundary.

[0020] Another video data processing method according to the present disclosure performs a neural network-based decoding process, an inverse quantization process, and an entropy decoding process, and controls the decoding process so that it does not cross a slice boundary.

[0021] A video data processing program according to the present disclosure causes a computer to execute encoding processing based on a neural network, quantization processing, and entropy coding, and controls the encoding processing so as not to cross slice boundaries.

[0022] Another video data processing program according to the present disclosure causes a computer to execute a neural network-based decoding process, an inverse quantization process, and an entropy decoding process, and controls the decoding process so that it does not cross a slice boundary.

[0023] According to the present invention, neural network based video coding provides a slicing function similar to that of video coding based on predictive coding in the coding unit.

[0024] FIG. 1 is an explanatory diagram showing an algorithm of an autoencoder. FIG. 2 is a block diagram showing a video data processing system that encodes and decodes each picture constituting video data based on a neural network. FIG. 3 is an explanatory diagram showing an example of slice division in a video data processor on the encoding side. FIG. 4 is a flowchart showing an example of operation of a video data processor on the encoding side. FIG. 5 is an explanatory diagram showing an example of slice division in a video data processor on the decoding side. FIG. 6 is a flowchart showing an example of operation of a video data processor on the decoding side. FIG. 7 is a block diagram showing another example of a video data processing system that encodes and decodes each picture constituting video data based on a neural network. FIG. 8 is a flowchart showing another example of operation of a video data processor on the encoding side. FIG. 9 is a flowchart showing another example of operation of a video data processor on the decoding side. FIG. 10 is a block diagram showing example configurations of an encoder and a decoder. FIG. 11 is a block diagram showing another example configuration of a video data processor on the encoding side. FIG. 12 is a block diagram showing another example configuration of a video data processor on the decoding side. FIG. 13 is an explanatory diagram showing an example of slice division in a video data processor on the encoding side. FIG. 14 is a block diagram showing an example configuration of an information processing system. FIG. 15 is a block diagram showing main parts of a video data processing device. FIG. 16 is a block diagram showing main parts of a video data processing device of another aspect.

[0025] Hereinafter, an embodiment will be described with reference to the drawings.

[0026] Embodiment 1. Figure 2 is a block diagram showing video data processors 101 and 201 that encode and decode each picture constituting video data based on a neural network. The video data processor 101 is a video data processor on the encoding side that performs video processing (e.g., encoding processing) based on a neural network. The video data processor 201 is a video data processor on the decoding side that performs video processing (e.g., decoding processing) based on a neural network. A system including the video data processors 101 and 201 is called a video data processing system.

[0027] The video data processor 101 includes an encoder 1001 , a quantizer 1002 , an entropy encoder 1003 , a first control unit 1004 , a second control unit 1005 , and a multiplexer 1006 .

[0028] The video data processor 201 includes a decoder 2001 , an inverse quantizer 2002 , an entropy decoder 2003 , a first control unit 2004 , a second control unit 2005 , and a demultiplexer 2006 .

[0029] 2 simply indicates the direction of signal (data) flow, but does not exclude bidirectionality. This also applies to other block diagrams.

[0030] [Description on the Encoding Side] In the video data processor 101, the first control unit 1004 obtains feature vector domain slice boundaries based on input slice division information (information for identifying slice boundary information). Alternatively, the first control unit 1004 obtains information for identifying feature vector domain slice boundaries. The information for identifying feature vector domain slice boundaries may be, for example, image domain slice boundaries of each slice area. The first control unit 1004 controls the encoding process (hereinafter simply referred to as "encoding") by the encoder 1001 so that it does not cross image domain slice boundaries. Furthermore, the first control unit 1004 supplies position information of the feature vector domain slice boundaries as slice boundary information to the multiplexer 1006 via the second control unit 1005.

[0031] The encoder 1001 extracts features from the image of the input picture. Specifically, the encoder 1001 obtains a feature vector from the image of the input picture. However, under the control of the first control unit 1004, the encoder 1001 does not use input images of slice areas that are not the target of processing when obtaining the feature vector of the slice area to be processed. In other words, the feature vector of the slice area to be processed will be the same as the feature vector obtained by encoding the input images of the slice area to be processed as a single picture.

[0032] The quantizer 1002 quantizes the feature vector supplied from the encoder 1001 to obtain a quantized value.

[0033] The entropy encoder 1003 entropy encodes the quantized value supplied from the quantizer 1002 to obtain entropy-coded data.

[0034] The second control unit 1005 monitors the entropy encoder 1003 and causes the entropy encoder 1003 to terminate its processing after entropy encoding the last quantized value of each slice region.

[0035] The end of entropy coding means that the output of the entropy-coded data is data-aligned. That is, the entropy encoder 1003 adds alignment data for data alignment to the entropy-coded data. Furthermore, if the entropy coding is based on adaptive arithmetic coding, the entropy encoder 1003 also resets the most probable symbols and their probabilities, and the arithmetic coding engine, in order to end entropy coding.

[0036] The multiplexer 1006 multiplexes the slice boundary information and the entropy-encoded data supplied from the entropy encoder 1003, and outputs the multiplexed data as a bit stream.

[0037] Next, the operation of the video data processor 101 will be described. Here, the operation of the video data processor 101 will be described using an example in which an input picture is divided equally into three slices in the vertical direction. Fig. 3 is an explanatory diagram showing an example of slice division in the video data processor 101. Fig. 4 is a flowchart showing an example of the operation of the video data processor 101.

[0038] In Fig. 3, W_img indicates the width (horizontal size) of the input picture, H_img indicates the height (vertical size) of the input picture, W_tensor indicates the row size of the feature vector, and H_tensor indicates the column size of the feature vector.

[0039] Note that the example in which the input picture is divided equally vertically into three slices is merely an example, and the division method is not limited to this. For example, the input picture may be divided unevenly. The number of divisions may be two, three, or more.

[0040] In the video data processor 101, the encoder 1001 divides an input picture into a plurality of slices (step S101). In this embodiment, the input picture is divided equally into three slices in the vertical direction (see FIG. 3).

[0041] The encoder 1001 extracts features from each slice and generates a feature vector (step S102).

[0042] The quantizer 1002 quantizes the feature vector for each slice to generate a quantized value (step S103).

[0043] The entropy encoder 1003 entropy encodes the quantized value to generate entropy-encoded data (step S104).

[0044] Then, the entropy encoder 1003 terminates the process (step S105). As described above, the termination is performed under the control of the second control unit 1005.

[0045] The multiplexer 1006 multiplexes the slice boundary information and the entropy-coded data and outputs the multiplexed data as a bit stream (step S106).

[0046] [Description on the Decoding Side] In the video data processor 201, the demultiplexer 2006 demultiplexes the bitstream to obtain slice boundary information and entropy-encoded data. The demultiplexer 2006 supplies the slice boundary information to the second control unit 2005. The demultiplexer 2006 supplies the entropy-encoded data to the entropy decoder 2003.

[0047] The entropy decoder 2003 entropy decodes the entropy-encoded data supplied from the demultiplexer 2006 to obtain quantized values.

[0048] The inverse quantizer 2002 inversely quantizes the quantized values ​​supplied from the entropy decoder 2003 to obtain a reconstructed feature vector.

[0049] The first control unit 2004 controls the decoding process (hereinafter simply referred to as "decoding") by the decoder 2001 based on slice boundary information obtained via the second control unit 2005 so that it does not cross a feature vector domain slice boundary.

[0050] The decoder 2001 obtains a reconstructed image of a decoded picture from the reconstructed feature vector. However, under the control of the first control unit 2004, when obtaining a reconstructed image of a slice region to be processed, the decoder 2001 does not use the reconstructed feature vector of a slice region that is not the target of processing. In other words, the reconstructed image of a slice region to be processed will be the same as a reconstructed image obtained by decoding the reconstructed feature vector of the slice region to be processed as a single picture.

[0051] The second control unit 2005 monitors the entropy decoder 2003 and causes the entropy decoder 2003 to terminate its processing after entropy decoding the last quantized value of each slice region.

[0052] The end of entropy decoding means data alignment of the entropy-coded data, i.e., reading alignment data for data alignment from the entropy-coded data. Furthermore, if the entropy coding is based on adaptive arithmetic coding, the end of entropy decoding also involves resetting the most probable symbols and their probabilities, and resetting the arithmetic coding engine.

[0053] Next, the operation of the video data processor 201 will be described. Here, the operation of the video data processor 201 will be described using an example in which an input picture is divided equally into three slices in the vertical direction. Fig. 5 is an explanatory diagram showing an example of slice division in the video data processor 201. Fig. 6 is a flowchart showing an example of the operation of the video data processor 201.

[0054] In Fig. 5, W_tensor indicates the row size of the feature vector, H_tensor indicates the column size of the feature vector, W_img indicates the width (horizontal size) of the decoded picture, and H_img indicates the height (vertical size) of the decoded picture.

[0055] In the video data processor 201, the demultiplexer 2006 demultiplexes the bitstream (step S201). The demultiplexer 2006 obtains slice boundary information and entropy-encoded data through demultiplexing.

[0056] The entropy decoder 2003 entropy decodes the entropy-encoded data to obtain quantized values ​​(step S202).

[0057] The inverse quantizer 2002 inversely quantizes the quantized value (step S203). The inverse quantizer 2002 obtains a reconstructed feature vector through the inverse quantization.

[0058] The decoder 2001 obtains a reconstructed image of the decoded picture from the reconstructed feature vector (step S204). As described above, the decoder 2001 does not use the reconstructed feature vector of a slice area that is not the target of processing when obtaining a reconstructed image of the target slice area under the control of the first control unit 2004. In other words, the decoder 2001 performs decoding based on slice boundary information so as not to cross a feature vector domain slice boundary.

[0059] Then, the entropy decoder 2003 terminates the process (step S205). As described above, the termination is performed under the control of the second control unit 2005. Note that the video data processor 201 may perform the decoding process starting from step S201 when it has confirmed that the bitstream contains information indicating the termination or information indicating that such division has been performed.

[0060] In this embodiment, the control of the first control unit 1004 and the first control unit 2004 prevents encoding on the encoding side and decoding on the decoding side from crossing slice boundaries, thereby ensuring independence in neural network-based video coding.

[0061] Furthermore, by controlling the termination of the second control unit 1005 and the second control unit 2005, the entropy encoding process on the encoding side and the entropy decoding process on the decoding side are no longer continuous at slice boundaries, thereby ensuring divisibility in video encoding based on neural networks.

[0062] Embodiment 2 The video data processor 101 of the first embodiment may increase the amount of processing, decrease the coding efficiency, or both, due to MTU (Maximum Transfer Unit) size matching when transmitting the bitstream of each slice in packets.

[0063] The above possibilities arise as follows: 1) If the bitstream of a slice processed by division based on the set slice division information does not satisfy the MTU size (i.e., if it becomes larger than the MTU size), the video coding itself must be restarted, increasing the amount of processing. 2) If the number of slice divisions is increased from the beginning, division loss occurs, reducing coding efficiency.

[0064] FIG. 7 is a block diagram showing a video data processor 102 and a video data processor 202 that can eliminate the above-mentioned possibility, and that encode and decode each picture that constitutes video data based on a neural network.

[0065] The video data processor 102 includes an encoder 1001 , a quantizer 1002 , an entropy encoder 1003 A, a first control unit 1004 , a second control unit 1005 A, and a multiplexer 1006 .

[0066] The video data processor 202 includes a decoder 2001 , an inverse quantizer 2002 , an entropy decoder 2003 A, a first control unit 2004 , a second control unit 2005 A, and a demultiplexer 2006 .

[0067] [Description of the Encoding Side] In the video data processor 102, the encoder 1001, quantizer 1002, first control unit 1004, and multiplexer 1006 have the same configurations and functions as those in the first embodiment.

[0068] The entropy encoder 1003A entropy encodes the quantized value supplied from the quantizer 1002 and the end identification information supplied from the second control unit 1005A, to obtain entropy-coded data.

[0069] In addition to the functions of the second control unit 1005 in the first embodiment, the second control unit 1005A has a function of determining the value of termination identification information (e.g., 0: not terminated, 1: terminated) indicating whether or not to terminate entropy coding within a slice area at a predetermined interval (e.g., W_tensor, which is the width of one line of a feature vector) that is finer than the granularity of the slice area. The second control unit 1005A may use a value of 0 indicating termination and 1 indicating no termination as the value of the termination identification information.

[0070] For example, when the second control unit 1005A monitors the number of output bits of the entropy encoder 1003A and predicts that the size of the bitstream of the slice area to be processed will be larger than the MTU size, or when the last quantized value of the slice area has been entropy encoded, the second control unit 1005A sets the value of the end identification information to 1. In other cases, the second control unit 1005A sets the value of the end identification information to 0. When the value of the end identification information is 1, the second control unit 1005A causes the entropy encoder 1003A to terminate its processing.

[0071] Next, a description will be given of the operation of the video data processor 102. Fig. 8 is a flowchart showing an example of the operation of the video data processor 102. In this embodiment, as in the first embodiment, the operation of the video data processor 102 will be described using an example in which an input picture is divided equally into three slices in the vertical direction.

[0072] The processes in steps S101 to S103 are the same as those in the first embodiment.

[0073] In step S104A, the entropy encoder 1003A entropy encodes the quantized value supplied from the quantizer 1002 and the end identification information supplied from the second control unit 1005A to obtain entropy-coded data.

[0074] The processes in steps S105 and S106 are the same as those in the first embodiment. In this embodiment, the entropy-encoded data multiplexed in the process of step S106 also includes data relating to the end identification information.

[0075] [Description of Decoding Side] In the video data processor 202, the decoder 2001, inverse quantizer 2002, first control unit 2004, and demultiplexer 2006 have the same configurations and functions as those in the first embodiment.

[0076] The entropy decoder 2003A entropy decodes the entropy-encoded data supplied from the demultiplexer 2006 to obtain quantized values ​​and end identification information.

[0077] In addition to the functions of the second control unit 2005 in the first embodiment, the second control unit 2005A has the function of monitoring the termination identification information supplied from the entropy decoder 2003A and causing the entropy decoder 2003A to terminate its processing when the value is 1.

[0078] Next, a description will be given of the operation of the video data processor 202. Fig. 9 is a flowchart showing an example of the operation of the video data processor 202. In this embodiment, as in the first embodiment, the operation of the video data processor 202 will be described using an example in which an input picture is divided equally into three slices in the vertical direction.

[0079] The process of step S201 is the same as that in the first embodiment. Note that in this embodiment, the entropy-encoded data demultiplexed in the process of step S201 also includes data relating to end identification information.

[0080] In step S202A, the entropy decoder 2003A entropy decodes the entropy-encoded data supplied from the demultiplexer 2006 to obtain quantized values ​​and end identification information.

[0081] The processes in steps S203 and S204 are the same as those in the first embodiment.

[0082] In step S205A, the second control unit 2005A monitors the termination identification information supplied from the entropy decoder 2003A, and when the value is 1, causes the entropy decoder 2003A to terminate its processing.

[0083] In this embodiment, the bitstream of each slice can be terminated internally through processing by the entropy encoder 1003A, the second control unit 1005A, the entropy decoder 2003A, and the second control unit 2005A. That is, the bitstream can be divided into units smaller than the division specified in the slice division information, making it possible to reduce the slice size to less than the MTU size. As a result, it is possible to reduce the increase in the amount of video encoding processing and the decrease in encoding efficiency.

[0084] [Encoder and Decoder Configuration] Fig. 10 is a block diagram showing an example configuration of the encoder 1001 and decoder 2001 in the first and second embodiments. In Fig. 10, a "downward arrow (↓) 2" indicates subsampling by 1 / 2 (also called pooling). An "upward arrow (↑) 2" indicates upsampling by 2.

[0085] In the example shown in Fig. 10, the encoder 1001 is composed of four residual blocks and one convolution block. Each residual block is composed of two convolutional blocks and one shortcut link. Each convolutional block is composed of one convolution layer and one activation function.

[0086] The decoder 2001 consists of four residual blocks and one pixel shuffle convolution layer. The pixel shuffler is a mechanism proposed as sub-pixel convolution. The pixel shuffler rearranges input feature vectors and outputs high-resolution feature vectors.

[0087] It should be noted that the configuration shown in FIG. 10 is just an example, and the configurations of the encoder 1001 and the decoder 2001 are not limited to the configuration shown in FIG.

[0088] Embodiment 3 Fig. 11 is a block diagram showing another example of the configuration of a video data processor on the encoding side, and Fig. 12 is a block diagram showing another example of the configuration of a video data processor on the decoding side.

[0089] The video data processor 103 shown in FIG. 11 includes a switch 3000, an NN video encoder 3001, an NN video decoder 3003, a CP video encoder 3002, a CP video decoder 3004, a decoded picture buffer 3005, and a multiplexer 4000 that performs multiplexing processing of entropy-encoded data and other information.

[0090] Note that "NN" stands for neural network, and "CP" stands for coding unit-based prediction.

[0091] The CP video encoder 3002 performs video encoding processing using a video encoding method based on predictive coding for each coding unit. The CP video decoder 3004 performs decoding processing using a video encoding method based on predictive coding for each coding unit. The decoded picture buffer 3005 is a storage unit that stores decoded pictures (reconstructed pictures). As described above, video encoding methods that comply with H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc. can be used.

[0092] The video data processor 101 or the video data processor 102 in each of the above embodiments can be used as the NN video encoder 3001. The video data processor 201 or the video data processor 202 in each of the above embodiments can be used as the NN video decoder 3003.

[0093] The video data processor 203 shown in FIG. 12 includes a demultiplexer 5000 that demultiplexes a bitstream, an NN video decoder 3003 , a CP video decoder 3004 , and a decoded picture buffer 3005 .

[0094] In other words, video data processor 103 shown in FIG. 11 and video data processor 203 shown in FIG. 12 are video data processors in which the above-mentioned video data processor 101 or video data processor 102 is used as an NN video encoder 3001, the above-mentioned video data processor 201 or video data processor 202 is used as an NN video decoder 3003, and these are combined with a CP video encoder 3002 and a CP video decoder 3004 that are based on a video encoding method based on predictive encoding in the encoding unit.

[0095] [Description of the Encoding Side] In the video data processor 103 shown in FIG. 11, a switch 3000 supplies an input picture to either an NN video encoder 3001 or a CP video encoder 3002 .

[0096] The NN video encoder 3001 operates in the same manner as the video data processor 101 of the first embodiment or the video data processor 102 of the second embodiment to generate a bitstream (also referred to as an NN bitstream).

[0097] The NN video decoder 3003 receives the NN bitstream supplied from the NN video encoder 3001, and operates in the same manner as the video data processor 201 of the first embodiment or the video data processor 202 of the second embodiment to obtain decoded pictures (also referred to as NN decoded pictures). The NN video decoder 3003 stores the NN decoded pictures in the decoded picture buffer 3005.

[0098] The CP video encoder 3002 uses the input picture and the decoded picture stored in the decoded picture buffer 3005 to perform video encoding based on predictive encoding in the encoding unit, and generates a bitstream (also called a CP bitstream).

[0099] The CP video decoder 3004 receives the CP bitstream supplied from the CP video encoder 3002, performs entropy decoding, and then performs decoding based on predictive coding in the coding unit to obtain a decoded picture (also referred to as a CP decoded picture). The CP video decoder 3004 stores the CP decoded picture in the decoded picture buffer 3005.

[0100] The decoded pictures stored in the decoded picture buffer 3005 are used as reference pictures.

[0101] In the configuration shown in Fig. 11, the NN video decoder 3003 receives an NN bitstream from the NN video encoder 3001 and performs entropy decoding. The NN video decoder 3003 then obtains NN-decoded pictures from the encoded data obtained by entropy decoding. However, the NN video decoder 3003 may also be configured to receive intermediate data (e.g., quantized values) before entropy encoding from the NN video encoder 3001 and obtain NN-decoded pictures from the intermediate data. In this case, the NN video decoder 3003 does not need to perform entropy decoding.

[0102] 11 , the CP video decoder 3004 receives the CP bitstream from the CP video encoder 3002 and performs entropy decoding. The CP video decoder 3004 then obtains CP-decoded pictures from the encoded data obtained by entropy decoding. However, the CP video decoder 3004 may also be configured to receive intermediate data (e.g., quantized values) before entropy encoding from the CP video encoder 3002 and obtain CP-decoded pictures from the intermediate data. In this case, the CP video decoder 3004 does not need to perform entropy decoding.

[0103] 12, the NN video decoder 3003 operates in the same manner as the video data processor 201 of the first embodiment or the video data processor 202 of the second embodiment, based on the coded data obtained by demultiplexing the NN bitstream, to obtain an NN-decoded picture. The NN video decoder 3003 stores the NN-decoded picture in a decoded picture buffer 3005.

[0104] The CP video decoder 3004 demultiplexes the CP bitstream, performs entropy decoding, and obtains CP decoded pictures based on the resulting coded data. The CP video decoder 3004 stores the CP decoded pictures in the decoded picture buffer 3005.

[0105] The video data processor 203 outputs the NN decoded picture or the CP decoded picture stored in the decoded picture buffer 3005 as a decoded picture.

[0106] [Variation 1] In each of the above embodiments, both slice independence and divisibility are satisfied. However, only slice divisibility may be satisfied. In that case, the first control unit 1004 may not be provided in the video data processors 101 and 102. Furthermore, the first control unit 2004 may not be provided in the video data processors 201 and 202.

[0107] [Variation 2] Furthermore, if complete slice independence is not required, the first control unit 1004 in the video data processors 101 and 102 may relax the slice boundaries that must not be straddled. For example, when encoding the feature vector of a slice region to be processed, a configuration may be considered in which the input image of the slice region corresponding to the subsequent slice in the bitstream is not straddled. That is, the feature vector of slice region 1 (see FIG. 3 ) is generated using the input image of slice region 1. The feature vector of slice region 2 is generated using the input images of slice region 1 and slice region 2. The feature vector of slice region 3 is generated using the input images of slice regions 1, 2, and 3. Although this configuration (Variation 2) does not achieve complete independence, it can mitigate the decrease in coding efficiency associated with partitioning loss.

[0108] In addition, in a video data processor on the decoding side corresponding to the video data processor of Modification 2, when decoding the reconstructed feature vector of the slice area to be processed, the first control unit 2004 controls so that the reconstructed feature vector does not cross over the reconstructed feature vector of the slice area corresponding to the subsequent slice in the bitstream. That is, when obtaining a reconstructed image of slice area 1, the decoder 2001 uses the reconstructed feature vector of slice area 1 (see FIG. 5 ). When obtaining a reconstructed image of slice area 2, the decoder 2001 uses the reconstructed feature vectors of slice area 1 and slice area 2. When obtaining a feature reconstructed image of slice area 3, the decoder 2001 uses the reconstructed feature vectors of slice areas 1, 2, and 3.

[0109] [Variation 3] When complete independence of slices is not required, the first control unit 1004 in the video encoders 101 and 102 may relax the slice boundaries that cannot be spanned. For example, when encoding the feature vector of a slice region to be processed, a conceivable mode is to limit the range in which an input image of a slice region corresponding to a subsequent slice in the bitstream is spanned.

[0110] FIG. 13 is an explanatory diagram showing an example of slice division in a video data processor on the encoding side in Modification 3. In FIG.

[0111] The encoder 1001 generates the feature vector of slice area 1 in FIG. 13 using the input images of slice area 1 and a predetermined area adjacent to the slice area (hereinafter referred to as the "look-ahead area": ​​the area shaded in FIG. 13). The encoder 1001 generates the feature vector of slice area 2 using the input images of slice area 1, slice area 2, and a look-ahead area adjacent to slice area 2. When generating the feature vector of slice area 3, the encoder 1001 does not use the look-ahead area adjacent to slice area 3 because there is no look-ahead area adjacent to slice area 3. In other words, the encoder 1001 generates the feature vector of slice area 3 using the input images of slice area 1, slice area 2, and slice area 3.

[0112] In this example, complete independence cannot be achieved, but the decrease in encoding efficiency due to partitioning loss can be alleviated more effectively than in Modification 2. However, the processing delay is longer than in Modification 2 by the time required to wait for input of the look-ahead region adjacent to the current slice. Naturally, if the height ph of the look-ahead region is set to the maximum value of the range referenced by the encoding process, the feature vector of the entire picture will match that in the case without slice partitioning. In the example encoder configuration shown in FIG. 10, if the kernel size of the convolution constituting each block is 3 × 3, the matching ph is 29.

[0113] The operation of a video decoder corresponding to the video encoder of Modification 3 is the same as the operation of the video decoder of Modification 2. That is, in the video decoder of this modification, when decoding the reconstructed feature vector of a slice area to be processed, the first control unit 2004 controls the decoding so that the reconstructed feature vector of a slice area corresponding to a subsequent slice in the bitstream does not overlap. Specifically, when obtaining a reconstructed image of slice area 1, the decoder 2001 uses the reconstructed feature vector of slice area 1 (see FIG. 5 ). When obtaining a reconstructed image of slice area 2, the decoder 2001 uses the reconstructed feature vectors of slice area 1 and slice area 2. When obtaining a feature reconstructed image of slice area 3, the decoder 2001 uses the reconstructed feature vectors of slice areas 1, 2, and 3.

[0114] [Variation 4] In the second and third embodiments, the multiplexer 1006 implicitly multiplexes termination identification information, which indicates whether entropy coding is to terminate within a slice region, into the bitstream at predetermined intervals (W_tensor). However, the multiplexer 1006 may explicitly embed the predetermined intervals in the bitstream. Furthermore, the predetermined intervals may be something other than W_tensor, as long as the granularity is finer than that of a slice region.

[0115] The slice boundary information described above can be configured as follows: When a frame is divided into slices of equal size, the information includes the number of slice divisions and the coordinates of the upper left corner of each slice or the slice number. When a frame is divided into slices of non-uniform size, the information includes the coordinates of the upper left corner of each slice and the horizontal and vertical widths of each slice. In the example shown in FIG. 3 , the slice numbers are reference numbers assigned to three slices (slice regions) in order from top to bottom. Slice numbers may also be assigned in order from bottom to top.

[0116] Each of the above embodiments can be configured by hardware, but can also be realized by a computer program.

[0117] The information processing system shown in Fig. 14 includes a processor 701 such as one or more CPUs (Central Processing Units), a program memory 702, a storage medium 703 for storing video data, and a storage medium 704 for storing a bitstream. The storage medium 703 and the storage medium 704 may be separate storage media or may be storage areas formed by the same storage medium. A magnetic storage medium such as a hard disk can be used as the storage medium.

[0118] In the information processing system, a program memory 702 stores a program (video data processing program) for realizing the functions of each block shown in each of the above embodiments.

[0119] The processor 701 executes processing in accordance with the program stored in the program memory 702, thereby realizing the functions of the video data processors 101, 102, and 103 and the video data processors 201, 202, and 203 shown in each embodiment.

[0120] For example, the functions of the video data processors 101, 102, and 103 are realized by the processor 701 executing processing in accordance with a video data processing program for realizing the functions of each block (except the decoded picture buffer 3005) in the video data processors 101, 102, and 103 shown in Figures 2, 7, and 11. Also, the functions of the video data processors 201, 202, and 203 are realized by the processor 701 executing processing in accordance with a video data processing program for realizing the functions of each block (except the decoded picture buffer 3005) in the video data processors 201, 202, and 203 shown in Figures 2, 7, and 12.

[0121] At least the program memory 702 is a non-transitory computer-readable medium. However, the program may be stored in various types of transitory computer-readable medium. The program is supplied to the transitory computer-readable medium, for example, via a wired or wireless communication channel, i.e., via an electrical signal, an optical signal, or an electromagnetic wave.

[0122] 15 is a block diagram showing the main components of a video data processing device. The video data processing device 10 (implemented by video data processors 101, 102, and 103 in the embodiment) shown in Fig. 15 includes a neural network 11 (implemented by an encoder 1001 in the embodiment), a quantization means 12 (implemented by a quantizer 1002 in the embodiment), and an entropy coding means 13 (implemented by entropy coders 1003 and 1003A in the embodiment), as well as a first control means 14 (implemented by a first control unit 1004 in the embodiment) that prevents the encoding process based on the neural network 11 from spanning a slice boundary.

[0123] 16 is a block diagram showing the main components of another aspect of a video data processing device. A video data processing device 20 (implemented by video data processors 201, 202, and 203 in this embodiment) shown in FIG. 16 includes a neural network 21 (implemented by a decoder 2001 in this embodiment), an inverse quantization unit 22 (implemented by an inverse quantizer 2002 in this embodiment), and an entropy decoding unit 23 (implemented by entropy decoders 2003 and 2003A in this embodiment), as well as a first control unit 24 (implemented by a first control unit 2004 in this embodiment) that prevents the decoding process based on the neural network 21 from spanning a slice boundary.

[0124] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes.

[0125] (Supplementary Note 1) A video data processing device comprising a neural network, a quantization means, and an entropy encoding means, the video data processing device further comprising a first control means for preventing an encoding process based on the neural network from crossing a slice boundary.

[0126] (Supplementary Note 2) A video data processing device according to Supplementary Note 1, further comprising a second control means (in the embodiment, realized by a second control unit 1005, 1005A) for terminating entropy coding within a slice, wherein the entropy coding means entropy codes identification information indicating whether the entropy coding has terminated.

[0127] (Supplementary Note 3) The video data processing device of Supplementary Note 1 or Supplementary Note 2, further comprising predictive coding means (implemented by the CP video coder 3002 in the embodiment) that performs predictive coding on a coding unit basis.

[0128] (Supplementary Note 4) A video data processing device comprising a neural network, an inverse quantization means, and an entropy decoding means, further comprising first control means for preventing a decoding process based on the neural network from crossing a slice boundary.

[0129] (Appendix 5) The video data processing device of Appendix 4, wherein the entropy decoding means entropy decodes identification information indicating whether entropy coding has terminated within the slice, and includes second control means (in the embodiment, realized by the second control unit 2005, 2005A) that terminates entropy decoding within the slice based on the identification information.

[0130] (Supplementary Note 6) The video data processing device of Supplementary Note 4 or Supplementary Note 5, further comprising predictive decoding means (implemented by the CP video decoder 3004 in the embodiment) that performs predictive decoding on a coding unit basis.

[0131] (Supplementary Note 7) A video data processing method that performs encoding based on a neural network, quantization, and entropy coding, and controls the encoding so as not to cross a slice boundary.

[0132] (Supplementary Note 8) The video data processing method of Supplementary Note 7, wherein control is performed to terminate entropy coding within a slice, and identification information indicating whether entropy coding has terminated is entropy coded.

[0133] (Supplementary Note 9) The video data processing method according to Supplementary Note 7 or Supplementary Note 8, wherein predictive coding processing is performed on a coding unit basis.

[0134] (Supplementary Note 10) A video data processing method that executes a neural network-based decoding process, an inverse quantization process, and an entropy decoding process, the video data processing method controlling the decoding process so as not to cross a slice boundary.

[0135] (Supplementary Note 11) A video data processing method according to Supplementary Note 10, which entropy-decodes identification information indicating whether entropy coding has terminated within a slice, and controls to terminate entropy decoding within the slice based on the identification information.

[0136] (Supplementary Note 12) The video data processing method according to Supplementary Note 10 or Supplementary Note 11, wherein predictive decoding is performed on a coding unit basis.

[0137] (Supplementary Note 13) A video data processing program for causing a computer to execute encoding processing, quantization processing, and entropy coding based on a neural network, and for controlling the encoding processing so as not to cross a slice boundary.

[0138] (Appendix 14) The video data processing program of Appendix 13, which causes a computer to perform control to terminate entropy coding within a slice, and to entropy code identification information indicating whether entropy coding has terminated.

[0139] (Supplementary Note 15) The video data processing program of Supplementary Note 13 or Supplementary Note 14, which causes a computer to execute predictive coding processing on a coding unit basis.

[0140] (Supplementary Note 16) A video data processing program that causes a computer to execute a neural network-based decoding process, an inverse quantization process, and an entropy decoding process, and controls the decoding process so that it does not cross a slice boundary.

[0141] (Appendix 17) A video data processing program according to Appendix 16, which causes a computer to entropy decode identification information indicating whether entropy coding has terminated within a slice, and to control the termination of entropy decoding within the slice based on the identification information.

[0142] (Supplementary Note 18) The video data processing program of Supplementary Note 16 or Supplementary Note 17, which causes a computer to execute predictive decoding processing on a coding unit basis.

[0143] (Supplementary Note 19) A storage medium for storing a bitstream generated by a video data processing device comprising a neural network, a quantization means, and an entropy coding means, the video data processing device comprising a first control means for preventing the encoding process based on the neural network from crossing a slice boundary.

[0144] (Supplementary Note 20) A storage medium for storing a bitstream generated by a video data processing method that performs encoding processing based on a neural network, quantization processing, and entropy coding, and controls the encoding processing so that it does not cross a slice boundary.

[0145] (Supplementary Note 21) A video data processing device comprising a neural network, a quantization means, and an entropy coding means, further comprising second control means for terminating entropy coding within a slice, wherein the entropy coding means entropy codes identification information indicating whether the entropy coding has terminated. Note that this supplementary note corresponds to the above-mentioned variant 1.

[0146] (Supplementary Note 22) The video data processing device according to Supplementary Note 21, wherein the encoding process based on the neural network does not span only slice boundaries of subsequent slice regions in bitstream order. This Supplementary Note corresponds to the above-mentioned Modification 2.

[0147] (Supplementary Note 23) The video data processing device of Supplementary Note 22, wherein a predetermined interval is multiplexed into a bit stream, and the entropy encoding means entropy encodes identification information indicating whether or not the second control means has terminated entropy encoding at the predetermined interval. Note that this supplementary note corresponds to the above-mentioned Variation 3.

[0148] (Supplementary Note 24) A video data processing device comprising a neural network, an inverse quantization means, and an entropy decoding means, wherein the entropy decoding means entropy decodes identification information indicating whether entropy coding has terminated within a slice, and the video data processing device comprises second control means for terminating the entropy decoding within the slice based on the identification information. Note that this supplementary note corresponds to the above-mentioned variant example 1.

[0149] (Supplementary Note 25) The video data processing device of Supplementary Note 24, wherein the decoding process based on the neural network does not cross only a slice boundary of a slice area that follows in bitstream order. Note that this supplementary note corresponds to the above-mentioned modified example 2.

[0150] (Supplementary Note 26) The video data processing device of Supplementary Note 25, wherein a predetermined interval is demultiplexed from the bit stream, and the entropy decoding means entropy decodes identification information indicating whether entropy coding has terminated at the predetermined interval. Note that this supplementary note corresponds to the above-mentioned variant example 3.

[0151] This application claims priority based on Japanese Patent Application No. 2023-155647 filed on September 21, 2023, and Japanese Patent Application No. 2024-035447 filed on March 8, 2024, the disclosures of which are incorporated herein in their entireties.

[0152] REFERENCE SIGNS LIST 10 Video data processing device 11 Neural network 12 Quantization means 13 Entropy encoding means 14 First control means 20 Video data processing device 21 Neural network 22 Inverse quantization means 23 Entropy decoding means 24 First control means 101, 102, 103 Video data processor 201, 202, 203 Video data processor 701 Processor 702 Program memory 703, 704 Storage medium 1001 Encoder 1002 Quantizer 1003, 1003A Entropy encoder 1004 First control unit 1005, 1005A Second control unit 1006 Multiplexer 2001 Decoder 2002 Inverse quantizer 2003, 2003A Entropy decoder 2004 First control unit 2005, 2005A Second control unit 2006 Demultiplexer 3000 Switch 3001 NN video encoder 3002 CP video encoder 3003 NN video decoder 3004 CP video decoder 3005 Decoded picture buffer 4000 Multiplexer 5000 Demultiplexer

Claims

1. A video data processing device comprising a neural network, a quantization means, and an entropy coding means, the video data processing device further comprising a first control means for preventing an encoding process based on the neural network from spanning a slice boundary.

2. A video data processing device according to claim 1, further comprising a second control means for terminating entropy coding within a slice, said entropy coding means entropy coding identification information indicating whether or not said entropy coding has been terminated.

3. The video data processing device according to claim 1 or 2, further comprising predictive coding means for performing predictive coding on a coding unit basis.

4. A video data processing device comprising a neural network, an inverse quantization means, and an entropy decoding means, the video data processing device further comprising a first control means for preventing a decoding process based on the neural network from spanning a slice boundary.

5. A video data processing device as described in claim 4, wherein the entropy decoding means entropy decodes identification information indicating whether or not entropy encoding has terminated within the slice, and further comprises second control means for terminating entropy decoding within the slice based on the identification information.

6. The video data processing device according to claim 4 or 5, further comprising predictive decoding means for performing predictive decoding on a coding unit basis.

7. A video data processing method that performs encoding based on a neural network, quantization, and entropy coding, and controls the encoding so as not to cross a slice boundary.

8. The video data processing method according to claim 7, further comprising the steps of: controlling to terminate entropy coding within a slice; and entropy coding identification information indicating whether or not entropy coding has terminated.

9. The video data processing method according to claim 7 or 8, wherein predictive coding processing is executed on a coding unit basis.

10. A video data processing method that executes a neural network-based decoding process, an inverse quantization process, and an entropy decoding process, and controls the decoding process so that it does not cross a slice boundary.

11. The video data processing method according to claim 10, further comprising: entropy decoding identification information indicating whether or not entropy encoding has terminated within a slice; and controlling to terminate entropy decoding within the slice based on the identification information.

12. A video data processing method according to claim 10 or 11, wherein predictive decoding processing is executed on a coding unit basis.

13. A video data processing program for causing a computer to execute encoding processing, quantization processing, and entropy coding based on a neural network, and for controlling the encoding processing so as not to cross slice boundaries.

14. The video data processing program according to claim 13, which causes a computer to perform control to terminate entropy coding within a slice, and to entropy code identification information indicating whether or not entropy coding has terminated.

15. A video data processing program as claimed in claim 13 or 14, which causes a computer to execute predictive coding processing on a coding unit basis.

16. A video data processing program that causes a computer to execute decoding processing, inverse quantization processing, and entropy decoding processing based on a neural network, and controls the decoding processing so that it does not cross a slice boundary.

17. A video data processing program as described in claim 16, which causes a computer to entropy decode identification information indicating whether or not entropy encoding has terminated within a slice, and to control the termination of entropy decoding within the slice based on the identification information.

18. A video data processing program as claimed in claim 16 or 17, which causes a computer to execute predictive decoding processing on an encoding unit basis.

19. A storage medium for storing a bitstream generated by a video data processing device having a neural network, a quantization means, and an entropy encoding means, the video data processing device having a first control means that prevents the encoding process based on the neural network from crossing slice boundaries.

20. A storage medium for storing a bitstream generated by a video data processing method that performs neural network-based encoding, quantization, and entropy coding, and controls the encoding so that it does not cross slice boundaries.

Images