Video data processing apparatus and video data processing method

Abstract

In order to ensure independence and separability in neural network-based video data processing, a video data processing apparatus includes a neural network, a quantization means, and an entropy encoding means, and further includes a first control means that prevents a neural network-based encoding process from crossing slice boundaries.

Description

Technical Field

[0001] This invention relates to video data processing equipment and video data processing methods. Background Technology

[0002] In order to efficiently transmit or record video, video encoding devices are used to generate an encoded representation (hereinafter referred to as a bitstream) obtained by encoding the input video, and video decoding devices are used to decode the bitstream to generate a decoded video.

[0003] Video coding based on predictive coding in coding units

[0004] Examples of video coding scheme standards include H.264 / AVC (Advanced Video Coding), H.265 / HEVC (High Efficiency Video Coding), and H.266 / VVC (Multi-Functional Video Coding), which are standardized by ITU-T SG16 and ISO / IEC / SC29. As a more recent video coding technology, the technology described in NPL 1 exists.

[0005] In these video coding schemes, video data is managed in a hierarchical structure and is encoded and decoded. The hierarchical structure includes, for example, pictures that constitute the video data, slices obtained by segmenting (dividing) the pictures (or tiles), coding tree units (CTUs) obtained by segmenting the slices, and coding units (CUs) obtained by segmenting the coding tree units.

[0006] The input image for processing the target CU is typically encoded relative to the target CU in the past, and is predictively encoded based on a prediction image generated from the decoded image. That is, a prediction error image obtained by subtracting the prediction image from the input image is encoded and decoded. Predictive encoding includes intra-frame prediction (intra-prediction) using the decoded image included in the picture at the same display time as the target CU's display time, and inter-frame prediction (inter-prediction) using the decoded image included in the picture at different display times than the target CU's display time.

[0007] The prediction error image is encoded using frequency transform, quantization, and entropy coding. The prediction error image is decoded using entropy decoding, inverse quantization, and inverse frequency transform. The frequency transform value of the quantized prediction error image is called the quantization value.

[0008] The functionality provided by slicing includes independent decoding and data segmentation. Independent decoding is a function used to decode without using the decoding results of another slice from the same image. Data segmentation is a function used to divide the bitstream into segments of any size.

[0009] [Neural Network-Based Video Coding]

[0010] NPL2 discloses a novel video coding technique that combines autoencoders, quantization, and entropy coding, and is one of the neural network techniques.

[0011] An autoencoder compresses the input data into a low-dimensional feature vector so that only the important features are included. The autoencoder then generates reconstructed data by reconstructing the low-dimensional feature vector back to the original dimensions. Figure 1 This is an explanatory diagram illustrating the algorithm of an automatic encoder.

[0012] exist Figure 1 In this diagram, the circular parts are called nodes, and the arrows are called edges. The process of discarding the low-dimensional feature vectors (the first half) is called encoding. The process of generating the reconstructed data (the second half) is called decoding.

[0013] The autoencoder continues learning in order to minimize the reconstruction error (the difference between the input data and the reconstructed data). To obtain meaningful feature quantities, the autoencoder is designed to add constraints to the structure of the encoding or to add regularization terms to the network's loss function.

[0014] Reference List

[0015] Non-patent literature

[0016] NPL 1: "Algorithm description of Enhanced Compression Model 9 (ECM 9)", JVET-AD2025, 30th meeting of JVET and ISO / IEC JTC 1 / SC 29, ITU-T SG 16 WP 3, Antalya, TR, April 21-28, 2023.

[0017] NPL 2: J. Ball'e, V. Larpara and Esimoncelli, “End-to-end Optimized Image Compression”, presented as a conference paper at ICLR 2017. Summary of the Invention

[0018] Technical issues

[0019] Neural network-based video coding, like predictive coding in coding units, suffers from the inability to provide independent decoding and data segmentation slicing capabilities.

[0020] The reason why independent decoding cannot be provided is that the encoding on the encoding device side ( Figure 1 The encoding shown) and the decoding on the decoding device side ( Figure 1The decoding shown can be performed beyond (across) slice boundaries. The reason why data segmentation cannot be provided is that the entropy encoding process on the encoding side and the entropy decoding process on the decoding side may be continuous at slice boundaries.

[0021] A series of entropy coding processes implies that the output is not data aligned. Furthermore, in the case where entropy coding is based on adaptive arithmetic coding, the series of entropy coding processes also includes not resetting the more likely symbols and their probabilities, and not resetting the arithmetic coding engine. Simultaneously, a series of entropy decoding processes implies that the data used for data alignment is not decoded. Furthermore, in the case where entropy coding is based on adaptive arithmetic coding, the series of entropy coding processes also includes not resetting the more likely symbols and their probabilities, and not resetting the arithmetic decoding engine.

[0022] Neural network-based video coding suffers from increased processing power, reduced coding efficiency, or both, when segmenting an image into multiple slices.

[0023] The purpose of this invention is to provide a video data processing device and a video data processing method that provide slicing functionality similar to predictive coding in coding units in neural network-based video coding.

[0024] Solution to the problem

[0025] The video data processing apparatus according to this disclosure includes a neural network, a quantization device, and an entropy coding device, and includes a first control device for preventing neural network-based coding processing from crossing slice boundaries.

[0026] According to another aspect of this disclosure, a video data processing apparatus includes a neural network, an inverse quantization device, and an entropy decoding device, and includes a first control device for preventing neural network-based decoding processing from crossing slice boundaries.

[0027] The video data processing method disclosed herein performs neural network-based encoding processing, quantization processing, and entropy encoding, and performs control to ensure that the encoding processing does not cross slice boundaries.

[0028] According to another aspect of the video data processing method of this disclosure, neural network-based decoding processing, inverse quantization processing, and entropy decoding processing are performed, and control is exercised to ensure that the decoding processing does not cross slice boundaries.

[0029] The video data processing procedure according to this disclosure enables a computer to perform neural network-based encoding processing, quantization processing, and entropy encoding, and performs control such that the encoding processing does not cross slice boundaries.

[0030] According to another aspect of this disclosure, a video data processing program causes a computer to perform neural network-based decoding, inverse quantization, and entropy decoding, and performs control such that the decoding process does not cross slice boundaries.

[0031] Beneficial effects of the present invention

[0032] According to the present invention, in neural network-based video coding, a slicing function similar to that of video coding based on predictive coding in coding units is provided. Attached Figure Description

[0033] Figure 1 This is an explanatory diagram depicting the algorithm of an automatic encoder.

[0034] Figure 2 It is a block diagram depicting a video data processing system that encodes and decodes each image that constitutes video data based on a neural network.

[0035] Figure 3 This is an illustrative diagram depicting an example of slice segmentation in a video data processor on the encoding side.

[0036] Figure 4 This is a flowchart illustrating an example of the operation of a video data processor on the encoding side.

[0037] Figure 5 This is an illustrative diagram depicting an example of slice segmentation in a video data processor on the decoding side.

[0038] Figure 6 This is a flowchart illustrating an example of the operation of a video data processor on the decoding side.

[0039] Figure 7 This is a block diagram illustrating another example of a video data processing system that encodes and decodes each image that constitutes video data based on a neural network.

[0040] Figure 8 This is a flowchart illustrating another operational example of a video data processor on the encoding side.

[0041] Figure 9 This is a flowchart illustrating another operational example of a video data processor on the decoding side.

[0042] Figure 10 It is a block diagram depicting an example configuration of the encoder and decoder.

[0043] Figure 11 This is a block diagram depicting another configuration example of a video data processor on the encoding side.

[0044] Figure 12This is a block diagram illustrating another configuration example of a video data processor on the decoding side.

[0045] Figure 13 This is an illustrative diagram depicting an example of slice segmentation in a video data processor on the encoding side.

[0046] Figure 14 It is a block diagram depicting an example configuration of an information processing system.

[0047] Figure 15 It is a block diagram depicting the main parts of a video data processing device.

[0048] Figure 16 It is a block diagram depicting the main parts of another side of the video data processing device. Detailed Implementation

[0049] In the following description, exemplary embodiments will be illustrated with reference to the accompanying drawings.

[0050] First example embodiment.

[0051] Figure 2 This is a block diagram illustrating video data processors 101 and 201 that encode and decode each image constituting video data based on a neural network. Video data processor 101 is a video data processor on the encoding side that performs video processing (e.g., encoding processing) based on a neural network. Video data processor 201 is a video data processor on the decoding side that performs video processing (e.g., decoding processing) based on a neural network. A system including video data processors 101 and 201 is called a video data processing system.

[0052] The video data processor 101 includes an encoder 1001, a quantizer 1002, an entropy encoder 1003, a first control unit 1004, a second control unit 1005, and a multiplexer 1006.

[0053] The video data processor 201 includes a decoder 2001, an inverse quantizer 2002, an entropy decoder 2003, a first control unit 2004, a second control unit 2005, and a demultiplexer 2006.

[0054] Figure 2 The arrows clearly indicate the direction of signal (data) flow, but do not exclude bidirectionality. The same applies to other block diagrams.

[0055] [Description of the encoding side]

[0056] In the video data processor 101, the first control unit 1004 obtains the slice boundaries in the feature vector domain based on the input slice segmentation information (information used to specify slice boundary information). Alternatively, the first control unit 1004 obtains information used to specify the slice boundaries in the feature vector domain. This information may be, for example, the slice boundaries in the image domain for each slice region. The first control unit 1004 performs control to ensure that the encoding process (hereinafter referred to as encoding) by the encoder 1001 does not cross the slice boundaries in the image domain. Furthermore, the first control unit 1004 supplies the position information of the slice boundaries in the feature vector domain as slice boundary information to the multiplexer 1006 via the second control unit 1005.

[0057] Encoder 1001 extracts feature values ​​from the image of the input image. Specifically, encoder 1001 obtains feature vectors from the image of the input image. However, when obtaining the feature vector of the slice region to be processed under the control of the first control unit 1004, encoder 1001 does not use the input image of slice regions that are not the processing target. That is, the feature vector of the slice region to be processed is similar to the feature vector obtained by encoding the input image of the slice region to be processed into an image.

[0058] The quantizer 1002 quantizes the feature vector supplied by the encoder 1001 to obtain the quantized value.

[0059] The entropy encoder 1003 entropy-encodes the quantized values ​​supplied from the quantizer 1002 to obtain entropy-encoded data.

[0060] The second control unit 1005 monitors the entropy encoder 1003 and terminates the entropy encoder 1003 after performing entropy encoding on the last quantized value of each slice region.

[0061] The termination of the entropy coding device signifies the alignment of the entropy-coded data output. That is, the entropy encoder 1003 adds alignment data for data alignment to the entropy-coded data. Furthermore, in the case where entropy coding is based on adaptive arithmetic coding, the entropy encoder 1003 further resets the more likely symbols and their probabilities, as well as the arithmetic coding engine used for the termination of entropy coding.

[0062] Multiplexer 1006 multiplexes the slice boundary information and entropy-encoded data supplied from entropy encoder 1003 and outputs the multiplexed data as a bit stream.

[0063] Next, the operation of the video data processor 101 will be described. Here, the operation of the video data processor 101 will be described using an example where the input image is equally divided into three slices in the vertical direction. Figure 3 This is an illustrative diagram showing an example of slice segmentation in the video data processor 101. Figure 4This is a flowchart illustrating an operational example of the video data processor 101.

[0064] exist Figure 3 In the diagram, W_img indicates the width (horizontal size) of the input image. H_img indicates the height (vertical size) of the input image. W_tensor indicates the size of the feature vector in the row direction. H_tensor indicates the size of the feature vector in the column direction.

[0065] The example of dividing the input image into three equal slices vertically is one example, and the method of segmentation is not limited to this. For instance, the input image may be segmented unevenly. The number of segments can be two, three, or more.

[0066] In the video data processor 101, the encoder 1001 segments the input image into multiple slices (step S101). In this example embodiment, the input image is equally segmented into three slices in the vertical direction (see...). Figure 3 ).

[0067] Encoder 1001 extracts the feature values ​​of each slice and generates a feature vector (step S102).

[0068] Quantizer 1002 quantizes the feature vector of each slice to generate quantized values ​​(step S103).

[0069] Entropy encoder 1003 performs entropy encoding on the quantized value to generate entropy encoded data (step S104).

[0070] Then, the entropy encoder 1003 terminates the process (step S105). As described above, termination is performed under the control of the second control unit 1005.

[0071] Multiplexer 1006 multiplexes slice boundary information and entropy-encoded data, and outputs multiplexed data as a bit stream (step S106).

[0072] [Description of the decoding side]

[0073] In the video data processor 201, the demultiplexer 2006 demultiplexes the bitstream to obtain slice boundary information and entropy-coded data. The demultiplexer 2006 supplies the slice boundary information to the second control unit 2005. The demultiplexer 2006 supplies the entropy-coded data to the entropy decoder 2003.

[0074] The entropy decoder 2003 performs entropy decoding on the entropy-encoded data supplied from the demultiplexer 2006 to obtain quantized values.

[0075] The inverse quantizer 2002 inverse quantizes the quantized values ​​supplied by the entropy decoder 2003 to obtain the reconstructed feature vector.

[0076] The first control unit 2004 performs control based on the slice boundary information obtained via the second control unit 2005, so that the decoding process (hereinafter referred to as decoding) by the decoder 2001 does not cross the slice boundary in the feature vector domain.

[0077] Decoder 2001 obtains the reconstructed image of the decoded image from the reconstructed feature vectors. However, when obtaining the reconstructed image of the slice region to be processed based on the control of the first control unit 2004, the reconstructed feature vectors of the non-slice regions are not used. That is, the reconstructed image of the slice region to be processed is similar to the reconstructed image obtained by decoding the reconstructed feature vectors of the slice region to be processed into an image.

[0078] The second control unit 2005 monitors the entropy decoder 2003, performs entropy decoding on the last quantized value of each slice region, and then terminates the processing of the entropy decoder 2003.

[0079] The termination of entropy decoding means aligning the entropy-encoded data. That is, reading aligned data from the entropy-encoded data. Furthermore, in the case where entropy coding is based on adaptive arithmetic coding, the termination of entropy decoding resets the more probable symbols and their probabilities, as well as the arithmetic coding engine.

[0080] Next, the operation of the video data processor 201 will be described. Here, the operation of the video data processor 201 will be described using an example where the input image is equally divided into three slices in the vertical direction. Figure 5 This is an illustrative diagram showing an example of slice segmentation in the video data processor 201. Figure 6 This is a flowchart illustrating an operational example of the video data processor 201.

[0081] exist Figure 5 In the diagram, W_tensor indicates the size of the feature vector along the row direction. H_tensor indicates the size of the feature vector along the column direction. W_img indicates the width (horizontal size) of the decoded image. H_img indicates the height (vertical size) of the decoded image.

[0082] In the video data processor 201, the demultiplexer 2006 demultiplexes the bitstream (step S201). The demultiplexer 2006 obtains slice boundary information and entropy-coded data through demultiplexing.

[0083] Entropy decoder 2003 performs entropy decoding on the entropy-encoded data to obtain quantized values ​​(step S202).

[0084] Inverse quantizer 2002 inverse quantizes the quantized value (step S203). Inverse quantizer 2002 obtains the reconstructed feature vector through inverse quantization.

[0085] Decoder 2001 obtains the reconstructed image of the decoded image from the reconstructed feature vector (step S204). As described above, when obtaining the reconstructed image of the slice region to be processed based on the control of the first control unit 2004, decoder 2001 does not use the reconstructed feature vector of the slice region that is not the processing target. In other words, decoder 2001 performs decoding based on slice boundary information so as not to cross the slice boundary in the feature vector domain.

[0086] Then, the entropy decoder 2003 terminates the process (step S205). As described above, termination is performed under the control of the second control unit 2005. If the video data processor 201 confirms that the bitstream includes information indicating termination or information indicating that such segmentation has been performed, it can perform the decoding process that started from step S201.

[0087] In this example embodiment, under the control of the first control unit 1004 and the first control unit 2004, the encoding on the encoding side and the decoding on the decoding side do not cross the slice boundary, thus ensuring independent decoding in neural network-based video coding.

[0088] Through control related to the termination of the second control unit 1005 and the second control unit 2005, the entropy encoding processing on the encoding side and the entropy decoding processing on the decoding side are discontinuous at the slice boundaries, and therefore, data segmentation is guaranteed in neural network-based video coding.

[0089] Second example embodiment.

[0090] The video data processor 101 of the first example embodiment may result in an increase in the amount of processing in matching the maximum transmission unit (MTU) size during the packet transmission of the bitstream of each slice, a decrease in coding efficiency, or both.

[0091] The above possibilities are as follows.

[0092] 1) If the bitstream processed by segmentation based on the set slice segmentation information does not meet the MTU size (i.e., if the bitstream is larger than the MTU size), the video must be re-encoded, and the processing volume increases.

[0093] 2) When the number of slices is increased from the beginning, segmentation loss occurs and coding efficiency is reduced.

[0094] Figure 7 This is a block diagram illustrating video data processor 102 and video data processor 202, which can eliminate the above-mentioned possibilities. Video data processor 102 and video data processor 202 encode and decode each picture that constitutes video data based on a neural network.

[0095] The video data processor 102 includes an encoder 1001, a quantizer 1002, an entropy encoder 1003A, a first control unit 1004, a second control unit 1005A, and a multiplexer 1006.

[0096] The video data processor 202 includes a decoder 2001, an inverse quantizer 2002, an entropy decoder 2003A, a first control unit 2004, a second control unit 2005A, and a demultiplexer 2006.

[0097] [Description of the encoding side]

[0098] In the video data processor 102, the encoder 1001, quantizer 1002, first control unit 1004, and multiplexer 1006 are configured and function the same as in the first example embodiment.

[0099] The entropy encoder 1003A performs entropy encoding on the quantized value supplied from the quantizer 1002 and the termination identification information supplied from the second control unit 1005A to obtain entropy encoded data.

[0100] In addition to the functions of the second control unit 1005 in the first example embodiment, the second control unit 1005A has the function of determining the value of termination flag information (e.g., 0: no termination, 1: termination), which indicates whether to terminate entropy encoding in the slice region at a predetermined interval with a finer granularity than the slice region (e.g., W_tensor, the width of a row of a feature vector). The second control unit 1005A can use a value indicating 0 for termination and 1 for no termination as the value of the termination flag information.

[0101] For example, when the second control unit 1005A monitors the number of output bits of the entropy encoder 1003A and predicts that the size of the bitstream of the slice region to be processed will become greater than the MTU size, or when entropy encoding is performed on the last quantized value of the slice region, the value of the termination flag information is set to 1. Otherwise, the second control unit 1005A sets the value of the termination flag information to 0. When the value of the termination flag information is 1, the second control unit 1005A causes the entropy encoder 1003A to terminate processing.

[0102] The operation of the video data processor 102 will be described next. Figure 8 This is a flowchart illustrating an example of the operation of the video data processor 102. Similarly, in this example embodiment, as in the first example embodiment, the operation of the video data processor 102 will be described using the example of an input image being equally divided into three slices in the vertical direction.

[0103] The processing in steps S101 to S103 is the same as that in the first example embodiment.

[0104] In step S104 A, the entropy encoder 1003A performs entropy encoding on the quantized value supplied from the quantizer 1002 and the termination identifier information supplied from the second control unit 1005 A to obtain entropy encoded data.

[0105] The processing in steps S105 and S106 is the same as that in the first example embodiment. In this example embodiment, the entropy-coded data reused in step S106 also includes data related to the termination identifier information.

[0106] [Description of the decoding side]

[0107] In the video data processor 202, the configuration and function of the decoder 2001, the inverse quantizer 2002, the first control unit 2004, and the demultiplexer 2006 are the same as in the first example embodiment.

[0108] The entropy decoder 2003A performs entropy decoding on the entropy-encoded data supplied from the demultiplexer 2006 to obtain quantization values ​​and termination identification information.

[0109] In addition to the functions of the second control unit 2005 in the first example embodiment, the second control unit 2005A also has the function of monitoring the termination flag information supplied from the entropy decoder 2003A and causing the entropy decoder 2003A to terminate processing when the value is 1.

[0110] The operation of the video data processor 202 will be described next. Figure 9 This is a flowchart illustrating an example of the operation of the video data processor 202. Similarly, in this example embodiment, as in the first example embodiment, the operation of the video data processor 202 will be described using the example of an input image being equally divided into three slices in the vertical direction.

[0111] The processing in step S201 is the same as that in the first example embodiment. In this example embodiment, the demultiplexed entropy-coded data in the processing of step S201 also includes data related to termination identification information.

[0112] In step S202 A, the entropy decoder 2003A performs entropy decoding on the entropy-encoded data supplied from the demultiplexer 2006 to obtain quantization values ​​and termination identification information.

[0113] The processing in steps S203 and S204 is the same as that in the first example embodiment.

[0114] In step S205A, the second control unit 2005A monitors the termination flag information supplied by the entropy decoder 2003A, and when the value is 1, causes the entropy decoder 2003A to terminate the process.

[0115] In this example embodiment, the bitstream of each slice can be terminated internally through the processing of the entropy encoder 1003A, the second control unit 1005A, the entropy decoder 2003A, and the second control unit 2005A. That is, the bitstream can be divided in finer units than the segmentation of the aggregate slice segmentation information, so that the slices are equal to or smaller than the MTU size. Therefore, it is possible to reduce the increase in video encoding processing power and the decrease in encoding efficiency.

[0116] [Encoder and decoder configuration]

[0117] Figure 10 This is a block diagram illustrating configuration examples of encoder 1001 and decoder 2001 according to the first and second example embodiments. Figure 10 In the diagram, "down arrow (↓) 2" represents a 1 / 2 subsample (also known as pooling). "up arrow (↑) 2" represents 2 samples.

[0118] exist Figure 10 In the example shown, encoder 1001 includes four residual blocks and one convolutional block. The residual blocks consist of two convolutional blocks and a shortcut link. Each convolutional block includes a convolutional layer and an activation function.

[0119] The decoder 2001 consists of four residual blocks and a pixel-shuffle convolutional layer. The pixel-shuffler is a mechanism proposed as sub-pixel convolution. The pixel-shuffler rearranges the input feature vector and outputs a high-resolution feature vector.

[0120] Figure 10 The configuration shown is an example, and the configuration of encoder 1001 and decoder 2001 is not limited to this. Figure 10 The configuration shown.

[0121] Third example embodiment.

[0122] Figure 11 This is a block diagram illustrating another configuration example of a video data processor on the encoding side. Figure 12 This is a block diagram illustrating another configuration example of a video data processor on the decoding side.

[0123] Figure 11 The video data processor 103 shown includes a switch 3000, an NN video encoder 3001, an NN video decoder 3003, a CP video encoder 3002, a CP video decoder 3004, a decoded image buffer 3005, and a multiplexer 4000 that performs multiplexing processing of entropy-encoded data and other information.

[0124] “NN” stands for Neural Network. “CP” stands for Prediction Based on Encoding Units.

[0125] The CP video encoder 3002 performs video encoding processing based on predictive coding at the coding unit level using a video coding scheme. The CP video decoder 3004 performs decoding processing based on predictive coding at the coding unit level using a video coding scheme. The decoded image buffer 3005 is a storage unit for storing decoded images (reconstructed images). As described above, schemes conforming to H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc., can be used as video coding schemes.

[0126] As the NN video encoder 3001, either the video data processor 101 or the video data processor 102 of each of the above example embodiments can be used. As the NN video decoder 3003, either the video data processor 201 or the video data processor 202 of each of the above example embodiments can be used.

[0127] Figure 12 The video data processor 203 shown includes a demultiplexer 5000 for demultiplexing bitstreams, an NN video decoder 3003, a CP video decoder 3004, and a decoded image buffer 3005.

[0128] In other words, Figure 11 The video data processor 103 shown and Figure 12 The video data processor 203 shown is a video data processor in which the aforementioned video data processor 101 or 102 is an NN video encoder 3001, the aforementioned video data processor 201 or 202 is an NN video decoder 3003, and the video data processor is combined with a CP video encoder 3002 and a CP video decoder 3004 according to a video coding scheme based on predictive coding in the coding unit.

[0129] [Description of the encoding side]

[0130] exist Figure 11 In the video data processor 103 shown, switch 3000 supplies the input image to NN video encoder 3001 or CP video encoder 3002.

[0131] The NN video encoder 3001 operates similarly to the video data processor 101 of the first example embodiment or the video data processor 102 of the second example embodiment to generate a bitstream (also known as an NN bitstream).

[0132] The NN video decoder 3003 receives input from the NN bitstream supplied by the NN video encoder 3001 and operates similarly to the video data processor 201 of the first example embodiment or the video data processor 202 of the second example embodiment to obtain a decoded picture (also referred to as an NN decoded picture). The NN video decoder 3003 stores the NN decoded picture in the decoded picture buffer 3005.

[0133] The CP video encoder 3002 performs video encoding based on prediction coding in the coding unit by using the input image and the decoded image stored in the decoded image buffer 3005, and generates a bitstream (also known as the CP bitstream).

[0134] The CP video decoder 3004 takes the CP bitstream supplied from the CP video encoder 3002 as input, performs entropy decoding processing, and then performs decoding processing based on predictive coding in the coding unit to obtain a decoded picture (also called a CP decoded picture). The CP video decoder 3004 stores the CP decoded picture in the decoded picture buffer 3005.

[0135] The decoded image stored in the decoded image buffer 3005 is used as a reference image.

[0136] exist Figure 11 In the configuration shown, the NN video decoder 3003 receives an NN bitstream from the NN video encoder 3001 and performs entropy decoding processing. Then, the NN video decoder 3003 obtains the NN decoded image from the encoded data obtained through entropy decoding. However, the NN video decoder 3003 can be configured to receive intermediate data (e.g., quantization values) from the NN video encoder 3001 before entropy encoding and obtain the NN decoded image from the intermediate data. In this case, the NN video decoder 3003 may not perform entropy decoding processing.

[0137] exist Figure 11 In the configuration shown, the CP video decoder 3004 receives the CP bitstream from the CP video encoder 3002 and performs entropy decoding. Then, the CP video decoder 3004 obtains the CP decoded image from the encoded data obtained through entropy decoding. However, the CP video decoder 3004 can be configured to receive intermediate data (e.g., quantization values) prior to entropy encoding from the CP video encoder 3002 and obtain the CP decoded image from the intermediate data. In this case, the CP video decoder 3004 may not perform entropy decoding.

[0138] [Description of the decoding side]

[0139] exist Figure 12In the video data processor 203 shown, based on the encoded data obtained by demultiplexing the NN bitstream, the NN video decoder 3003 operates similarly to the video data processor 201 of the first example embodiment or the video data processor 202 of the second example embodiment to obtain NN decoded images. The NN video decoder 3003 stores the NN decoded images in the decoded image buffer 3005.

[0140] The CP video decoder 3004 demultiplexes the CP bitstream, performs entropy decoding processing, and obtains the CP decoded image based on the acquired encoded data. The CP video decoder 3004 stores the CP decoded image in the decoded image buffer 3005.

[0141] The video data processor 203 outputs either the NN decoded image or the CP decoded image stored in the decoded image buffer 3005 as the decoded image.

[0142] [First Variation]

[0143] In each of the above example embodiments, slice-independent decoding and data segmentation are satisfied. However, only slice-independent data segmentation may be satisfied. In this case, the first control unit 1004 may not be provided in the video data processors 101 and 102. The first control unit 2004 may not be provided in the video data processors 201 and 202.

[0144] [Second Variation]

[0145] In cases where completely independent decoding of slices is not required, the first control unit 1004 in video data processors 101 and 102 can relax the slice boundaries that cannot be crossed. For example, when encoding the feature vector of the slice region to be processed, the form of not allowing the input image of the slice region in the bitstream to intersect with subsequent slices can be considered. That is, the feature vector of slice region 1 is generated using the input image of slice region 1 (see...). Figure 3 The feature vector of slice region 2 is generated using the input images of slice regions 1 and 2. The feature vector of slice region 3 is generated using the input images of slice regions 1, 2, and 3. In this form (the second variant), completely independent decoding cannot be obtained, but the reduction in coding efficiency due to segmentation loss can be relaxed.

[0146] In the video data processor on the decoding side associated with the second variant of the video data processor, when decoding the reconstructed feature vector of the slice region to be processed, the first control unit 2004 performs control to prevent crossing the reconstructed feature vector of the slice region associated with subsequent slices in the bitstream. That is, when obtaining the reconstructed image of slice region 1, the decoder 2001 uses the reconstructed feature vector of slice region 1 (see...). Figure 5Decoder 2001 uses the reconstructed feature vectors of slice region 1 and slice region 2 when obtaining the reconstructed image of slice region 2. Decoder 2001 uses the reconstructed feature vectors of slice region 1, slice region 2, and slice region 3 when obtaining the feature reconstructed image of slice region 3.

[0147] [Third Variation]

[0148] In cases where completely independent decoding of slices is not required, the first control unit 1004 in video data processors 101 and 102 can relax the slice boundaries that cannot be crossed. For example, when encoding the feature vectors of the slice region to be processed, it is conceivable to limit the range of input images that are allowed to intersect with the slice region associated with subsequent slices in the bitstream.

[0149] Figure 13 This is an illustrative diagram showing an example of slice segmentation in a video data processor on the encoding side according to the third variant.

[0150] Encoder 1001 uses slice region 1 and a predetermined region adjacent to the slice region (hereinafter referred to as the "look-ahead region"): Figure 13 The input image (shaded area in the image) is used to generate Figure 13 The encoder 1001 uses the input images of slice region 1, slice region 2, and the look-ahead region adjacent to slice region 2 to generate the feature vector of slice region 2. When generating the feature vector of slice region 3, the encoder 1001 does not use the look-ahead region because there is no look-ahead region adjacent to slice region 3. That is, the encoder 1001 uses the input images of slice region 1, slice region 2, and slice region 3 to generate the feature vector of slice region 3.

[0151] In this example, completely independent decoding cannot be achieved, but the reduction in coding efficiency due to segmentation loss can be relaxed compared to the second variant. However, the processing latency is longer than in the second variant due to the time spent waiting for input from the look-ahead region adjacent to the target slice. Of course, when the maximum value of the range referenced by the encoding process is set to the height ph of the look-ahead region, the feature vector of the entire image matches the result without slice segmentation. Figure 10 In the example encoder configuration shown, with a kernel size of 3x3 for each block of convolution, the matching ph is 29.

[0152] The operation of the video data processor on the decoding side associated with the third variant of the video data processor is the same as that of the second variant of the video data processor. That is, in the video data processor according to this variant, when decoding the reconstruction feature vector of the slice region to be processed, the first control unit 2004 performs control to prevent crossing the reconstruction feature vector of the slice region associated with subsequent slices in the bitstream. Specifically, when obtaining the reconstructed image of slice region 1, the decoder 2001 uses the reconstruction feature vector of slice region 1 (see...) Figure 5 When obtaining the reconstructed image of slice region 2, decoder 2001 uses the reconstructed feature vectors of slice region 1 and slice region 2. When obtaining the feature reconstructed image of slice region 3, decoder 2001 uses the reconstructed feature vectors of slice region 1, slice region 2, and slice region 3.

[0153] [Fourth Variation]

[0154] In the second and third example embodiments described above, the multiplexer 1006 implicitly multiplexes termination identifier information, indicating whether entropy encoding is terminated in the slice region, into the bitstream at a predetermined interval (W_tensor). However, the multiplexer 1006 may explicitly embed the predetermined interval into the bitstream. The predetermined interval may not be W_tensor, as long as the granularity is finer than that of the slice region.

[0155] The slice boundary information can be configured as follows.

[0156] When frames are sliced ​​into equal sizes, the number of slices and the coordinates of the top-left corner or slice number of each slice.

[0157] In the case where the frame is sliced ​​into non-uniform sizes, the coordinates of the top-left corner of each slice and the horizontal and vertical widths of each slice.

[0158] exist Figure 3 In the example shown, the slice numbers are reference numbers assigned sequentially from the top to the three slices (slice regions). Slice numbers can be assigned sequentially from the bottom.

[0159] Each of the above example embodiments can be configured by hardware, but it can also be implemented by a computer program.

[0160] Figure 14 The information processing system shown includes a processor 701 such as one or more central processing units (CPUs), a program memory 702, a storage medium 703 for storing video data, and a storage medium 704 for storing bitstreams. Storage mediums 703 and 704 can be separate storage media or a storage area comprised of the same storage medium. Magnetic storage media such as hard disks can be used as the storage medium.

[0161] In the information processing system, the program (video encoding program or video decoding program) for implementing the function of each block described in each of the above example embodiments is stored in the program memory 702.

[0162] Then, the processor 701 performs the functions of the video data processors 101, 102, and 103 and the video data processors 201, 202, and 203 described in each example embodiment by executing processing according to the program stored in the program memory 702.

[0163] For example, the functions of video data processors 101, 102, and 103 are implemented by processor 701 performing processing according to a video processing program, which is used to implement... Figure 2 , Figure 7 and Figure 11 The functions of the blocks (excluding the decoded image buffer 3005) in the video data processors 101, 102, and 103 shown are illustrated. For example, the functions of the video data processors 201, 202, and 203 are implemented by the processor 701 performing processing according to a video processing program, which is used to implement... Figure 2 , Figure 7 and Figure 12 The functionality of each block in the video data processors 201, 202, and 203 shown (excluding the decoded image buffer 3005).

[0164] At least the program memory 702 is a non-transitory computer-readable medium. However, programs can be stored in various types of transient computer-readable media. Transitory computer-readable media are supplied with programs via, for example, wired or wireless communication paths (i.e., via electrical signals, optical signals, or electromagnetic waves).

[0165] Figure 15 This is a block diagram showing the main parts of a video data processing device. Figure 15 The video data processing device 10 shown (implemented by video data processors 101, 102, and 103 in the example embodiment) includes a neural network 11 (implemented by encoder 1001 in the example embodiment), a quantization device 12 (implemented by quantizer 1002 in the example embodiment), and an entropy encoding device 13 (implemented by entropy encoders 1003 and 1003A in the example embodiment), and includes a first control device 14 (implemented by first control unit 1004 in the example embodiment) for disallowing encoding processing based on neural network 11 from crossing slice boundaries.

[0166] Figure 16 This is a block diagram showing the main parts of another side of the video data processing device. Figure 16The video data processing device 20 shown (implemented by video data processors 201, 202, and 203 in the example embodiment) includes a neural network 21 (implemented by decoder 2001 in the example embodiment), an inverse quantization device 22 (implemented by inverse quantizer 2002 in the example embodiment), and an entropy decoding device 23 (implemented by entropy decoders 2003 and 2003A in the example embodiment), and includes a first control device 24 (implemented by first control unit 2004 in the example embodiment) for disallowing decoding processing based on neural network 21 from crossing slice boundaries.

[0167] Some or all of the above example embodiments may be described as the following supplementary notes, but are not limited to them.

[0168] (Supplementary Note 1) A video data processing device includes a neural network, a quantization device, and an entropy coding device. Video data processing equipment also includes: The first control device is used to prevent neural network-based encoding processing from crossing slice boundaries.

[0169] (Supplementary Note 2) The video data processing device according to Supplementary Note 1 also includes

[0170] The second control device (implemented in the example embodiment by second control units 1005 and 1005A) is used to terminate entropy coding in the slice. in The entropy encoding device performs entropy encoding on the identification information indicating whether to terminate entropy encoding.

[0171] (Supplementary Note 3) The video data processing device according to Supplementary Note 1 or 2 also includes

[0172] A predictive coding apparatus (implemented by CP video encoder 3002 in the example embodiment) is used to perform predictive coding on a unit-by-unit basis.

[0173] (Supplementary Note 4) A video data processing device includes a neural network, an inverse quantization device, and an entropy decoding device. The video data processing equipment also includes The first control device is used to prevent neural network-based decoding processing from crossing slice boundaries.

[0174] (Supplementary Note 5) The video data processing device according to Supplementary Note 4, wherein, The entropy decoding device is used to perform entropy decoding on the identification information indicating whether to terminate entropy encoding in a slice. The video data processing equipment also includes The second control device (implemented by the second control unit 2005 and 2005A in the example embodiment) is used to terminate entropy decoding in the slice based on the identification information.

[0175] (Supplementary Note 6) The video data processing device according to Supplementary Note 4 or 5 also includes

[0176] A predictive decoding apparatus (implemented by CP video decoder 3004 in the example embodiment) is used to perform predictive decoding on a unit-by-unit basis.

[0177] (Supplementary Note 7) A video data processing method for performing quantization and entropy coding based on neural network encoding. The video data processing methods include: Execution control ensures that encoding processes do not cross slice boundaries.

[0178] (Supplementary Note 8) The video data processing method described in Supplementary Note 7 further includes: Execute control to terminate entropy encoding in the slice; and Perform entropy encoding on the identification information indicating whether to terminate entropy encoding.

[0179] (Supplementary Note 9) The video data processing method according to Supplementary Note 7 or Supplementary Note 8 further includes: Perform predictive coding processing on a unit-by-unit basis.

[0180] (Supplementary Note 10) A video data processing method for performing neural network-based decoding, inverse quantization, and entropy decoding. The video data processing methods include: Execution control ensures that decoding processing does not cross slice boundaries.

[0181] (Supplementary Note 11) The video data processing method described in Supplementary Note 10 further includes: Entropy decoding is performed on the identifier indicating whether entropy encoding should be terminated in the slice; and Control is performed based on the identification information to terminate entropy decoding in the slice.

[0182] (Supplementary Note 12) The video data processing method according to Supplementary Note 10 or Supplementary Note 11 further includes: Perform predictive decoding processing on a unit-by-unit basis.

[0183] (Supplementary Note 13) A video data processing program for enabling a computer to: Perform neural network-based encoding, quantization, and entropy encoding; and Execution control ensures that encoding processes do not cross slice boundaries.

[0184] (Supplementary Note 14) According to the video data processing procedure described in Supplementary Note 13, Among these features, the video data processing program also enables the computer to: Execute control to terminate entropy encoding in the slice; and Perform entropy encoding on the identification information indicating whether to terminate entropy encoding.

[0185] (Supplementary Note 15) According to the video data processing program in Supplementary Note 13 or Supplementary Note 14, the computer also: Perform predictive coding processing on a unit-by-unit basis.

[0186] (Supplementary Note 16) A video data processing program for enabling a computer to: Perform neural network-based decoding, inverse quantization, and entropy decoding; and Execution control ensures that decoding processing does not cross slice boundaries.

[0187] (Supplementary Note 17) Based on the video data processing procedure in Supplementary Note 16, Among these features, the video data processing program also enables the computer to: Entropy decoding is performed on the identifier indicating whether entropy encoding should be terminated in the slice; and Control is performed based on the identification information to terminate entropy decoding in the slice.

[0188] (Supplementary Note 18) According to the video data processing procedure in Supplementary Note 16 or Supplementary Note 17, the computer also: Perform predictive decoding processing on a unit-by-unit basis.

[0189] (Supplementary Note 19) A storage medium for storing a bitstream generated by a video data processing device including a neural network, a quantization device, and an entropy coding device. The video data processing device includes a first control device for preventing the encoding processing based on the neural network from crossing the slice boundary.

[0190] (Supplementary Note 20) A storage medium for storing a bitstream generated by a video data processing method for performing neural network-based encoding, quantization, and entropy coding. Among them, the video data processing method execution control ensures that the encoding process does not cross the slice boundary.

[0191] (Supplementary Note 21) A video data processing device includes a neural network, a quantization device, and an entropy coding device, comprising...

[0192] The second control device is used to terminate entropy coding in the slice. in The entropy encoding device performs entropy encoding on the identification information indicating whether to terminate entropy encoding.

[0193] This supplementary note relates to the first variant mentioned above.

[0194] (Supplementary Note 22) The video data processing device according to Supplementary Note 21, wherein, In neural network-based encoding processing, only the slice boundaries of subsequent slice regions in the bitstream sequence do not intersect.

[0195] This supplementary note relates to the second variant mentioned above.

[0196] (Supplementary Note 23) The video data processing device according to Supplementary Note 22, wherein, Multiplexing a predetermined interval into a bit stream; and The entropy encoding device performs entropy encoding on the identification information that indicates whether the second control device should terminate entropy encoding at a predetermined interval.

[0197] This supplementary note relates to the third variant mentioned above.

[0198] (Supplementary Note 24) A video data processing device includes a neural network, an inverse quantization device, and an entropy decoding device, wherein...

[0199] The entropy decoding device performs entropy decoding on the identification information indicating whether to terminate entropy encoding in the slice. The video data processing device also includes...

[0200] The second control device is used to terminate entropy decoding in the slice based on the identification information.

[0201] This supplementary note relates to the first variant mentioned above.

[0202] (Supplementary Note 25) The video data processing device according to Supplementary Note 24, wherein, In neural network-based decoding, only the slice boundaries of subsequent slice regions in the bitstream sequence are not crossed.

[0203] This supplementary note relates to the second variant mentioned above.

[0204] (Supplementary Note 26) The video data processing device according to Supplementary Note 25, wherein, From the bitstream demultiplexing predetermined interval, and The entropy decoding device performs entropy decoding on the identification information indicating whether to terminate entropy encoding at predetermined intervals.

[0205] This supplementary note relates to the third variant mentioned above.

[0206] This application is based on and claims the benefit of priority from Japanese Patent Application No. 2023-155647, filed on September 21, 2023, and Japanese Patent Application No. 2024-035447, filed on March 8, 2024, the disclosure of which is incorporated herein by reference in its entirety.

[0207] List of reference numerals

[0208] 10. Video data processing equipment

[0209] 11 Neural Networks

[0210] 12 Quantization device

[0211] 13 Entropy Coding Device

[0212] 14 First Control Device

[0213] 20 Video data processing equipment

[0214] 21 Neural Networks

[0215] 22 Inverse Quantization Device

[0216] 23 Entropy Decoding Device

[0217] 24 First Control Device

[0218] 101, 102, 103 Video Data Processors

[0219] 201, 202, 203 Video Data Processors

[0220] 701 processor

[0221] 702 Program Memory

[0222] 703 and 704 storage media

[0223] 1001 Encoder

[0224] 1002 quantizer

[0225] 1003, 1003A Entropy Encoder

[0226] 1004 First Control Department

[0227] 1005, 1005A Second Control Unit

[0228] 1006 Multiplexer

[0229] 2001 Decoder

[0230] 2002 Inverse Quantizer

[0231] 2003, 2003A Entropy Decoder

[0232] 2004 First Control Department

[0233] 2005, 2005A Second Control Section

[0234] 2006 Demultiplexer

[0235] 3000 switch

[0236] 3001 NN Video Encoder

[0237] 3002 CP Video Encoder

[0238] 3003 NN Video Decoder

[0239] 3004 CP Video Decoder

[0240] 3005 Decoding Image Buffer

[0241] 4000 Multiplexer

[0242] 5000 Demultiplexer

Claims

1. A video data processing device, comprising a neural network, a quantization device, and an entropy coding device. The video data processing device also includes: A first control device is used to prevent encoding processing based on the neural network from crossing slice boundaries.

2. The video data processing device according to claim 1, further comprising: A second control device is used to terminate entropy encoding in the slice. in, The entropy encoding device performs entropy encoding on the identification information indicating whether to terminate the entropy encoding.

3. The video data processing device according to claim 1 or 2, further comprising: A predictive coding apparatus for performing predictive coding on a unit-by-unit basis.

4. A video data processing device, comprising a neural network, an inverse quantization device, and an entropy decoding device. in, The video data processing device also includes: A first control device is used to prevent decoding processing based on the neural network from crossing slice boundaries.

5. The video data processing device according to claim 4, wherein, The entropy decoding device performs entropy decoding on the identifier information indicating whether to terminate entropy encoding in the slice. The video data processing device further includes: A second control device is used to terminate entropy decoding in the slice based on the identification information.

6. The video data processing apparatus according to claim 4 or 5, further comprising: A predictive decoding apparatus for performing predictive decoding on a unit-by-unit basis.

7. A video data processing method for performing neural network-based encoding, quantization, and entropy coding. in, The video data processing method includes: Execution control ensures that the encoding process does not cross slice boundaries.

8. The video data processing method according to claim 7, further comprising: Execute control to terminate entropy encoding in the slice; as well as Entropy encoding is performed on the identification information indicating whether to terminate the entropy encoding.

9. The video data processing method according to claim 7 or 8, further comprising: Perform predictive coding processing on a unit-by-unit basis.

10. A video data processing method for performing neural network-based decoding, inverse quantization, and entropy decoding. in, The video data processing method includes: The execution control ensures that the decoding process does not cross slice boundaries.

11. The video data processing method according to claim 10, further comprising: Perform entropy decoding on the identifier indicating whether to terminate entropy encoding in the slice; as well as Control is performed based on the identification information to terminate entropy decoding in the slice.

12. The video data processing method according to claim 10 or 11, further comprising: Perform predictive decoding processing on a unit-by-unit basis.

13. A video data processing program for enabling a computer to: Perform neural network-based encoding, quantization, and entropy encoding; and Execution control ensures that the encoding process does not cross slice boundaries.

14. The video data processing program according to claim 13, in, The video data processing program also enables the computer to: Execute control to terminate entropy encoding in the slice; and Entropy encoding is performed on the identification information indicating whether to terminate the entropy encoding.

15. The video data processing program according to claim 13 or 14 further enables the computer to: Perform predictive coding processing on a unit-by-unit basis.

16. A video data processing program for enabling a computer to: Perform neural network-based decoding, inverse quantization, and entropy decoding; and The execution control ensures that the decoding process does not cross slice boundaries.

17. The video data processing program according to claim 16, in, The video data processing program also enables the computer to: Entropy decoding is performed on the identifier indicating whether entropy encoding should be terminated in the slice; and Control is performed based on the identification information to terminate entropy decoding in the slice.

18. The video data processing program according to claim 16 or 17 further enables the computer to: Perform predictive decoding processing on a unit-by-unit basis.

19. A storage medium for storing a bitstream generated by a video data processing device, said video data processing device comprising a neural network, a quantization device, and an entropy coding device. in, The video data processing device includes a first control device for preventing the encoding processing based on the neural network from crossing the slice boundary.

20. A storage medium for storing a bitstream generated by a video data processing method, the video data processing method being used to perform neural network-based encoding processing, quantization processing, and entropy encoding. in, The video data processing method is controlled to ensure that the encoding process does not cross slice boundaries.

Images