Video encoder, video decoder, video encoding method, and video decoding method

Abstract

In order to ensure independence and separability in neural network-based video encoding, a video encoding apparatus includes a neural network, a quantization means, and an entropy encoding means, and further includes a first control means that prevents a neural network-based encoding process from crossing slice boundaries.

Description

Video encoder, video decoder, video encoding method, and video decoding method Technical Field

[0001] This invention relates to video encoding equipment, video decoding equipment, video encoding method, and video decoding method. Background Technology

[0002] In order to efficiently transmit or record video, video encoding devices are used to generate an encoded representation (hereinafter referred to as a bitstream) obtained by encoding the input video, and video decoding devices are used to decode the bitstream to generate a decoded video.

[0003] Video coding based on predictive coding in coding units

[0004] Examples of video coding scheme standards include H.264 / AVC (Advanced Video Coding), H.265 / HEVC (High Efficiency Video Coding), and H.266 / VVC (Multi-Functional Video Coding), which are standardized by ITU-T SG16 and ISO / IEC / SC29. As a more recent video coding technology, the technology described in NPL 1 exists.

[0005] In these video coding schemes, video data is managed in a hierarchical structure and is encoded and decoded. The hierarchical structure includes, for example, pictures that constitute the video data, slices obtained by segmenting (dividing) the pictures (or tiles), coding tree units (CTUs) obtained by segmenting the slices, and coding units (CUs) obtained by segmenting the coding tree units.

[0006] The input image for processing the target CU is typically encoded relative to the target CU in the past, and is predictively encoded based on a prediction image generated from the decoded image. That is, a prediction error image obtained by subtracting the prediction image from the input image is encoded and decoded. Predictive encoding includes intra-frame prediction (intra-prediction) using the decoded image included in the picture at the same display time as the target CU's display time, and inter-frame prediction (inter-prediction) using the decoded image included in the picture at different display times than the target CU's display time.

[0007] The prediction error image is encoded using frequency transform, quantization, and entropy coding. The prediction error image is decoded using entropy decoding, inverse quantization, and inverse frequency transform. The frequency transform value of the quantized prediction error image is called the quantization value.

[0008] The functionality provided by slicing includes independent decoding and data segmentation. Independent decoding is a function used to decode without using the decoding results of another slice from the same image. Data segmentation is a function used to divide the bitstream into segments of any size.

[0009] [Neural Network-Based Video Coding]

[0010] NPL2 discloses a novel video coding technique that combines autoencoders, quantization, and entropy coding, and is one of the neural network techniques.

[0011] An autoencoder compresses the input data into a low-dimensional feature vector so that only important features are included. The autoencoder then generates reconstructed data by reconstructing the low-dimensional feature vector back to the original dimensions. Figure 1 is an illustrative diagram showing the algorithm of the autoencoder.

[0012] In Figure 1, the circular portions are called nodes, and the arrows are called edges. The process of discarding the low-dimensional feature vectors (the first half) is called encoding. The process of generating the reconstructed data (the second half) is called decoding.

[0013] The autoencoder continues learning in order to minimize the reconstruction error (the difference between the input data and the reconstructed data). To obtain meaningful feature quantities, the autoencoder is designed to add constraints to the structure of the encoding or to add regularization terms to the network's loss function.

[0014] Reference List

[0015] Non-patent literature

[0016] NPL 1: "Algorithm description of Enhanced Compression Model 9 (ECM 9)", JVET-AD2025, 30th meeting of JVET and ISO / IEC JTC 1 / SC 29, ITU-T SG 16 WP 3, Antalya, TR, April 21-28, 2023.

[0017] NPL 2: J. Ball'e, V. Larpara and Esimoncelli, “End-to-end Optimized Image Compression”, presented as a conference paper at ICLR 2017. Summary of the Invention

[0018] Technical issues

[0019] Neural network-based video coding, like predictive coding in coding units, suffers from the inability to provide independent decoding and data segmentation slicing capabilities.

[0020] The reason why independent decoding cannot be provided is that encoding on the encoding device side (encoding shown in Figure 1) and decoding on the decoding device side (decoding shown in Figure 1) can be performed beyond (across) slice boundaries. The reason why data segmentation cannot be provided is that entropy encoding processing on the encoding side and entropy decoding processing on the decoding side may be continuous at slice boundaries.

[0021] A series of entropy coding processes implies that the output is not data aligned. Furthermore, in the case where entropy coding is based on adaptive arithmetic coding, the series of entropy coding processes also includes not resetting the more likely symbols and their probabilities, and not resetting the arithmetic coding engine. Simultaneously, a series of entropy decoding processes implies that the data used for data alignment is not decoded. Furthermore, in the case where entropy coding is based on adaptive arithmetic coding, the series of entropy coding processes also includes not resetting the more likely symbols and their probabilities, and not resetting the arithmetic decoding engine.

[0022] Neural network-based video coding suffers from increased processing power, reduced coding efficiency, or both, when segmenting an image into multiple slices.

[0023] The purpose of this invention is to provide a video encoding device, a video decoding device, a video encoding method, and a video decoding method that provide slicing functionality similar to predictive coding in coding units in neural network-based video coding.

[0024] Solution to the problem

[0025] The video encoding apparatus according to this disclosure includes a neural network, a quantization device, and an entropy encoding device, and includes a first control device for preventing neural network-based encoding processing from crossing slice boundaries.

[0026] The video decoding device according to this disclosure includes a neural network, an inverse quantization device, and an entropy decoding device, and includes a first control device for preventing neural network-based decoding processing from crossing slice boundaries.

[0027] The video coding method disclosed herein performs neural network-based coding processing, quantization processing, and entropy coding, and performs control to ensure that the coding processing does not cross slice boundaries.

[0028] The video decoding method disclosed herein performs neural network-based decoding, inverse quantization, and entropy decoding, and controls the decoding process to prevent it from crossing slice boundaries.

[0029] The video encoding program according to this disclosure enables a computer to perform neural network-based encoding processing, quantization processing, and entropy encoding, and performs control such that the encoding processing does not cross slice boundaries.

[0030] The video decoding program according to this disclosure enables a computer to perform neural network-based decoding, inverse quantization, and entropy decoding, and performs control such that the decoding process does not cross slice boundaries.

[0031] Beneficial effects of the present invention

[0032] According to the present invention, in neural network-based video coding, a slicing function similar to that of video coding based on predictive coding in coding units is provided. Attached Figure Description

[0033] Figure 1 is an illustrative diagram depicting the algorithm of an autoencoder.

[0034] Figure 2 is a block diagram depicting a video encoder and a video decoder that encode and decode each image that constitutes the video data based on a neural network.

[0035] Figure 3 is an illustrative diagram depicting an example of slice segmentation in a video encoder.

[0036] Figure 4 is a flowchart depicting an example of the operation of a video encoder.

[0037] Figure 5 is an illustrative diagram depicting an example of slice segmentation in a video decoder.

[0038] Figure 6 is a flowchart depicting an example of the operation of a video decoder.

[0039] Figure 7 is a block diagram depicting another example of a video encoder and video decoder that encodes and decodes each image that constitutes video data based on a neural network.

[0040] Figure 8 is a flowchart depicting another example of the operation of a video encoder.

[0041] Figure 9 is a flowchart depicting another operational example of a video decoder.

[0042] Figure 10 is a block diagram depicting an example configuration of the encoder and decoder.

[0043] Figure 11 is a block diagram depicting another configuration example of a video encoder.

[0044] Figure 12 is a block diagram depicting another configuration example of a video decoder.

[0045] Figure 13 is an illustrative diagram depicting an example of slice segmentation in a video encoder.

[0046] Figure 14 is a block diagram depicting an example configuration of an information processing system.

[0047] Figure 15 is a block diagram depicting the main parts of a video encoding device.

[0048] Figure 16 is a block diagram depicting the main parts of a video decoding device. Detailed Implementation

[0049] In the following description, exemplary embodiments will be illustrated with reference to the accompanying drawings.

[0050] First example embodiment.

[0051] Figure 2 is a block diagram showing a video encoder 101 and a video decoder 201 that encode and decode each image constituting the video data based on a neural network.

[0052] The video encoder 101 includes an encoder 1001, a quantizer 1002, an entropy encoder 1003, a first control unit 1004, a second control unit 1005, and a multiplexer 1006.

[0053] The video decoder 201 includes a decoder 2001, an inverse quantizer 2002, an entropy decoder 2003, a first control unit 2004, a second control unit 2005, and a demultiplexer 2006.

[0054] The arrows in Figure 2 clearly indicate the direction of signal (data) flow, but do not exclude bidirectionality. The same applies to other block diagrams.

[0055] [Description of the encoding side]

[0056] In the video encoder 101, the first control unit 1004 obtains the slice boundaries in the feature vector domain based on the input slice segmentation information (information used to specify slice boundary information). Alternatively, the first control unit 1004 obtains information used to specify the slice boundaries in the feature vector domain. This information may be, for example, the slice boundaries in the image domain for each slice region. The first control unit 1004 performs control to ensure that the encoding process (hereinafter referred to as encoding) by the encoder 1001 does not cross the slice boundaries in the image domain. Furthermore, the first control unit 1004 supplies the position information of the slice boundaries in the feature vector domain as slice boundary information to the multiplexer 1006 via the second control unit 1005.

[0057] Encoder 1001 extracts feature values ​​from the image of the input image. Specifically, encoder 1001 obtains feature vectors from the image of the input image. However, when obtaining the feature vector of the slice region to be processed under the control of the first control unit 1004, encoder 1001 does not use the input image of slice regions that are not the processing target. That is, the feature vector of the slice region to be processed is similar to the feature vector obtained by encoding the input image of the slice region to be processed into an image.

[0058] The quantizer 1002 quantizes the feature vector supplied by the encoder 1001 to obtain the quantized value.

[0059] The entropy encoder 1003 entropy-encodes the quantized values ​​supplied from the quantizer 1002 to obtain entropy-encoded data.

[0060] The second control unit 1005 monitors the entropy encoder 1003 and terminates the entropy encoder 1003 after performing entropy encoding on the last quantized value of each slice region.

[0061] The termination of the entropy coding device signifies the alignment of the entropy-coded data output. That is, the entropy encoder 1003 adds alignment data for data alignment to the entropy-coded data. Furthermore, in the case where entropy coding is based on adaptive arithmetic coding, the entropy encoder 1003 further resets the more probable symbols and their probabilities, as well as the arithmetic coding engine used for the termination of entropy coding.

[0062] Multiplexer 1006 multiplexes the slice boundary information and entropy-encoded data supplied from entropy encoder 1003 and outputs the multiplexed data as a bit stream.

[0063] Next, the operation of the video encoder 101 will be described. Here, the operation of the video encoder 101 will be described using an example of the input image being equally divided into three slices in the vertical direction. Figure 3 is an explanatory diagram showing an example of slice division in the video encoder 101. Figure 4 is a flowchart showing an example of the operation of the video encoder 101.

[0064] In Figure 3, W_img indicates the width (horizontal size) of the input image. H_img indicates the height (vertical size) of the input image. W_tensor indicates the size of the feature vector in the row direction. H_tensor indicates the size of the feature vector in the column direction.

[0065] The example of dividing the input image into three equal slices vertically is one example, and the method of segmentation is not limited to this. For instance, the input image may be segmented unevenly. The number of segments can be two, three, or more.

[0066] In the video encoder 101, the encoder 1001 divides the input image into multiple slices (step S101). In this example embodiment, the input image is divided into three slices equally in the vertical direction (see Figure 3).

[0067] Encoder 1001 extracts the feature values ​​of each slice and generates a feature vector (step S102).

[0068] Quantizer 1002 quantizes the feature vector of each slice to generate quantized values ​​(step S103).

[0069] Entropy encoder 1003 performs entropy encoding on the quantized value to generate entropy encoded data (step S104).

[0070] Then, the entropy encoder 1003 terminates the process (step S105). As described above, termination is performed under the control of the second control unit 1005.

[0071] Multiplexer 1006 multiplexes slice boundary information and entropy-encoded data, and outputs multiplexed data as a bit stream (step S106).

[0072] [Description of the decoding side]

[0073] In the video decoder 201, the demultiplexer 2006 demultiplexes the bitstream to obtain slice boundary information and entropy-coded data. The demultiplexer 2006 supplies the slice boundary information to the second control unit 2005. The demultiplexer 2006 supplies the entropy-coded data to the entropy decoder 2003.

[0074] The entropy decoder 2003 performs entropy decoding on the entropy-encoded data supplied from the demultiplexer 2006 to obtain quantized values.

[0075] The inverse quantizer 2002 inverse quantizes the quantized values ​​supplied by the entropy decoder 2003 to obtain the reconstructed feature vector.

[0076] The first control unit 2004 performs control based on the slice boundary information obtained via the second control unit 2005, so that the decoding process (hereinafter referred to as decoding) by the decoder 2001 does not cross the slice boundary in the feature vector domain.

[0077] Decoder 2001 obtains the reconstructed image of the decoded image from the reconstructed feature vectors. However, when obtaining the reconstructed image of the slice region to be processed based on the control of the first control unit 2004, the reconstructed feature vectors of the non-slice regions are not used. That is, the reconstructed image of the slice region to be processed is similar to the reconstructed image obtained by decoding the reconstructed feature vectors of the slice region to be processed into an image.

[0078] The second control unit 2005 monitors the entropy decoder 2003, performs entropy decoding on the last quantized value of each slice region, and then terminates the processing of the entropy decoder 2003.

[0079] The termination of entropy decoding means aligning the entropy-encoded data. That is, reading aligned data from the entropy-encoded data. Furthermore, in the case where entropy coding is based on adaptive arithmetic coding, the termination of entropy decoding resets the more probable symbols and their probabilities, as well as the arithmetic coding engine.

[0080] Next, the operation of the video decoder 201 will be described. Here, the operation of the video decoder 201 will be described using an example of the input image being equally divided into three slices in the vertical direction. Figure 5 is an explanatory diagram showing an example of slice division in the video decoder 201. Figure 6 is a flowchart showing an example of the operation of the video decoder 201.

[0081] In Figure 5, W_tensor indicates the size of the feature vector in the row direction. H_tensor indicates the size of the feature vector in the column direction. W_img indicates the width (horizontal size) of the decoded image. H_img indicates the height (vertical size) of the decoded image.

[0082] In the video decoder 201, the demultiplexer 2006 demultiplexes the bitstream (step S201). The demultiplexer 2006 obtains slice boundary information and entropy-coded data through demultiplexing.

[0083] Entropy decoder 2003 performs entropy decoding on the entropy-encoded data to obtain quantized values ​​(step S202).

[0084] Inverse quantizer 2002 inverse quantizes the quantized value (step S203). Inverse quantizer 2002 obtains the reconstructed feature vector through inverse quantization.

[0085] Decoder 2001 obtains the reconstructed image of the decoded image from the reconstructed feature vector (step S204). As described above, when obtaining the reconstructed image of the slice region to be processed based on the control of the first control unit 2004, decoder 2001 does not use the reconstructed feature vector of the slice region that is not the processing target. In other words, decoder 2001 performs decoding based on slice boundary information so as not to cross the slice boundary in the feature vector domain.

[0086] Then, the entropy decoder 2003 terminates the process (step S205). As described above, termination is performed under the control of the second control unit 2005. If the video decoder 201 confirms that the bitstream includes information indicating termination or information indicating that such segmentation has been performed, it can perform the decoding process that started from step S201.

[0087] In this example embodiment, under the control of the first control unit 1004 and the first control unit 2004, the encoding on the encoding side and the decoding on the decoding side do not cross the slice boundary, thus ensuring independent decoding in neural network-based video coding.

[0088] Through control related to the termination of the second control unit 1005 and the second control unit 2005, the entropy encoding processing on the encoding side and the entropy decoding processing on the decoding side are discontinuous at the slice boundaries, and therefore, data segmentation is guaranteed in neural network-based video coding.

[0089] Second example embodiment.

[0090] The video encoder 101 of the first example embodiment may result in an increase in the amount of processing in the maximum transmission unit (MTU) size matching during the packet transmission of the bitstream of each slice, a decrease in coding efficiency, or both.

[0091] The above possibilities are as follows.

[0092] 1) If the bitstream processed by segmentation based on the set slice segmentation information does not meet the MTU size (i.e., if the bitstream is larger than the MTU size), the video must be re-encoded, and the processing volume increases.

[0093] 2) When the number of slices is increased from the beginning, segmentation loss occurs and coding efficiency is reduced.

[0094] Figure 7 is a block diagram illustrating a video encoder 102 and a video decoder 202 that can eliminate the above-mentioned possibilities. The video encoder 102 and the video decoder 202 encode and decode each picture that constitutes the video data based on a neural network.

[0095] The video encoder 102 includes an encoder 1001, a quantizer 1002, an entropy encoder 1003A, a first control unit 1004, a second control unit 1005A, and a multiplexer 1006.

[0096] The video decoder 202 includes a decoder 2001, an inverse quantizer 2002, an entropy decoder 2003A, a first control unit 2004, a second control unit 2005A, and a demultiplexer 2006.

[0097] [Description of the encoding side]

[0098] In the video encoder 102, the encoder 1001, quantizer 1002, first control unit 1004 and multiplexer 1006 are configured and function the same as in the first example embodiment.

[0099] The entropy encoder 1003A performs entropy encoding on the quantized value supplied from the quantizer 1002 and the termination identification information supplied from the second control unit 1005A to obtain entropy encoded data.

[0100] In addition to the functions of the second control unit 1005 in the first example embodiment, the second control unit 1005A has the function of determining the value of termination flag information (e.g., 0: no termination, 1: termination), which indicates whether to terminate entropy encoding in the slice region at a predetermined interval with a finer granularity than the slice region (e.g., W_tensor, the width of a row of a feature vector). The second control unit 1005A can use a value indicating 0 for termination and 1 for no termination as the value of the termination flag information.

[0101] For example, when the second control unit 1005A monitors the number of output bits of the entropy encoder 1003A and predicts that the size of the bitstream of the slice region to be processed will become greater than the MTU size, or when entropy encoding is performed on the last quantized value of the slice region, the value of the termination flag information is set to 1. Otherwise, the second control unit 1005A sets the value of the termination flag information to 0. When the value of the termination flag information is 1, the second control unit 1005A causes the entropy encoder 1003A to terminate processing.

[0102] Next, the operation of the video encoder 102 will be described. Figure 8 is a flowchart illustrating an example of the operation of the video encoder 102. Similarly, in this example embodiment, as in the first example embodiment, the operation of the video encoder 102 will be described using an example where the input image is equally divided into three slices in the vertical direction.

[0103] The processing in steps S101 to S103 is the same as that in the first example embodiment.

[0104] In step S104 A, the entropy encoder 1003A performs entropy encoding on the quantized value supplied from the quantizer 1002 and the termination identifier information supplied from the second control unit 1005 A to obtain entropy encoded data.

[0105] The processing in steps S105 and S106 is the same as that in the first example embodiment. In this example embodiment, the entropy-coded data reused in step S106 also includes data related to the termination identifier information.

[0106] [Description of the decoding side]

[0107] In the video decoder 202, the configuration and function of the decoder 2001, the inverse quantizer 2002, the first control unit 2004, and the demultiplexer 2006 are the same as in the first example embodiment.

[0108] The entropy decoder 2003A performs entropy decoding on the entropy-encoded data supplied from the demultiplexer 2006 to obtain quantization values ​​and termination identification information.

[0109] In addition to the functions of the second control unit 2005 in the first example embodiment, the second control unit 2005A also has the function of monitoring the termination flag information supplied from the entropy decoder 2003A and causing the entropy decoder 2003A to terminate processing when the value is 1.

[0110] Next, the operation of the video decoder 202 will be described. Figure 9 is a flowchart illustrating an example of the operation of the video decoder 202. Similarly, in this example embodiment, as in the first example embodiment, the operation of the video decoder 202 will be described using an example where the input image is equally divided into three slices in the vertical direction.

[0111] The processing in step S201 is the same as that in the first example embodiment. In this example embodiment, the demultiplexed entropy-coded data in the processing of step S201 also includes data related to termination identification information.

[0112] In step S202 A, the entropy decoder 2003A performs entropy decoding on the entropy-encoded data supplied from the demultiplexer 2006 to obtain quantization values ​​and termination identification information.

[0113] The processing in steps S203 and S204 is the same as that in the first example embodiment.

[0114] In step S205A, the second control unit 2005A monitors the termination flag information supplied by the entropy decoder 2003A, and when the value is 1, causes the entropy decoder 2003A to terminate the process.

[0115] In this example embodiment, the bitstream of each slice can be terminated internally through the processing of the entropy encoder 1003A, the second control unit 1005A, the entropy decoder 2003A, and the second control unit 2005A. That is, the bitstream can be divided in finer units than the segmentation of the aggregate slice segmentation information, so that the slices are equal to or smaller than the MTU size. Therefore, it is possible to reduce the increase in video encoding processing power and the decrease in encoding efficiency.

[0116] [Encoder and decoder configuration]

[0117] Figure 10 is a block diagram illustrating configuration examples of encoder 1001 and decoder 2001 according to the first and second example embodiments. In Figure 10, "down arrow (↓) 2" represents 1 / 2 subsample (also known as pooling). "up arrow (↑) 2" represents 2 upsamples.

[0118] In the example shown in Figure 10, encoder 1001 includes four residual blocks and one convolutional block. The residual blocks consist of two convolutional blocks and a shortcut link. Each convolutional block includes a convolutional layer and an activation function.

[0119] The decoder 2001 consists of four residual blocks and a pixel-shuffle convolutional layer. The pixel-shuffler is a mechanism proposed as sub-pixel convolution. The pixel-shuffler rearranges the input feature vector and outputs a high-resolution feature vector.

[0120] As an activation function, a parametric modified linear unit (parameter ReLU) can be used, where the output value is obtained by multiplying the input value by α (where α is a parameter and is determined through learning) when the input value is less than 0, and the output value is the same as the input value when the input value is greater than 0.

[0121] The configuration shown in Figure 10 is an example, and the configuration of encoder 1001 and decoder 2001 is not limited to the configuration shown in Figure 10.

[0122] Third example embodiment.

[0123] Figure 11 is a block diagram illustrating another configuration example of a video encoder. Figure 12 is a block diagram illustrating another configuration example of a video decoder.

[0124] The video encoder 103 shown in Figure 11 includes a switch 3000, an NN video encoder 3001, an NN video decoder 3003, a CP video encoder 3002, a CP video decoder 3004, a decoded image buffer 3005, and a multiplexer 4000 that performs multiplexing processing of entropy-encoded data and other information.

[0125] “NN” stands for Neural Network. “CP” stands for Prediction Based on Encoding Units.

[0126] The CP video encoder 3002 performs video encoding processing based on predictive coding at the coding unit level using a video coding scheme. The CP video decoder 3004 performs decoding processing based on predictive coding at the coding unit level using a video coding scheme. The decoded image buffer 3005 is a storage unit for storing decoded images (reconstructed images). As described above, schemes conforming to H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc., can be used as video coding schemes.

[0127] As the NN video encoder 3001, either video encoder 101 or video encoder 102 from each of the above example embodiments can be used. As the NN video decoder 3003, either video decoder 201 or video decoder 202 from each of the above example embodiments can be used.

[0128] The video decoder 203 shown in Figure 12 includes a demultiplexer 5000 for demultiplexing bitstreams, an NN video decoder 3003, a CP video decoder 3004, and a decoding image buffer 3005.

[0129] In other words, the video encoder 103 shown in FIG11 and the video decoder 203 shown in FIG12 are video encoders and video decoders as follows: wherein the video encoder 101 or the video encoder 102 is an NN video encoder 3001, the video decoder 201 or the video decoder 202 is an NN video decoder 3003, and the video encoder and the video decoder are combined with a CP video encoder 3002 and a CP video decoder 3004 according to a video coding scheme based on prediction coding in the coding unit.

[0130] [Description of the encoding side]

[0131] In the video encoder 103 shown in Figure 11, switch 3000 supplies the input image to NN video encoder 3001 or CP video encoder 3002.

[0132] The NN video encoder 3001 operates similarly to the video encoder 101 of the first example embodiment or the video encoder 102 of the second example embodiment to generate a bitstream (also known as an NN bitstream).

[0133] The NN video decoder 3003 receives input from the NN bitstream supplied by the NN video encoder 3001 and operates similarly to the video decoder 201 of the first example embodiment or the video decoder 202 of the second example embodiment to obtain a decoded picture (also referred to as an NN decoded picture). The NN video decoder 3003 stores the NN decoded picture in the decoded picture buffer 3005.

[0134] The CP video encoder 3002 performs video encoding based on prediction coding in the coding unit by using the input image and the decoded image stored in the decoded image buffer 3005, and generates a bitstream (also known as the CP bitstream).

[0135] The CP video decoder 3004 takes the CP bitstream supplied from the CP video encoder 3002 as input, performs entropy decoding processing, and then performs decoding processing based on predictive coding in the coding unit to obtain a decoded picture (also called a CP decoded picture). The CP video decoder 3004 stores the CP decoded picture in the decoded picture buffer 3005.

[0136] The decoded image stored in the decoded image buffer 3005 is used as a reference image.

[0137] In the configuration shown in Figure 11, the NN video decoder 3003 receives an NN bitstream from the NN video encoder 3001 and performs entropy decoding. Then, the NN video decoder 3003 obtains the NN decoded image from the encoded data obtained through entropy decoding. However, the NN video decoder 3003 can be configured to receive intermediate data (e.g., quantization values) from the NN video encoder 3001 before entropy encoding and obtain the NN decoded image from the intermediate data. In this case, the NN video decoder 3003 may not perform entropy decoding.

[0138] In the configuration shown in Figure 11, the CP video decoder 3004 receives the CP bitstream from the CP video encoder 3002 and performs entropy decoding. Then, the CP video decoder 3004 obtains the CP decoded image from the encoded data obtained through entropy decoding. However, the CP video decoder 3004 can be configured to receive intermediate data (e.g., quantization values) prior to entropy encoding from the CP video encoder 3002 and obtain the CP decoded image from the intermediate data. In this case, the CP video decoder 3004 may not perform entropy decoding.

[0139] [Description of the decoding side]

[0140] In the video decoder 203 shown in Figure 12, based on the encoded data obtained by demultiplexing the NN bitstream, the NN video decoder 3003 operates similarly to the video decoder 201 of the first example embodiment or the video decoder 202 of the second example embodiment to obtain NN decoded images. The NN video decoder 3003 stores the NN decoded images in the decoded image buffer 3005.

[0141] The CP video decoder 3004 demultiplexes the CP bitstream, performs entropy decoding, and obtains the CP decoded image based on the acquired encoded data. The CP video decoder 3004 stores the CP decoded image in the decoded image buffer 3005.

[0142] The video decoder 203 outputs either the NN decoded image or the CP decoded image stored in the decoded image buffer 3005 as the decoded image.

[0143] [First Variation]

[0144] In each of the above example embodiments, slice-independent decoding and data segmentation are satisfied. However, only slice-independent data segmentation may be satisfied. In this case, the first control unit 1004 may not be provided in the video encoders 101 and 102. The first control unit 2004 may not be provided in the video decoders 201 and 202.

[0145] [Second Variation]

[0146] In cases where completely independent decoding of slices is not required, the first control unit 1004 in video encoders 101 and 102 can relax the slice boundaries that cannot be crossed. For example, when encoding the feature vector of the slice region to be processed, consider a form where the input images of slice regions in the bitstream that are not allowed to intersect with those of subsequent slices are not permitted. That is, the feature vector of slice region 1 is generated using the input image of slice region 1 (see Figure 3). The feature vector of slice region 2 is generated using the input images of slice regions 1 and 2. The feature vector of slice region 3 is generated using the input images of slice regions 1, 2, and 3. In this form (second variant), completely independent decoding cannot be obtained, but the reduction in coding efficiency due to segmentation loss can be relaxed.

[0147] In the video decoder associated with the second variant of the video encoder, when decoding the reconstructed feature vector of the slice region to be processed, the first control unit 2004 performs control to prevent crossing the reconstructed feature vector of the slice region associated with subsequent slices in the bitstream. That is, when obtaining the reconstructed image of slice region 1, the decoder 2001 uses the reconstructed feature vector of slice region 1 (see Figure 5). The decoder 2001 uses the reconstructed feature vectors of slice region 1 and slice region 2 when obtaining the reconstructed image of slice region 2. The decoder 2001 uses the reconstructed feature vectors of slice region 1, slice region 2, and slice region 3 when obtaining the feature reconstructed image of slice region 3.

[0148] [Third Variation]

[0149] In cases where completely independent decoding of slices is not required, the first control unit 1004 in video encoders 101 and 102 can relax the slice boundaries that cannot be crossed. For example, when encoding the feature vectors of the slice region to be processed, it is conceivable to limit the range of input images that are allowed to intersect with the slice region associated with subsequent slices in the bitstream.

[0150] Figure 13 is an illustrative diagram showing an example of slice segmentation according to a third variant of the video encoder.

[0151] Encoder 1001 uses the input image of slice region 1 and a predetermined region adjacent to the slice region (hereinafter referred to as the "look-ahead region": the shaded area in Figure 13) to generate the feature vector of slice region 1 in Figure 13. Encoder 1001 uses the input images of slice region 1, slice region 2, and the look-ahead region adjacent to slice region 2 to generate the feature vector of slice region 2. When generating the feature vector of slice region 3, encoder 1001 does not use the look-ahead region because there is no look-ahead region adjacent to slice region 3. That is, encoder 1001 uses the input images of slice region 1, slice region 2, and slice region 3 to generate the feature vector of slice region 3.

[0152] In this example, completely independent decoding cannot be achieved, but the reduction in coding efficiency due to segmentation loss can be relaxed compared to the second variant. However, the processing latency is longer than in the second variant by waiting for the input of the look-ahead region adjacent to the target slice. Of course, when the maximum value of the range referenced by the encoding process is set to the height ph of the look-ahead region, the feature vector of the entire image matches as it would without slice segmentation. In the encoder configuration of the example shown in Figure 10, with a kernel size of 3x3 for the convolutions constituting each block, the matching ph is 29.

[0153] The operation of the video decoder associated with the third variant of the video encoder is the same as that of the second variant of the video decoder. That is, in the video decoder according to this variant, when decoding the reconstruction feature vector of the slice region to be processed, the first control unit 2004 performs control to prevent crossing the reconstruction feature vector of the slice region associated with subsequent slices in the bitstream. Specifically, when obtaining the reconstructed image of slice region 1, the decoder 2001 uses the reconstruction feature vector of slice region 1 (see Figure 5). When obtaining the reconstructed image of slice region 2, the decoder 2001 uses the reconstruction feature vectors of slice region 1 and slice region 2. When obtaining the feature reconstruction image of slice region 3, the decoder 2001 uses the reconstruction feature vectors of slice region 1, slice region 2, and slice region 3.

[0154] [Fourth Variation]

[0155] In the second and third example embodiments described above, the multiplexer 1006 implicitly multiplexes termination identifier information, indicating whether entropy encoding is terminated in the slice region, into the bitstream at a predetermined interval (W_tensor). However, the multiplexer 1006 may explicitly embed the predetermined interval into the bitstream. The predetermined interval may not be W_tensor, as long as the granularity is finer than that of the slice region.

[0156] The slice boundary information can be configured as follows.

[0157] When frames are sliced ​​into equal sizes, the number of slices and the coordinates of the top-left corner or slice number of each slice.

[0158] In the case where the frame is sliced ​​into non-uniform sizes, the coordinates of the top-left corner of each slice and the horizontal and vertical widths of each slice.

[0159] In the example shown in Figure 3, the slice numbers are reference numbers assigned sequentially from the top to the three slices (slice regions). Slice numbers can be assigned sequentially from the bottom.

[0160] Each of the above example embodiments can be configured by hardware, but it can also be implemented by a computer program.

[0161] The information processing system shown in Figure 14 includes a processor 701 such as one or more central processing units (CPUs), a program memory 702, a storage medium 703 for storing video data, and a storage medium 704 for storing bitstreams. Storage mediums 703 and 704 can be separate storage media or a storage area comprised of the same storage medium. Magnetic storage media such as hard disks can be used as the storage medium.

[0162] In the information processing system, the program (video encoding program or video decoding program) for implementing the function of each block described in each of the above example embodiments is stored in the program memory 702.

[0163] Then, the processor 701 implements the functions of the video encoders 101, 102, and 103 and the video decoders 201, 202, and 203 described in each example embodiment by performing processing according to the program stored in the program memory 702.

[0164] For example, the functions of video encoders 101, 102 and 103 are implemented by processor 701 according to the video encoding program, which performs the functions of the blocks (excluding the decoded image buffer 3005) in video encoders 101, 102 and 103 shown in Figures 2, 7 and 11.

[0165] For example, the functions of video decoders 201, 202 and 203 are implemented by processor 701 according to the video decoding program, which is used to implement the functions of each block (excluding the decoded image buffer 3005) in video decoders 201, 202 and 203 shown in Figures 2, 7 and 12.

[0166] At least the program memory 702 is a non-transitory computer-readable medium. However, programs can be stored in various types of transient computer-readable media. Transitory computer-readable media are supplied with programs via, for example, wired or wireless communication paths (i.e., via electrical signals, optical signals, or electromagnetic waves).

[0167] Figure 15 is a block diagram showing the main parts of a video encoding device. The video encoding device 10 shown in Figure 15 (implemented by video encoders 101, 102, and 103 in an example embodiment) includes a neural network 11 (implemented by encoder 1001 in an example embodiment), a quantization device 12 (implemented by quantizer 1002 in an example embodiment), and an entropy encoding device 13 (implemented by entropy encoders 1003 and 1003A in an example embodiment), and includes a first control device 14 (implemented by a first control unit 1004 in an example embodiment) for disallowing encoding processing based on the neural network 11 from crossing slice boundaries.

[0168] Figure 16 is a block diagram showing the main parts of a video decoding device. The video decoding device 20 shown in Figure 16 (implemented by video decoders 201, 202, and 203 in an example embodiment) includes a neural network 21 (implemented by decoder 2001 in an example embodiment), an inverse quantization device 22 (implemented by inverse quantizer 2002 in an example embodiment), and an entropy decoding device 23 (implemented by entropy decoders 2003 and 2003A in an example embodiment), and includes a first control device 24 (implemented by a first control unit 2004 in an example embodiment) for disallowing decoding processing based on neural network 21 from crossing slice boundaries.

[0169] Some or all of the above example embodiments may be described as the following supplementary notes, but are not limited to them.

[0170] (Supplementary Note 1) A video encoding device includes a neural network, a quantization device and an entropy encoding device. The video encoding device further includes: a first control device for preventing neural network-based encoding processing from crossing slice boundaries.

[0171] (Supplementary Note 2) The video encoding device according to Supplementary Note 1 also includes

[0172] The second control device (implemented by the second control units 1005 and 1005A in the example embodiment) is used to terminate entropy coding in the slice, wherein the entropy coding device performs entropy coding on the identification information indicating whether to terminate entropy coding.

[0173] (Supplementary Note 3) The video encoding device according to Supplementary Note 1 or 2 also includes

[0174] A predictive coding apparatus (implemented by CP video encoder 3002 in the example embodiment) is used to perform predictive coding on a unit-by-unit basis.

[0175] (Supplementary Note 4) A video decoding device includes a neural network, an inverse quantization device, and an entropy decoding device, wherein the video decoding device further includes a first control device for preventing the neural network-based decoding process from crossing slice boundaries.

[0176] (Supplementary Note 5) According to the video decoding device described in Supplementary Note 4, the entropy decoding device is used to perform entropy decoding on the identification information indicating whether to terminate entropy encoding in the slice, wherein the video decoding device further includes a second control device (implemented by the second control unit 2005 and 2005A in the example embodiment) for terminating entropy decoding in the slice based on the identification information.

[0177] (Supplementary Note 6) The video decoding device according to Supplementary Note 4 or 5 also includes

[0178] A predictive decoding apparatus (implemented by CP video decoder 3004 in the example embodiment) is used to perform predictive decoding on a unit-by-unit basis.

[0179] (Supplementary Note 7) A video coding method for performing quantization and entropy coding based on neural network coding processes, wherein the video coding method includes: performing control such that the coding process does not cross slice boundaries.

[0180] (Supplementary Note 8) The video coding method according to Supplementary Note 7 further includes: performing control to terminate entropy coding in a slice; and performing entropy coding on identification information indicating whether to terminate entropy coding.

[0181] (Supplementary Note 9) The video coding method according to Supplementary Note 7 or Supplementary Note 8 further includes: performing predictive coding processing on a unit basis.

[0182] (Supplementary Note 10) A video decoding method for performing neural network-based decoding, inverse quantization, and entropy decoding, wherein the video decoding method includes: performing control such that the decoding process does not cross slice boundaries.

[0183] (Supplementary Note 11) The video decoding method according to Supplementary Note 10 further includes: performing entropy decoding on identification information indicating whether to terminate entropy coding in a slice; and performing control based on the identification information to terminate entropy decoding in a slice.

[0184] (Supplementary Note 12) The video decoding method according to Supplementary Note 10 or Supplementary Note 11 further includes: performing predictive decoding processing on a unit basis.

[0185] (Supplementary Note 13) A video coding program for enabling a computer to: perform neural network-based coding processing, quantization processing, and entropy coding; and to perform control such that the coding processing does not cross slice boundaries.

[0186] (Supplementary Note 14) According to the video encoding procedure described in Supplementary Note 13, the video encoding procedure further causes the computer to: perform control to terminate entropy encoding in a slice; and perform entropy encoding on identification information indicating whether to terminate entropy encoding.

[0187] (Supplementary Note 15) According to the video coding procedure of Supplementary Note 13 or Supplementary Note 14, the computer is also made to: perform predictive coding processing on a unit-by-unit basis.

[0188] (Supplementary Note 16) A video decoding program for enabling a computer to: perform neural network-based decoding processing, inverse quantization processing, and entropy decoding processing; and to perform control such that the decoding processing does not cross slice boundaries.

[0189] (Supplementary Note 17) According to the video decoding procedure of Supplementary Note 16, the video decoding procedure also causes the computer to: perform entropy decoding on identification information indicating whether to terminate entropy decoding in the slice; and perform control based on the identification information to terminate entropy decoding in the slice.

[0190] (Supplementary Note 18) According to the video decoding procedure of Supplementary Note 16 or Supplementary Note 17, the computer is also made to: perform predictive decoding processing on a unit-by-unit basis.

[0191] (Supplementary Note 19) A storage medium for storing a bitstream generated by a video encoding device including a neural network, a quantization device, and an entropy encoding device, wherein the video encoding device includes a first control device for preventing encoding processing based on the neural network from crossing slice boundaries.

[0192] (Supplementary Note 20) A storage medium for storing a bitstream generated by a video coding method for performing neural network-based coding, quantization, and entropy coding, wherein the video coding method performs control such that the coding process does not cross slice boundaries.

[0193] (Supplementary Note 21) A video coding device includes a neural network, a quantization device, and an entropy coding device, comprising...

[0194] The second control device is used to terminate entropy coding in the slice, wherein the entropy coding device performs entropy coding on the identification information indicating whether to terminate entropy coding.

[0195] This supplementary note relates to the first variant mentioned above.

[0196] (Supplementary Note 22) The video encoding device according to Supplementary Note 21, wherein, in the neural network-based encoding process, only the slice boundaries of subsequent slice regions in the bit stream sequence do not intersect.

[0197] This supplementary note relates to the second variant mentioned above.

[0198] (Supplementary Note 23) The video encoding apparatus according to Supplementary Note 22, wherein a predetermined interval is multiplexed into a bit stream; and the entropy encoding device performs entropy encoding on identification information indicating whether the second control device terminates entropy encoding at a predetermined interval.

[0199] This supplementary note relates to the third variant mentioned above.

[0200] (Supplementary Note 24) A video decoding device includes a neural network, an inverse quantization device, and an entropy decoding device, wherein...

[0201] The entropy decoding device performs entropy decoding on the identification information indicating whether to terminate entropy encoding in the slice. The video decoding device also includes...

[0202] The second control device is used to terminate entropy decoding in the slice based on the identification information.

[0203] This supplementary note relates to the first variant mentioned above.

[0204] (Supplementary Note 25) According to the video decoding device described in Supplementary Note 24, in the neural network-based decoding process, only the slice boundaries of slice regions that follow the bitstream sequence are not crossed.

[0205] This supplementary note relates to the second variant mentioned above.

[0206] (Supplementary Note 26) The video encoding apparatus according to Supplementary Note 25, wherein the bitstream is demultiplexed at a predetermined interval, and the entropy decoding apparatus performs entropy decoding on the identification information indicating whether to terminate entropy encoding at the predetermined interval.

[0207] This supplementary note relates to the third variant mentioned above.

[0208] This application is based on and claims the benefit of priority from Japanese Patent Application No. 2023-155646 filed on September 21, 2023 and Japanese Patent Application No. 2024-035446 filed on March 8, 2024, the disclosure of which is incorporated herein by reference in its entirety.

[0209] List of reference numerals

[0210] 10. Video encoding equipment

[0211] 11 Neural Networks

[0212] 12 Quantization device

[0213] 13 Entropy Coding Device

[0214] 14 First Control Device

[0215] 20 Video decoding equipment

[0216] 21 Neural Networks

[0217] 22 Inverse Quantization Device

[0218] 23 Entropy Decoding Device

[0219] 24 First Control Device

[0220] 101, 102, 103 Video encoders

[0221] 201, 202, 203 Video decoders

[0222] 701 processor

[0223] 702 Program Memory

[0224] 703 and 704 storage media

[0225] 1001 Encoder

[0226] 1002 quantizer

[0227] 1003, 1003A Entropy Encoder

[0228] 1004 First Control Department

[0229] 1005, 1005A Second Control Unit

[0230] 1006 Multiplexer

[0231] 2001 Decoder

[0232] 2002 Inverse Quantizer

[0233] 2003, 2003A Entropy Decoder

[0234] 2004 First Control Department

[0235] 2005, 2005A Second Control Section

[0236] 2006 Demultiplexer

[0237] 3000 switch

[0238] 3001 NN Video Encoder

[0239] 3002 CP Video Encoder

[0240] 3003 NN Video Decoder

[0241] 3004 CP Video Decoder

[0242] 3005 Decoding Image Buffer

[0243] 4000 multiplexer

[0244] 5000 Demultiplexer

Claims

1. A video encoding device, comprising a neural network, a quantization device, and an entropy encoding device, wherein the video encoding device further comprises: A first control device is used to prevent encoding processing based on the neural network from crossing slice boundaries.

2. The video encoding device according to claim 1, further comprising: A second control device is used to terminate entropy coding in a slice, wherein the entropy coding device performs entropy coding on identification information indicating whether to terminate the entropy coding.

3. The video encoding device according to claim 1 or 2, further comprising: A predictive coding apparatus for performing predictive coding on a unit-by-unit basis.

4. A video decoding device, comprising a neural network, an inverse quantization device, and an entropy decoding device, wherein, The video decoding device further includes a first control device, which is used to prevent decoding processing based on the neural network from crossing slice boundaries.

5. The video decoding device according to claim 4, wherein, The entropy decoding device performs entropy decoding on the identification information indicating whether to terminate entropy encoding in the slice. The video decoding device further includes a second control device, which is used to terminate entropy decoding in the slice based on the identification information.

6. The video decoding device according to claim 4 or 5, further comprising: A predictive decoding apparatus for performing predictive decoding on a unit-by-unit basis.

7. A video coding method for performing neural network-based coding, quantization, and entropy coding, wherein, The video encoding method includes: performing control to ensure that the encoding process does not cross slice boundaries.

8. The video encoding method according to claim 7, further comprising: Execute control to terminate entropy encoding in the slice; And perform entropy encoding on the identification information indicating whether to terminate the entropy encoding.

9. The video encoding method according to claim 7 or 8, further comprising: Perform predictive coding processing on a unit-by-unit basis.

10. A video decoding method for performing neural network-based decoding, inverse quantization, and entropy decoding, wherein, The video decoding method includes: performing control to ensure that the decoding process does not cross slice boundaries.

11. The video decoding method according to claim 10, further comprising: Perform entropy decoding on the identifier indicating whether to terminate entropy encoding in the slice; And based on the identification information, control is performed to terminate entropy decoding in the slice.

12. The video decoding method according to claim 10 or 11, further comprising: Perform predictive decoding processing on a unit-by-unit basis.

13. A video encoding program for enabling a computer to: perform neural network-based encoding processing, quantization processing, and entropy encoding; and to perform control such that the encoding processing does not cross slice boundaries.

14. The video encoding program according to claim 13, wherein, The video encoding program also causes the computer to: perform control to terminate entropy encoding in a slice; and perform entropy encoding on identification information indicating whether to terminate the entropy encoding.

15. The video encoding program according to claim 13 or 14 further causes the computer to: perform predictive coding processing on a unit-by-unit basis.

16. A video decoding program for causing a computer to: perform neural network-based decoding processing, inverse quantization processing, and entropy decoding processing; and perform control such that the decoding processing does not cross slice boundaries.

17. The video decoding program according to claim 16, wherein, The video decoding program also causes the computer to: perform entropy decoding on identification information indicating whether to terminate entropy decoding in a slice; and perform control based on the identification information to terminate entropy decoding in the slice.

18. The video decoding program according to claim 16 or 17 further causes the computer to: perform predictive decoding processing on a unit-by-unit basis.

19. A storage medium for storing a bitstream generated by a video encoding device, the video encoding device comprising a neural network, a quantization device, and an entropy encoding device, wherein, The video encoding device includes a first control device for preventing encoding processes based on the neural network from crossing slice boundaries.

20. A storage medium for storing a bitstream generated by a video coding method, said video coding method being used to perform neural network-based coding processing, quantization processing, and entropy coding, wherein, The video encoding method is controlled to ensure that the encoding process does not cross slice boundaries.

Images