Video encoding device, video decoding device, video encoding method, and video decoding method

Abstract

To prevent a reduction in memory access efficiency.SOLUTION: A video encoding device 10 comprises: division means 51 that divides a tensor to be encoded into a plurality of rectangular areas; reduction means 52 that, for every rectangular area, generates a plurality of reduction areas from the rectangular area; entropy encoding means 53 that, for every rectangular area, subjects a tensor of the rectangular area and the tensors of the plurality of reduction areas to entropy encoding by using probability distributions, and generates a bit stream; and multiplexing means 54 that multiplexes the bit stream. The entropy encoding means 53 uses the probability distribution estimated from the tensor of the rectangular area to be encoded or the reduction area having a smaller size than the reduction area, and uses the probability distribution related to the minimum reduction area in the already-encoded rectangular area as the probability distribution related to the minimum reduction area.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] The present disclosure relates to a video encoding device, a video decoding device, a video encoding method, and a video decoding method using a neural network.

Background Art

[0002] A new video encoding technique that combines an auto-encoder, which is a type of neural network, quantization, and entropy encoding is described in Non-Patent Document 1.

[0003] An auto-encoder compresses input data into a low-dimensional feature tensor so as to include only important features. Then, the auto-encoder generates reconstructed data by reconstructing the low-dimensional feature tensor to the original dimension. The process of dropping the data into the low-dimensional feature tensor (the first half) is called encoding. The process of generating the reconstructed data (the second half) is called decoding.

[0004] The learning of the auto-encoder is advanced so as to minimize the reconstruction error (the difference between the input data and the reconstructed data). The auto-encoder is designed to impose constraints on the structure of the encoding or to add a regularization term to the loss function of the network so as to obtain meaningful feature amounts.

[0005] Non-Patent Document 2 describes a method of inputting a predetermined tensor and entropy encoding the input tensor using a probability model (that is, a probability distribution of prediction). In this method, first, the input tensor is downsampled to obtain a tensor with a scale smaller than the scale of the input tensor. A probability distribution is estimated from the obtained tensor, and the tensor is entropy encoded. Then, the input tensor is entropy encoded using the estimated probability distribution. Such a method is called MSPSM (Multi-Scale Progressive Statistical Model) entropy encoding. MSPSM entropy encoding is simply denoted as MSPSM.

[0006] Hereinafter, as a predetermined tensor, a feature tensor obtained by applying quantization to a tensor output by an encoder in an autoencoder will be taken as an example.

Prior Art Documents

Non-Patent Documents

[0007]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0008] FIG. 1 is a block diagram showing an implementation example of MSPSM. Let the configuration shown in FIG. 1 be an MSPSM system.

[0009] The MSPSM system downsamples the input feature tensor from the encoder. The MSPSM system further downsamples the feature tensor obtained by the downsampling. The MSPSM system repeats the downsampling of the feature tensor obtained by the downsampling. The feature tensor obtained by the i-th downsampling is called the feature tensor of scale i. i is called the number of scales.

[0010] The feature tensor of scale i is y imay be expressed as. The probability distribution estimated from the feature tensor at scale i is denoted as p i and the context information when estimating p i is denoted as z i .

[0011] FIG. 1 shows a configuration example of the MSPSM system for the case of i = 4. The MSPSM system illustrated in FIG. 1 includes downsamplers 11 to 14 corresponding to each of scales i (i = 1 to 4), probability models (probability model estimators 21 to 24) for estimating a probability distribution using the outputs of the downsamplers 11 to 14, and entropy encoders 30 to 34 for entropy-encoding a feature tensor using the probability distribution estimated by the probability model estimators 21 to 24.

[0012] The downsamplers 11 to 14 downsample the input feature tensor at a predetermined interval. For example, the downsamplers 11 to 14 downsample the input feature tensor by a factor of 1 / 2. That is, the number of samples is reduced to 1 / 2 both vertically and horizontally. Specifically, for example, when the downsamplers 11 to 14 perform downsampling by a factor of 1 / 2, they extract the lower-right element of a 2×2 element (feature value).

[0013] The probability model estimators 21 to 23 estimate the probability model p i+2 from the feature tensor at scale (i + 1) and the context information z i+1 . For example, the probability model p i+1 is obtained by inputting the feature tensor and the context information into a convolutional neural network. Note that the probability model estimator 24 estimates the probability model p 4 from the context information with a predetermined initial value set and the feature tensor at scale (i + 1). Here, the predetermined initial value is, for example, a zero value.

[0014] The entropy encoders 31 to 33 encode the feature tensor at scale i using the probability model p i+1Use it to perform entropy encoding. Note that the entropy encoder 30 uses the input feature tensor from the encoder as the probability model p 1 to perform entropy encoding. Also, the entropy encoder 34 performs entropy encoding on the feature tensor at scale 4 using a predetermined probability distribution.

[0015] The bitstream generator 40 uses the outputs of the entropy encoders 30 to 34 as the bitstream.

[0016] FIG. 2 is an explanatory diagram showing an example of the state of downsampling. In FIG. 2, as an example, the state of reduction of a feature tensor of 32×32 elements is illustrated. The small rectangles in FIG. 2 correspond to the elements. The markings inside the small rectangles are provided to facilitate understanding of the correspondence relationship of the elements between the scales.

[0017] The y illustrated in FIG. 2 1 is obtained by downsampling y 0 (input feature tensor). The y 2 is obtained by downsampling y 1 The y 3 is obtained by downsampling y 2 The y 4 is obtained by downsampling y 3 is obtained by downsampling y

[0018] As shown in FIG. 3, each element of the feature tensor y i at scale i is entropy encoded using the probability model estimated from the feature tensor at scale (i + 1). That is, in the MSPSM, when the entropy encoders 31 to 33 encode the elements of the feature tensor at a certain scale, they use the probability distribution estimated from the feature tensor of a smaller size (for example, one size smaller) than that scale. Note that the entropy encoder 34 uses the feature tensor y 4For the (minimum feature tensor), entropy encoding is performed using a predetermined probability distribution. The predetermined probability distribution may be referred to as a fixed probability distribution. The fixed probability distribution is shared between the video encoding device and the video decoding device. The entropy encoder 30 uses the probability model estimated from the feature tensor y 1 to entropy-encode each element of the input feature tensor y 0 .

[0019] Each element of the feature tensor y i at scale i is entropy-encoded using the probability model estimated from the feature tensor at scale (i + 1). Therefore, in the MSPSM system, the entropy encoder 33 performs entropy encoding when the processing of the probability model estimator 24 is completed. The entropy encoder 32 performs entropy encoding when the processing of the probability model estimator 23 is completed. The entropy encoder 31 performs entropy encoding when the processing of the probability model estimator 22 is completed. The entropy encoder 30 performs entropy encoding when the processing of the probability model estimator 21 is completed.

[0020] Note that the entropy encoders 31 to 34 do not perform entropy encoding on the elements that have been entropy-encoded at a scale (a scale with a larger value of i for scale i) that was the target of entropy encoding previously.

[0021] The entropy encoders 30 to 34 may operate on the values of each element of the feature tensor input to the entropy encoder before the entropy encoding process. For example, there may be rounding to integer values. In such a case, the probability model estimators 21 to 24 apply the process for aligning with the values to be entropy-decoded to the feature tensor at scale (i + 1), and then execute the estimation process using the feature tensor after the application.

[0022] As described above, the feature tensor y iEach element is entropy-coded using a probability model estimated from the feature tensor at scale (i + 1). Therefore, when entropy-coding the elements of the feature tensor y at scale i i when entropy-coding the elements of i , the probability model used for the elements to be coded reflects not only the elements preceding the element to be coded in raster scan order at scale (i + 1) (the upper and left elements at each scale in Fig. 2), but also the elements following in raster scan order (the right and lower elements at each scale in Fig. 2).

[0023] Hereinafter, when estimating the probability model, considering the elements following in raster scan order may be expressed as look-ahead of elements, or preprocessing of elements.

[0024] Note that the probability model is updated as appropriate. For example, the probability model is updated each time entropy-coding of elements is performed.

[0025] Since the probability model reflecting the elements following in raster scan order is used, the probability distribution used for entropy-coding becomes closer to the probability distribution of the entire screen. As a result, even when the probability distribution varies depending on the region on the screen, the coding performance is improved. From a different perspective on look-ahead, when entropy-coding each element of the feature tensor y at scale i i since the probability model based on the elements discretely arranged within the screen is used, it can be said that the probability distribution used for entropy-coding becomes closer to the probability distribution of the entire screen.

[0026] Generally, when video coding is performed using a neural network, the feature tensor is temporarily stored in the memory within the video coding device.

[0027] The look-ahead of elements contributes to the improvement of encoding performance and the reduction of the number of updates of the probability model. However, when obtaining the feature tensor of scale (i + 1) from the feature tensor of scale i, discontinuous areas of memory are accessed. Then, the memory access efficiency decreases. That is, the overall memory access time becomes longer.

[0028] An object of the present invention is to provide a video encoding device, a video decoding device, a video encoding method, and a video decoding method that can suppress a decrease in memory access efficiency.

Means for Solving the Problem

[0029] A video encoding device according to the present disclosure includes a dividing unit that divides an encoding target tensor into a plurality of rectangular regions, a reducing unit that generates a plurality of reduced regions from the rectangular region for each rectangular region, and for each rectangular region, entropy encoding the tensor of the rectangular region and the tensors of the plurality of reduced regions using a probability distribution to generate a bitstream, and a multiplexing unit that multiplexes the bitstreams. The entropy encoding unit uses a probability distribution estimated from a tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be encoded, and uses the probability distribution regarding the smallest reduced region in the already encoded rectangular region as the probability distribution regarding the smallest reduced region.

[0030] The video decoding apparatus according to the present disclosure includes demultiplexing means for demultiplexing a multiplexed bitstream in which bitstreams corresponding to respective rectangular regions obtained by dividing an encoding target tensor are multiplexed, and for each rectangular region, entropy decoding is performed using a probability distribution for the rectangular region and a plurality of reduced regions generated from the rectangular region, and a tensor of the rectangular region is output, entropy decoding means, and combining means for combining tensors of the rectangular regions obtained by entropy decoding. The entropy decoding means uses a probability distribution estimated from a tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be decoded, and uses, as the probability distribution regarding the smallest reduced region, the probability distribution regarding the smallest reduced region in the already encoded rectangular region acquired by the demultiplexing means.

[0031] The video encoding method according to the present disclosure divides an encoding target tensor into a plurality of rectangular regions, generates a plurality of reduced regions from each rectangular region, and for each rectangular region, entropy encodes the tensor of the rectangular region and the tensors of the plurality of reduced regions using a probability distribution to generate a bitstream, multiplexes the bitstreams, and when performing entropy encoding, uses a probability distribution estimated from a tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be encoded, and uses, as the probability distribution regarding the smallest reduced region, the probability distribution regarding the smallest reduced region in the already encoded rectangular region.

[0032] The video decoding method according to the present disclosure demultiplexes a multiplexed bitstream in which bitstreams corresponding to each of rectangular regions obtained by dividing an encoding target tensor are multiplexed, and for each rectangular region, entropy decoding is performed using a probability distribution for the rectangular region and a plurality of reduced regions generated from the rectangular region to output a tensor of the rectangular region, synthesizes the tensors of the rectangular regions obtained by entropy decoding, and when performing entropy decoding, uses a probability distribution estimated from a tensor of a reduced region having a size smaller than the rectangular region or reduced region to be decoded, and uses, as the probability distribution regarding the smallest reduced region, the probability distribution regarding the smallest reduced region in the already encoded rectangular region obtained by demultiplexing.

[0033] The video encoding program according to the present disclosure causes a computer to execute a process of dividing an encoding target tensor into a plurality of rectangular regions, a process of generating a plurality of reduced regions from each rectangular region, a process of performing entropy encoding on the tensor of the rectangular region and the tensors of the plurality of reduced regions using a probability distribution to generate a bitstream, and a process of multiplexing the bitstream, and when performing entropy encoding, causes the probability distribution estimated from the tensor of the reduced region having a size smaller than the rectangular region or reduced region to be encoded to be used, and causes the probability distribution regarding the smallest reduced region in the already encoded rectangular region to be used as the probability distribution regarding the smallest reduced region.

[0034] The video decoding program based on the present disclosure causes a computer to perform a process of demultiplexing a multiplexed bitstream in which bitstreams corresponding to each of rectangular regions obtained by dividing an encoding target tensor are multiplexed, and for each rectangular region, perform entropy decoding using a probability distribution for the rectangular region and a plurality of reduced regions generated from the rectangular region to output a tensor of the rectangular region, and a process of synthesizing tensors of the rectangular regions obtained by entropy decoding. When performing entropy decoding, use a probability distribution estimated from a tensor of a reduced region having a size smaller than the rectangular region or reduced region to be decoded, and use, as the probability distribution regarding the smallest reduced region, the probability distribution regarding the smallest reduced region in the already encoded rectangular region obtained by demultiplexing.

Advantages of the Invention

[0035] According to the present invention, a decrease in memory access efficiency can be suppressed.

Brief Description of the Drawings

[0036]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Embodiments for Carrying Out the Invention

[0037] Hereinafter, embodiments will be described with reference to the drawings. In the embodiments described below, as an example of a tensor, a feature tensor output by an encoder in an autoencoder will be used.

[0038] FIG. 4 is an explanatory diagram for explaining region division processing executed by the video encoding device according to the first embodiment.

[0039] In the first embodiment, the video encoding device divides the input feature tensor into a plurality of rectangular regions. In FIG. 4(A), an example in which the feature tensor is divided into two rectangular regions is shown. In FIG. 4(B), an example in which it is divided into 16 rectangular regions is shown. However, the examples shown in FIG. 4 are just examples, and the shape of the rectangular regions obtained by the division is arbitrary.

[0040] For each of the plurality of rectangular regions obtained by the division, the video encoding device performs entropy encoding based on MSPSM and generates each encoded sequence (hereinafter referred to as a bitstream). The video encoding device multiplexes the generated bitstreams to generate a multiplexed bitstream.

[0041] When performing entropy encoding on a rectangular region, the video encoding device uses the probability distribution based on the elements within that rectangular region and does not use the probability distribution based on the elements within other rectangular regions.

[0042] The video decoding device demultiplexes the multiplexed bitstream to obtain bitstreams corresponding to each of a plurality of rectangular regions. The video decoding device performs entropy decoding on each bitstream and reproduces each rectangular region (each element of the rectangular region). The video decoding device combines each rectangular region to restore the feature tensor.

[0043] The video encoding device according to the first embodiment can improve the memory access efficiency by applying MSPSM in units of rectangular regions. For example, if the video encoding device transfers only the elements of the rectangular region from the main memory to the cache memory (cache) in advance and executes the prefetching process on the elements in the cache, the memory access efficiency is improved.

[0044] Compare the case of encoding the elements of the entire screen with the case of encoding the elements in the feature tensor of the rectangular region. As an example, compare the case of encoding the elements in the feature tensor of 32×32 elements (reference in FIG. 2) with the case of encoding the elements in the feature tensor of a 32×16 element rectangular region (reference in FIG. 4(A)). Let the number of elements in the horizontal direction of the feature tensor be W and the number of elements in the vertical direction be H. 0 Compare the case of encoding the elements in the feature tensor of 32×32 elements (see FIG. 2) with the case of encoding the elements in the feature tensor of a 32×16 element rectangular region (see FIG. 4(A)). Let the number of elements in the horizontal direction of the feature tensor be W and the number of elements in the vertical direction be H.

[0045] When encoding the elements in the 32×32 element feature tensor (see FIG. 2), the number of elements stored in the memory when executing MSPSM is W×H+(1 / 4)×(W×H)+(1 / 16)×(W×H)+(1 / 64)×(W×H)+(1 / 256)×(W×H)=(341 / 256)×(W×H)=1364.

[0046] When encoding the elements in the feature tensor of the 32×16 element rectangular region, the memory amount required when executing MSPSM is (1 / 2)×(W×H)+(1 / 8)×(W×H)+(1 / 32)×(W×H)+(1 / 128)×(W×H)+(1 / 512)×(W×H)=(341 / 512)×(W×H)=682.

[0047] When encoding the elements in the feature tensor of the rectangular region, the number of elements stored in memory is less than the number of elements stored in memory when encoding the elements of the entire screen. Therefore, the required memory amount also decreases, and the memory access efficiency is improved. In other words, compared with the case of encoding the entire screen as a unit, the memory amount required for one unit of encoding (encoding for one rectangular region) is less, so the memory access efficiency is improved.

[0048] FIG. 5 is an explanatory diagram for explaining the probability distribution reference executed by the video encoding device of the second embodiment.

[0049] In the second embodiment, the video encoding device divides the input feature tensor into a plurality of rectangular regions in the same manner as the video encoding device in the first embodiment. FIG. 5 shows an example in which the input feature tensor is divided into 16 rectangular regions.

[0050] The video encoding device does not use a fixed probability distribution when entropy encoding the smallest feature tensor y 4 but uses a probability distribution estimated from a feature tensor of a rectangular region that has already been encoded, for example, y 4 in the already encoded rectangular region. That is, the video encoding device can use a probability distribution in which the elements in the encoded rectangular region are considered when entropy encoding the feature tensor y 4 As a result, it can be expected that the encoding performance will be further improved.

[0051] Note that the video encoding device determines the size of the rectangular region so as to satisfy the constraint of the minimum size of the MSPSM. FIG. 6 is an explanatory diagram for explaining the constraint of the size of the rectangular region.

[0052] In order to maintain the effect of look-ahead, the rectangular region has a minimum scale (in the example shown in FIG. 2 (scale number = 4), y 4It is preferable that a constraint that one or more of the elements included in [[ID=]] (the small rectangles marked with horizontal lines in FIGS. 2 and 6) are included is satisfied. The size of the rectangular area under such a constraint is 16×16 elements. For example, if the size of the rectangular area is 16×15 elements, since the rectangular area does not include the element of y 4 it is not preferable as the size of the rectangular area.

[0053] In other words, the size of the rectangular area is determined so that the number of elements at the minimum scale (for example, scale 4) does not become zero. For example, assuming that the scale number i = 4 and the downsamplers 11 to 14 shown in FIG. 1 are configured to extract the lower right element of 2×2 elements (feature amounts) and downsample it to 1 / 2, for the feature tensor y of 16×16 elements 0 the feature tensors y 1 ~y 4 each contain one or more elements. However, for the feature tensor y of 16×15 elements 0 the number of elements of the feature tensor y 4 is zero. Since there are no elements at the minimum scale, the probability distribution cannot be estimated from the feature tensor y 4 As a result, it is difficult to realize the MSPSM.

[0054] Embodiment 1. FIG. 7 is a block diagram showing a configuration example of a video encoding device and a video decoding device.

[0055] The video encoding device 100 shown in FIG. 7 includes a region division unit 101, a downsampling unit 102, a probability model estimator 103, an entropy encoder 104, and a multiplexer 105.

[0056] The video decoding device 200 shown in FIG. 7 includes a tensor reconstruction unit 201, a probability model estimator 203, an entropy decoder 204, and a demultiplexer 205.

[0057] Note that the arrows in FIG. 7 clearly indicate the direction of the signal (data) flow, but do not exclude bidirectionality. This also applies to other block diagrams.

[0058] [Description of Video Encoding Device] In the video encoding device 100, the region division unit 101 divides the tensor to be encoded (encoding target tensor) into a plurality of rectangular regions (see FIG. 4). Hereinafter, as an example, the encoding target tensor is assumed to be a feature tensor obtained by applying quantization to the tensor output from an encoder (not shown) in an autoencoder.

[0059] The downsampling unit 102 generates one or more reduced regions for each of the plurality of rectangular regions. For example, the downsampling unit 102 downsamples the feature tensor of each rectangular region input from the region division unit 101 at a predetermined interval. The downsampling unit 102 can be configured by, for example, the downsamplers 11 to 14 shown in FIG. 1.

[0060] The probability model estimator 103 can be configured by, for example, the probability model estimators 21 to 24 shown in FIG. 1. The probability model estimator 103 estimates a probability distribution in the same manner as the probability model estimators 21 to 24.

[0061] The entropy encoder 104 can be configured by, for example, the entropy encoders 30 to 34 shown in FIG. 1. The entropy encoder 104 entropy-encodes the feature tensor of each of the plurality of rectangular regions using the probability distribution estimated by the probability model estimator 103. Specifically, the entropy encoder 104 entropy-encodes each element of the feature tensor to generate each bit stream.

[0062] The multiplexer 105 multiplexes the generated bit streams to generate a multiplexed bit stream.

[0063] Next, an example of the specific operation of the video encoding device 100 will be described with reference to the flowchart of FIG. 8.

[0064] The region division unit 101 divides the input feature tensor into a plurality of rectangular regions (step S101).

[0065] The downsampling unit 102 downsamples the feature tensor at a predetermined interval for each of the plurality of rectangular regions input from the region division unit 101 (step S102). By downsampling, feature tensors of a plurality of scales as exemplified in FIG. 2, for example, are obtained from each of the plurality of rectangular regions.

[0066] The probability model estimator 103 estimates a probability distribution from the feature tensors of each scale for each of the plurality of rectangular regions (step S103).

[0067] The entropy encoder 104 entropy-encodes the feature tensor of each of the plurality of rectangular regions to generate a bitstream (step S104). Note that the entropy encoder 104 generates a bitstream for each scale for each of the rectangular regions.

[0068] For one rectangular region, when the entropy encoder 104 encodes the elements of the feature tensor of a certain scale, it uses the probability distribution estimated from the feature tensor of a size smaller (for example, one size smaller) than the scale.

[0069] Specifically, taking the example shown in FIG. 2, the probability model estimator 103 calculates the probability distribution from the feature tensor y 4 generated by the downsampling unit 102. The entropy encoder 104 entropy-encodes the feature tensor y 3 using the probability distribution. The probability model estimator 103 calculates the probability distribution from the feature tensor y 3 generated by the downsampling unit 102. The entropy encoder 104 entropy-encodes the feature tensor y2 Perform entropy encoding. The probability model estimator 103 calculates a probability distribution from the feature tensor y generated by the downsampling unit 102. 2 The entropy encoder 104 uses the probability distribution to perform entropy encoding on the feature tensor y. 1 Perform entropy encoding. The probability model estimator 103 calculates a probability distribution from the feature tensor y generated by the downsampling unit 102. 1 The entropy encoder 104 uses the probability distribution to perform entropy encoding on the feature tensor y. 0 Perform entropy encoding. The entropy encoder 104 outputs the encoded data of the feature tensor of each scale to the multiplexer 105.

[0070] Note that the entropy encoder 104 uses a fixed probability distribution to perform entropy encoding on the feature tensor y. 4 Perform entropy encoding.

[0071] In the flowchart shown in FIG. 8, in the processes of steps S102 and S103, feature tensors of multiple scales are generated and probability distributions are estimated for multiple rectangular regions. However, steps S102 and S103 may be configured to perform the process on one rectangular region. In that case, the processes of steps S102 and S103 are executed for the number of rectangular regions generated in the process of step S101.

[0072] The entropy encoder 104 multiplexes the multiple bitstreams generated in the process of step S104 to generate a multiplexed bitstream (step S105).

[0073] [Description of video decoding device] In the video decoding device 200, the demultiplexer 205 demultiplexes the multiplexing of the multiplexed bitstream from the video encoding device 100 to obtain bitstreams corresponding to each of the multiple rectangular regions.

[0074] For each bit stream, that is, for each rectangular region, the entropy decoder 204 entropy-decodes the elements of the feature tensor at each scale with reference to the probability distribution from the probability model estimator 203.

[0075] The probability model estimator 203 operates in the same manner as the probability model estimator 103. That is, the probability model estimator 203 can be composed of, for example, the probability model estimators 21 to 24 shown in FIG. 1. Similar to the probability model estimators 21 to 24, the probability model estimator 203 estimates a probability distribution.

[0076] The tensor reconstruction unit 201 synthesizes the feature tensors of each rectangular region output by the entropy decoder 204 to restore the feature tensor input to the video encoding device 100. That is, the tensor reconstruction unit 201 connects the feature tensors of each rectangular region to obtain the feature tensor input to the video encoding device 100.

[0077] Next, an example of the specific operation of the video decoding device 200 will be described with reference to the flowchart of FIG. 9.

[0078] The demultiplexer 205 demultiplexes the multiplexing of the multiplexed bit stream from the video encoding device 100 to obtain a plurality of bit streams (step S201). Each bit stream corresponds to each of the plurality of rectangular regions.

[0079] The probability model estimator 203 estimates a probability distribution from the feature tensors at each scale for each rectangular region (step S202).

[0080] The entropy decoder 204 entropy-decodes the plurality of bit streams to obtain the feature tensors of each rectangular region (step S203).

[0081] For a rectangular region, when the entropy decoder 204 decodes the elements of the feature tensor at a certain scale, it uses the probability distribution estimated from the feature tensor with a size smaller than that scale (for example, one level smaller).

[0082] Specifically, taking the example shown in FIG. 2, for a rectangular region, when the probability model estimator 203 outputs the feature tensor y 4 the probability model estimator 203 calculates the probability distribution from the feature tensor y 4 . The entropy decoder 204 uses the probability distribution to entropy-decode the feature tensor y 3 and outputs the feature tensor y 3 . When the probability model estimator 203 outputs the feature tensor y 3 the probability model estimator 203 calculates the probability distribution from the feature tensor y 3 . The entropy decoder 204 uses the probability distribution to entropy-decode the feature tensor y 2 and outputs the feature tensor y 2 . When the probability model estimator 203 outputs the feature tensor y 2 the probability model estimator 203 calculates the probability distribution from the feature tensor y 2 . The entropy decoder 204 uses the probability distribution to entropy-decode the feature tensor y 1 and outputs the feature tensor y 1 . When the probability model estimator 203 outputs the feature tensor y 1 the probability model estimator 203 calculates the probability distribution from the feature tensor y 1 . The entropy decoder 204 uses the probability distribution to entropy-decode the feature tensor y 0 and outputs the feature tensor y 0 .

[0083] The feature tensor y 0 is supplied to the tensor reconstruction unit 201.

[0084] Note that the entropy decoder 204 uses a fixed probability distribution to entropy-decode the feature tensor y4 Perform entropy decoding.

[0085] When the above processes are executed for all rectangular regions, the tensor reconstruction unit 201 can obtain the feature tensors of all rectangular regions. The tensor reconstruction unit 201 synthesizes the feature tensors of all rectangular regions to restore the feature tensor input to the video encoding apparatus 100 (step S204).

[0086] Note that the entropy decoder 204 may supply the feature vector of one rectangular region to the tensor reconstruction unit 201 each time it is obtained, or may supply all of them to the tensor reconstruction unit 201 in a batch when the feature vectors of all rectangular regions are obtained.

[0087] Embodiment 2. The video encoding apparatus according to the second embodiment is configured in the same manner as the video encoding apparatus according to the first embodiment shown in FIG. 7. However, the operations of the probability model estimator 103, the entropy encoder 104, and the multiplexer 105 in the video encoding apparatus 100 are different from those operations in the first embodiment.

[0088] [Description of Video Encoding Apparatus] In the first embodiment, the entropy encoder 104 uses a fixed probability distribution to perform entropy encoding on the feature tensor y 4 In the second embodiment, when one or more rectangular regions have been encoded, the entropy encoder 104 does not use a fixed probability distribution.

[0089] In the second embodiment, the probability model estimator 103 estimates a probability distribution from y 4 in the already encoded rectangular regions (see FIG. 5). That is, the probability model estimator 103 calculates a probability distribution from y 4 in the already encoded rectangular regions. The probability distribution is supplied to the entropy encoder 104. The entropy encoder 104 uses the probability distribution to perform entropy encoding on the feature tensor y 4

[0090] The multiplexer 105 generates a multiplexed bit stream as in the first embodiment. However, in the second embodiment, the multiplexer 105 also performs a process of multiplexing the probability distribution from y 4 in the already encoded rectangular region into the multiplexed bit stream as auxiliary information, for example.

[0091] Other processes of the video encoding device 100 are the same as those in the first embodiment.

[0092] In the second embodiment, in addition to the effects of the first embodiment, an effect that the encoding performance is further improved can be obtained.

[0093] [Description of video decoding device] The video decoding device of the second embodiment is configured in the same manner as the decoding device of the first embodiment shown in FIG. 7. However, the operations of the demultiplexer 205 and the entropy decoder 204 in the video decoding device 200 are different from those operations in the first embodiment.

[0094] In the second embodiment, the demultiplexer 205 also demultiplexes the information on the probability distribution estimated from y 4 in the already encoded rectangular region from the multiplexed bit stream.

[0095] In the first embodiment, the entropy decoder 204 entropy-decodes the feature tensor y 4 using a fixed probability distribution. However, in the second embodiment, the entropy decoder 204 entropy-decodes the feature tensor y 4 using the probability distribution estimated from y 4 in the already encoded rectangular region, which is the information demultiplexed by the demultiplexer 205.

[0096] Other processes of the video decoding device 200 are the same as those in the first embodiment.

[0097] Although each of the above-described embodiments can be configured by hardware, it can also be realized by a computer program.

[0098] The information processing system shown in FIG. 10 includes a processor 701 such as a CPU (Central Processing Unit), a program memory 702, a storage medium 703 for storing temporary data, and a storage medium 704 for storing a bit stream. The storage medium 703 and the storage medium 704 may be separate storage media or may be storage areas composed of the same storage medium. As the storage medium, a magnetic storage device such as a hard disk can be used. As the storage medium, SRAM (Static Random Access Memory) or flash ROM (Read Only Memory) may be used.

[0099] Examples of the temporary data include the feature tensors used by the video encoding device 100 and the video decoding device 200.

[0100] In the information processing system, a program (video encoding program or video decoding program) for realizing the functions of each block shown in each of the above embodiments is stored in the program memory 702.

[0101] The processor 701 realizes each function in the above embodiment by executing processing according to the program (software element: code) stored in the program memory 702.

[0102] That is, the processor 701 realizes the functions of the video encoding device 100 and the video decoding device 200 shown in each embodiment by executing processing according to the program stored in the program memory 702.

[0103] For example, the functions of the video encoding apparatus 100 are realized when the processor 701 executes processing according to a video encoding program for realizing the functions of each block in the video encoding apparatus 100 shown in FIG. 7.

[0104] Also, for example, the functions of the video decoding apparatus 200 are realized when the processor 701 executes processing according to a video decoding program for realizing the functions of each block in the video decoding apparatus 200 shown in FIG. 7.

[0105] Note that at least the program memory 702 is a non-transitory computer readable medium. The non-transitory computer readable medium includes various types of tangible storage media. Specific examples of the non-transitory computer readable medium include magnetic recording media (e.g., hard disks), magneto-optical recording media (e.g., magneto-optical disks), CD-ROM (Compact Disc-Read Only Memory), CD-R (Compact Disc-Recordable), CD-R / W (Compact Disc-ReWritable), and semiconductor memories (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM). However, the program may be stored in various types of transitory computer readable media.

[0106] The transitory computer readable medium is supplied with a program, for example, via a wired communication path or a wireless communication path, that is, via an electrical signal, an optical signal, or an electromagnetic wave.

[0107] FIG. 11 is a block diagram showing a main part of a video encoding apparatus. The video encoding apparatus 50 (realized by the video encoding apparatus 100 in the embodiment) shown in FIG. 11 includes a dividing means 51 (realized by the region dividing unit 101 in the embodiment) that divides a tensor to be encoded into a plurality of rectangular regions, and for each rectangular region, a reducing means 52 (realized by the downsampling unit 102 in the embodiment) that generates a plurality of reduced regions (for example, regions based on the feature tensor of scale i) from the rectangular region, and for each rectangular region, an entropy encoding means 53 (realized by the entropy encoder 104 in the embodiment) that entropy-encodes the tensor of the rectangular region and the tensors of the plurality of reduced regions using a probability distribution to generate a bit stream, and a multiplexing means 54 (realized by the multiplexer 105 in the embodiment) that multiplexes the bit stream. The entropy encoding means 53 uses a probability distribution estimated from the tensor of a reduced region (for example, a region based on the feature tensor of scale (i + 1) with respect to a region based on the feature tensor of scale i) having a size smaller than the rectangular region or the reduced region to be encoded, and uses the probability distribution regarding the smallest reduced region in the already encoded rectangular region as the probability distribution regarding the smallest reduced region.

[0108] FIG. 12 is a block diagram showing the main part of the video decoding apparatus. The video decoding apparatus 60 (realized by the video decoding apparatus 200 in the embodiment) shown in FIG. 12 includes a demultiplexing means 61 (realized by the demultiplexer 205 in the embodiment) for demultiplexing a multiplexed bitstream in which bitstreams corresponding to each of the rectangular regions obtained by dividing the tensor to be encoded are multiplexed, and for each rectangular region, entropy decoding is performed using a probability distribution for the rectangular region and a plurality of reduced regions generated from the rectangular region, and entropy decoding means 62 (realized by the entropy decoder 204 in the embodiment) for outputting the tensor of the rectangular region, and combining means 63 (realized by the tensor reconstruction unit 201 in the embodiment) for combining the tensors of the rectangular regions obtained by entropy decoding. The entropy decoding means 62 uses a probability distribution estimated from the tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be decoded, and uses, as the probability distribution regarding the smallest reduced region, the probability distribution regarding the smallest reduced region in the already encoded rectangular region acquired by the demultiplexing means.

[0109] Some or all of the above embodiments may be described as follows in the following supplementary notes, but are not limited thereto.

[0110] (Supplementary Note 1) A dividing means for dividing the tensor to be encoded into a plurality of rectangular regions, For each rectangular region, a reducing means for generating a plurality of reduced regions from the rectangular region, For each rectangular region, entropy encoding means for entropy encoding the tensor of the rectangular region and the tensors of the plurality of reduced regions using a probability distribution to generate a bitstream, A multiplexing means for multiplexing the bitstream, The entropy encoding means uses a probability distribution estimated from the tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be encoded. Video encoding apparatus.

[0111] (Appendix 2) For each rectangular region, there is provided a probability distribution estimation means for estimating the probability distribution of the tensor of the reduced region (in the embodiment, realized by the probability model estimator 103). The video encoding device according to Appendix 1.

[0112] (Appendix 3) The entropy encoding means uses the probability distribution estimated from the tensor of the reduced region having the size closest to the size of the rectangular region or the reduced region to be encoded (for example, the region by the feature tensor of scale (i + 1) with respect to the region by the feature tensor of scale i). The video encoding device according to Appendix 2.

[0113] (Appendix 4) The entropy encoding means performs entropy encoding on the tensor of the smallest reduced region using a predetermined fixed probability distribution. The video encoding device according to Appendix 3.

[0114] (Appendix 5) Demultiplexing means for demultiplexing the multiplexed bitstream in which the bitstreams corresponding to each of the rectangular regions obtained by dividing the tensor to be encoded are multiplexed, For each rectangular region, entropy decoding is performed using a probability distribution for the rectangular region and a plurality of reduced regions generated from the rectangular region, and entropy decoding means for outputting the tensor of the rectangular region, Comprising synthesis means for synthesizing the tensors of the rectangular regions obtained by the entropy decoding, The entropy decoding means uses the probability distribution estimated from the tensor of the reduced region having a size smaller than the rectangular region or the reduced region to be decoded. Video decoding device.

[0115] (Appendix 6) For each rectangular region, there is provided a probability distribution estimation means for estimating the probability distribution of the tensor of the reduced region. The video decoding device according to Appendix 5.

[0116] (Appendix 7) Divide the tensor to be encoded into a plurality of rectangular regions, For each rectangular region, generate a plurality of reduced regions from the rectangular region. For each rectangular region, entropy-encode the tensor of the rectangular region and the tensors of the plurality of reduced regions using a probability distribution to generate a bitstream. Multiplex the bitstreams. When performing the entropy encoding, use a probability distribution estimated from tensors of reduced regions having a size smaller than the rectangular region or the reduced regions to be encoded. Video encoding method.

[0117] (Appendix 8) Demultiplex the multiplexed bitstream in which the bitstreams corresponding to each of the rectangular regions obtained by dividing the tensor to be encoded are multiplexed. For each rectangular region, perform entropy decoding on the rectangular region and a plurality of reduced regions generated from the rectangular region using a probability distribution to output the tensor of the rectangular region. Synthesize the tensors of the rectangular regions obtained by the entropy decoding. When performing the entropy decoding, use a probability distribution estimated from tensors of reduced regions having a size smaller than the rectangular region or the reduced regions to be decoded. Video decoding method.

[0118] (Appendix 9) Cause a computer to perform a process of dividing the tensor to be encoded into a plurality of rectangular regions, for each rectangular region, perform a process of generating a plurality of reduced regions from the rectangular region, for each rectangular region, perform a process of entropy-encoding the tensor of the rectangular region and the tensors of the plurality of reduced regions using a probability distribution to generate a bitstream, perform a process of multiplexing the bitstreams, and when performing the entropy encoding, cause to use a probability distribution estimated from tensors of reduced regions having a size smaller than the rectangular region or the reduced regions to be encoded. Video encoding program therefor.

[0119] (Appendix 10) Cause a computer to perform a process of demultiplexing a multiplexed bitstream in which bitstreams corresponding to respective rectangular regions obtained by dividing an object tensor to be encoded are multiplexed, for each rectangular region, perform entropy decoding using a probability distribution on the rectangular region and a plurality of reduced regions generated from the rectangular region, and output a tensor of the rectangular region, perform a process of synthesizing tensors of rectangular regions obtained by the entropy decoding, and when performing the entropy decoding, use a probability distribution estimated from a tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be decoded video decoding program therefor.

[0120] (Appendix 11) A dividing unit that divides an object tensor to be encoded into a plurality of rectangular regions, a reducing unit that generates a plurality of reduced regions from the rectangular region for each rectangular region, an entropy encoding unit that performs entropy encoding on the tensor of the rectangular region and the tensors of the plurality of reduced regions using a probability distribution for each rectangular region to generate a bitstream, a multiplexing unit that multiplexes the bitstream, wherein the entropy encoding unit uses a probability distribution estimated from a tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be encoded, and uses, as the probability distribution regarding the smallest reduced region, the probability distribution regarding the smallest reduced region in the already encoded rectangular region video encoding device.

[0121] (Appendix 12) Include a probability distribution estimation unit that estimates a probability distribution of the tensor of the reduced region for each rectangular region The video encoding device of Appendix 11.

[0122] (Appendix 13) The entropy encoding means uses a probability distribution estimated from a tensor of a reduced region having a size closest to the size of the rectangular region or the reduced region to be encoded. The video encoding apparatus according to Appendix 12.

[0123] (Appendix 14) Demultiplexing means for demultiplexing a multiplexed bitstream in which bitstreams corresponding to respective rectangular regions obtained by dividing an encoding target tensor are multiplexed, Entropy decoding means for performing entropy decoding using a probability distribution for each rectangular region on the rectangular region and a plurality of reduced regions generated from the rectangular region, and outputting a tensor of the rectangular region, Comprising synthesis means for synthesizing tensors of rectangular regions obtained by the entropy decoding, The entropy decoding means, uses a probability distribution estimated from a tensor of a reduced region smaller than the rectangular region or the reduced region to be decoded, As the probability distribution regarding the smallest reduced region, the probability distribution regarding the smallest reduced region in the already encoded rectangular region acquired by the demultiplexing means is used. Video decoding apparatus.

[0124] (Appendix 15) Each rectangular region is provided with probability distribution estimation means for estimating the probability distribution of the tensor of the reduced region. The video decoding apparatus according to Appendix 14.

[0125] (Appendix 16) The probability distribution estimated from the tensor of the reduced region having the size closest to the size of the rectangular region or the reduced region to be encoded is used. The video decoding apparatus according to Appendix 15.

[0126] (Appendix 17) Divide the encoding target tensor into a plurality of rectangular regions, For each rectangular region, generate a plurality of reduced regions from the rectangular region, For each rectangular region, entropy-encode the tensor of the rectangular region and the tensors of the plurality of reduced regions using a probability distribution to generate a bit stream. Multiplex the bit stream. When performing the entropy encoding, Use a probability distribution estimated from the tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be encoded. As the probability distribution for the smallest reduced region, use the probability distribution for the smallest reduced region in the already encoded rectangular region. Video encoding method.

[0127] (Appendix 18) Demultiplex the multiplexed bit stream in which the bit streams corresponding to each of the rectangular regions obtained by dividing the tensor to be encoded are multiplexed. For each rectangular region, perform entropy decoding on the rectangular region and a plurality of reduced regions generated from the rectangular region using a probability distribution, and output the tensor of the rectangular region. Synthesize the tensors of the rectangular regions obtained by the entropy decoding. When performing the entropy decoding, Use a probability distribution estimated from the tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be decoded. As the probability distribution for the smallest reduced region, use the probability distribution for the smallest reduced region in the already encoded rectangular region obtained by demultiplexing. Video decoding method.

[0128] (Appendix 19) Cause a computer to perform a process of dividing the tensor to be encoded into a plurality of rectangular regions, perform a process of generating a plurality of reduced regions from the rectangular region for each rectangular region, perform a process of entropy-encoding the tensor of the rectangular region and the tensors of the plurality of reduced regions using a probability distribution to generate a bit stream for each rectangular region, perform a process of multiplexing the bit stream. When performing the entropy encoding, use the probability distribution estimated from the tensor of the reduced region having a size smaller than the rectangular region or the reduced region to be encoded, and use the probability distribution regarding the smallest reduced region in the already encoded rectangular region as the probability distribution regarding the smallest reduced region. Video encoding program for

[0129] (Appendix 20) Cause a computer to perform a process of demultiplexing the multiplexed bitstream in which the bitstreams corresponding to each of the rectangular regions obtained by dividing the tensor to be encoded are multiplexed, and for each rectangular region, perform entropy decoding using a probability distribution for the rectangular region and a plurality of reduced regions generated from the rectangular region, and output the tensor of the rectangular region; perform a process of synthesizing the tensors of the rectangular regions obtained by the entropy decoding, When performing the entropy decoding, use the probability distribution estimated from the tensor of the reduced region having a size smaller than the rectangular region or the reduced region to be decoded, and use the probability distribution regarding the smallest reduced region in the already encoded rectangular region obtained by demultiplexing as the probability distribution regarding the smallest reduced region. Video decoding program for.

Explanation of Signs

[0130] 11, 12, 13, 14 Downsampler 21, 22, 23, 24 Probability model estimator 30, 31, 32, 33, 34 Entropy encoder 40 Bitstream generator 50 Video encoding device 51 Division means 52 Reduction means 53 Entropy encoding means 54 Multiplexing means 60 Video decoding device 61 Demultiplexing means 62 Entropy decoding means 63 Synthesis means 100 Video encoding device 101 Region division section 102 Downsampling section 103 Probability model estimator 104 Entropy encoder 105 Multiplexer 200 Video decoding device 201 Tensor reconstruction section 203 Probability model estimator 204 Entropy decoder 205 Demultiplexer 701 Processor 702 Program memory 703, 704 Storage media

Claims

1. Division means for dividing the tensor to be encoded into a plurality of rectangular regions, Reduction means for generating a plurality of reduced regions from the rectangular region for each rectangular region, Entropy encoding means for entropy encoding the tensor of the rectangular region and the tensors of the plurality of reduced regions for each rectangular region using a probability distribution to generate a bit stream, Multiplexing means for multiplexing the bit stream, The entropy encoding means, Uses a probability distribution estimated from a tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be encoded, As the probability distribution regarding the smallest reduced region, uses the probability distribution regarding the smallest reduced region in the already encoded rectangular region, Video encoding device.

2. Comprises probability distribution estimation means for estimating the probability distribution of the tensor of the reduced region for each rectangular region, The video encoding device according to claim 1.

3. The entropy encoding means uses a probability distribution estimated from a tensor of a reduced region having a size closest to the size of the rectangular region or the reduced region to be encoded, The video encoding device according to claim 2.

4. Demultiplexing means for demultiplexing the multiplexed bit stream in which the bit streams corresponding to each of the rectangular regions obtained by dividing the tensor to be encoded are multiplexed, Entropy decoding means for performing entropy decoding on the rectangular region and the plurality of reduced regions generated from the rectangular region for each rectangular region using a probability distribution and outputting the tensor of the rectangular region, Comprises combining means for combining the tensors of the rectangular regions obtained by the entropy decoding, The entropy decoding means, Uses a probability distribution estimated from a tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be decoded, As the probability distribution regarding the smallest reduced region, uses the probability distribution regarding the smallest reduced region in the already encoded rectangular region acquired by the demultiplexing means, Video decoding device.

5. Comprises probability distribution estimation means for estimating the probability distribution of the tensor of the reduced region for each rectangular region, The video decoding device according to claim 4.

6. Uses a probability distribution estimated from a tensor of a reduced region having a size closest to the size of the rectangular region or the reduced region to be decoded, The video decoding device according to claim 5.

7. Divides the tensor to be encoded into a plurality of rectangular regions, For each rectangular region, generate a plurality of reduced regions from the rectangular region, For each rectangular region, entropy-encode the tensor of the rectangular region and the tensors of the plurality of reduced regions using a probability distribution to generate a bitstream, Multiplex the bitstreams, When performing the entropy encoding, Use a probability distribution estimated from a tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be encoded, Use, as the probability distribution for the smallest reduced region, the probability distribution for the smallest reduced region in the already encoded rectangular region, Video encoding method.

8. Demultiplex a multiplexed bitstream in which bitstreams corresponding to each of the rectangular regions obtained by dividing the tensor to be encoded are multiplexed, For each rectangular region, perform entropy decoding on the rectangular region and a plurality of reduced regions generated from the rectangular region using a probability distribution to output the tensor of the rectangular region, Synthesize the tensors of the rectangular regions obtained by the entropy decoding, When performing the entropy decoding, Use a probability distribution estimated from a tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be decoded, Use, as the probability distribution for the smallest reduced region, the probability distribution for the smallest reduced region in the already encoded rectangular region obtained by demultiplexing, Video decoding method.

9. On a computer, A process of dividing the tensor to be encoded into a plurality of rectangular regions, For each rectangular region, a process of generating a plurality of reduced regions from the rectangular region, For each rectangular region, a process of entropy-encoding the tensor of the rectangular region and the tensors of the plurality of reduced regions using a probability distribution to generate a bitstream, Execute a process of multiplexing the bitstreams, When performing the entropy encoding, use a probability distribution estimated from a tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be encoded, and use, as the probability distribution for the smallest reduced region, the probability distribution for the smallest reduced region in the already encoded rectangular region, Video encoding program for this purpose.

10. On a computer, A process of demultiplexing a multiplexed bitstream in which bitstreams corresponding to each of the rectangular regions obtained by dividing the tensor to be encoded are multiplexed, For each rectangular area, perform entropy decoding using a probability distribution on the rectangular area and a plurality of reduced areas generated from the rectangular area, and output a tensor of the rectangular area; Execute a process of synthesizing tensors of rectangular areas obtained by the entropy decoding; When performing the entropy decoding, use a probability distribution estimated from a tensor of a reduced area having a size smaller than the rectangular area or the reduced area to be decoded, and use, as the probability distribution regarding the smallest reduced area, the probability distribution regarding the smallest reduced area in the already encoded rectangular area obtained by demultiplexing; A video decoding program for this purpose.

Images