Video encoding device, video decoding device, video encoding method, and video decoding method

Abstract

To prevent a reduction in memory access efficiency.SOLUTION: A video encoding device 10 comprises: division means 51 that divides a tensor to be encoded into a plurality of rectangular areas; reduction means 52 that, for every rectangular area, generates a plurality of reduction areas from the rectangular area; entropy encoding means 53 that, for every rectangular area, subjects a tensor of the rectangular area and the tensors of the plurality of reduction areas to entropy encoding by using probability distributions, and generates a bit stream; and multiplexing means 54 that multiplexes the bit stream. The entropy encoding means 53 uses the probability distribution estimated from the tensor of the rectangular area to be encoded or the reduction area having a smaller size than the reduction area.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] The present disclosure relates to a video encoding apparatus, a video decoding apparatus, a video encoding method, and a video decoding method that utilize a neural network.

Background Art

[0002] A new video encoding technique that combines an auto-encoder, which is one type of neural network, quantization, and entropy encoding is described in Non-Patent Document 1.

[0003] An auto-encoder compresses input data into a low-dimensional feature tensor so as to include only important features. Then, the auto-encoder generates reconstructed data by reconstructing the low-dimensional feature tensor to the original dimension. The process of dropping down to the low-dimensional feature tensor (the first half) is called encoding. The process of generating the reconstructed data (the second half) is called decoding.

[0004] The learning of the auto-encoder is advanced so as to minimize the reconstruction error (the difference between the input data and the reconstructed data). The auto-encoder is designed to impose constraints on the structure of the encoding or to add a regularization term to the loss function of the network so as to obtain meaningful feature amounts.

[0005] Non-Patent Document 2 describes a method of inputting a predetermined tensor and entropy encoding the input tensor using a probability model (that is, a probability distribution of prediction). In this method, first, the input tensor is downsampled to obtain a tensor having a scale smaller than the scale of the input tensor. A probability distribution is estimated from the obtained tensor, and the tensor is entropy encoded. Then, the input tensor is entropy encoded using the estimated probability distribution. Such a method is called MSPSM (Multi-Scale Progressive Statistical Model) entropy encoding. MSPSM entropy encoding is simply denoted as MSPSM.

[0006] Hereinafter, as a predetermined tensor, a feature tensor obtained by applying quantization to the tensor output by the encoder in the autoencoder will be taken as an example.

Prior Art Documents

Non-Patent Documents

[0007]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0008] FIG. 1 is a block diagram showing an implementation example of the MSPSM. The configuration shown in FIG. 1 is taken as the MSPSM system.

[0009] The MSPSM system downsamples the input feature tensor from the encoder. The MSPSM system further downsamples the feature tensor obtained by the downsampling. The MSPSM system repeats the downsampling of the feature tensor obtained by the downsampling. The feature tensor obtained by the i-th downsampling is referred to as the feature tensor of scale i. i is referred to as the number of scales.

[0010] The feature tensor of scale i is y imay be expressed as. Let \(p\) be the probability distribution estimated from the feature tensor at scale \(i\). i be expressed as, and let \(z\) be the context information when estimating \(p\). i i be expressed as.

[0011] FIG. 1 shows a configuration example of the MSPSM system for \(i = 4\). The MSPSM system illustrated in FIG. 1 includes downsamplers 11 to 14 corresponding to each of scales \(i\) (\(i=1\) to 4), probability models (probability model estimators 21 to 24) that estimate a probability distribution using the outputs of the downsamplers 11 to 14, and entropy encoders 30 to 34 that entropy-encode a feature tensor using the probability distribution estimated by the probability model estimators 21 to 24.

[0012] The downsamplers 11 to 14 downsample the input feature tensor at a predetermined interval. For example, the downsamplers 11 to 14 downsample the input feature tensor by a factor of 1 / 2. That is, the number of samples is reduced to 1 / 2 both vertically and horizontally. Specifically, for example, when the downsamplers 11 to 14 perform downsampling by a factor of 1 / 2, they extract the lower-right element of a \(2\times2\) element (feature quantity).

[0013] The probability model estimators 21 to 23 estimate the probability model \(p\) from the feature tensor at scale \((i + 1)\) and the context information \(z\). i+2 i+1 For example, the probability model \(p\). i+1 is obtained by inputting the feature tensor and the context information into a convolutional neural network. Note that the probability model estimator 24 estimates the probability model \(p\) from the context information with a predetermined initial value set and the feature tensor at scale \((i + 1)\). Here, the predetermined initial value is, for example, a zero value. 4

[0014] The entropy encoders 31 to 33 entropy-encode the feature tensor at scale \(i\) using the probability model \(p\) estimated from the feature tensor at scale \((i + 1)\). i+1 ​​​Use it for entropy encoding. Note that the entropy encoder 30 uses the input feature tensor from the encoder as the probability model p 1 to perform entropy encoding. Also, the entropy encoder 34 performs entropy encoding on the feature tensor at scale 4 using a predefined probability distribution.

[0015] The bitstream generator 40 uses the outputs of the entropy encoders 30 to 34 as the bitstream.

[0016] Figure 2 is an explanatory diagram showing an example of the downsampling state. In Figure 2, as an example, the state of a 32×32 element feature tensor being reduced is illustrated. The small rectangles in Figure 2 correspond to the elements. The markings inside the small rectangles are applied to make it easier to grasp the correspondence of the elements between scales.

[0017] The y illustrated in Figure 2 1 is obtained by downsampling y 0 (input feature tensor). y 2 is obtained by downsampling y 1 is obtained by downsampling y 3 is obtained by downsampling y 2 is obtained by downsampling y 4 is obtained by downsampling y 3 is obtained by downsampling y

[0018] As shown in Figure 3, each element of the feature tensor y i at scale i is entropy encoded using the probability model estimated from the feature tensor at scale (i + 1). That is, in the MSPSM, when the entropy encoders 31 to 33 encode the elements of the feature tensor at a certain scale, they use the probability distribution estimated from the feature tensor with a smaller size (for example, one size smaller) than that scale. Note that the entropy encoder 34 uses the feature tensor y 4For the (minimum feature tensor), entropy encoding is performed using a predetermined probability distribution. The predetermined probability distribution may be referred to as a fixed probability distribution. The fixed probability distribution is shared between the video encoding device and the video decoding device. The entropy encoder 30 uses the probability model estimated from the feature tensor y 1 to entropy-encode each element of the input feature tensor y 0 .

[0019] For each element of the feature tensor y i at scale i, since it is entropy-encoded using the probability model estimated from the feature tensor at scale (i + 1), in the MSPSM system, the entropy encoder 33 performs entropy encoding after the processing of the probability model estimator 24 is completed. The entropy encoder 32 performs entropy encoding after the processing of the probability model estimator 23 is completed. The entropy encoder 31 performs entropy encoding after the processing of the probability model estimator 22 is completed. The entropy encoder 30 performs entropy encoding after the processing of the probability model estimator 21 is completed.

[0020] Note that the entropy encoders 31 to 34 do not perform entropy encoding on the elements that have been entropy-encoded at a previous scale (a scale with a larger i value for scale i).

[0021] The entropy encoders 30 to 34 may operate on the values of each element of the feature tensor input to the entropy encoder before the entropy encoding process. For example, there may be rounding to integer values. In such a case, the probability model estimators 21 to 24 apply the process for aligning with the values to be entropy-decoded to the feature tensor at scale (i + 1), and then execute the estimation process using the applied feature tensor.

[0022] As described above, for the feature tensor y iEach element is entropy-encoded using a probability model estimated from the feature tensor at scale (i + 1). Therefore, when entropy-encoding the elements of the feature tensor y at scale i i when entropy-encoding the elements of i , the probability model used reflects not only the previous elements (the upper and left elements at each scale in Fig. 2) in raster scan order with respect to the element to be encoded at scale (i + 1), but also the subsequent elements (the right and lower elements at each scale in Fig. 2) in raster scan order with respect to the element to be encoded.

[0023] Hereinafter, when estimating the probability model, considering the subsequent elements in raster scan order may be expressed as look-ahead of elements, or look-ahead processing of elements.

[0024] Note that the probability model is updated as appropriate. For example, the probability model is updated each time entropy-encoding of an element is performed.

[0025] Since the probability model that reflects the subsequent elements in raster scan order is used, the probability distribution used during entropy-encoding becomes closer to the probability distribution of the entire screen. As a result, even when the probability distribution varies depending on the region on the screen, the encoding performance is improved. From a different perspective on look-ahead, when entropy-encoding each element of the feature tensor y at scale i i a probability model based on the elements discretely arranged within the screen is used, so it can be said that the probability distribution used during entropy-encoding becomes closer to the probability distribution of the entire screen.

[0026] Generally, when video encoding is performed using a neural network, the feature tensor is temporarily stored in the memory within the video encoding device.

[0027] The look-ahead of elements contributes to the improvement of encoding performance and the reduction of the number of updates of the probability model. However, when obtaining the feature tensor of scale (i + 1) from the feature tensor of scale i, discontinuous areas of memory are accessed. Then, the memory access efficiency decreases. That is, the overall memory access time becomes longer.

[0028] An object of the present invention is to provide a video encoding apparatus, a video decoding apparatus, a video encoding method, and a video decoding method that can suppress a decrease in memory access efficiency.

Means for Solving the Problems

[0029] A video encoding apparatus according to the present disclosure includes a dividing unit that divides an encoding target tensor into a plurality of rectangular regions, a reducing unit that generates a plurality of reduced regions from the rectangular region for each rectangular region, and for each rectangular region, entropy encoding the tensor of the rectangular region and the tensors of the plurality of reduced regions using a probability distribution to generate a bit stream, and a multiplexing unit that multiplexes the bit stream. The entropy encoding unit uses a probability distribution estimated from a tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be encoded.

[0030] A video decoding apparatus according to the present disclosure includes a demultiplexing unit that demultiplexes a multiplexed bit stream in which bit streams corresponding to each of the rectangular regions obtained by dividing the encoding target tensor are multiplexed, an entropy decoding unit that performs entropy decoding on the rectangular region and a plurality of reduced regions generated from the rectangular region for each rectangular region using a probability distribution and outputs a tensor of the rectangular region, and a synthesizing unit that synthesizes the tensors of the rectangular regions obtained by entropy decoding. The entropy decoding unit uses a probability distribution estimated from a tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be decoded.

[0031] The video encoding method according to the present disclosure divides an encoding target tensor into a plurality of rectangular regions, generates a plurality of reduced regions from each rectangular region for each rectangular region, and for each rectangular region, entropy-encodes the tensor of the rectangular region and the tensors of the plurality of reduced regions using a probability distribution to generate a bitstream, multiplexes the bitstreams, and uses a probability distribution estimated from the tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be encoded when performing entropy encoding.

[0032] The video decoding method according to the present disclosure demultiplexes a multiplexed bitstream in which bitstreams corresponding to each of the rectangular regions obtained by dividing the encoding target tensor are multiplexed, performs entropy decoding on each rectangular region for the rectangular region and a plurality of reduced regions generated from the rectangular region using a probability distribution to output the tensor of the rectangular region, synthesizes the tensors of the rectangular regions obtained by entropy decoding, and uses a probability distribution estimated from the tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be decoded when performing entropy decoding.

[0033] The video encoding program according to the present disclosure causes a computer to execute a process of dividing an encoding target tensor into a plurality of rectangular regions, a process of generating a plurality of reduced regions from each rectangular region for each rectangular region, a process of entropy-encoding the tensor of the rectangular region and the tensors of the plurality of reduced regions using a probability distribution to generate a bitstream, and a process of multiplexing the bitstreams, and causes the computer to use a probability distribution estimated from the tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be encoded when performing entropy encoding.

[0034] The video decoding program based on the present disclosure causes a computer to perform a process of demultiplexing a multiplexed bitstream in which bitstreams corresponding to each of rectangular regions obtained by dividing an encoding target tensor are multiplexed, and for each rectangular region, perform entropy decoding using a probability distribution on the rectangular region and a plurality of reduced regions generated from the rectangular region to output a tensor of the rectangular region, and a process of synthesizing tensors of the rectangular regions obtained by entropy decoding. When performing entropy decoding, a probability distribution estimated from a tensor of a reduced region having a size smaller than the rectangular region or reduced region to be decoded is used.

Advantages of the Invention

[0035] According to the present invention, a decrease in memory access efficiency can be suppressed.

Brief Description of the Drawings

[0036]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Embodiment for Carrying Out the Invention

[0037] Hereinafter, embodiments will be described with reference to the drawings. Also in the embodiments described below, as an example of a tensor, a feature tensor output by an encoder in an autoencoder will be used.

[0038] FIG. 4 is an explanatory diagram for explaining region division processing executed by the image encoding device of the first embodiment.

[0039] In the first embodiment, the image encoding device divides the input feature tensor into a plurality of rectangular regions. In FIG. 4(A), an example in which the feature tensor is divided into two rectangular regions is shown. In FIG. 4(B), an example in which it is divided into 16 rectangular regions is shown. However, the example shown in FIG. 4 is just an example, and the shape of the rectangular regions obtained by the division is arbitrary.

[0040] For each of the plurality of rectangular regions obtained by the division, the image encoding device performs entropy encoding based on MSPSM to generate each encoded sequence (hereinafter referred to as a bitstream). The image encoding device multiplexes the generated bitstreams to generate a multiplexed bitstream.

[0041] When performing entropy encoding on a rectangular region, the image encoding device uses the probability distribution based on the elements within that rectangular region and does not use the probability distribution based on the elements within other rectangular regions.

[0042] The image decoding device demultiplexes the multiplexed bitstream to obtain the bitstreams corresponding to each of the plurality of rectangular regions. The image decoding device performs entropy decoding on each bitstream to reproduce each rectangular region (each element of the rectangular region). The image decoding device combines each rectangular region to restore the feature tensor.

[0043] The video encoding device according to the first embodiment can improve the memory access efficiency by applying the MSPSM in units of rectangular regions. For example, if the video encoding device transfers only the elements of the rectangular region from the main memory to the cache memory (cache) in advance and performs a prefetch process on the elements in the cache, the memory access efficiency is improved.

[0044] Compare the case of encoding the elements of the entire screen with the case of encoding the elements in the feature tensor of the rectangular region. As an example, compare the case of encoding the elements in a feature tensor of 32×32 elements (see y in FIG. 2) 0 reference) with the case of encoding the elements in the feature tensor of a rectangular region of 32×16 elements (see FIG. 4(A)). Let the number of elements in the horizontal direction of the feature tensor be W and the number of elements in the vertical direction be H.

[0045] When encoding the elements in a feature tensor of 32×32 elements (see FIG. 2), the number of elements stored in the memory when executing the MSPSM is W×H+(1 / 4)×(W×H)+(1 / 16)×(W×H)+(1 / 64)×(W×H)+(1 / 256)×(W×H)=(341 / 256)×(W×H)=1364.

[0046] When encoding the elements in the feature tensor of a rectangular region of 32×16 elements, the amount of memory required when executing the MSPSM is (1 / 2)×(W×H)+(1 / 8)×(W×H)+(1 / 32)×(W×H)+(1 / 128)×(W×H)+(1 / 512)×(W×H)=(341 / 512)×(W×H)=682.

[0047] When encoding the elements in the feature tensor of the rectangular region, the number of elements stored in memory is less than the number of elements stored in memory when encoding the elements of the entire screen. Therefore, the required memory amount also decreases, and the memory access efficiency is improved. In other words, compared with the case of encoding the entire screen as a unit, the memory amount required for one unit of encoding (encoding for one rectangular region) is less, so the memory access efficiency is improved.

[0048] FIG. 5 is an explanatory diagram for explaining the probability distribution reference executed by the video encoding device of the second embodiment.

[0049] In the second embodiment, the video encoding device divides the input feature tensor into a plurality of rectangular regions in the same manner as the video encoding device in the first embodiment. FIG. 5 shows an example in which the input feature tensor is divided into 16 rectangular regions.

[0050] The video encoding device does not use a fixed probability distribution when entropy-encoding the smallest feature tensor y 4 but uses a probability distribution estimated from a feature tensor of a rectangular region that has already been encoded, for example, y 4 in the already-encoded rectangular region. That is, when entropy-encoding the feature tensor y 4 the video encoding device can use a probability distribution in which the elements in the already-encoded rectangular region are considered. As a result, it can be expected that the encoding performance will be further improved.

[0051] Note that the video encoding device determines the size of the rectangular region so as to satisfy the constraint of the minimum size of the MSPSM. FIG. 6 is an explanatory diagram for explaining the constraint of the size of the rectangular region.

[0052] In order to maintain the effect of look-ahead, the rectangular region has a minimum scale (in the example shown in FIG. 2 (scale number = 4), y 4It is preferable that a constraint that one or more of the elements included in (in FIGS. 2 and 6, small rectangles with horizontal lines) are included is satisfied. The size of the rectangular area under such a constraint is 16×16 elements. For example, if the size of the rectangular area is 16×15 elements, there are no elements of y 4 in the rectangular area, so it is not preferable as the size of the rectangular area.

[0053] In other words, the size of the rectangular area is determined so that the number of elements at the minimum scale (for example, scale 4) does not become zero. For example, assuming that the scale number i = 4 and the downsamplers 11 to 14 shown in FIG. 1 are configured to extract the lower right element of 2×2 elements (feature amounts) and downsample it by 1 / 2, for the feature tensor y of 16×16 elements 0 with respect to the feature tensor y 1 ~y 4 each contains one or more elements. However, for the feature tensor y of 16×15 elements 0 with respect to the feature tensor y 4 the number of elements is zero. Since there are no elements at the minimum scale, a probability distribution cannot be estimated from the feature tensor y 4 . As a result, it is difficult to implement MSPSM.

[0054] Embodiment 1. FIG. 7 is a block diagram showing a configuration example of a video encoding device and a video decoding device.

[0055] The video encoding device 100 shown in FIG. 7 includes a region division unit 101, a downsampling unit 102, a probability model estimator 103, an entropy encoder 104, and a multiplexer 105.

[0056] The video decoding device 200 shown in FIG. 7 includes a tensor reconstruction unit 201, a probability model estimator 203, an entropy decoder 204, and a demultiplexer 205.

[0057] Note that the arrows in Fig. 7 clearly indicate the direction of the signal (data) flow, but do not rule out bidirectionality. The same applies to other block diagrams.

[0058] [Description of Video Encoding Device] In the video encoding device 100, the region division unit 101 divides the tensor to be encoded (encoding target tensor) into a plurality of rectangular regions (see Fig. 4). Hereinafter, as an example, the encoding target tensor is assumed to be a feature tensor obtained by applying quantization to the tensor output from an encoder (not shown) in an autoencoder.

[0059] The downsampling unit 102 generates one or more reduced regions for each of the plurality of rectangular regions. For example, the downsampling unit 102 downsamples the feature tensor of each rectangular region input from the region division unit 101 at a predetermined interval. The downsampling unit 102 can be configured by, for example, the downsamplers 11 to 14 shown in Fig. 1.

[0060] The probability model estimator 103 can be configured by, for example, the probability model estimators 21 to 24 shown in Fig. 1. Similar to the probability model estimators 21 to 24, the probability model estimator 103 estimates a probability distribution.

[0061] The entropy encoder 104 can be configured by, for example, the entropy encoders 30 to 34 shown in Fig. 1. Using the probability distribution estimated by the probability model estimator 103, the entropy encoder 104 performs entropy encoding on the feature tensor of each of the plurality of rectangular regions. Specifically, the entropy encoder 104 performs entropy encoding on each element of the feature tensor to generate each bit stream.

[0062] The multiplexer 105 multiplexes the generated bit streams to generate a multiplexed bit stream.

[0063] Next, an example of the specific operation of the video encoding device 100 will be described with reference to the flowchart of FIG. 8.

[0064] The region division unit 101 divides the input feature tensor into a plurality of rectangular regions (step S101).

[0065] The downsampling unit 102 downsamples the feature tensor at a predetermined interval for each of the plurality of rectangular regions input from the region division unit 101 (step S102). By downsampling, feature tensors of a plurality of scales as exemplified in FIG. 2 are obtained from each of the plurality of rectangular regions.

[0066] The probability model estimator 103 estimates a probability distribution from the feature tensors of each scale for each of the plurality of rectangular regions (step S103).

[0067] The entropy encoder 104 entropy-encodes the feature tensors of each of the plurality of rectangular regions to generate a bitstream (step S104). Note that the entropy encoder 104 generates a bitstream for each scale for each of the rectangular regions.

[0068] For one rectangular region, when the entropy encoder 104 encodes the elements of the feature tensor of a certain scale, it uses the probability distribution estimated from the feature tensor of a size smaller (for example, one level smaller) than the scale.

[0069] Specifically, taking the example shown in FIG. 2, the probability model estimator 103 calculates the probability distribution from the feature tensor y 4 generated by the downsampling unit 102. The entropy encoder 104 entropy-encodes the feature tensor y 3 using the probability distribution. The probability model estimator 103 calculates the probability distribution from the feature tensor y 3 generated by the downsampling unit 102. The entropy encoder 104 entropy-encodes the feature tensor y2 Perform entropy encoding. The probability model estimator 103 calculates a probability distribution from the feature tensor y generated by the downsampling unit 102. 2 The entropy encoder 104 uses the probability distribution to perform entropy encoding on the feature tensor y. 1 Perform entropy encoding. The probability model estimator 103 calculates a probability distribution from the feature tensor y generated by the downsampling unit 102. 1 The entropy encoder 104 uses the probability distribution to perform entropy encoding on the feature tensor y. 0 Perform entropy encoding. The entropy encoder 104 outputs the encoded data of the feature tensor of each scale to the multiplexer 105.

[0070] Note that the entropy encoder 104 performs entropy encoding on the feature tensor y using a fixed probability distribution. 4 Perform entropy encoding.

[0071] In the flowchart shown in FIG. 8, in the processes of steps S102 and S103, feature tensors of multiple scales are generated and probability distributions are estimated for multiple rectangular regions. However, steps S102 and S103 may be configured to perform the process on one rectangular region. In that case, the processes of steps S102 and S103 are executed for the number of rectangular regions generated in the process of step S101.

[0072] The entropy encoder 104 multiplexes the multiple bitstreams generated in the process of step S104 to generate a multiplexed bitstream (step S105).

[0073] [Description of video decoder] In the video decoder 200, the demultiplexer 205 demultiplexes the multiplexing of the multiplexed bitstream from the video encoder 100 to obtain bitstreams corresponding to each of the multiple rectangular regions.

[0074] For each bit stream, that is, for each rectangular region, the entropy decoder 204 entropy-decodes the elements of the feature tensor at each scale with reference to the probability distribution from the probability model estimator 203.

[0075] The probability model estimator 203 operates in the same manner as the probability model estimator 103. That is, the probability model estimator 203 can be composed of, for example, the probability model estimators 21 to 24 shown in FIG. 1. The probability model estimator 203 estimates a probability distribution in the same way as the probability model estimators 21 to 24.

[0076] The tensor reconstruction unit 201 synthesizes the feature tensors of each rectangular region output by the entropy decoder 204 to restore the feature tensor input to the video encoding device 100. That is, the tensor reconstruction unit 201 connects the feature tensors of each rectangular region to obtain the feature tensor input to the video encoding device 100.

[0077] Next, an example of the specific operation of the video decoding device 200 will be described with reference to the flowchart of FIG. 9.

[0078] The demultiplexer 205 demultiplexes the multiplexing of the multiplexed bit stream from the video encoding device 100 to obtain a plurality of bit streams (step S201). Each bit stream corresponds to each of the plurality of rectangular regions.

[0079] The probability model estimator 203 estimates a probability distribution from the feature tensors at each scale for each rectangular region (step S202).

[0080] The entropy decoder 204 entropy-decodes the plurality of bit streams to obtain the feature tensors of each rectangular region (step S203).

[0081] For a single rectangular region, when the entropy decoder 204 decodes the elements of the feature tensor at a certain scale, it uses the probability distribution estimated from the feature tensor with a size smaller than that scale (for example, one level smaller).

[0082] Specifically, taking the example shown in FIG. 2, for a single rectangular region, when the probability model estimator 203 outputs the feature tensor y 4 the probability model estimator 203 calculates the probability distribution from the feature tensor y 4 . The entropy decoder 204 uses the probability distribution to entropy-decode the feature tensor y 3 and outputs the feature tensor y 3 . When the entropy decoder 204 outputs the feature tensor y 3 the probability model estimator 203 calculates the probability distribution from the feature tensor y 3 . The entropy decoder 204 uses the probability distribution to entropy-decode the feature tensor y 2 and outputs the feature tensor y 2 . When the entropy decoder 204 outputs the feature tensor y 2 the probability model estimator 203 calculates the probability distribution from the feature tensor y 2 . The entropy decoder 204 uses the probability distribution to entropy-decode the feature tensor y 1 and outputs the feature tensor y 1 . When the entropy decoder 204 outputs the feature tensor y 1 the probability model estimator 203 calculates the probability distribution from the feature tensor y 1 . The entropy decoder 204 uses the probability distribution to entropy-decode the feature tensor y 0 and outputs the feature tensor y 0 .

[0083] The feature tensor y 0 is supplied to the tensor reconstruction unit 201.

[0084] Note that the entropy decoder 204 uses a fixed probability distribution to entropy-decode the feature tensor y4 Perform entropy decoding.

[0085] When the above processes are executed for all rectangular regions, the tensor reconstruction unit 201 can obtain the feature tensors of all rectangular regions. The tensor reconstruction unit 201 synthesizes the feature tensors of all rectangular regions to restore the feature tensor input to the video encoding device 100 (step S204).

[0086] Note that the entropy decoder 204 may supply the feature vector of one rectangular region to the tensor reconstruction unit 201 each time it is obtained, or may supply all of them to the tensor reconstruction unit 201 collectively when the feature vectors of all rectangular regions are obtained.

[0087] Embodiment 2. The video encoding device according to the second embodiment is configured in the same manner as the video encoding device according to the first embodiment shown in FIG. 7. However, the operations of the probability model estimator 103, the entropy encoder 104, and the multiplexer 105 in the video encoding device 100 are different from those operations in the first embodiment.

[0088] [Description of Video Encoding Device] In the first embodiment, the entropy encoder 104 uses a fixed probability distribution to perform entropy encoding on the feature tensor y 4 In the second embodiment, when one or more rectangular regions have been encoded, the entropy encoder 104 does not use a fixed probability distribution.

[0089] In the second embodiment, the probability model estimator 103 estimates a probability distribution from y 4 in the already encoded rectangular regions (see FIG. 5). That is, the probability model estimator 103 calculates a probability distribution from y 4 in the already encoded rectangular regions. The probability distribution is supplied to the entropy encoder 104. The entropy encoder 104 uses the probability distribution to perform entropy encoding on the feature tensor y 4

[0090] The multiplexer 105 generates a multiplexed bitstream as in the first embodiment. However, in the second embodiment, the multiplexer 105 also performs a process of multiplexing the probability distribution from y 4 in the already encoded rectangular region into the multiplexed bitstream as, for example, auxiliary information.

[0091] Other processes of the video encoding apparatus 100 are the same as those in the first embodiment.

[0092] In the second embodiment, in addition to the effects of the first embodiment, an effect that the encoding performance is further improved can be obtained.

[0093] [Explanation of video decoding apparatus] The video decoding apparatus of the second embodiment is configured in the same manner as the decoding apparatus of the first embodiment shown in FIG. 7. However, the operations of the demultiplexer 205 and the entropy decoder 204 in the video decoding apparatus 200 are different from those operations in the first embodiment.

[0094] In the second embodiment, the demultiplexer 205 also demultiplexes the information on the probability distribution estimated from y 4 in the already encoded rectangular region from the multiplexed bitstream.

[0095] In the first embodiment, the entropy decoder 204 entropy-decodes the feature tensor y 4 using a fixed probability distribution. However, in the second embodiment, the entropy decoder 204 entropy-decodes the feature tensor y 4 using the probability distribution estimated from y 4 in the already encoded rectangular region, which is the information demultiplexed by the demultiplexer 205.

[0096] Other processes of the video decoding apparatus 200 are the same as those in the first embodiment.

[0097] Although each of the above-described embodiments can be configured by hardware, it can also be realized by a computer program.

[0098] The information processing system shown in FIG. 10 includes a processor 701 such as a CPU (Central Processing Unit), a program memory 702, a storage medium 703 for storing temporary data, and a storage medium 704 for storing a bit stream. The storage medium 703 and the storage medium 704 may be separate storage media or may be storage areas consisting of the same storage medium. As the storage medium, a magnetic storage device such as a hard disk can be used. As the storage medium, an SRAM (Static Random Access Memory) or a flash ROM (Read Only Memory) may be used.

[0099] Examples of the temporary data include the feature tensors used by the video encoding device 100 and the video decoding device 200.

[0100] In the information processing system, a program (video encoding program or video decoding program) for realizing the functions of each block shown in each of the above embodiments is stored in the program memory 702.

[0101] The processor 701 realizes each function in the above embodiment by executing processing according to the program (software element: code) stored in the program memory 702.

[0102] That is, the processor 701 realizes the functions of the video encoding device 100 and the video decoding device 200 shown in each embodiment by executing processing according to the program stored in the program memory 702.

[0103] For example, the functions of the video encoding apparatus 100 are realized when the processor 701 executes processing according to a video encoding program for realizing the functions of each block in the video encoding apparatus 100 shown in FIG. 7.

[0104] Also, for example, the functions of the video decoding apparatus 200 are realized when the processor 701 executes processing according to a video decoding program for realizing the functions of each block in the video decoding apparatus 200 shown in FIG. 7.

[0105] Note that at least the program memory 702 is a non-transitory computer readable medium. The non-transitory computer readable medium includes various types of tangible storage media. Specific examples of the non-transitory computer readable medium include magnetic recording media (e.g., hard disks), magneto-optical recording media (e.g., magneto-optical disks), CD-ROM (Compact Disc-Read Only Memory), CD-R (Compact Disc-Recordable), CD-R / W (Compact Disc-ReWritable), and semiconductor memories (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM). However, the program may be stored in various types of transitory computer readable media.

[0106] The transitory computer readable medium is supplied with the program via, for example, a wired communication path or a wireless communication path, that is, via an electrical signal, an optical signal, or an electromagnetic wave.

[0107] FIG. 11 is a block diagram showing the main part of a video encoding device. The video encoding device 50 (realized by the video encoding device 100 in the embodiment) shown in FIG. 11 includes a dividing means 51 (realized by the region dividing unit 101 in the embodiment) that divides an encoding target tensor into a plurality of rectangular regions, and for each rectangular region, a reducing means 52 (realized by the downsampling unit 102 in the embodiment) that generates a plurality of reduced regions (for example, regions by the feature tensor of scale i) from the rectangular region, and for each rectangular region, an entropy encoding means 53 (realized by the entropy encoder 104 in the embodiment) that entropy-encodes the tensor of the rectangular region and the tensors of the plurality of reduced regions using a probability distribution to generate a bitstream, and a multiplexing means 54 (realized by the multiplexer 105 in the embodiment) that multiplexes the bitstream output by the entropy encoding means 53. The entropy encoding means 53 uses a probability distribution estimated from the tensor of a reduced region (for example, a region by the feature tensor of scale (i + 1) for a region by the feature tensor of scale i) having a size smaller than the rectangular region or the reduced region to be encoded.

[0108] FIG. 12 is a block diagram showing the main part of a video decoding device. The video decoding device 60 (realized by the video decoding device 200 in the embodiment) shown in FIG. 12 includes a demultiplexing means 61 (realized by the demultiplexer 205 in the embodiment) that demultiplexes a multiplexed bitstream in which bitstreams corresponding to each of the rectangular regions obtained by dividing the encoding target tensor are multiplexed, and for each rectangular region, an entropy decoding means 62 (realized by the entropy decoder 204 in the embodiment) that performs entropy decoding on the rectangular region and a plurality of reduced regions generated from the rectangular region using a probability distribution and outputs the tensor of the rectangular region, and a synthesizing means 63 (realized by the tensor reconstruction unit 201 in the embodiment) that synthesizes the tensors of the rectangular regions obtained by the entropy decoding. The entropy decoding means 62 uses a probability distribution estimated from the tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be decoded.

[0109] Some or all of the above embodiments may be described as follows in the appended claims, but are not limited thereto.

[0110] (Appendix 1) A dividing means for dividing a tensor to be encoded into a plurality of rectangular regions, For each rectangular region, a reducing means for generating a plurality of reduced regions from the rectangular region, For each rectangular region, an entropy encoding means for entropy encoding the tensor of the rectangular region and the tensors of the plurality of reduced regions using a probability distribution to generate a bit stream, A multiplexing means for multiplexing the bit stream, The entropy encoding means uses a probability distribution estimated from a tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be encoded. Video encoding device.

[0111] (Appendix 2) For each rectangular region, a probability distribution estimation means for estimating the probability distribution of the tensor of the reduced region (in the embodiment, realized by the probability model estimator 103). The video encoding device of Appendix 1.

[0112] (Appendix 3) The entropy encoding means uses a probability distribution estimated from a tensor of a reduced region having a size closest to the size of the rectangular region or the reduced region to be encoded (for example, a region by a feature tensor of scale (i + 1) with respect to a region by a feature tensor of scale i). The video encoding device of Appendix 2.

[0113] (Appendix 4) The entropy encoding means entropy encodes the tensor of the smallest reduced region using a predetermined fixed probability distribution. The video encoding device of Appendix 3.

[0114] (Appendix 5) A demultiplexing means for demultiplexing the multiplexed bit stream in which the bit streams corresponding to each of the rectangular regions obtained by dividing the tensor to be encoded are multiplexed, For each rectangular region, entropy decoding is performed using a probability distribution for the rectangular region and a plurality of reduced regions generated from the rectangular region, and entropy decoding means for outputting a tensor of the rectangular region comprising synthesis means for synthesizing tensors of rectangular regions obtained by the entropy decoding The entropy decoding means uses a probability distribution estimated from a tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be decoded Video decoding apparatus

[0115] (Appendix 6) Probability distribution estimation means for estimating the probability distribution of the tensor of the reduced region for each rectangular region The video decoding apparatus according to Appendix 5

[0116] (Appendix 7) Divide the tensor to be encoded into a plurality of rectangular regions For each rectangular region, generate a plurality of reduced regions from the rectangular region For each rectangular region, entropy encode the tensor of the rectangular region and the tensors of the plurality of reduced regions using a probability distribution to generate a bit stream Multiplex the bit streams When performing the entropy encoding, use a probability distribution estimated from a tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be encoded Video encoding method

[0117] (Appendix 8) Demultiplex a multiplexed bit stream in which bit streams corresponding to each of the rectangular regions obtained by dividing the tensor to be encoded are multiplexed For each rectangular region, perform entropy decoding using a probability distribution for the rectangular region and a plurality of reduced regions generated from the rectangular region, and output a tensor of the rectangular region Synthesize tensors of rectangular regions obtained by the entropy decoding When performing the entropy decoding, use the probability distribution estimated from the tensor of the reduced region having a size smaller than the rectangular region or the reduced region to be decoded. Video decoding method.

[0118] (Appendix 9) Cause a computer to perform a process of dividing the tensor to be encoded into a plurality of rectangular regions, for each rectangular region, perform a process of generating a plurality of reduced regions from the rectangular region, for each rectangular region, perform entropy encoding on the tensor of the rectangular region and the tensors of the plurality of reduced regions using a probability distribution to generate a bit stream, perform a process of multiplexing the bit streams, When performing the entropy encoding, use the probability distribution estimated from the tensor of the reduced region having a size smaller than the rectangular region or the reduced region to be encoded. Video encoding program therefor.

[0119] (Appendix 10) Cause a computer to perform a process of demultiplexing the multiplexed bit stream in which the bit streams corresponding to each of the rectangular regions obtained by dividing the tensor to be encoded are multiplexed, for each rectangular region, perform entropy decoding on the rectangular region and the plurality of reduced regions generated from the rectangular region using a probability distribution to output the tensor of the rectangular region, perform a process of synthesizing the tensors of the rectangular regions obtained by the entropy decoding, When performing the entropy decoding, use the probability distribution estimated from the tensor of the reduced region having a size smaller than the rectangular region or the reduced region to be decoded. Video decoding program therefor.

[0120] (Appendix 11) Dividing means for dividing the tensor to be encoded into a plurality of rectangular regions, Reducing means for generating a plurality of reduced regions from the rectangular region for each rectangular region, For each rectangular region, entropy encoding means for performing entropy encoding on the tensor of the rectangular region and the tensors of the plurality of reduced regions using a probability distribution to generate a bit stream; multiplexing means for multiplexing the bit stream; The entropy encoding means uses a probability distribution estimated from a tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be encoded, and uses, as the probability distribution regarding the smallest reduced region, the probability distribution regarding the smallest reduced region in the already encoded rectangular region. A video encoding apparatus.

[0121] (Appendix 12) Comprising probability distribution estimation means for estimating the probability distribution of the tensor of the reduced region for each rectangular region The video encoding apparatus of Appendix 11.

[0122] (Appendix 13) The entropy encoding means uses a probability distribution estimated from a tensor of a reduced region having a size closest to the size of the rectangular region or the reduced region to be encoded. The video encoding apparatus of Appendix 12.

[0123] (Appendix 14) Demultiplexing means for demultiplexing the multiplexed bit stream in which the bit streams corresponding to each of the rectangular regions obtained by dividing the tensor to be encoded are multiplexed, For each rectangular region, entropy decoding means for performing entropy decoding on the rectangular region and the plurality of reduced regions generated from the rectangular region using a probability distribution and outputting the tensor of the rectangular region, and combining means for combining the tensors of the rectangular regions obtained by the entropy decoding; The entropy decoding means uses a probability distribution estimated from a tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be decoded, As the probability distribution regarding the smallest reduction area, use the probability distribution regarding the smallest reduction area in the already encoded rectangular area acquired by the demultiplexing means. Video decoding apparatus.

[0124] (Appendix 15) For each rectangular area, include probability distribution estimation means for estimating the probability distribution of the tensor of the reduction area. Video decoding apparatus of Appendix 14.

[0125] (Appendix 16) Use the probability distribution estimated from the tensor of the reduction area with the size closest to the size of the rectangular area or the reduction area to be encoded. Video decoding apparatus of Appendix 15.

[0126] (Appendix 17) Divide the tensor to be encoded into a plurality of rectangular areas, For each rectangular area, generate a plurality of reduction areas from the rectangular area, For each rectangular area, entropy encode the tensor of the rectangular area and the tensors of the plurality of reduction areas using a probability distribution to generate a bit stream, Multiplex the bit stream, When performing the entropy encoding, Use the probability distribution estimated from the tensor of the reduction area smaller than the rectangular area or the reduction area to be encoded, As the probability distribution regarding the smallest reduction area, use the probability distribution regarding the smallest reduction area in the already encoded rectangular area. Video encoding method.

[0127] (Appendix 18) Demultiplex the multiplexed bit stream in which the bit streams corresponding to each of the rectangular areas obtained by dividing the tensor to be encoded are multiplexed, For each rectangular area, perform entropy decoding on the rectangular area and the plurality of reduction areas generated from the rectangular area using a probability distribution to output the tensor of the rectangular area, Synthesize the tensors of the rectangular areas obtained by the entropy decoding, When performing the entropy decoding, using the probability distribution estimated from the tensor of the reduced region having a size smaller than the rectangular region or the reduced region to be decoded, as the probability distribution regarding the smallest reduced region, using the probability distribution regarding the smallest reduced region in the already encoded rectangular region obtained by demultiplexing Video decoding method.

[0128] (Appendix 19) causing a computer to perform a process of dividing the tensor to be encoded into a plurality of rectangular regions, for each rectangular region, perform a process of generating a plurality of reduced regions from the rectangular region, for each rectangular region, perform entropy encoding on the tensor of the rectangular region and the tensors of the plurality of reduced regions using a probability distribution to generate a bit stream, perform a process of multiplexing the bit stream, and when performing the entropy encoding, cause to use the probability distribution estimated from the tensor of the reduced region having a size smaller than the rectangular region or the reduced region to be encoded, and cause to use the probability distribution regarding the smallest reduced region in the already encoded rectangular region as the probability distribution regarding the smallest reduced region Video encoding program therefor

[0129] (Appendix 20) causing a computer to perform a process of demultiplexing the multiplexed bit stream in which the bit streams corresponding to each of the rectangular regions obtained by dividing the tensor to be encoded are multiplexed, for each rectangular region, perform entropy decoding on the rectangular region and the plurality of reduced regions generated from the rectangular region using a probability distribution to output the tensor of the rectangular region, perform a process of synthesizing the tensors of the rectangular regions obtained by the entropy decoding, and When performing the entropy decoding, use the probability distribution estimated from the tensor of the reduced region having a size smaller than the rectangular region or the reduced region to be decoded, and use, as the probability distribution regarding the smallest reduced region, the probability distribution regarding the smallest reduced region in the already encoded rectangular region obtained by demultiplexing. Video decoding program therefor.

Explanation of Signs

[0130] 11, 12, 13, 14 Downsampler 21, 22, 23, 24 Probability model estimator 30, 31, 32, 33, 34 Entropy encoder 40 Bit stream generator 50 Video encoding device 51 Division means 52 Reduction means 53 Entropy encoding means 54 Multiplexing means 60 Video decoding device 61 Demultiplexing means 62 Entropy decoding means 63 Composition means 100 Video encoding device 101 Region division section 102 Downsampling section 103 Probability model estimator 104 Entropy encoder 105 Multiplexer 200 Video decoding device 201 Tensor reconstruction section 203 Probability model estimator 204 Entropy decoder 205 Demultiplexer 701 Processor 702 Program memory 703, 704 Storage media

Claims

1. Partitioning means for partitioning a tensor to be coded into a plurality of rectangular regions; Reduction means for generating a plurality of reduced regions from the rectangular region for each rectangular region; Entropy coding means for entropy coding the tensor of the rectangular region and the tensors of the plurality of reduced regions for each rectangular region using a probability distribution to generate a bit stream; Multiplexing means for multiplexing the bit stream, wherein the entropy coding means uses a probability distribution estimated from a tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be coded; A video coding apparatus.

2. Probability distribution estimation means for estimating the probability distribution of the tensor of the reduced region for each rectangular region is provided. The video coding apparatus according to claim 1.

3. The entropy coding means uses a probability distribution estimated from a tensor of a reduced region having a size closest to the size of the rectangular region or the reduced region to be coded. The video coding apparatus according to claim 2.

4. The entropy coding means entropy-codes the tensor of the smallest reduced region using a predetermined fixed probability distribution. The video coding apparatus according to claim 3.

5. Demultiplexing means for demultiplexing a multiplexed bit stream in which bit streams corresponding to each of the rectangular regions obtained by dividing the tensor to be coded are multiplexed; Entropy decoding means for performing entropy decoding on the rectangular region and a plurality of reduced regions generated from the rectangular region for each rectangular region using a probability distribution to output the tensor of the rectangular region; Combining means for combining the tensors of the rectangular regions obtained by the entropy decoding; wherein the entropy decoding means uses a probability distribution estimated from a tensor of a reduced region having a size smaller than the rectangular region or the reduced region to be decoded; A video decoding apparatus.

6. Probability distribution estimation means for estimating the probability distribution of the tensor of the reduced region for each rectangular region is provided. The video decoding apparatus according to claim 5.

7. Partitioning a tensor to be coded into a plurality of rectangular regions; Generating a plurality of reduced regions from the rectangular region for each rectangular region; Entropy coding the tensor of the rectangular region and the tensors of the plurality of reduced regions for each rectangular region using a probability distribution to generate a bit stream; Multiplexing the bit stream; When performing the entropy encoding, use the probability distribution estimated from the tensor of the reduced region having a size smaller than the rectangular region or the reduced region to be encoded. Video encoding method.

8. Demultiplex the multiplexed bitstream in which the bitstreams corresponding to each of the rectangular regions obtained by dividing the tensor to be encoded are multiplexed. For each rectangular region, perform entropy decoding using the probability distribution for the rectangular region and a plurality of reduced regions generated from the rectangular region, and output the tensor of the rectangular region. Combine the tensors of the rectangular regions obtained by the entropy decoding. When performing the entropy decoding, use the probability distribution estimated from the tensor of the reduced region having a size smaller than the rectangular region or the reduced region to be decoded. Video decoding method.

9. On a computer, Perform a process of dividing the tensor to be encoded into a plurality of rectangular regions, For each rectangular region, perform a process of generating a plurality of reduced regions from the rectangular region, For each rectangular region, perform entropy encoding on the tensor of the rectangular region and the tensors of the plurality of reduced regions using the probability distribution to generate a bitstream, Perform a process of multiplexing the bitstream, When performing the entropy encoding, use the probability distribution estimated from the tensor of the reduced region having a size smaller than the rectangular region or the reduced region to be encoded. Video encoding program therefor.

10. On a computer, Perform a process of demultiplexing the multiplexed bitstream in which the bitstreams corresponding to each of the rectangular regions obtained by dividing the tensor to be encoded are multiplexed, For each rectangular region, perform entropy decoding using the probability distribution for the rectangular region and a plurality of reduced regions generated from the rectangular region, and output the tensor of the rectangular region, Perform a process of combining the tensors of the rectangular regions obtained by the entropy decoding, When performing the entropy decoding, use the probability distribution estimated from the tensor of the reduced region having a size smaller than the rectangular region or the reduced region to be decoded. Video decoding program therefor.

Images