Encoding Device, Decoding Device, and Storage Medium

Abstract

The present invention provides an encoding device, a decoding device, and a storage medium. The encoding device uses a block segmentation pattern set obtained by combining one or more block segmentation patterns to segment into a plurality of blocks; the block segmentation pattern set is composed of a first and a second block segmentation pattern. The first block segmentation pattern defines a segmentation direction and a segmentation number for segmenting a first block, and the second block segmentation pattern defines a segmentation direction and a segmentation number for segmenting a second block obtained by segmenting one of the blocks obtained by segmenting the first block; the parameter for identifying the second block segmentation pattern includes a first flag indicating in which of the horizontal and vertical directions the block is segmented; when the segmentation number of the first block segmentation pattern is 3 and the second block is the central block, when the second block segmentation pattern indicated by the first flag is the same as the segmentation direction of the first block segmentation pattern, the second block segmentation pattern only includes a block segmentation pattern with a segmentation number of 3, and when the segmentation directions are different, the second block segmentation pattern includes a block segmentation pattern with a segmentation number of 2.

Description

[0001] This application is a divisional of the patent application for an invention titled "Coding Device, Decoding Device, Coding Method, Decoding Method, and Picture Compression Program" with an application date of May 9, 2019, an application number of 201980033751.4. Technical Field

[0002] The present disclosure relates to methods and apparatuses for encoding and decoding images and pictures using block partitioning. Background Art

[0003] In conventional image and video coding methods, an image is generally divided into blocks, and coding and decoding processes are performed at the block level. In recent video standard development, in addition to typical sizes of 8×8 or 16×16, coding and decoding processes can be performed with various block sizes. For image coding and decoding processes, a range of sizes from 4×4 to 256×256 can be used.

[0004] Prior Art Documents

[0005] Non-Patent Documents

[0006] Non-Patent Document 1: H.265 (ISO / IEC 23008-2 HEVC (High Efficiency Video Coding)) Summary of the Invention

[0007] Problems to be Solved by the Invention

[0008] In order to represent a range of sizes from 4×4 to 256×256, block partitioning information such as block partitioning patterns (e.g., quadtrees, binary trees, and ternary trees) and partitioning flags (e.g., split flag) is determined and signaled for use in blocks. The overhead of this signaling increases as the depth of partitioning increases. Moreover, the increased overhead reduces the video compression efficiency.

[0009] Accordingly, an encoding device according to one aspect of the present disclosure provides an encoding device and the like capable of improving compression efficiency in the encoding of block partitioning information.

[0010] Means for Solving the Problems

[0011] An encoding apparatus according to an aspect of the present disclosure encodes an image, and includes: a processor; and a memory; the processor performs the following processes: using a set of block segmentation patterns obtained by combining one or more block segmentation patterns, the image read from the memory is segmented into a plurality of blocks, the block segmentation patterns define segmentation types; and encoding the plurality of blocks; the set of block segmentation patterns includes a first block segmentation pattern and a second block segmentation pattern, the first block segmentation pattern defines a segmentation direction and a segmentation number for segmenting a first block, the second block segmentation pattern defines a segmentation direction and a segmentation number for segmenting a second block, which is one of the blocks obtained by segmenting the first block; a parameter for identifying the second block segmentation pattern includes a first flag indicating in which direction, among the horizontal direction and the vertical direction, the block is segmented; when the segmentation number of the first block segmentation pattern is 3, the second block is the central block among the blocks obtained by segmenting the first block, and the segmentation direction of the second block segmentation pattern indicated by the first flag is the same as the segmentation direction of the first block segmentation pattern, the second block segmentation pattern includes only a block segmentation pattern with a segmentation number of 3; when the segmentation number of the first block segmentation pattern is 3, the second block is the central block among the blocks obtained by segmenting the first block, and the segmentation direction of the second block segmentation pattern indicated by the first flag is different from the segmentation direction of the first block segmentation pattern, the second block segmentation pattern includes a block segmentation pattern with a segmentation number of 2.

[0012] A decoding device according to an aspect of the present disclosure decodes an encoded signal, and includes: a processor; and a memory; the processor performs the following processes: using a block segmentation pattern set obtained by combining one or more block segmentation patterns, segment the encoded signal read from the memory into a plurality of blocks, the block segmentation pattern defines a segmentation type; and decode the plurality of blocks; the block segmentation pattern set is composed of a first block segmentation pattern and a second block segmentation pattern, the first block segmentation pattern defines a segmentation direction and a segmentation number for segmenting a first block, the second block segmentation pattern defines a segmentation direction and a segmentation number for segmenting a second block, which is one of the blocks obtained after segmentation of the first block; a parameter for identifying the second block segmentation pattern includes a first flag indicating in which direction, among the horizontal direction and the vertical direction, the block is segmented; when the segmentation number of the first block segmentation pattern is 3, the second block is the central block among the blocks obtained after segmentation of the first block, and the segmentation direction of the second block segmentation pattern indicated by the first flag is the same as the segmentation direction of the first block segmentation pattern, the second block segmentation pattern only includes a block segmentation pattern with a segmentation number of 3; when the segmentation number of the first block segmentation pattern is 3, the second block is the central block among the blocks obtained after segmentation of the first block, and the segmentation direction of the second block segmentation pattern indicated by the first flag is different from the segmentation direction of the first block segmentation pattern, the second block segmentation pattern includes a block segmentation pattern with a segmentation number of 2.

[0013] A non - transitory storage medium according to an aspect of the present disclosure is a non - transitory storage medium that stores a bitstream and is readable by a computer, wherein the bitstream includes information for causing a computer that receives the bitstream to perform a decoding process; the information is for causing the computer to perform the following processes: using a set of block partitioning patterns obtained by combining one or more block partitioning patterns, dividing an encoded signal read from a memory into a plurality of blocks, where the block partitioning patterns define a division type; and decoding the plurality of blocks; the set of block partitioning patterns consists of a first block partitioning pattern and a second block partitioning pattern, the first block partitioning pattern defines a division direction and a division quantity for dividing a first block, and the second block partitioning pattern defines a division direction and a division quantity for dividing a second block, which is one of the blocks obtained after dividing the first block; a parameter for identifying the second block partitioning pattern includes a first flag indicating in which direction, among the horizontal direction and the vertical direction, the block is divided; when the division quantity of the first block partitioning pattern is 3, the second block is the central block among the blocks obtained after dividing the first block, and the division direction of the second block partitioning pattern indicated by the first flag is the same as the division direction of the first block partitioning pattern, the second block partitioning pattern only includes a block partitioning pattern with a division quantity of 3; when the division quantity of the first block partitioning pattern is 3, the second block is the central block among the blocks obtained after dividing the first block, and the division direction of the second block partitioning pattern indicated by the first flag is different from the division direction of the first block partitioning pattern, the second block partitioning pattern includes a block partitioning pattern with a division quantity of 2.

[0014] In addition, these inclusive or specific forms can also be implemented by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer - readable CD - ROM, or can be implemented by any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0015] Advantages of the Invention

[0016] According to the present invention, it is possible to improve the compression efficiency in the encoding of block partitioning information. Brief Description of the Drawings

[0017] Figure 1 It is a block diagram showing the functional structure of an encoding device according to Embodiment 1.

[0018] Figure 2 It is a diagram showing an example of block partitioning in Embodiment 1.

[0019] Figure 3 It is a table showing transform basis functions corresponding to respective transform types.

[0020] Figure 4A It is a diagram showing an example of the shape of the filter used in the ALF.

[0021] Figure 4B It is a diagram showing another example of the shape of the filter used in the ALF.

[0022] Figure 4C It is a diagram showing another example of the shape of the filter used in the ALF.

[0023] Figure 5A It is a diagram showing 67 intra prediction modes of intra prediction.

[0024] Figure 5B It is a flowchart for explaining the outline of the predicted image correction process based on the OBMC process.

[0025] Figure 5C It is a conceptual diagram for explaining the outline of the predicted image correction process based on the OBMC process.

[0026] Figure 5D It is a diagram showing an example of FRUC.

[0027] Figure 6 It is a diagram for explaining the pattern matching (bidirectional matching) between two blocks along the motion trajectory.

[0028] Figure 7 It is a diagram for explaining the pattern matching (template matching) between a template in the current picture and a block in the reference picture.

[0029] Figure 8 It is a diagram for explaining a model assuming uniform linear motion.

[0030] Figure 9A It is a diagram for explaining the derivation of the motion vector in sub-block units based on the motion vectors of multiple adjacent blocks.

[0031] Figure 9B It is a diagram for explaining the outline of the motion vector derivation process based on the merge mode.

[0032] Figure 9C It is a conceptual diagram for explaining the outline of the DMVR process.

[0033] Figure 9D It is a diagram for explaining the outline of the predicted image generation method using the luminance correction process based on the LIC process.

[0034] Figure 10 It is a block diagram showing the functional structure of the decoding device according to Embodiment 1.

[0035] Figure 11It is a flowchart showing the video encoding process related to Embodiment 2.

[0036] Figure 12 It is a flowchart showing the video decoding process related to Embodiment 2.

[0037] Figure 13 It is a flowchart showing the video encoding process related to Embodiment 3.

[0038] Figure 14 It is a flowchart showing the video decoding process related to Embodiment 3.

[0039] Figure 15 It is a block diagram showing the structure of the video / image encoding apparatus related to Embodiment 2 or 3.

[0040] Figure 16 It is a block diagram showing the structure of the video / image decoding apparatus related to Embodiment 2 or 3.

[0041] Figure 17 It is a diagram showing an example of the possible positions of the first parameter in the compressed video stream in Embodiment 2 or 3.

[0042] Figure 18 It is a diagram showing an example of the possible positions of the second parameter in the compressed video stream in Embodiment 2 or 3.

[0043] Figure 19 It is a diagram showing an example of the second parameter following the first parameter in Embodiment 2 or 3.

[0044] Figure 20 It is a diagram showing an example of not selecting the second block mode for the division of a 2N×N pixel block as shown in step (2c) in Embodiment 2.

[0045] Figure 21 It is a diagram showing an example of not selecting the second block mode for the division of an N×2N pixel block as shown in step (2c) in Embodiment 2.

[0046] Figure 22 It is a diagram showing an example of not selecting the second block mode for the division of an N×N pixel block as shown in step (2c) in Embodiment 2.

[0047] Figure 23 It is a diagram showing an example of not selecting the second block mode for the division of an N×N pixel block as shown in step (2C) in Embodiment 2.

[0048] Figure 24This is a diagram showing an example of dividing a 2N×N pixel block using the block mode selected when not selecting the second block mode as shown in step (3) in Embodiment 2.

[0049] Figure 25 This is a diagram showing an example of dividing an N×2N pixel block using the block mode selected when not selecting the second block mode as shown in step (3) in Embodiment 2.

[0050] Figure 26 This is a diagram showing an example of dividing an N×N pixel block using the block mode selected when not selecting the second block mode as shown in step (3) in Embodiment 2.

[0051] Figure 27 This is a diagram showing an example of dividing an N×N pixel block using the block mode selected when not selecting the second block mode as shown in step (3) in Embodiment 2.

[0052] Figure 28 This is a diagram showing an example of the block mode for dividing an N×N pixel block in Embodiment 2. Figure 28 (a) to (h) thereof are diagrams showing mutually different block modes.

[0053] Figure 29 This is a diagram showing an example of the block type and block direction for dividing an N×N pixel block in Embodiment 3. (1), (2), (3), and (4) are different block types, (1a), (2a), (3a), and (4a) are block modes with different block types in the vertical block direction, and (1b), (2b), (3b), and (4b) are block modes with different block types in the horizontal block direction.

[0054] Figure 30 This is a diagram showing the advantage of encoding the block type before the block direction compared to encoding the block direction before the block type in Embodiment 3.

[0055] Figure 31A This is a diagram showing an example of dividing a block into sub - blocks using a block mode set that uses fewer binary (bin) numbers in the encoding of the block mode.

[0056] Figure 31B This is a diagram showing an example of dividing a block into sub - blocks using a block mode set that uses fewer binary numbers in the encoding of the block mode.

[0057] Figure 32A This is a diagram showing an example of dividing a block into sub - blocks using the block mode set that first appears in a prescribed order of multiple block mode sets.

[0058] Figure 32B This is a diagram showing an example of dividing a block into sub - blocks by using the block pattern set that first appears in a specified order among multiple block pattern sets.

[0059] Figure 32C This is a diagram showing an example of dividing a block into sub - blocks by using the block pattern set that first appears in a specified order among multiple block pattern sets.

[0060] Figure 33 This is an overall structure diagram of a content supply system that implements a content distribution service.

[0061] Figure 34 This is a diagram showing an example of the coding structure in scalable coding.

[0062] Figure 35 This is a diagram showing an example of the coding structure in scalable coding.

[0063] Figure 36 This is a diagram showing an example of the display screen of a web page.

[0064] Figure 37 This is a diagram showing an example of the display screen of a web page.

[0065] Figure 38 This is a diagram showing an example of a smart phone.

[0066] Figure 39 This is a block diagram showing an example of the structure of a smart phone.

[0067] Figure 40 This is a diagram showing an example of the constraints of a block - splitting pattern that divides a rectangular block into 3 sub - blocks.

[0068] Figure 41 This is a diagram showing an example of the constraints of a block - splitting pattern that divides a block into 2 sub - blocks.

[0069] Figure 42 This is a diagram showing an example of the constraints of a block - splitting pattern that divides a square block into 3 sub - blocks.

[0070] Figure 43 This is a diagram showing an example of the constraints of a block - splitting pattern that divides a rectangular block into 2 sub - blocks.

[0071] Figure 44 This is a diagram showing an example of the constraints based on the splitting direction of a block - splitting pattern that divides a non - rectangular block into 2 sub - blocks.

[0072] Figure 45 This is a diagram showing an example of an effective splitting direction for dividing a non - rectangular block into 2 sub - blocks. Detailed implementation manners

[0073] Hereinafter, the implementation manners will be specifically described with reference to the drawings.

[0074] In addition, the implementation manners described below all represent inclusive or specific examples. The numerical values, shapes, materials, constituent elements, configurations and connection forms of the constituent elements, steps, order of steps, etc. shown in the following implementation manners are examples and do not limit the meaning of the claims. In addition, among the constituent elements of the following implementation manners, the constituent elements not described in the independent claims representing the most general concept are described as arbitrary constituent elements.

[0075] (Embodiment 1)

[0076] First, as an example of an encoding device and a decoding device that can apply the processes and / or structures described in each aspect of the present invention described below, an outline of Embodiment 1 will be described. However, Embodiment 1 is merely an example of an encoding device and a decoding device that can apply the processes and / or structures described in each aspect of the present invention, and the processes and / or structures described in each aspect of the present invention can also be implemented in encoding devices and decoding devices different from Embodiment 1.

[0077] When applying the processes and / or structures described in each aspect of the present invention to Embodiment 1, for example, any of the following may be performed.

[0078] (1) For the encoding device or decoding device of Embodiment 1, replace the constituent elements corresponding to the constituent elements described in each aspect of the present invention among the multiple constituent elements constituting the encoding device or decoding device with the constituent elements described in each aspect of the present invention;

[0079] (2) For the encoding device or decoding device of Embodiment 1, after arbitrarily changing the addition, replacement, deletion, etc. of the functions or processes implemented on a part of the multiple constituent elements constituting the encoding device or decoding device, replace the constituent elements corresponding to the constituent elements described in each aspect of the present invention with the constituent elements described in each aspect of the present invention;

[0080] (3) For the method implemented by the encoding device or decoding device of Embodiment 1, add processes, and / or arbitrarily change the replacement, deletion, etc. of a part of the multiple processes included in the method, and then replace the processes corresponding to the processes described in each aspect of the present invention with the processes described in each aspect of the present invention;

[0081] (4) A part of the constituent elements of the encoding device or decoding device constituting Embodiment 1 is combined with the constituent elements described in each aspect of the present invention, the constituent elements having a part of the functions possessed by the constituent elements described in each aspect of the present invention, or the constituent elements implementing a part of the processes implemented by the constituent elements described in each aspect of the present invention, and then implemented;

[0082] (5) A constituent element having a part of the functions possessed by a part of the constituent elements of the encoding device or decoding device constituting Embodiment 1, or a constituent element implementing a part of the processes implemented by a part of the constituent elements of the encoding device or decoding device constituting Embodiment 1, is combined with the constituent elements described in each aspect of the present invention, the constituent elements having a part of the functions possessed by the constituent elements described in each aspect of the present invention, or the constituent elements implementing a part of the processes implemented by the constituent elements described in each aspect of the present invention, and then implemented;

[0083] (6) For the method implemented by the encoding device or decoding device of Embodiment 1, the process corresponding to the processes described in each aspect of the present invention among the multiple processes included in the method is replaced with the processes described in each aspect of the present invention;

[0084] (7) A part of the processes included in the method implemented by the encoding device or decoding device of Embodiment 1 is combined with the processes described in each aspect of the present invention and then implemented.

[0085] In addition, the implementation manners of the processes and / or structures described in each aspect of the present invention are not limited to the above examples. For example, it can also be implemented in a device used for a purpose different from the moving image / image encoding device or moving image / image decoding device disclosed in Embodiment 1, or the processes and / or structures described in each aspect can be implemented independently. In addition, the processes and / or structures described in different aspects can be combined and implemented.

[0086] [Outline of Encoding Device]

[0087] First, the outline of the encoding device of Embodiment 1 will be described. Figure 1 FIG. is a block diagram showing the functional structure of the encoding device 100 of Embodiment 1. The encoding device 100 is a moving image / image encoding device that encodes moving images / images in units of blocks.

[0088] As Figure 1As shown in the figure, the encoding device 100 is a device that encodes an image in units of blocks, and includes a segmentation unit 102, a subtraction unit 104, a transformation unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse transformation unit 114, an addition unit 116, a block memory 118, a loop filter unit 120, a frame memory 122, an intra prediction unit 124, an inter prediction unit 126, and a prediction control unit 128.

[0089] The encoding device 100 is implemented by, for example, a general-purpose processor and a memory. In this case, when a software program stored in the memory is executed by the processor, the processor functions as the segmentation unit 102, the subtraction unit 104, the transformation unit 106, the quantization unit 108, the entropy encoding unit 110, the inverse quantization unit 112, the inverse transformation unit 114, the addition unit 116, the loop filter unit 120, the intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128. In addition, the encoding device 100 can also be implemented as one or more dedicated electronic circuits corresponding to the segmentation unit 102, the subtraction unit 104, the transformation unit 106, the quantization unit 108, the entropy encoding unit 110, the inverse quantization unit 112, the inverse transformation unit 114, the addition unit 116, the loop filter unit 120, the intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128.

[0090] Hereinafter, each component included in the encoding device 100 will be described.

[0091] [Segmentation Unit]

[0092] The segmentation unit 102 divides each picture included in the input moving image into a plurality of blocks and outputs each block to the subtraction unit 104. For example, the segmentation unit 102 first divides the picture into blocks of a fixed size (e.g., 128×128). Such blocks of a fixed size are sometimes referred to as coding tree units (CTUs). And the segmentation unit 102 divides each block of a fixed size into blocks of a variable size (e.g., 64×64 or less) based on recursive quadtree and / or binary tree block partitioning. Such blocks of a variable size are sometimes referred to as coding units (CUs), prediction units (PUs), or transform units (TUs). In addition, in the present embodiment, it is not necessary to distinguish between CUs, PUs, and TUs, and a part or all of the blocks in the picture can also be used as the processing units for CUs, PUs, and TUs.

[0093] Figure 2 is a diagram showing an example of block partitioning in Embodiment 1. In Figure 2 it, solid lines represent block boundaries based on quadtree block partitioning, and dashed lines represent block boundaries based on binary tree block partitioning.

[0094] Here, block 10 is a square block of 128×128 pixels (128×128 block). This 128×128 block 10 is first divided into 4 square 64×64 blocks (quad-tree block division).

[0095] The upper-left 64×64 block is further vertically divided into 2 rectangular 32×64 blocks, and the left 32×64 block is further vertically divided into 2 rectangular 16×64 blocks (binary-tree block division). As a result, the upper-left 64×64 block is divided into 2 16×64 blocks 11, 12 and a 32×64 block 13.

[0096] The upper-right 64×64 block is horizontally divided into 2 rectangular 64×32 blocks 14, 15 (binary-tree block division).

[0097] The lower-left 64×64 block is divided into 4 square 32×32 blocks (quad-tree block division). The upper-left block and the lower-right block among the 4 32×32 blocks are further divided. The upper-left 32×32 block is vertically divided into 2 rectangular 16×32 blocks, and the right 16×32 block is further horizontally divided into 2 16×16 blocks (binary-tree block division). The lower-right 32×32 block is horizontally divided into 2 32×16 blocks (binary-tree block division). As a result, the lower-left 64×64 block is divided into a 16×32 block 16, 2 16×16 blocks 17, 18, 2 32×32 blocks 19, 20, and 2 32×16 blocks 21, 22.

[0098] The lower-right 64×64 block 23 is not divided.

[0099] As described above, in Figure 2 , block 10 is divided into 13 variable-size blocks 11 to 23 based on recursive quad-tree and binary-tree block division. Such a division is sometimes called QTBT (quad-tree plus binary tree) division.

[0100] In addition, in Figure 2 , 1 block is divided into 4 or 2 blocks (quad-tree or binary-tree block division), but the division is not limited to this. For example, 1 block can also be divided into 3 blocks (ternary-tree division). A division including such a ternary-tree division is sometimes called MBT (multi type tree) division.

[0101] [Subtraction unit]

[0102] The subtraction unit 104 subtracts the predicted signal (predicted sample) from the original signal (original sample) in units of blocks divided by the division unit 102. That is, the subtraction unit 104 calculates the prediction error (also referred to as the residual) of the block to be encoded (hereinafter referred to as the current block). And the subtraction unit 104 outputs the calculated prediction error to the transformation unit 106.

[0103] The original signal is the input signal of the encoding device 100 and is a signal representing the images of each picture constituting the moving image (for example, a luma signal and two chroma signals). Hereinafter, there are cases where a signal representing an image is also referred to as a sample.

[0104] [Transformation unit]

[0105] The transformation unit 106 transforms the prediction error in the spatial domain into transform coefficients in the frequency domain and outputs the transform coefficients to the quantization unit 108. Specifically, the transformation unit 106, for example, performs a preset discrete cosine transform (DCT) or discrete sine transform (DST) on the prediction error in the spatial domain.

[0106] In addition, the transformation unit 106 can also adaptively select a transformation type from multiple transformation types, use a transform basis function corresponding to the selected transformation type, and transform the prediction error into transform coefficients. Such a transformation is called EMT (explicit multiple core transform, multi-core transform) or AMT (adaptive multiple transform, adaptive multi-transform) in some cases.

[0107] The multiple transformation types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII. Figure 3 is a table showing the transform basis functions corresponding to each transformation type. In Figure 3 where N represents the number of input pixels. The selection of the transformation type from these multiple transformation types can depend on, for example, the type of prediction (intra prediction and inter prediction), or can also depend on the intra prediction mode.

[0108] Information indicating whether to apply such EMT or AMT (for example, called the AMT flag) and information indicating the selected transformation type are signaled at the CU level. In addition, the signaling of this information does not need to be limited to the CU level and can also be other levels (for example, sequence level, picture level, slice level, tile level, or CTU level).

[0109] In addition, the transformation unit 106 can also perform a re - transformation on the transformation coefficients (transformation results). Such re - transformations include cases known as AST (adaptive secondary transform) or NSST (non - separable secondary transform). For example, the transformation unit 106 performs a re - transformation on each sub - block (e.g., 4×4 sub - block) included in the block of transformation coefficients corresponding to the intra - prediction error. Information indicating whether to apply NSST and information related to the transformation matrix used in NSST are signaled at the CU level. Additionally, the signaling of this information does not need to be limited to the CU level and can also be at other levels (e.g., sequence level, picture level, slice level, tile level, or CTU level).

[0110] Here, a Separable transformation refers to a method of performing multiple transformations by separating in each direction according to the number of dimensions of the input. A Non - Separable transformation refers to a method of treating two or more dimensions as one dimension and performing a transformation together when the input is multi - dimensional.

[0111] For example, as an example of a Non - Separable transformation, when the input is a 4×4 block, it can be regarded as a permutation with 16 elements, and a transformation process is performed on this permutation using a 16×16 transformation matrix.

[0112] In addition, similarly, a method of performing multiple Givens rotations on the permutation after regarding a 4×4 input block as a permutation with 16 elements (Hypercube Givens Transform) is also an example of a Non - Separable transformation.

[0113] [Quantization unit]

[0114] The quantization unit 108 quantizes the transformation coefficients output from the transformation unit 106. Specifically, the quantization unit 108 scans the transformation coefficients of the current block in a prescribed scan order and quantizes the transformation coefficients based on the quantization parameter (QP) corresponding to the scanned transformation coefficients. And the quantization unit 108 outputs the quantized transformation coefficients (hereinafter referred to as quantization coefficients) of the current block to the entropy coding unit 110 and the inverse quantization unit 112.

[0115] The prescribed order is the order for quantization / inverse quantization of transformation coefficients. For example, the prescribed scan order is defined by ascending frequency (from low frequency to high frequency) or descending frequency (from high frequency to low frequency).

[0116] A quantization parameter refers to a parameter that defines the quantization step (quantization width). For example, if the value of the quantization parameter increases, the quantization step also increases. That is, if the value of the quantization parameter increases, the quantization error increases.

[0117] [Entropy encoding section]

[0118] The entropy encoding section 110 generates an encoded signal (encoded bit stream) by performing variable-length encoding on the quantized coefficients that are the input from the quantization section 108. Specifically, the entropy encoding section 110 binarizes the quantized coefficients, for example, and performs arithmetic encoding on the binary signal.

[0119] [Inverse quantization section]

[0120] The inverse quantization section 112 performs inverse quantization on the quantized coefficients that are the input from the quantization section 108. Specifically, the inverse quantization section 112 performs inverse quantization on the quantized coefficients of the current block in a prescribed scan order. And the inverse quantization section 112 outputs the inverse-quantized transform coefficients of the current block to the inverse transform section 114.

[0121] [Inverse transform section]

[0122] The inverse transform section 114 restores the prediction error by performing an inverse transform on the transform coefficients that are the input from the inverse quantization section 112. Specifically, the inverse transform section 114 restores the prediction error of the current block by performing an inverse transform corresponding to the transform of the transform section 106 on the transform coefficients. And the inverse transform section 114 outputs the restored prediction error to the addition section 116.

[0123] In addition, since information is lost due to quantization in the restored prediction error, it does not match the prediction error calculated by the subtraction section 104. That is, the restored prediction error includes a quantization error.

[0124] [Addition section]

[0125] The addition section 116 reconstructs the current block by adding the prediction error that is the input from the inverse transform section 114 and the prediction sample that is the input from the prediction control section 128. And the addition section 116 outputs the reconstructed block to the block memory 118 and the loop filter section 120. The reconstructed block is sometimes referred to as a local decoded block.

[0126] [Block memory]

[0127] The block memory 118 is a storage section for storing blocks within the picture to be encoded (hereinafter referred to as the current picture) that are referred to in intra prediction. Specifically, the block memory 118 stores the reconstructed block output from the addition section 116.

[0128] [Loop filter section]

[0129] The loop filter unit 120 performs loop filtering on the block reconstructed by the adder unit 116 and outputs the filtered reconstructed block to the frame memory 122. Loop filtering refers to the filtering used within the coding loop (in-loop filtering), for example, including deblocking filter (DF), sample adaptive offset (SAO), and adaptive loop filter (ALF), etc.

[0130] In ALF, a least squares error filter for removing coding distortion is adopted. For example, for each 2×2 sub-block within the current block, one filter selected from multiple filters is adopted based on the direction and activity of the locality-based gradient.

[0131] Specifically, first, sub-blocks (e.g., 2×2 sub-blocks) are classified into multiple classes (e.g., 15 or 25 classes). The classification of sub-blocks is performed based on the direction and activity of the gradient. For example, using the direction value D of the gradient (e.g., 0 to 2 or 0 to 4) and the activity value A of the gradient (e.g., 0 to 4), the classification value C is calculated (e.g., C = 5D + A). And based on the classification value C, the sub-blocks are classified into multiple classes (e.g., 15 or 25 classes).

[0132] The direction value D of the gradient is derived, for example, by comparing the gradients in multiple directions (e.g., horizontal, vertical, and two diagonal directions). In addition, the activity value A of the gradient is derived, for example, by adding the gradients in multiple directions and quantifying the added result.

[0133] Based on the result of such classification, the filter for the sub-block is determined from among multiple filters.

[0134] As the shape of the filter used in ALF, for example, a circularly symmetric shape is used. Figures 4A to 4C It is a diagram showing multiple examples of the shape of the filter used in ALF. Figure 4A It represents a 5×5 diamond-shaped filter, Figure 4B It represents a 7×7 diamond-shaped filter, Figure 4C It represents a 9×9 diamond-shaped filter. The information representing the shape of the filter is signaled at the picture level. Additionally, the signaling of the information representing the shape of the filter does not need to be limited to the picture level and can also be other levels (e.g., sequence level, slice level, tile level, CTU level, or CU level).

[0135] The on / off of ALF is determined, for example, at the picture level or CU level. For example, for luminance, it is determined at the CU level whether to adopt ALF, and for chrominance difference, it is determined at the picture level whether to adopt ALF. The information representing the on / off of ALF is signaled at the picture level or CU level. Additionally, the signaling of the information representing the on / off of ALF does not need to be limited to the picture level or CU level and can also be other levels (e.g., sequence level, slice level, tile level, or CTU level).

[0136] Coefficient sets of multiple selectable filters (e.g., up to 15 or 25 filters) are signaled at the picture level. In addition, the signaling of the coefficient sets does not need to be limited to the picture level and can also be at other levels (e.g., sequence level, slice level, tile level, CTU level, CU level, or sub-block level).

[0137] [Frame memory]

[0138] Frame memory 122 is a storage unit for storing reference pictures used in inter-frame prediction and is sometimes referred to as a frame buffer. Specifically, frame memory 122 stores the reconstructed blocks filtered by loop filter 120.

[0139] [Intra-frame prediction unit]

[0140] Intra-frame prediction unit 124 performs intra-frame prediction (also called intra-picture prediction) of the current block by referring to the block in the current picture stored in block memory 118, thereby generating a prediction signal (intra-frame prediction signal). Specifically, intra-frame prediction unit 124 generates an intra-frame prediction signal by performing intra-frame prediction with reference to samples (e.g., luminance values, chrominance difference values) of blocks adjacent to the current block and outputs the intra-frame prediction signal to prediction control unit 128.

[0141] For example, intra-frame prediction unit 124 performs intra-frame prediction using one of a plurality of predefined intra-frame prediction modes. The plurality of intra-frame prediction modes include one or more non-directional prediction modes and a plurality of directional prediction modes.

[0142] One or more non-directional prediction modes include, for example, the Planar (plane) prediction mode and the DC prediction mode defined by the H.265 / HEVC (High-Efficiency Video Coding) standard (Non-Patent Document 1).

[0143] The plurality of directional prediction modes include, for example, 33-direction prediction modes defined by the H.265 / HEVC standard. In addition, the plurality of directional prediction modes may also include 32-direction prediction modes (a total of 65 directional prediction modes) in addition to the 33 directions. Figure 5A It is a diagram showing 67 intra-frame prediction modes (2 non-directional prediction modes and 65 directional prediction modes) in intra-frame prediction. The solid arrows represent 33 directions defined by the H.265 / HEVC standard, and the dashed arrows represent the additional 32 directions.

[0144] In addition, in the intra prediction of the chrominance blocks, the luminance blocks may also be referred to. That is, the chrominance components of the current block may also be predicted based on the luminance component of the current block. Such intra prediction is sometimes called CCLM (cross-component linear model) prediction. The intra prediction mode of the chrominance blocks that refer to the luminance blocks (e.g., called the CCLM mode) may also be added as one of the intra prediction modes of the chrominance blocks.

[0145] The intra prediction unit 124 may also correct the intra-predicted pixel value based on the gradients of the reference pixels in the horizontal / vertical directions. The intra prediction accompanied by such correction is sometimes called PDPC (position dependent intraprediction combination). Information indicating whether PDPC is used (e.g., called the PDPC flag) is signaled at the CU level, for example. In addition, the signaling of this information is not limited to the CU level and may also be at other levels (e.g., sequence level, picture level, slice level, tile level, or CTU level).

[0146] [Inter prediction unit]

[0147] The inter prediction unit 126 generates a prediction signal (inter prediction signal) by performing inter prediction (also called inter-picture prediction) of the current block with reference to a reference picture different from the current picture stored in the frame memory 122. The inter prediction is performed in units of the current block or sub-blocks (e.g., 4×4 blocks) within the current block. For example, the inter prediction unit 126 performs motion estimation within the reference picture for the current block or sub-block. Then, the inter prediction unit 126 performs motion compensation using the motion information (e.g., motion vector) obtained by the motion estimation to generate the inter prediction signal of the current block or sub-block. And the inter prediction unit 126 outputs the generated inter prediction signal to the prediction control unit 128.

[0148] The motion information used in the motion compensation is signaled. In the signaling of the motion vector, a motion vector predictor may also be used. That is, the difference between the motion vector and the predicted motion vector may also be signaled.

[0149] Alternatively, it may be that not only the motion information of the current block obtained by motion estimation is used, but also the motion information of adjacent blocks is used to generate an inter-frame prediction signal. Specifically, the prediction signal based on the motion information obtained by motion estimation may be weighted and added to the prediction signal based on the motion information of adjacent blocks, thereby generating an inter-frame prediction signal in units of sub-blocks within the current block. Such inter-frame prediction (motion compensation) is sometimes referred to as OBMC (overlapped block motion compensation).

[0150] In such an OBMC mode, information indicating the size of the sub-blocks used for OBMC (e.g., referred to as the OBMC block size) is signaled at the sequence level. In addition, information indicating whether the OBMC mode is adopted (e.g., referred to as the OBMC flag) is signaled at the CU level. Additionally, the levels at which these pieces of information are signaled do not need to be limited to the sequence level and the CU level, and may also be other levels (e.g., picture level, slice level, tile level, CTU level, or sub-block level).

[0151] A more specific description of the OBMC mode will be given. Figure 5B and Figure 5C are a flowchart and a conceptual diagram for explaining the outline of the predicted image correction process based on OBMC processing.

[0152] First, using the motion vector (MV) assigned to the block to be encoded, a predicted image (Pred) obtained by normal motion compensation is acquired.

[0153] Next, the motion vector (MV_L) of the encoded left adjacent block is adopted for the block to be encoded to obtain a predicted image (Pred_L), and the first correction of the predicted image is performed by weighted superposition of the above predicted image and Pred_L.

[0154] Similarly, the motion vector (MV_U) of the encoded upper adjacent block is adopted for the block to be encoded to obtain a predicted image (Pred_U), and the second correction of the predicted image is performed by weighted superposition of the predicted image after the above first correction and Pred_U, and this is used as the final predicted image.

[0155] In addition, the method of two-stage correction using the left adjacent block and the upper adjacent block is described here, but it may also be configured to perform more than two-stage corrections using the right adjacent block and the lower adjacent block.

[0156] In addition, the area for superposition may not be the entire pixel area of the block, but only a partial area near the block boundary.

[0157] In addition, the prediction image correction process based on one reference image has been described here. However, the same applies to the case of correcting the prediction image based on multiple reference images. After obtaining the corrected prediction images according to the respective reference images, the obtained prediction images are further superimposed to obtain the final prediction image.

[0158] In addition, the processing target block described above may be in units of prediction blocks or in units of sub-blocks obtained by further dividing the prediction blocks.

[0159] As a method for determining whether to use the OBMC process, for example, there is a method of using a signal indicating whether to use the OBMC process, namely, obmc_flag. As a specific example, in an encoding device, it is determined whether the encoding target block belongs to a region with complex motion. If it belongs to a region with complex motion, a value of 1 is set as obmc_flag and encoding is performed using the OBMC process. If it does not belong to a region with complex motion, a value of 0 is set as obmc_flag and encoding is performed without using the OBMC process. On the other hand, in a decoding device, by decoding the obmc_flag described in the stream, whether to use the OBMC process is switched according to its value to perform decoding.

[0160] In addition, the motion information may not be signaled but derived on the decoding device side. For example, the merge mode defined by the H.265 / HEVC standard may be used. In addition, for example, the motion information may be derived by performing motion estimation on the decoding device side. In this case, motion estimation is performed without using the pixel values of the current block.

[0161] Here, the mode of performing motion estimation on the decoding device side will be described. This mode of performing motion estimation on the decoding device side is called the PMMVD (pattern matched motion vector derivation) mode or the FRUC (frame rate up-conversion) mode.

[0162] In Figure 5D shows an example of the FRUC process. First, referring to the motion vectors of the encoded blocks adjacent to the current block in space or time, a list of multiple candidates each having a predicted motion vector is generated (which may be shared with the merge list). Then, the best candidate MV is selected from among the multiple candidate MVs registered in the candidate list. For example, the evaluation value of each candidate included in the candidate list is calculated, and one candidate is selected based on the evaluation value.

[0163] And, based on the motion vectors of the selected candidates, a motion vector for the current block is derived. Specifically, for example, the motion vector of the selected candidate (the best candidate MV) is directly derived as the motion vector for the current block. In addition, for example, the motion vector for the current block may also be derived by performing pattern matching in a peripheral region of the position within the reference picture corresponding to the motion vector of the selected candidate. That is, the peripheral region of the best candidate MV may be searched by the same method, and in the case where there is an MV with a better evaluation value, the best candidate MV is updated to the above MV, and it is used as the final MV of the current block. Alternatively, a configuration may be made where this process is not performed.

[0164] The exact same process may also be performed when processing is carried out in units of sub-blocks.

[0165] In addition, regarding the evaluation value, it is calculated by obtaining a difference value of the reconstructed image through pattern matching between the region within the reference picture corresponding to the motion vector and a specified region. Alternatively, in addition to the difference value, other information may also be used to calculate the evaluation value.

[0166] As the pattern matching, the first pattern matching or the second pattern matching is used. In some cases, the first pattern matching and the second pattern matching are respectively referred to as bilateral matching and template matching.

[0167] In the first pattern matching, pattern matching is performed between two blocks along the motion trajectory of the current block in two different reference pictures. Therefore, in the first pattern matching, as the specified region for calculating the evaluation value of the candidate, a region within another reference picture along the motion trajectory of the current block is used.

[0168] Figure 6 It is a diagram for explaining an example of pattern matching (bilateral matching) between two blocks along the motion trajectory. As Figure 6 shown, in the first pattern matching, by searching for the most matching pair among pairs of two blocks in two different reference pictures (Ref0, Ref1) along the motion trajectory of the current block (Cur block), two motion vectors (MV0, MV1) are derived. Specifically, for the current block, the difference between the reconstructed image at the specified position within the first encoded reference picture (Ref0) specified by the candidate MV and the reconstructed image at the specified position within the second encoded reference picture (Ref1) specified by the symmetric MV obtained by scaling the above candidate MV by the display time interval is obtained, and the obtained difference value is used to calculate the evaluation value. The candidate MV with the best evaluation value among multiple candidate MVs may be selected as the final MV.

[0169] Under the assumption of a continuous motion trajectory, the motion vectors (MV0, MV1) indicating two reference blocks are proportional to the temporal distances (TD0, TD1) between two reference pictures (Ref0, Ref1) with respect to the current picture (Cur Pic). For example, when the current picture is temporally located between two reference pictures and the temporal distances from the current picture to the two reference pictures are equal, in the first pattern matching, a mirror-symmetric bidirectional motion vector is derived.

[0170] In the second pattern matching, pattern matching is performed between a template within the current picture (a block adjacent to the current block within the current picture (e.g., an upper and / or left adjacent block)) and a block within the reference picture. Therefore, in the second pattern matching, as the specified region for calculating the evaluation value for the above candidates, a block adjacent to the current block within the current picture is used.

[0171] Figure 7 is a diagram for explaining an example of pattern matching (template matching) between a template within the current picture and a block within the reference picture. As Figure 7 shown, in the second pattern matching, by searching for the block that best matches the block adjacent to the current block (Cur block) within the current picture (Cur Pic) within the reference picture (Ref0), the motion vector of the current block is derived. Specifically, for the current block, the difference between the reconstructed images of the left adjacent and upper adjacent encoded regions or one of them and the reconstructed image at the same position within the encoded reference picture (Ref0) specified by the candidate MV is derived, and the evaluation value is calculated using the obtained difference value. Among multiple candidate MVs, the candidate MV with the best evaluation value is selected as the best candidate MV.

[0172] Information indicating whether to adopt the FRUC mode (e.g., called the FRUC flag) is signaled at the CU level. In addition, when the FRUC mode is adopted (e.g., when the FRUC flag is true), information indicating the method of pattern matching (the first pattern matching or the second pattern matching) (e.g., called the FRUC mode flag) is signaled at the CU level. Additionally, the signaling of this information does not need to be limited to the CU level and can also be at other levels (e.g., sequence level, picture level, slice level, tile level, CTU level, or sub-block level).

[0173] Here, a mode of deriving a motion vector based on a model assuming uniform linear motion is described. This mode has a case called BIO (bi-directional optical flow).

[0174] Figure 8 is a diagram for explaining a model assuming uniform linear motion. InFigure 8 Among them, (v x , v y ) represents the velocity vector, and τ0 and τ1 respectively represent the temporal distances between the current picture (Cur Pic) and two reference pictures (Ref0, Ref1). (MVx0, MVy0) represents the motion vector corresponding to the reference picture Ref0, and (MVx1, MVy1) represents the motion vector corresponding to the reference picture Ref1.

[0175] At this time, under the assumption of uniform linear motion of the velocity vector (v x , v y ), (MVx0, MVy0) and (MVx1, MVy1) are respectively expressed as (v x τ0, v y τ0) and (-v x τ1, -v y τ1), and the following optical flow equation (1) holds.

[0176] [Equation 1]

[0177]

[0178] Here, I (k) represents the luminance value of the reference image k (k = 0, 1) after motion compensation. This optical flow equation means that the sum of (i) the temporal differential of the luminance value, (ii) the product of the horizontal velocity and the horizontal component of the spatial gradient of the reference image, and (iii) the product of the vertical velocity and the vertical component of the spatial gradient of the reference image is equal to zero. Based on the combination of this optical flow equation and Hermite interpolation, the motion vectors in block units obtained from a merge list, etc. are corrected in pixel units.

[0179] In addition, the motion vector can also be derived on the decoder side by a method different from the derivation of the motion vector based on the model assuming uniform linear motion. For example, the motion vector can also be derived in sub-block units based on the motion vectors of multiple adjacent blocks.

[0180] Here, the mode of deriving the motion vector in sub-block units based on the motion vectors of multiple adjacent blocks will be described. This mode is called the affine motion compensation prediction mode in some cases.

[0181] Figure 9A is a diagram for explaining the derivation of the motion vector in sub-block units based on the motion vectors of multiple adjacent blocks. In Figure 9AAmong them, the current block includes 16 4×4 sub-blocks. Here, based on the motion vectors of adjacent blocks, the motion vector v0 of the upper-left control point of the current block is derived, and based on the motion vectors of adjacent sub-blocks, the motion vector v1 of the upper-right control point of the current block is derived. And, using the two motion vectors v0 and v1, through the following formula (2), the motion vectors (v x , v y ) of each sub-block within the current block are derived.

[0182] [Equation 2]

[0183]

[0184] Here, x and y respectively represent the horizontal position and vertical position of the sub-block, and w represents a preset weight coefficient.

[0185] In such an affine motion compensation prediction mode, several modes with different methods for deriving the motion vectors of the upper-left and upper-right control points may also be included. Information indicating such an affine motion compensation prediction mode (for example, called an affine flag) is signaled at the CU level. In addition, the signaling of the information indicating the affine motion compensation prediction mode does not need to be limited to the CU level, and may also be other levels (for example, sequence level, picture level, slice level, tile level, CTU level or sub-block level).

[0186] [Prediction control unit]

[0187] The prediction control unit 128 selects one of the intra-prediction signal and the inter-prediction signal, uses the selected signal as the prediction signal, and outputs it to the subtraction unit 104 and the addition unit 116.

[0188] Here, an example of deriving the motion vector of the coded object picture through the merge mode is described. Figure 9B is a diagram for explaining the outline of the motion vector derivation process based on the merge mode.

[0189] First, a prediction MV list registering candidates of prediction MVs is generated. As candidates of prediction MVs, there are spatial neighboring prediction MVs which are the MVs of multiple coded blocks spatially located around the coded object block, temporal neighboring prediction MVs which are the MVs of blocks near the position where the coded object block in the coded reference picture is projected, combined prediction MVs which are MVs generated by combining the MV values of spatial neighboring prediction MVs and temporal neighboring prediction MVs, and zero prediction MVs whose values are zero, etc.

[0190] Next, by selecting one prediction MV from the multiple prediction MVs registered in the prediction MV list, it is determined as the MV of the coded object block.

[0191] Further, in the variable-length coding section, the merge_idx, which is a signal indicating which predicted MV is selected, is described in the stream and encoded.

[0192] In addition, Figure 9B The predicted MVs registered in the predicted MV list described in is an example, and the number may be different from the number in the figure, or the structure may not include some types of the predicted MVs in the figure, or the structure may be appended with predicted MVs other than the types of the predicted MVs in the figure.

[0193] In addition, the MV of the coding target block derived by the merge mode may be used for the subsequent DMVR processing to determine the final MV.

[0194] Here, an example of determining the MV using the DMVR processing will be described.

[0195] Figure 9C is a conceptual diagram for explaining the outline of the DMVR processing.

[0196] First, the optimal MVP set for the processing target block is used as the candidate MV. According to the above candidate MV, reference pixels are obtained from the first reference picture of the processed pictures in the L0 direction and the second reference picture of the processed pictures in the L1 direction respectively, and a template is generated by taking the average of each reference pixel.

[0197] Next, using the above template, the peripheral areas of the candidate MVs in the first reference picture and the second reference picture are searched respectively, and the MV with the minimum cost is determined as the final MV. In addition, for the cost value, it is calculated using the difference values between the pixel values of the template and the pixel values of the search area and the MV value, etc.

[0198] In addition, in the encoding device and the decoding device, the outline of the processing described here is basically common.

[0199] In addition, even if it is not the processing itself described here, as long as it is a processing that can search the periphery of the candidate MV and derive the final MV, other processing can also be used.

[0200] Here, the mode of generating a predicted image using the LIC processing will be described.

[0201] Figure 9D is a diagram for explaining the outline of the predicted image generation method using the luminance correction processing based on the LIC processing.

[0202] First, an MV for obtaining a reference image corresponding to the coding target block from the reference picture as the encoded picture is derived.

[0203] Next, for the coded object block, using the luminance pixel values of the left-adjacent and upper-adjacent coded surrounding reference regions and the luminance pixel values at the corresponding positions in the reference picture specified by the MV, information indicating how the luminance values change in the reference picture and the coded object picture is extracted, and a luminance correction parameter is calculated.

[0204] By performing a luminance correction process on the reference image in the reference picture specified by the MV using the above luminance correction parameter, a prediction image for the coded object block is generated.

[0205] In addition, Figure 9D the shape of the above surrounding reference region in Figure 9D is an example, and other shapes may also be used.

[0206] Furthermore, the process of generating a prediction image based on one reference picture is described here, but the same applies when generating a prediction image based on multiple reference pictures. After performing a luminance correction process on the reference images obtained from each reference picture in the same manner, a prediction image is generated.

[0207] As a method for determining whether to perform LIC processing, for example, there is a method of using lic_flag, which is a signal indicating whether to perform LIC processing. As a specific example, in the coding device, it is determined whether the coded object block belongs to a region where luminance change has occurred. If it belongs to a region where luminance change has occurred, the value 1 is set as lic_flag, and coding is performed using LIC processing. If it does not belong to a region where luminance change has occurred, the value 0 is set as lic_flag, and coding is performed without using LIC processing. On the other hand, in the decoding device, by decoding the lic_flag described in the stream, decoding is performed by switching whether to perform LIC processing according to its value.

[0208] As another method for determining whether to perform LIC processing, for example, there is also a method of determining according to whether LIC processing has been performed on the surrounding blocks. As a specific example, in the case where the coded object block is in the merge mode, it is determined whether the surrounding coded blocks selected when deriving the MV in the merge mode processing have been coded using LIC processing, and according to the result, coding is performed by switching whether to perform LIC processing. In addition, in this example, the processing during decoding is exactly the same.

[0209] [Outline of the decoding device]

[0210] Next, an outline of the decoding device that can decode the coded signal (coded bitstream) output from the above coding device 100 will be described. Figure 10FIG. 0 is a block diagram showing the functional configuration of the decoding device 200 according to Embodiment 1. The decoding device 200 is a moving image / image decoding device that decodes moving images / images in units of blocks.

[0211] As Figure 10 shown, the decoding device 200 includes an entropy decoding unit 202, an inverse quantization unit 204, an inverse transform unit 206, an addition unit 208, a block memory 210, a loop filter unit 212, a frame memory 214, an intra prediction unit 216, an inter prediction unit 218, and a prediction control unit 220.

[0212] The decoding device 200 is implemented by, for example, a general-purpose processor and a memory. In this case, when the software program stored in the memory is executed by the processor, the processor functions as the entropy decoding unit 202, the inverse quantization unit 204, the inverse transform unit 206, the addition unit 208, the loop filter unit 212, the intra prediction unit 216, the inter prediction unit 218, and the prediction control unit 220. In addition, the decoding device 200 may be implemented as one or more dedicated electronic circuits corresponding to the entropy decoding unit 202, the inverse quantization unit 204, the inverse transform unit 206, the addition unit 208, the loop filter unit 212, the intra prediction unit 216, the inter prediction unit 218, and the prediction control unit 220.

[0213] Hereinafter, each component included in the decoding device 200 will be described.

[0214] [Entropy Decoding Unit]

[0215] The entropy decoding unit 202 performs entropy decoding on the encoded bitstream. Specifically, the entropy decoding unit 202, for example, arithmetic decodes the encoded bitstream into a binary signal. Then, the entropy decoding unit 202 de-binarizes the binary signal. Thereby, the entropy decoding unit 202 outputs the quantization coefficients to the inverse quantization unit 204 in units of blocks.

[0216] [Inverse Quantization Unit]

[0217] The inverse quantization unit 204 performs inverse quantization on the quantization coefficients of the block to be decoded (hereinafter referred to as the current block) as the input from the entropy decoding unit 202. Specifically, for the quantization coefficients of the current block, the inverse quantization unit 204 performs inverse quantization on each quantization coefficient based on the quantization parameter corresponding to the quantization coefficient. And the inverse quantization unit 204 outputs the inverse quantized quantization coefficients (i.e., transform coefficients) of the current block to the inverse transform unit 206.

[0218] [Inverse Transform Unit]

[0219] The inverse transform unit 206 restores the prediction error by performing an inverse transform on the transform coefficients as the input from the inverse quantization unit 204.

[0220] For example, in a case where the information decoded from the coded bitstream indicates the use of EMT or AMT (e.g., the AMT flag is true), the inverse transform unit 206 performs an inverse transform on the transform coefficients of the current block based on the decoded information indicating the transform type.

[0221] Furthermore, for example, in a case where the information decoded from the coded bitstream indicates the use of NSST, the inverse transform unit 206 applies an inverse re - transform to the transform coefficients.

[0222] [Addition unit]

[0223] The addition unit 208 reconstructs the current block by adding the prediction error, which is the input from the inverse transform unit 206, to the prediction sample, which is the input from the prediction control unit 220. Also, the addition unit 208 outputs the reconstructed block to the block memory 210 and the loop filter unit 212.

[0224] [Block memory]

[0225] The block memory 210 is a storage unit for storing blocks within the decoded picture (hereinafter referred to as the current picture) that are referenced in intra - prediction. Specifically, the block memory 210 stores the reconstructed blocks output from the addition unit 208.

[0226] [Loop filter unit]

[0227] The loop filter unit 212 applies loop filtering to the block reconstructed by the addition unit 208 and outputs the filtered reconstructed block to the frame memory 214, the display device, etc.

[0228] In a case where the information decoded from the coded bitstream indicating the on / off of ALF indicates that ALF is on, one filter is selected from a plurality of filters based on the direction and activity of the local gradient, and the selected filter is applied to the reconstructed block.

[0229] [Frame memory]

[0230] The frame memory 214 is a storage unit for storing reference pictures used in inter - prediction, and is sometimes referred to as a frame buffer. Specifically, the frame memory 214 stores the reconstructed blocks filtered by the loop filter unit 212.

[0231] [Intra - prediction unit]

[0232] The intra prediction unit 216 performs intra prediction with reference to the blocks within the current picture stored in the block memory 210 based on the intra prediction mode decoded from the encoded bitstream, thereby generating a prediction signal (intra prediction signal). Specifically, the intra prediction unit 216 generates an intra prediction signal by performing intra prediction with reference to the samples (e.g., luminance values, chrominance differences) of the blocks adjacent to the current block, and outputs the intra prediction signal to the prediction control unit 220.

[0233] In addition, when the intra prediction mode of referring to the luminance block is selected in the intra prediction of the chrominance block, the intra prediction unit 216 may also predict the chrominance component of the current block based on the luminance component of the current block.

[0234] Furthermore, when the information decoded from the encoded bitstream indicates the adoption of PDPC, the intra prediction unit 216 corrects the pixel value after intra prediction based on the gradients of the reference pixels in the horizontal / vertical directions.

[0235] [Inter prediction unit]

[0236] The inter prediction unit 218 predicts the current block with reference to the reference pictures stored in the frame memory 214. The prediction is performed in units of the current block or sub-blocks (e.g., 4×4 blocks) within the current block. For example, the inter prediction unit 218 performs motion compensation using the motion information (e.g., motion vector) decoded from the encoded bitstream, thereby generating an inter prediction signal for the current block or sub-block, and outputs the inter prediction signal to the prediction control unit 220.

[0237] In addition, when the information decoded from the encoded bitstream indicates the adoption of the OBMC mode, the inter prediction unit 218 generates an inter prediction signal using not only the motion information of the current block obtained through motion estimation but also the motion information of the adjacent blocks.

[0238] Furthermore, when the information decoded from the encoded bitstream indicates the adoption of the FRUC mode, the inter prediction unit 218 performs motion estimation according to the pattern matching method (bidirectional matching or template matching) decoded from the encoded stream, thereby deriving the motion information. And the inter prediction unit 218 performs motion compensation using the derived motion information.

[0239] In addition, the inter prediction unit 218 derives a motion vector based on a model assuming uniform linear motion when adopting the BIO mode. In addition, when the information decoded from the encoded bitstream indicates the adoption of the affine motion compensation prediction mode, the inter prediction unit 218 derives a motion vector in units of sub-blocks based on the motion vectors of multiple adjacent blocks.

[0240] [Prediction control unit]

[0241] The prediction control unit 220 selects one of an intra prediction signal and an inter prediction signal, and outputs the selected signal as a prediction signal to the adder 208.

[0242] (Embodiment 2)

[0243] Regarding the encoding process and the decoding process related to Embodiment 2, refer to Figure 11 and Figure 12 Specifically, regarding the encoding device and the decoding device related to Embodiment 2, refer to Figure 15 and Figure 16 Specifically described.

[0244] [Encoding Process]

[0245] Figure 11 Indicates the video encoding process related to Embodiment 2.

[0246] First, in step S1001, a first parameter for identifying a partitioning mode for partitioning a first block into a plurality of sub-blocks from among a plurality of partitioning modes is written into the bitstream. If the partitioning mode is used, the block is partitioned into a plurality of sub-blocks. If different partitioning modes are used, the block is partitioned into a plurality of sub-blocks having different shapes, different heights, or different widths.

[0247] Figure 28 Shows an example of a partitioning mode for partitioning an N×N pixel block in Embodiment 2. In Figure 28 , (a) to (h) represent mutually different partitioning modes. As Figure 28As shown, if the block mode (a) is used, a block of N×N pixels (for example, 16×16 pixels, and "N" can take any value that is an integer multiple of 4 from 8 to 128) is divided into two sub - blocks of N / 2×N pixels (for example, 8×16 pixels). If the block mode (b) is used, a block of N×N pixels is divided into a sub - block of N / 4×N pixels (for example, 4×16 pixels) and a sub - block of 3N / 4×N pixels (for example, 12×16 pixels). If the block mode (c) is used, a block of N×N pixels is divided into a sub - block of 3N / 4×N pixels (for example, 12×16 pixels) and a sub - block of N / 4×N pixels (for example, 4×16 pixels). If the block mode (d) is used, a block of N×N pixels is divided into a sub - block of (N / 4)×N pixels (for example, 4×16 pixels), a sub - block of N / 2×N pixels (for example, 8×16 pixels), and a sub - block of N / 4×N pixels (for example, 4×16 pixels). If the block mode (e) is used, a block of N×N pixels is divided into two sub - blocks of N×N / 2 pixels (for example, 16×8 pixels). If the block mode (f) is used, a block of N×N pixels is divided into a sub - block of N×N / 4 pixels (for example, 16×4 pixels) and a sub - block of N×3N / 4 pixels (for example, 16×12 pixels). If the block mode (g) is used, a block of N×N pixels is divided into a sub - block of N×3N / 4 pixels (for example, 16×12 pixels) and a sub - block of N×N / 4 pixels (for example, 16×4 pixels). If the block mode (h) is used, a block of N×N pixels is divided into a sub - block of N×N / 4 pixels (for example, 16×4 pixels), a sub - block of N×N / 2 pixels (for example, 16×8 pixels), and a sub - block of N×N / 4 pixels (for example, 16×4 pixels).

[0248] Next, in step S1002, it is determined whether the first parameter identifies the first block mode.

[0249] Next, in step S1003, based at least on the determination of whether the first parameter identifies the first block mode, it is determined whether the second block mode is not selected as a candidate for dividing the second block.

[0250] Two different sets of block modes may divide a block into sub - blocks of the same shape and size. For example, as Figure 31A shown, the sub - blocks of (1b) and (2c) have the same shape and size. One set of block modes can contain at least two block modes. For example, as Figure 31A shown in (1a) and (1b) of Figure 31AAs shown in (2a), (2b), and (2c), other block pattern sets can follow the vertical binary tree splitting to include the vertical binary tree splitting of two sub-blocks. Each block pattern set results in sub-blocks of the same shape and size.

[0251] When selecting between two block pattern sets that divide a block into sub-blocks of the same shape and size and have different binary numbers or different numbers of bits when encoded in the bitstream, select the block pattern set with fewer binary numbers or fewer bits. Additionally, the binary numbers and the number of bits correspond to the code amount.

[0252] When selecting between two block pattern sets that divide a block into sub-blocks of the same shape and size and have the same binary number or the same number of bits when encoded in the bitstream, select the block pattern set that appears first in a specified order of multiple block pattern sets. The specified order can be, for example, the order based on the number of block patterns within each block pattern set.

[0253] Figure 31A and Figure 31B is a diagram showing an example of dividing a block into sub-blocks using a block pattern set with fewer binary numbers in the encoding of the block pattern. In this example, when the left N×N pixel block is vertically divided into two sub-blocks, the second block pattern for the right N×N pixel block is not selected in step (2c). This is because, in Figure 31B the encoding method of the block pattern, the second block pattern set (2a, 2b, 2c) requires more binary numbers for encoding the block pattern compared to the first block pattern set (1a, 1b).

[0254] Figures 32A to 32C is a diagram showing an example of dividing a block into sub-blocks using the block pattern set that appears first in a specified order of multiple block pattern sets. In this example, when the 2N×N / 2 pixel block is vertically divided into three sub-blocks, the second block pattern for the lower 2N×N / 2 pixel block is not selected in step (2c). This is because, in Figure 32B the encoding method of the block pattern, the second block pattern set (2a, 2b, 2c) has the same binary number as the first block pattern set (1a, 1b, 1c, 1d), and in Figure 32C the specified order of the block pattern sets shown, it appears after the first block pattern set (1a, 1b, 1c, 1d). The specified order of multiple block pattern sets can also be fixed and signaled within the bitstream.

[0255] Figure 20This shows an example in Embodiment 2 where the second block division mode is not selected for the division of a 2N×N pixel block as shown in step (2c). As Figure 20 shown, it is possible to use the first division method (i) to equally divide a 2N×2N pixel (e.g., 16×16 pixel) block into 4 sub-blocks of N×N pixels (e.g., 8×8 pixels) as in step (1a). Also, it is possible to use the second division method (ii) to horizontally equally divide a 2N×2N pixel block into 2 sub-blocks of 2N×N pixels (e.g., 16×8 pixels) as in step (2a). Here, in the second division method (ii), when the upper 2N×N pixel block (the first block) is vertically divided into 2 sub-blocks of N×N pixels by the first block division mode as in step (2b), in step (2c), the second block division mode for vertically dividing the lower 2N×N pixel block (the second block) into 2 sub-blocks of N×N pixels is not selected as a candidate for possible block division modes. This is because sub-block sizes identical to those obtained by the four-fold division using the first division method (i) are generated.

[0256] As described above, in Figure 20 , when the first block is vertically equally divided into 2 sub-blocks if the first block division mode is used, and the second block adjacent to the first block in the vertical direction is vertically equally divided into 2 sub-blocks if the second block division mode is used, the second block division mode is not selected as a candidate.

[0257] Figure 21 This shows an example in Embodiment 2 where the second block division mode is not selected for the division of an N×2N pixel block as shown in step (2c). As Figure 21 shown, it is possible to use the first division method (i) to equally divide a 2N×2N pixel block into 4 sub-blocks of N×N pixels as in step (1a). Also, it is possible to use the second division method (ii) to vertically equally divide a 2N×2N pixel block into 2 sub-blocks of 2N×N pixels (e.g., 8×16 pixels) as in step (2a). In the second division method (ii), when the left N×2N pixel block (the first block) is horizontally divided into 2 sub-blocks of N×N pixels by the first block division mode as in step (2b), in step (2c), the second block division mode for horizontally dividing the right N×2N pixel block (the second block) into 2 sub-blocks of N×N pixels is not selected as a candidate for possible block division modes. This is because sub-block sizes identical to those obtained by the four-fold division using the first division method (i) are generated.

[0258] As described above, in Figure 21In the case where, if the first block mode is used, the first block is equally divided into two sub - blocks horizontally, and if the second block mode is used, the second block adjacent to the first block horizontally is equally divided into two sub - blocks horizontally, the second block mode is not selected as a candidate.

[0259] Figure 40 Indicates an example of dividing a 4N×2N block in Figure 20 into three parts in a ratio of 1:2:1, such as N×2N, 2N×2N, N×2N. Here, when the upper block is divided into three parts, the block - dividing mode of dividing the lower block into three parts in a ratio of 1:2:1 is not selected as a candidate for possible block - dividing modes. The three - part division can also be in a ratio different from 1:2:1. Furthermore, it can be divided into more than three parts, or it can be divided into two parts, or it can be in a ratio different from 1:1, such as 1:2 or 1:3. Figure 40 This is an example of dividing first horizontally, but the same constraints can also be applied when dividing first vertically.

[0260] Figure 41 and Figure 42 Indicates an example of applying the same constraint when the first block is rectangular.

[0261] Figure 43 This is the second constraint example when a square is divided into three parts vertically and then equally divided into two parts horizontally. When applying the Figure 43 constraint, in Figure 40 , the block - dividing mode of dividing the lower block of 4N×2N into three parts in a ratio of 1:2:1 can be selected. It is also possible to separately encode the information indicating which of the Figure 40 constraint and Figure 43 constraint is applied into the header information, etc. Or, constraints can also be applied to reduce the code amount of the information representing the block division. For example, if it is assumed that the code amounts of the information representing the block division in Case 1 and Case 2 are as follows, the division in Case 1 is set to be valid and the division in Case 2 is set to be invalid. That is, the Figure 43 constraint is applied.

[0262] (Case 1) (1) Divide the square into two parts horizontally, and then (2) divide the upper and lower two rectangular blocks vertically into three parts respectively: (1) Direction information: 1 bit, division quantity information: 1 bit, (2) (Direction information: 1 bit, division quantity information: 1 bit)×2, a total of 6 bits

[0263] (Case 2) (1) Divide the square vertically, and then (2) divide the left, middle, and right rectangular blocks horizontally into two parts respectively: (1) Direction information: 1 bit, division quantity information: 1 bit, (2) (Direction information: 1 bit, division quantity information: 1 bit)×3, a total of 8 bits

[0264] Alternatively, there is a case where the optimal block is determined while selecting block patterns in a prescribed order during encoding. For example, 2-way splitting may be tried first, followed by 3-way or 4-way splitting (bisecting horizontally and vertically), etc. At this time, before the attempt of 3-way splitting as in Figure 43 , the attempt starting from 2-way splitting as in the example of Figure 40 has already been carried out. Thus, in the attempt starting from 2-way splitting, the block splitting of horizontally bisecting and then vertically trisecting the upper and lower two blocks has already been tried, so the constraint of Figure 43 is applied. In this way, the constraint method for selection can also be determined based on a prescribed encoding method.

[0265] In Figure 44 , an example is shown in which the block patterns that can be selected in the same direction as the first block pattern are restricted in the second block pattern. Here, the first block pattern is 3-way splitting in the vertical direction. At this time, 2-way splitting cannot be selected as the second block pattern. On the other hand, for the vertical direction which is a direction different from the first block pattern, 2-way splitting can be selected ( Figure 45 ).

[0266] Figure 22 An example is shown in which the second block pattern is not selected for the splitting of an N×N pixel block as shown in step (2c) in Embodiment 2. As shown in Figure 22 , the first splitting method (i) can be used to vertically split a 2N×N pixel block (e.g., 16×8 pixels, and any value that is an integer multiple of 4 from 8 to 128 can be taken as the value of "N") into sub-blocks of N / 2×N pixels, N×N pixels, and N / 2×N pixels (e.g., sub-blocks of 4×8 pixels, 8×8 pixels, and 4×8 pixels). Also, the second splitting method (ii) can be used to split a 2N×N pixel block into two N×N pixel sub-blocks as shown in step (2a). In the first splitting method (i), the central N×N pixel block can be vertically split into two N / 2×N pixel (e.g., 4×8 pixel) sub-blocks in step (1b). In the second splitting method (ii), when the left N×N pixel block (the first block) is vertically split into two N / 2×N pixel sub-blocks as shown in step (2b), the block pattern of vertically splitting the right N×N pixel block (the second block) into two N / 2×N pixel sub-blocks is not selected as a candidate for possible block patterns in step (2c). This is because sub-blocks of the same size as those obtained by the first splitting method (i) will be generated, i.e., four sub-blocks of N / 2×N pixels.

[0267] As described above, in Figure 22In the case where, if the first block mode is used, the first block is equally divided into two sub-blocks in the vertical direction, and if the second block mode is used, the second block adjacent to the first block in the horizontal direction is equally divided into two sub-blocks in the vertical direction, the second block mode is not selected as a candidate.

[0268] Figure 23 This shows an example of not selecting the second block mode for the division of an N×N pixel block as shown in step (2c) in Embodiment 2. As Figure 23 shown, the first division method (i) can be used to divide an N×2N pixel (for example, 8×16 pixels, and any value that is an integer multiple of 4 from 8 to 128 can be taken as the value of "N") into sub-blocks of N×N / 2 pixels, sub-blocks of N×N pixels, and sub-blocks of N×N / 2 pixels (for example, sub-blocks of 8×4 pixels, sub-blocks of 8×8 pixels, and sub-blocks of 8×4 pixels) as in step (1a). Also, the second division method can be used to divide it into two sub-blocks of N×N pixels as in step (2a). In the first division method (i), the central N×N pixel block can be divided into two sub-blocks of N×N / 2 pixels as in step (1b). In the second division method (ii), when the upper N×N pixel block (the first block) is horizontally divided into two sub-blocks of N×N / 2 pixels as in step (2b), in step (2c), the division mode of horizontally dividing the lower N×N pixel block (the second block) into two sub-blocks of N×N / 2 pixels is not selected as a candidate for possible division modes. This is because sub-blocks of the same size as those obtained by the first division method (i) will be generated, that is, four sub-blocks of N×N / 2 pixels.

[0269] As described above, in Figure 23 the case where, if the first block mode is used, the first block is equally divided into two sub-blocks in the horizontal direction, and if the second block mode is used, the second block adjacent to the first block in the vertical direction is equally divided into two sub-blocks in the horizontal direction, the second block mode is not selected as a candidate.

[0270] If it is determined that the second block mode is selected as a candidate for dividing the second block (No in S1003), then in step S1004, a block mode is selected from among a plurality of block modes including the second block mode as a candidate. In step S1005, a second parameter indicating the selection result is written into the bitstream.

[0271] If it is determined that the second block mode is not selected as a candidate for dividing the second block (Yes in S1003), then in step S1006, a block mode different from the second block mode is selected for dividing the second block. Among the block modes selected here, the block is divided into sub-blocks having a different shape or different size compared to the sub-blocks generated by the second block mode.

[0272] Figure 24 This shows an example of using the selected block mode when not selecting the second block mode as shown in step (3) in Embodiment 2 to divide a 2N×N pixel block. As Figure 24 shown, the selected block mode can divide the current 2N×N pixel block (the lower block in this example) as Figure 24 shown in (c) and (f) into 3 sub-blocks. The sizes of the 3 sub-blocks can be different. For example, among the 3 sub-blocks, the large sub-block can have a width / height that is 2 times that of the small sub-block. And for example, the selected block mode can also divide the current block as Figure 24 shown in (a), (b), (d), and (e) into 2 sub-blocks with different sizes (asymmetric binary tree). For example, when using an asymmetric binary tree, the large sub-block can have a width / height that is 3 times that of the small sub-block.

[0273] Figure 25 This shows an example of using the selected block mode when not selecting the second block mode as shown in step (3) in Embodiment 2 to divide an N×2N pixel block. As Figure 25 shown, the selected block mode can divide the current N×2N pixel block (the right block in this example) as Figure 25 shown in (c) and (f) into 3 sub-blocks. The sizes of the 3 sub-blocks can be different. For example, among the 3 sub-blocks, the large sub-block can have a width / height that is 2 times that of the small sub-block. And for example, the selected block mode can also divide the current block as Figure 25 shown in (a), (b), (d), and (e) into 2 sub-blocks with different sizes (asymmetric binary tree). For example, when using an asymmetric binary tree, the large sub-block can have a width / height that is 3 times that of the small sub-block.

[0274] Figure 26 This shows an example of using the selected block mode when not selecting the second block mode as shown in step (3) in Embodiment 2 to divide an N×N pixel block. As Figure 26 shown, in step (1), the 2N×N pixel block is vertically divided into 2 N×N pixel sub-blocks. In step (2), the left N×N pixel block is vertically divided into 2 N / 2×N pixel sub-blocks. In step (3), the selected block mode for the current N×N pixel block (the left block in this example) can be used to divide the current block as Figure 26 shown in (c) and (f) into 3 sub-blocks. The sizes of the 3 sub-blocks can be different. For example, among the 3 sub-blocks, the large sub-block can have a width / height that is 2 times that of the small sub-block. And for example, the selected block mode can also divide the current block as Figure 26is divided into two sub - blocks (asymmetric binary tree) with different sizes as shown in (a), (b), (d), and (e). For example, in the case of using an asymmetric binary tree, the large sub - block can have three times the width / height of the small sub - block.

[0275] Figure 27 This shows an example of dividing an N×N pixel block using the selected block - splitting pattern when not selecting the second block - splitting mode as shown in step (3) in Embodiment 2. As Figure 27 shown, in step (1), the N×2N pixel block is horizontally divided into two N×N pixel sub - blocks. In step (2), the upper N×N pixel block is horizontally divided into two N×N / 2 pixel sub - blocks. In step (3), the selected block - splitting pattern for the current N×N pixel block (the lower block in this example) can be used to divide the current block into three sub - blocks as shown in Figure 27 (c) and (f). The sizes of the three sub - blocks can be different. For example, among the three sub - blocks, the large sub - block can have twice the width / height of the small sub - block. And for example, the selected block - splitting pattern can also divide the current block into two sub - blocks (asymmetric binary tree) with different sizes as shown in Figure 27 (a), (b), (d), and (e). For example, in the case of using an asymmetric binary tree, the large sub - block can have three times the width / height of the small sub - block.

[0276] Figure 17 This shows the possible positions of the first parameter in the compressed video stream. As Figure 17 shown, the first parameter can be configured in the video parameter set, sequence parameter set, picture parameter set, slice header, or coding tree unit. The first parameter can represent the method of dividing a block into multiple sub - blocks. For example, the first parameter can include a flag indicating whether to divide the block horizontally or vertically. The first parameter can also include a parameter indicating whether to divide the block into more than two sub - blocks.

[0277] Figure 18 This shows the possible positions of the second parameter in the compressed video stream. As Figure 18 shown, the second parameter can be configured in the video parameter set, sequence parameter set, picture parameter set, slice header, or coding tree unit. The second parameter can represent the method of dividing a block into multiple sub - blocks. For example, the second parameter can include a flag indicating whether to divide the block horizontally or vertically. The second parameter can also include a parameter indicating whether to divide the block into more than two sub - blocks. As Figure 19 shown, the second parameter follows the first parameter in the bitstream.

[0278] The first block and the second block are different blocks. The first block and the second block may be included in the same frame. For example, the first block may be a block adjacent above the second block. And for example, the first block may also be a block adjacent to the left of the second block.

[0279] In step S1007, the second block is divided into sub-blocks using the selected block partitioning mode. In step S1008, the divided blocks are encoded.

[0280] [Encoding device]

[0281] Figure 15 is a block diagram showing the structure of the video / image encoding device according to Embodiment 2 or 3.

[0282] The video encoding device 5000 is a device for encoding an input video / image for each block to generate an encoded output bitstream. As Figure 15 shown, the video encoding device 5000 includes a transform unit 5001, a quantization unit 5002, an inverse quantization unit 5003, an inverse transform unit 5004, a block memory 5005, a frame memory 5006, an intra prediction unit 5007, an inter prediction unit 5008, an entropy encoding unit 5009, and a block segmentation determination unit 5010.

[0283] The input video is input to the adder, and the added value is output to the transform unit 5001. The transform unit 5001 transforms the added value into frequency coefficients based on the block partitioning mode derived by the block segmentation determination unit 5010, and outputs the frequency coefficients to the quantization unit 5002. The block partitioning mode can be associated with a block partitioning mode, a block partitioning type, or a block partitioning direction. The quantization unit 5002 quantizes the input quantization coefficients and outputs the quantized values to the inverse quantization unit 5003 and the entropy encoding unit 5009.

[0284] The inverse quantization unit 5003 inverse quantizes the quantized values output from the quantization unit 5002 and outputs the frequency coefficients to the inverse transform unit 5004. The inverse transform unit 5004 performs an inverse frequency transform on the frequency coefficients based on the block segmentation mode derived by the block segmentation determination unit 5010, transforms the frequency coefficients into sample values of the bitstream, and outputs the sample values to the adder.

[0285] The adder adds the sample values of the bitstream output from the inverse transform unit 5004 to the predicted video / image values output from the intra / inter prediction units 5007 and 5008, and outputs the added value to the block memory 5005 or the frame memory 5006 for further prediction. The block segmentation determination unit 5010 collects block information from the block memory 5005 or the frame memory 5006, and derives the block partitioning mode and parameters related to the block partitioning mode. If the derived block partitioning mode is used, the block is divided into a plurality of sub-blocks. The intra / inter prediction units 5007 and 5008 search among the video / images stored in the block memory 5005 or the video / images in the frame memory 5006 reconstructed by the block partitioning mode derived by the block segmentation determination unit 5010, and estimate, for example, the video / image region most similar to the input video / image to be predicted.

[0286] The entropy encoding unit 5009 encodes the quantization values output from the quantization unit 5002, encodes the parameters from the block segmentation determination unit 5010, and outputs a bitstream.

[0287] [Decoding process]

[0288] Figure 12 Represents the video decoding process related to Embodiment 2.

[0289] First, in step S2001, the first parameter is interpreted according to the bitstream. The first parameter identifies the partitioning mode for dividing the first block into sub-blocks from among a plurality of partitioning modes. If the partitioning mode is used, the block is divided into sub-blocks, and if a different partitioning mode is used, the block is divided into sub-blocks having different shapes, different heights, or different widths.

[0290] Figure 28 Represents an example of the partitioning mode for dividing an N×N pixel block in Embodiment 2. In Figure 28 ,(a) to (h) represent different partitioning modes. As Figure 28As shown, if the block mode (a) is used, a block of N×N pixels (for example, 16×16 pixels, and as the value of "N", any value that is an integer multiple of 4 from 8 to 128 can be taken) is divided into two sub-blocks of N / 2×N pixels (for example, 8×16 pixels). If the block mode (b) is used, the block of N×N pixels is divided into a sub-block of N / 4×N pixels (for example, 4×16 pixels) and a sub-block of 3N / 4×N pixels (for example, 12×16 pixels). If the block mode (c) is used, the block of N×N pixels is divided into a sub-block of 3N / 4×N pixels (for example, 12×16 pixels) and a sub-block of N / 4×N pixels (for example, 4×16 pixels). If the block mode (d) is used, the block of N×N pixels is divided into a sub-block of (N / 4)×N pixels (for example, 4×16 pixels), a sub-block of N / 2×N pixels (for example, 8×16 pixels), and a sub-block of N / 4×N pixels (for example, 4×16 pixels). If the block mode (e) is used, the block of N×N pixels is divided into two sub-blocks of N×N / 2 pixels (for example, 16×8 pixels). If the block mode (f) is used, the block of N×N pixels is divided into a sub-block of N×N / 4 pixels (for example, 16×4 pixels) and a sub-block of N×3N / 4 pixels (for example, 16×12 pixels). If the block mode (g) is used, the block of N×N pixels is divided into a sub-block of N×3N / 4 pixels (for example, 16×12 pixels) and a sub-block of N×N / 4 pixels (for example, 16×4 pixels). If the block mode (h) is used, the block of N×N pixels is divided into a sub-block of N×N / 4 pixels (for example, 16×4 pixels), a sub-block of N×N / 2 pixels (for example, 16×8 pixels), and a sub-block of N×N / 4 pixels (for example, 16×4 pixels).

[0291] Next, in step S2002, it is determined whether the first parameter identifies the first block mode.

[0292] Next, in step S2003, based at least on the determination of whether the first parameter identifies the first block mode, it is determined whether the second block mode is not selected as a candidate for dividing the second block.

[0293] Two different sets of block modes may divide a block into sub-blocks of the same shape and size. For example, as Figure 31A shown, the sub-blocks of (1b) and (2c) have the same shape and size. One set of block modes can include at least two block modes. For example, as Figure 31A shown in (1a) and (1b) of, one set of block modes can then include a vertical binary tree division of the central sub-block and a non-division of the other sub-blocks based on a vertical division of a ternary tree. And for example, as Figure 31AAs shown in (2a), (2b), and (2c), other block pattern sets can then include a binary tree vertical split that contains two sub-blocks by vertically splitting the binary tree. Any block pattern set results in sub-blocks of the same shape and size.

[0294] When selecting between two block pattern sets that split a block into sub-blocks of the same shape and size and that are different binary numbers or different numbers of bits when encoded in the bitstream, select the block pattern set with fewer binary numbers or fewer bits.

[0295] When selecting between two block pattern sets that split a block into sub-blocks of the same shape and size and that have the same number of bits or the same number of bits when encoded in the bitstream, select the block pattern set that first appears in a specified order of multiple block pattern sets. The specified order can be, for example, an order based on the number of block patterns within each block pattern set.

[0296] Figure 31A and Figure 31B is a diagram showing an example of splitting a block into sub-blocks using a block pattern set with fewer binary numbers in the encoding of the block pattern. In this example, when the left N×N pixel block is vertically split into 2 sub-blocks, the second block pattern for the right N×N pixel block is not selected in step (2c). This is because, in Figure 31B the encoding method of the block pattern, the second block pattern set (2a, 2b, 2c) requires more binary numbers for encoding the block pattern compared to the first block pattern set (1a, 1b).

[0297] Figure 32A is a diagram showing an example of splitting a block into sub-blocks using the block pattern set that first appears in a specified order of multiple block pattern sets. In this example, when the 2N×N / 2 pixel block is vertically split into 3 sub-blocks, the second block pattern for the lower 2N×N / 2 pixel block is not selected in step (2c). This is because, in Figure 32B the encoding method of the block pattern, the second block pattern set (2a, 2b, 2c) has the same binary number as the first block pattern set (1a, 1b, 1c, 1d), and in Figure 32C the specified order of the block pattern sets shown, it appears after the first block pattern set (1a, 1b, 1c, 1d). The specified order of multiple block pattern sets can also be fixed or signaled within the bitstream.

[0298] Figure 20 shows an example of not selecting the second block pattern for the split of the 2N×N pixel block as shown in step (2c) in Embodiment 2. As Figure 20As shown, it is possible to use the first segmentation method (i) to equally divide a block of 2N×2N pixels (e.g., 16×16 pixels) into 4 sub-blocks of N×N pixels (e.g., 8×8 pixels) as in step (1a). Also, it is possible to use the second segmentation method (ii) to horizontally equally divide a block of 2N×2N pixels into 2 sub-blocks of 2N×N pixels (e.g., 16×8 pixels) as in step (2a). In the second segmentation method (ii), when the upper 2N×N pixel block (the first block) is vertically divided into 2 sub-blocks of N×N pixels by the first block division mode as in step (2b), the second block division mode for vertically dividing the lower 2N×N pixel block (the second block) into 2 sub-blocks of N×N pixels is not selected as a candidate for the possible block division mode in step (2c). This is because sub-blocks of the same size as those obtained by the four-way division using the first segmentation method (i) are generated.

[0299] As described above, Figure 20 in a case where if the first block division mode is used, the first block is equally divided into 2 sub-blocks in the vertical direction, and if the second block division mode is used, the second block adjacent to the first block in the vertical direction is equally divided into 2 sub-blocks in the vertical direction, the second block division mode is not selected as a candidate.

[0300] Figure 21 This shows an example in Embodiment 2 where the second block division mode is not selected for the division of the N×2N pixel block as shown in step (2c). As Figure 21 shown, it is possible to use the first segmentation method (i) to equally divide a block of 2N×2N pixels into 4 sub-blocks of N×N pixels as in step (1a). Also, it is possible to use the second segmentation method (ii) to vertically equally divide a block of 2N×2N pixels into 2 sub-blocks of 2N×N pixels (e.g., 8×16 pixels) as in step (2a). In the second segmentation method (ii), when the left N×2N pixel block (the first block) is horizontally divided into 2 sub-blocks of N×N pixels by the first block division mode as in step (2b), the second block division mode for horizontally dividing the right N×2N pixel block (the second block) into 2 sub-blocks of N×N pixels is not selected as a candidate for the possible block division mode in step (2c). This is because sub-blocks of the same size as those obtained by the four-way division using the first segmentation method (i) are generated.

[0301] As described above, Figure 21 in a case where if the first block division mode is used, the first block is equally divided into 2 sub-blocks in the horizontal direction, and if the second block division mode is used, the second block adjacent to the first block in the horizontal direction is equally divided into 2 sub-blocks in the horizontal direction, the second block division mode is not selected as a candidate.

[0302] Figure 22 This shows an example in Embodiment 2 where the second block division mode is not selected for the division of an N×N pixel block as shown in step (2c). As Figure 22 shown, the first division method (i) can be used to vertically divide a 2N×N pixel block (e.g., 16×8 pixels, where "N" can be any value that is an integer multiple of 4 from 8 to 128) into N / 2×N pixel sub-blocks, N×N pixel sub-blocks, and N / 2×N pixel sub-blocks (e.g., 4×8 pixel sub-blocks, 8×8 pixel sub-blocks, and 4×8 pixel sub-blocks) as in step (1a). Also, the second division method (ii) can be used to divide a 2N×N pixel block into two N×N pixel sub-blocks as in step (2a). In the first division method (i), the central N×N pixel block can be vertically divided into two N / 2×N pixel (e.g., 4×8 pixel) sub-blocks in step (1b). In the second division method (ii), when the left N×N pixel block (the first block) is vertically divided into two N / 2×N pixel sub-blocks as in step (2b), the block division mode of vertically dividing the right N×N pixel block (the second block) into two N / 2×N pixel sub-blocks in step (2c) is not selected as a candidate for the possible block division modes. This is because sub-blocks of the same size as those obtained by the first division method (i) will be generated, i.e., four N / 2×N pixel sub-blocks.

[0303] As described above, Figure 22 in a case where if the first block division mode is used, the first block is equally divided into two sub-blocks in the vertical direction, and if the second block division mode is used, the second block adjacent to the first block in the horizontal direction is equally divided into two sub-blocks in the vertical direction, the second block division mode is not selected as a candidate.

[0304] Figure 23 This shows an example in Embodiment 2 where the second block division mode is not selected for the division of an N×N pixel block as shown in step (2c). As Figure 23As shown, it is possible to use the first splitting method (i) to split an N×2N pixel (e.g., 8×16 pixels, and any value that is an integer multiple of 4 from 8 to 128 can be taken as the value of "N") into sub-blocks of N×N / 2 pixels, sub-blocks of N×N pixels, and sub-blocks of N×N / 2 pixels (e.g., sub-blocks of 8×4 pixels, sub-blocks of 8×8 pixels, and sub-blocks of 8×4 pixels) as in step (1a). Also, it is possible to use the second splitting method to split it into two sub-blocks of N×N pixels as in step (2a). In the first splitting method (i), it is possible to split the central block of N×N pixels into two sub-blocks of N×N / 2 pixels as in step (1b). In the second splitting method (ii), when the upper N×N pixel block (the first block) is horizontally split into two sub-blocks of N×N / 2 pixels as in step (2b), the block splitting mode in which the lower N×N pixel block (the second block) is horizontally split into two sub-blocks of N×N / 2 pixels in step (2c) is not selected as a candidate for the possible block splitting mode. This is because sub-blocks of the same size as those obtained by the first splitting method (i) will be generated, i.e., four sub-blocks of N×N / 2 pixels.

[0305] As described above, Figure 23 in the case where if the first block splitting mode is used, the first block is equally split into two sub-blocks in the horizontal direction, and if the second block splitting mode is used, the second block adjacent to the first block in the vertical direction is equally split into two sub-blocks in the horizontal direction, the second block splitting mode is not selected as a candidate.

[0306] If it is determined that the second block splitting mode is selected as a candidate for splitting the second block (No in S2003), then in step S2004, the second parameter is decoded from the bitstream, and a block splitting mode is selected from among multiple block splitting modes including the second block splitting mode as a candidate.

[0307] If it is determined that the second block splitting mode is not selected as a candidate for splitting the second block (Yes in S2003), then in step S2005, a block splitting mode different from the second block splitting mode is selected for splitting the second block. The block splitting mode selected here splits the block into sub-blocks having a different shape or different size compared to the sub-blocks generated by the second block splitting mode.

[0308] Figure 24 An example is shown in Embodiment 2 of splitting a 2N×N pixel block using the block splitting mode selected when the second block splitting mode is not selected as in step (3). As Figure 24 shown, the selected block splitting mode can split the current 2N×N pixel block (the lower block in this example) as Figure 24is divided into three sub - blocks as shown in (c) and (f). The sizes of the three sub - blocks can be different. For example, among the three sub - blocks, the large sub - block can have twice the width / height of the small sub - block. And for example, the selected block - partitioning pattern can also partition the current block as Figure 24 is divided into two sub - blocks with different sizes (asymmetric binary tree) as shown in (a), (b), (d) and (e). For example, in the case of using an asymmetric binary tree, the large sub - block can have three times the width / height of the small sub - block.

[0309] Figure 25 represents an example of using the selected block - partitioning pattern to partition a block of N×2N pixels without selecting the second block - partitioning pattern as shown in step (3) in Embodiment 2. As Figure 25 shown, the selected block - partitioning pattern can partition the current block of N×2N pixels (the right block in this example) as Figure 25 is divided into three sub - blocks as shown in (c) and (f). The sizes of the three sub - blocks can be different. For example, among the three sub - blocks, the large sub - block can have twice the width / height of the small sub - block. And for example, the selected block - partitioning pattern can also partition the current block as Figure 25 is divided into two sub - blocks with different sizes (asymmetric binary tree) as shown in (a), (b), (d) and (e). For example, in the case of using an asymmetric binary tree, the large sub - block can have three times the width / height of the small sub - block.

[0310] Figure 26 represents an example of using the selected block - partitioning pattern to partition a block of N×N pixels without selecting the second block - partitioning pattern as shown in step (3) in Embodiment 2. As Figure 26 shown, in step (1), the block of 2N×N pixels is vertically divided into two sub - blocks of N×N pixels. In step (2), the left block of N×N pixels is vertically divided into two sub - blocks of N / 2×N pixels. In step (3), the selected block - partitioning pattern for the current block of N×N pixels (the left block in this example) can be used to divide the current block into three sub - blocks as Figure 26 is divided into three sub - blocks as shown in (c) and (f). The sizes of the three sub - blocks can be different. For example, among the three sub - blocks, the large sub - block can have twice the width / height of the small sub - block. And for example, the selected block - partitioning pattern can also partition the current block as Figure 26 is divided into two sub - blocks with different sizes (asymmetric binary tree) as shown in (a), (b), (d) and (e). For example, in the case of using an asymmetric binary tree, the large sub - block can have three times the width / height of the small sub - block.

[0311] Figure 27 represents an example of using the selected block - partitioning pattern to partition a block of N×N pixels without selecting the second block - partitioning pattern as shown in step (3) in Embodiment 2. As Figure 27As shown, in step (1), a block of N×2N pixels is horizontally divided into two sub-blocks of N×N pixels. In step (2), the upper block of N×N pixels is horizontally divided into two sub-blocks of N×N / 2 pixels. In step (3), the selected partitioning mode for the current block of N×N pixels (the lower block in this example) can be used to divide the current block into three sub-blocks as shown in (c) and (f) of Figure 27 . The sizes of the three sub-blocks can be different. For example, in the three sub-blocks, the large sub-block has twice the width / height of the small sub-block. And for example, the selected partitioning mode can also divide the current block into two sub-blocks of different sizes (asymmetric binary tree) as shown in (a), (b), (d), and (e) of Figure 27 . For example, in the case of using an asymmetric binary tree, the large sub-block can have three times the width / height of the small sub-block.

[0312] Figure 17 Indicates the possible positions of the first parameter within the compressed video stream. As shown in Figure 17 , the first parameter can be configured in the video parameter set, sequence parameter set, picture parameter set, slice header, or coding tree unit. The first parameter can represent a method for dividing a block into multiple sub-blocks. For example, the first parameter can include a flag indicating whether to divide the block horizontally or vertically. The first parameter can also include a parameter indicating whether to divide the block into more than two sub-blocks.

[0313] Figure 18 Indicates the possible positions of the second parameter within the compressed video stream. As shown in Figure 18 , the second parameter can be configured in the video parameter set, sequence parameter set, picture parameter set, slice header, or coding tree unit. The second parameter can represent a method for dividing a block into multiple sub-blocks. For example, the second parameter can include a flag indicating whether to divide the block horizontally or vertically. The second parameter can also include a parameter indicating whether to divide the block into more than two sub-blocks. As shown in Figure 19 , the second parameter is configured in the bitstream following the first parameter.

[0314] The first block and the second block are different blocks. The first block and the second block can also be included in the same frame. For example, the first block can be a block adjacent above the second block. And for example, the first block can also be a block adjacent to the left of the second block.

[0315] In step S2006, the second block is divided into sub-blocks using the selected partitioning mode. In step S2007, the divided blocks are decoded.

[0316] [Decoding device]

[0317] Figure 16It is a block diagram showing the configuration of the video / image decoding device according to Embodiment 2 or 3.

[0318] The video decoding device 6000 is a device for decoding an input encoded bitstream for each block and outputting a video / image. The video decoding device 6000 is as Figure 16 shown, and includes an entropy decoding unit 6001, an inverse quantization unit 6002, an inverse transform unit 6003, a block memory 6004, a frame memory 6005, an intra prediction unit 6006, an inter prediction unit 6007, and a block segmentation determination unit 6008.

[0319] The input encoded bitstream is input to the entropy decoding unit 6001. After the input encoded bitstream is input to the entropy decoding unit 6001, the entropy decoding unit 6001 decodes the input encoded bitstream, outputs the parameters to the block segmentation determination unit 6008, and outputs the decoded value to the inverse quantization unit 6002.

[0320] The inverse quantization unit 6002 performs inverse quantization on the decoded value and outputs the frequency coefficients to the inverse transform unit 6003. The inverse transform unit 6003 performs an inverse frequency transform on the frequency coefficients based on the block partitioning pattern derived by the block segmentation determination unit 6008, transforms the frequency coefficients into sample values, and outputs the sample values to the adder. The block partitioning pattern can be associated with a block partitioning pattern, a block partitioning type, or a block partitioning direction. The adder adds the sample values to the predicted video / image values output from the intra / inter prediction units 6006 and 6007, outputs the added value to the display, and outputs the added value to the block memory 6004 or the frame memory 6005 for further prediction. The block segmentation determination unit 6008 collects block information from the block memory 6004 or the frame memory 6005, and uses the parameters decoded by the entropy decoding unit 6001 to derive the block partitioning pattern. If the derived block partitioning pattern is used, the block is divided into a plurality of sub-blocks. Furthermore, the intra / inter prediction units 6006 and 6007 perform prediction on the video / image region of the block to be decoded based on the video / image stored in the block memory 6004 or the video / image in the frame memory 6005 reconstructed according to the block partitioning pattern derived by the block segmentation determination unit 6008.

[0321] (Embodiment 3)

[0322] Refer to Figure 13 and Figure 14 to specifically describe the encoding process and decoding process according to Embodiment 3. Refer to Figure 15 and Figure 16 to specifically describe the encoding device and decoding device according to Embodiment 3.

[0323] [Encoding Process]

[0324] Figure 13Indicates the video encoding process related to Embodiment 3.

[0325] First, in step S3001, a first parameter is written to the bitstream, and the first parameter identifies, from among a plurality of block types, the block type used to divide the first block into sub-blocks.

[0326] In the next step S3002, a second parameter indicating the block division direction is written to the bitstream. The second parameter is configured in the bitstream following the first parameter. The block type and the block division direction can form a block division pattern together. The divided block indicates the number and division ratio of sub-blocks used to divide the block.

[0327] Figure 29 Shows an example of the block type and block division direction used to divide an N×N pixel block in Embodiment 3. In Figure 29 , (1), (2), (3), and (4) are different block types, (1a), (2a), (3a), and (4a) are block division patterns with different block types in the vertical division direction, and (1b), (2b), (3b), and (4b) are block division patterns with different block types in the horizontal division direction. As Figure 29 shown, when the division ratio is 1:1 and the N×N pixel block is divided along the vertical direction by a symmetric binary tree (i.e., 2 sub-blocks), the N×N pixel block is divided using the block division pattern (1a). When the division ratio is 1:1 and the N×N pixel block is divided along the horizontal direction by a symmetric binary tree (i.e., 2 sub-blocks), the N×N pixel block is divided using the block division pattern (1b). When the division ratio is 1:3 and the N×N pixel block is divided along the vertical direction by an asymmetric binary tree (i.e., 2 sub-blocks), the N×N pixel block is divided using the block division pattern (2a). When the division ratio is 1:3 and the N×N pixel block is divided along the horizontal direction by an asymmetric binary tree (i.e., 2 sub-blocks), the N×N pixel block is divided using the block division pattern (2b). When the division ratio is 3:1 and the N×N pixel block is divided along the vertical direction by an asymmetric binary tree (i.e., 2 sub-blocks), the N×N pixel block is divided using the block division pattern (3a). When the division ratio is 3:1 and the N×N pixel block is divided along the horizontal direction by an asymmetric binary tree (i.e., 2 sub-blocks), the N×N pixel block is divided using the block division pattern (3b). When the division ratio is 1:2:1 and the N×N pixel block is divided along the vertical direction by a ternary tree (i.e., 3 sub-blocks), the N×N pixel block is divided using the block division pattern (4a). When the division ratio is 1:2:1 and the N×N pixel block is divided along the horizontal direction by a ternary tree (i.e., 3 sub-blocks), the N×N pixel block is divided using the block division pattern (4b).

[0328] Figure 17 Shows the possible positions of the first parameter in the compressed video stream. As Figure 17As shown, the first parameter can be configured within a video parameter set, a sequence parameter set, a picture parameter set, a slice header, or a coding tree unit. The first parameter can represent a method of dividing a block into multiple sub-blocks. For example, the first parameter can include a flag indicating whether to divide the block horizontally or vertically. The first parameter can also include a parameter indicating whether to divide the block into more than two sub-blocks.

[0329] Figure 18 Indicates the possible positions of the second parameter within the compressed video stream. As Figure 18 shown, the second parameter can be configured within a video parameter set, a sequence parameter set, a picture parameter set, a slice header, or a coding tree unit. The second parameter can represent a method of dividing a block into multiple sub-blocks. For example, the second parameter can include a flag indicating whether to divide the block horizontally or vertically. The second parameter can also include a parameter indicating whether to divide the block into more than two sub-blocks. As Figure 19 shown, the second parameter is configured in the bitstream following the first parameter.

[0330] Figure 30 Indicates the advantages of encoding the block type before the block direction compared to encoding the block direction before the block type. In this example, when the horizontal block direction is invalidated due to an unsupported size (16×2 pixels), there is no need to encode the block direction. In this example, the block direction is determined to be the vertical block direction, and the horizontal block direction is invalidated. When encoding the block type before the block direction, compared to encoding the block direction before the block type, the code bits introduced by encoding the block direction are suppressed.

[0331] In this way, it is also possible to determine whether a block can be divided horizontally and vertically based on pre-determined conditions for block divisibility or non-divisibility. Then, when it is determined that the block can be divided only in one of the horizontal and vertical directions, the writing of the block direction to the bitstream can be skipped. Furthermore, when it is determined that the block is not divisible in both the horizontal and vertical directions, in addition to skipping the writing of the block direction to the bitstream, the writing of the block type to the bitstream can also be skipped.

[0332] The pre-determined conditions for block divisibility or non-divisibility are defined, for example, by the size (number of pixels) or the number of divisions. The conditions for block divisibility or non-divisibility can also be pre-defined in the standard specification. Also, the conditions for block divisibility or non-divisibility can be included in a video parameter set, a sequence parameter set, a picture parameter set, a slice header, or a coding tree unit. The conditions for block divisibility or non-divisibility can be fixed for all blocks, or can be dynamically switched according to the characteristics of the block (e.g., luminance and chrominance blocks) or the characteristics of the picture (e.g., I, P, B pictures), etc.

[0333] In step S3003, the block is divided into sub-blocks using the identified block type and the indicated block direction. In step S3004, the divided blocks are encoded.

[0334] [Encoding device]

[0335] Figure 15 is a block diagram showing the structure of the video / image encoding device according to Embodiment 2 or 3.

[0336] The video encoding device 5000 is a device that encodes an input video / image for each block and generates an encoded output bitstream. As Figure 15 shown, the video encoding device 5000 includes a transform unit 5001, a quantization unit 5002, an inverse quantization unit 5003, an inverse transform unit 5004, a block memory 5005, a frame memory 5006, an intra prediction unit 5007, an inter prediction unit 5008, an entropy encoding unit 5009, and a block division determination unit 5010.

[0337] The input video is input to the adder, and the added value is output to the transform unit 5001. The transform unit 5001 transforms the added value into frequency coefficients based on the block division type and direction derived by the block division determination unit 5010, and outputs the frequency coefficients to the quantization unit 5002. The block division type and direction can be associated with the block division mode, block division type, or block division direction. The quantization unit 5002 quantizes the input quantization coefficients and outputs the quantization values to the inverse quantization unit 5003 and the entropy encoding unit 5009.

[0338] The inverse quantization unit 5003 inverse-quantizes the quantization values output from the quantization unit 5002 and outputs the frequency coefficients to the inverse transform unit 5004. The inverse transform unit 5004 performs an inverse frequency transform on the frequency coefficients based on the block division type and direction derived by the block division determination unit 5010, transforms the frequency coefficients into sample values of the bitstream, and outputs the sample values to the adder.

[0339] The adder adds the sample values of the bitstream output from the intra-frame / inter-frame prediction units 5007 and 5008 to the predicted video / image values, and outputs the added values to the block memory 5005 or the frame memory 5006 for further prediction. The block segmentation determination unit 5010 collects block information from the block memory 5005 or the frame memory 5006, and derives the block segmentation type and direction, as well as parameters related to the block segmentation type and direction. If the derived block segmentation type and direction are used, the block is divided into a plurality of sub-blocks. The intra-frame / inter-frame prediction units 5007 and 5008 search among the video / images stored in the block memory 5005 or the video / images in the frame memory 5006 reconstructed according to the block segmentation type and direction derived by the block segmentation determination unit 5010, and estimate, for example, the video / image region most similar to the input video / image to be predicted.

[0340] The entropy encoding unit 5009 encodes the quantization values output from the quantization unit 5002, and encodes the parameters from the block segmentation determination unit 5010, and outputs a bitstream.

[0341] [Decoding process]

[0342] Figure 14 Represents the video decoding process related to Embodiment 3.

[0343] First, in step S4001, the first parameter is read from the bitstream, and the first parameter identifies the segmentation type for dividing the first block into sub-blocks from among a plurality of segmentation types.

[0344] In the next step S4002, the second parameter indicating the segmentation direction is read from the bitstream. The second parameter follows the first parameter in the bitstream. The segmentation type may also form a segmentation pattern together with the segmentation direction. The segmentation type indicates the number and segmentation ratio of sub-blocks for dividing the block.

[0345] Figure 29 Shows an example of the segmentation type and segmentation direction for dividing an N×N pixel block in Embodiment 3. In Figure 29 ,(1), (2), (3) and (4) are different segmentation types, (1a), (2a), (3a) and (4a) are segmentation patterns with different segmentation types in the vertical segmentation direction, and (1b), (2b), (3b) and (4b) are segmentation patterns with different segmentation types in the horizontal segmentation direction. As Figure 29As shown, when the block ratio is 1:1 and the N×N pixel block is divided along the vertical direction by a symmetric binary tree (i.e., 2 sub-blocks), the N×N pixel block is divided using the block partitioning mode (1a). When the block ratio is 1:1 and the N×N pixel block is divided along the horizontal direction by a symmetric binary tree (i.e., 2 sub-blocks), the N×N pixel block is divided using the block partitioning mode (1b). When the block ratio is 1:3 and the N×N pixel block is divided along the vertical direction by an asymmetric binary tree (i.e., 2 sub-blocks), the N×N pixel block is divided using the block partitioning mode (2a). When the block ratio is 1:3 and the N×N pixel block is divided along the horizontal direction by an asymmetric binary tree (i.e., 2 sub-blocks), the N×N pixel block is divided using the block partitioning mode (2b). When the block ratio is 3:1 and the N×N pixel block is divided along the vertical direction by an asymmetric binary tree (i.e., 2 sub-blocks), the N×N pixel block is divided using the block partitioning mode (3a). When the block ratio is 3:1 and the N×N pixel block is divided along the horizontal direction by an asymmetric binary tree (i.e., 2 sub-blocks), the N×N pixel block is divided using the block partitioning mode (3b). When the block ratio is 1:2:1 and the N×N pixel block is divided along the vertical direction by a ternary tree (i.e., 3 sub-blocks), the N×N pixel block is divided using the block partitioning mode (4a). When the block ratio is 1:2:1 and the N×N pixel block is divided along the horizontal direction by a ternary tree (i.e., 3 sub-blocks), the N×N pixel block is divided using the block partitioning mode (4b).

[0346] Figure 17 Represents the possible positions of the first parameter within the compressed video stream. As Figure 17 shown, the first parameter can be configured in the video parameter set, sequence parameter set, picture parameter set, slice header, or coding tree unit. The first parameter can represent a method for dividing a block into multiple sub-blocks. For example, the first parameter can include an identifier for the above-mentioned block partitioning types. For example, the first parameter can include a flag indicating whether to divide the block horizontally or vertically. The first parameter can also include a parameter indicating whether to divide the block into more than 2 sub-blocks.

[0347] Figure 18 Represents the possible positions of the second parameter within the compressed video stream. As Figure 18 shown, the second parameter can be configured in the video parameter set, sequence parameter set, picture parameter set, slice header, or coding tree unit. The second parameter can represent a method for dividing a block into multiple sub-blocks. For example, the second parameter can include a flag indicating whether to divide the block horizontally or vertically. That is, the second parameter can include a parameter indicating the block partitioning direction. The second parameter can also include a parameter indicating whether to divide the block into more than 2 sub-blocks. As Figure 19 shown, the second parameter is configured in the bitstream following the first parameter.

[0348] Figure 30Shows the advantages of encoding the block type before the block direction compared to the case of encoding the block direction before the block type. In this example, when the block direction in the horizontal direction is invalidated due to an unsupported size (16×2 pixels), there is no need to encode the block direction. In this example, the block direction is determined to be the block direction in the vertical direction, and the block direction in the horizontal direction is invalidated. Encoding the block type before the block direction suppresses the code bits caused by the encoding of the block direction compared to the case of encoding the block direction before the block type.

[0349] In this way, it is also possible to determine whether a block can be divided in the horizontal and vertical directions respectively based on pre-determined conditions for block divisibility or indivisibility. Then, when it is determined that the block can be divided only in one of the horizontal and vertical directions, the reading of the block direction from the bitstream can also be skipped. Furthermore, when it is determined that the block cannot be divided in both the horizontal and vertical directions, in addition to skipping the reading of the block direction, the reading of the block type from the bitstream can also be skipped.

[0350] The pre-determined conditions for block divisibility or indivisibility are defined by, for example, size (number of pixels) or number of divisions. These conditions for block divisibility or indivisibility can also be pre-defined in the standard specification. Also, the conditions for block divisibility or indivisibility can be included in the video parameter set, sequence parameter set, picture parameter set, slice header, or coding tree unit. The conditions for block divisibility or indivisibility can be fixed for all blocks, or can be dynamically switched according to the characteristics of the block (e.g., luminance and chrominance blocks) or the characteristics of the picture (e.g., I, P, B pictures), etc.

[0351] In step S4003, the block is divided into sub-blocks using the identified block type and the indicated division direction. In step S4004, the divided block is decoded.

[0352] [Decoding device]

[0353] Figure 16 Is a block diagram showing the structure of the video / image decoding device according to Embodiment 2 or 3.

[0354] The video decoding device 6000 is a device for decoding an input encoded bitstream for each block and outputting a video / image. The video decoding device 6000 is as Figure 16 shown, and includes an entropy decoding unit 6001, an inverse quantization unit 6002, an inverse transform unit 6003, a block memory 6004, a frame memory 6005, an intra prediction unit 6006, an inter prediction unit 6007, and a block division determination unit 6008.

[0355] The input encoded bitstream is input to the entropy decoding unit 6001. After the input encoded bitstream is input to the entropy decoding unit 6001, the entropy decoding unit 6001 decodes the input encoded bitstream, outputs the parameters to the block segmentation determination unit 6008, and outputs the decoded values to the inverse quantization unit 6002.

[0356] The inverse quantization unit 6002 performs inverse quantization on the decoded values and outputs the frequency coefficients to the inverse transform unit 6003. The inverse transform unit 6003 performs an inverse frequency transform on the frequency coefficients based on the block partitioning type and direction derived by the block segmentation determination unit 6008, transforms the frequency coefficients into sample values, and outputs the sample values to the adder. The block partitioning type and direction can be associated with the block partitioning mode, block partitioning type, or block partitioning direction. The adder adds the sample values to the predicted video / image values output from the intra / inter prediction units 6006 and 6007, outputs the added values to the display, and outputs the added values to the block memory 6004 or the frame memory 6005 for further prediction. The block segmentation determination unit 6008 collects block information from the block memory 6004 or the frame memory 6005 and derives the block partitioning type and direction using the parameters decoded by the entropy decoding unit 6001. If the derived block partitioning type and direction are used, the block is divided into multiple sub-blocks. Furthermore, the intra / inter prediction units 6006 and 6007 perform prediction on the video / image region of the block to be decoded based on the video / image stored in the block memory 6004 or the video / image in the frame memory 6005 reconstructed according to the block partitioning type and direction derived by the block segmentation determination unit 6008.

[0357] (Embodiment 4)

[0358] In the above embodiments, each functional block can generally be implemented by an MPU and a memory, etc. In addition, the processing of each functional block is generally implemented by a program execution unit such as a processor reading and executing software (program) recorded in a recording medium such as a ROM. This software can be distributed by downloading, etc., or can be recorded in a recording medium such as a semiconductor memory for distribution. Additionally, of course, each functional block can also be implemented by hardware (special-purpose circuit).

[0359] In addition, the processing described in each embodiment can be implemented by centralized processing using a single device (system), or can also be implemented by distributed processing using multiple devices. In addition, the processor that executes the above program can be single or multiple. That is, either centralized processing or distributed processing can be performed.

[0360] The form of the present invention is not limited to the above embodiments, and various modifications can be made, and they are also included in the scope of the form of the present invention.

[0361] Next, application examples of the moving image encoding method (image encoding method) or moving image decoding method (image decoding method) shown in the above-described embodiments and a system using the same will be described. The system is characterized by including an image encoding device that uses the image encoding method, an image decoding device that uses the image decoding method, and an image encoding / decoding device that includes both. Regarding other configurations in the system, they can be appropriately changed as the case may be.

[0362] [Usage Example]

[0363] Figure 33 FIG. shows the overall configuration of a content supply system ex100 that implements a content distribution service. The provision of the communication service is divided into desired sizes, and base stations ex106, ex107, ex108, ex109, and ex110 that are fixed wireless stations are respectively provided in each unit.

[0364] In this content supply system ex100, various devices such as a computer ex111, a game machine ex112, a camera ex113, home appliances ex114, and a smart phone ex115 are connected via the Internet ex101 through an Internet service provider ex102 or a communication network ex104, and base stations ex106 to ex110. The content supply system ex100 may also connect by combining some of the above elements. Each device may be directly or indirectly connected to each other via a telephone network or short-range wireless without passing through the base stations ex106 to ex110 that are fixed wireless stations. In addition, a streaming media server ex103 is connected to various devices such as a computer ex111, a game machine ex112, a camera ex113, home appliances ex114, and a smart phone ex115 via the Internet ex101 and the like. In addition, the streaming media server ex103 is connected to terminals in a hotspot in an airplane ex117 via a satellite ex116.

[0365] In addition, a wireless access point or a hotspot or the like may be used instead of the base stations ex106 to ex110. In addition, the streaming media server ex103 may be directly connected to the communication network ex104 without passing through the Internet ex101 or the Internet service provider ex102, or may be directly connected to the airplane ex117 without passing through the satellite ex116.

[0366] The camera ex113 is a device such as a digital camera that can perform still image photography and moving image photography. In addition, the smart phone ex115 is a smart phone, a portable phone, or a PHS (Personal Handyphone System) or the like corresponding to the modes of mobile communication systems generally referred to as 2G, 3G, 3.9G, 4G, and 5G in the future.

[0367] The home appliance ex118 is a refrigerator or a device included in a household fuel cell cogeneration system, etc.

[0368] In the content supply system ex100, a terminal having a photographing function is connected to the streaming media server ex103 via a base station ex106 or the like, and thus live distribution or the like can be performed. In live distribution, terminals (such as a computer ex111, a game machine ex112, a camera ex113, a home appliance ex114, a smart phone ex115, and a terminal in an airplane ex117) perform the encoding process described in the above embodiments on the still image or moving image content captured by the user using the terminal, multiplex the video data obtained by encoding and the audio data obtained by encoding the sound corresponding to the video, and send the obtained data to the streaming media server ex103. That is, each terminal functions as an image encoding device according to one aspect of the present invention.

[0369] On the other hand, the streaming media server ex103 performs stream distribution on the content data sent by a requested client. The client is a computer ex111, a game machine ex112, a camera ex113, a home appliance ex114, a smart phone ex115, or a terminal in an airplane ex117 that can decode the data after the above encoding process. Each device that receives the distributed data performs a decoding process on the received data and reproduces it. That is, each device functions as an image decoding device according to one aspect of the present invention.

[0370] [Distributed processing]

[0371] In addition, the streaming media server ex103 may be multiple servers or multiple computers, and perform distributed processing or recording and distribution of data. For example, the streaming media server ex103 may be implemented by a CDN (Content Delivery Network), and content distribution is achieved through a network connecting many edge servers distributed in the world to each other. In a CDN, a physically closer edge server is dynamically allocated according to the client. And by caching and distributing the content to the edge server, the delay can be reduced. In addition, in the case of a certain error or a change in the communication state due to an increase in traffic or the like, the processing can be distributed among multiple edge servers, or the distribution entity can be switched to another edge server, or a part of the network with a fault can be bypassed and the distribution can continue, so high-speed and stable distribution can be achieved.

[0372] In addition, not limited to the distributed processing of distributing itself, the encoding process of the captured data can be performed by each terminal, on the server side, or can be shared among them. As an example, usually two processing loops are performed in the encoding process. In the first loop, the complexity or code amount of the image in units of frames or scenes is detected. In addition, in the second loop, a process is performed to improve the encoding efficiency while maintaining the image quality. For example, by performing the first encoding process by the terminal and the second encoding process by the server that receives the content, it is possible to reduce the processing load in each terminal while improving the quality and efficiency of the content. In this case, if there is a request to receive and decode almost in real time, the data completed by the first encoding performed by the terminal can also be received and reproduced by other terminals, so more flexible real-time distribution can also be performed.

[0373] As other examples, the camera ex113 etc. extracts feature amounts from the image, compresses the data regarding the feature amounts as metadata, and sends it to the server. The server, for example, judges the importance of the target based on the feature amounts and switches the quantization accuracy etc., and performs compression corresponding to the meaning of the image. The feature amount data is particularly effective for improving the accuracy and efficiency of motion vector prediction during re-compression in the server. In addition, simple encoding such as VLC (Variable Length Coding) can be performed by the terminal, and encoding with a large processing load such as CABAC (Context Adaptive Binary Arithmetic Coding) can be performed by the server.

[0374] As other examples, in a stadium, a shopping mall, a factory, etc., there are cases where there are multiple video data obtained by multiple terminals capturing substantially the same scene. In this case, the multiple terminals that performed the shooting are used, and other terminals and servers that did not perform the shooting are used as needed. For example, distributed processing is performed by respectively allocating the encoding process in units of GOP (Group of Picture), picture units, or tile units obtained by dividing the picture, etc. Thereby, it is possible to reduce the delay and better achieve real-time performance.

[0375] In addition, since the multiple video data are of substantially the same scene, the server can also manage and / or instruct to refer to the video data captured by each terminal with each other. Or, it can also be that the server receives the encoded data from each terminal and changes the reference relationship among the multiple data, or corrects or replaces the picture itself and re-encodes it. Thereby, it is possible to generate a stream with improved quality and efficiency of each data.

[0376] In addition, the server can also perform transcoding to change the encoding method of the video data and then distribute the video data. For example, the server can change the MPEG-like encoding method to the VP-like, or can change H.264 to H.265.

[0377] In this way, the encoding process can be performed by a terminal or one or more servers. Therefore, in the following descriptions, the "server" or "terminal" is used as the main body for performing the process. However, part or all of the processes performed by the server can also be performed by the terminal, and part or all of the processes performed by the terminal can also be performed by the server. In addition, the same applies to the decoding process for these.

[0378] [3D, Multi-angle]

[0379] In recent years, the situation of combining and using images or videos of different scenes captured by multiple cameras ex113 and / or terminals such as smartphones ex115 that are roughly synchronized with each other, or images or videos of the same scene captured from different angles has increased. The videos captured by each terminal are combined based on the relative position relationship between the terminals obtained separately, or the areas where the feature points included in the videos are consistent.

[0380] The server not only encodes two-dimensional moving images, but can also encode still images automatically or at a user-specified time based on scene analysis of the moving images and send them to the receiving terminal. When the server can obtain the relative position relationship between the shooting terminals, it can not only generate the three-dimensional shape of the scene based on the videos of the same scene captured from different angles, in addition to two-dimensional moving images. In addition, the server can also encode the three-dimensional data generated by a point cloud etc. separately, and can also select or reconstruct from the videos captured by multiple terminals based on the results of identifying or tracking a person or an object using the three-dimensional data to generate and send the video to the receiving terminal.

[0381] In this way, the user can not only arbitrarily select each video corresponding to each shooting terminal to view the scene, but also view the content of the video of an arbitrary viewpoint cut from the three-dimensional data reconstructed using multiple images or videos. Furthermore, similar to the video, the sound can also be collected from multiple different angles, and the server multiplexes and sends the sound from a specific angle or space with the video in accordance with the video.

[0382] In addition, in recent years, content that establishes a correspondence between the real world and the virtual world such as Virtual Reality (VR) and Augmented Reality (AR) has also been popularized. In the case of VR images, the server separately creates viewpoint images for the right eye and the left eye, and can perform encoding that allows reference between the viewpoint videos through Multi-View Coding (MVC) etc., or can encode them as different streams without referring to each other. When decoding different streams, it can be reproduced synchronously according to the user's viewpoint to reproduce a virtual three-dimensional space.

[0383] In the case of an AR image, it can also be that the server overlaps the virtual object information in the virtual space with the camera information in the real space based on the three-dimensional position or the movement of the user's viewpoint. The decoding device acquires or holds the virtual object information and the three-dimensional data, generates a two-dimensional image according to the movement of the user's viewpoint, and creates the overlapping data by smoothly connecting them. Alternatively, it can also be that the decoding device sends the movement of the user's viewpoint to the server in addition to the delegation of the virtual object information, and the server creates the overlapping data according to the three-dimensional data held in the server, matches the received movement of the viewpoint, encodes the overlapping data, and distributes it to the decoding device. In addition, the overlapping data has an α value representing the transmittance in addition to RGB, and the server sets the α value of the part other than the target created according to the three-dimensional data to 0, etc., and encodes it in a state where it is transmitted in this part. Alternatively, the server can also set the RGB value of a specified value as the background like chroma key and generate data with the part other than the target set as the background color.

[0384] Similarly, the decoding process of the distributed data can be performed by each terminal as a client, on the server side, or they can be shared with each other. As an example, it can also be that a certain terminal first sends a reception request to the server, and another terminal receives the content corresponding to the request and performs the decoding process, and sends the decoded signal to a device with a display. By dispersing the processing regardless of the performance of the communicable terminal itself and selecting appropriate content, it is possible to reproduce data with better image quality. In addition, as another example, a large-sized image data can be received by a TV or the like, and a personal terminal of the viewer decodes and displays a part of the area such as tiles after the picture is segmented. Thus, it is possible to share the overall image while confirming one's own responsible area or the area that one wants to confirm in more detail at hand.

[0385] In addition, it is envisioned that in the future, in a situation where multiple wireless communications of short, medium, or long distances can be used both indoors and outdoors, using a distribution system standard such as MPEG-DASH, seamless reception of content is performed while appropriately switching data for the connected communication. Thus, the user can not only freely select a decoding device or a display device such as a monitor installed indoors and outdoors with their own terminal, but also perform real-time switching. In addition, based on their own position information, etc., the decoding terminal and the display terminal can be switched for decoding. Thus, it is also possible to display map information on a part of the wall or floor of a building next to a displayable device while moving towards the destination. In addition, based on the ease of access to the encoded data on the network, such as the encoded data being cached in a server that can be accessed from the receiving terminal in a short time, or the encoded data being replicated in an edge server of the content distribution service, the bit rate of the received data can be switched.

[0386] [Scalable Coding]

[0387] Regarding the content switching, use Figure 34 As shown, a scalable stream compressed and encoded using the moving image encoding method shown in the above-described embodiments will be described. For the server, there may be multiple streams with the same content but different qualities as separate streams, or it may be a structure that switches content by utilizing the characteristics of a temporally / spatially scalable stream achieved by hierarchical encoding as shown in the figure. That is, the decoding side can freely switch between low-resolution content and high-resolution content for decoding by determining which layer to decode based on internal factors such as performance and external factors such as the state of the communication bandwidth. For example, when the subsequent video that was viewed on a smartphone ex115 while on the move is to be viewed later at home using a device such as an Internet TV, the device only needs to decode the same stream to different layers, thus reducing the burden on the server side.

[0388] Furthermore, in addition to the structure where pictures are encoded by each layer as described above to achieve the hierarchical nature where the enhancement layer exists above the base layer, it is also possible that the enhancement layer contains meta-information such as image-based statistical information, and the decoding side generates high-quality content by super-resolving the pictures in the base layer based on the meta-information. Super-resolution can be either an improvement in the signal-to-noise ratio at the same resolution or an expansion of the resolution. The meta-information includes information for determining linear or non-linear filter coefficients used in the super-resolution process, or information for determining parameter values in the filtering process, machine learning, or least squares operation used in the super-resolution process, etc.

[0389] Alternatively, it is also possible to divide pictures into tiles, etc. according to the meaning of objects, etc. within the image, and the decoding side decodes only a part of the area by selecting the tiles to be decoded. In addition, by saving the attributes of the object (person, car, ball, etc.) and the position within the image (coordinate position in the same image, etc.) as meta-information, the decoding side can determine the position of the desired object based on the meta-information and decide on the tiles including the object. For example, as Figure 35 shown, a data storage structure different from the pixel data, such as the SEI message in HEVC, is used to store the meta-information. This meta-information represents, for example, the position, size, or color of the main object.

[0390] In addition, the meta-information can also be stored in units composed of multiple pictures, such as streams, sequences, or random access units. Thereby, the decoding side can obtain the moment when a specific person appears within the video, etc., and by matching with the information of the picture unit, can determine the pictures where the object exists and the position of the object within the pictures.

[0391] [Optimization of Web Pages]

[0392] Figure 36 It is a diagram showing an example of a display screen of a web page in a computer ex111 or the like. Figure 37 It is a diagram showing an example of a display screen of a web page in a smartphone ex115 or the like. As Figure 36 and Figure 37 shown, there are cases where a web page contains multiple linked images that are links to image content, and the visible manner thereof varies depending on the viewing device. When multiple linked images can be seen on the screen, before the user explicitly selects a linked image, or before the linked image approaches near the center of the screen or the entire linked image enters the screen, the display device (decoding device) displays the still image or I-picture that each content has as a linked image, or displays an image such as a gif animation using multiple still images or I-pictures, or only receives the base layer and decodes and displays the image.

[0393] When a linked image is selected by the user, the display device decodes the base layer with the highest priority. In addition, if there is information indicating that the content is scalable in the HTML that constitutes the web page, the display device may also decode up to the enhancement layer. Furthermore, in order to ensure real-time performance or in cases where the communication bandwidth is extremely tight before selection, the display device can reduce the delay between the decoding time and the display time of the first picture (the delay from the start of content decoding to the start of display) by only decoding and displaying the pictures that are forward-referenced (I-pictures, P-pictures, B-pictures that are only forward-referenced). In addition, the display device can also forcibly ignore the reference relationship of the pictures and roughly decode all B-pictures and P-pictures as forward-referenced, and as the pictures received over time increase, perform normal decoding.

[0394] [Autonomous Driving]

[0395] In addition, when receiving still images or video data such as two-dimensional or three-dimensional map information for the autonomous driving or driving assistance of a vehicle, the receiving terminal can also receive information such as weather or construction information as meta-information in addition to the image data belonging to one or more layers, and decode them in correspondence. In addition, the meta-information can either belong to a layer or only be multiplexed with the image data.

[0396] In this case, since the vehicle, drone, or aircraft etc. that includes the receiving terminal is moving, the receiving terminal can switch the base stations ex106 to ex110 to perform seamless reception and decoding by sending the location information of the receiving terminal at the time of the reception request. In addition, the receiving terminal can dynamically switch the degree to which the meta-information is received or the degree to which the map information is updated according to the user's selection, the user's condition, or the state of the communication bandwidth.

[0397] As described above, in the content supply system ex100, the client can receive, decode, and reproduce the encoded information sent by the user in real time.

[0398] [Distribution of Personal Content]

[0399] In addition, in the content supply system ex100, not only high-quality, long-duration content provided by video distribution providers but also low-quality, short-duration content provided by individuals can be distributed via unicast or multicast. In addition, it is conceivable that such personal content will increase in the future. In order to make personal content better, the server can also perform an encoding process after an editing process. This can be achieved, for example, through the following structure.

[0400] During or after shooting in real time or cumulatively, the server performs recognition processing such as shooting error, scene search, meaning analysis, and object detection based on the original image or encoded data. And based on the recognition results, the server manually or automatically corrects focus deviation or camera shake, deletes scenes with low importance such as scenes with lower brightness or out-of-focus compared to other pictures, emphasizes the edges of the object, or changes the color tone, etc. The server encodes the edited data based on the editing results. In addition, it is known that the viewing rate decreases if the shooting time is too long. The server can also automatically crop scenes with low importance as described above and scenes with little movement, etc. based on the image processing results according to the shooting time to make the content within a specific time range. Or the server can also generate a summary and encode it based on the result of the meaning analysis of the scene.

[0401] In addition, there are cases where personal content in its original state contains content that infringes on copyright, the moral rights of the author, or the right of portrait, etc., and there are also inconvenient situations for individuals such as the sharing range exceeding the desired range. Therefore, for example, the server can also encode by forcibly changing the faces of people in the peripheral part of the screen or at home to out-of-focus images. In addition, the server can also identify whether a face of a person different from the pre-registered person is captured in the image to be encoded, and in the case of capture, perform processing such as applying a mosaic to the face part. Or, as pre-processing or post-processing of encoding, from the perspective of copyright, etc., the user designates the person or background area for which the image is to be processed, and the server performs processing such as replacing the designated area with another image or blurring the focus. In the case of a person, the image of the face part can be replaced while tracking the person in the moving image.

[0402] In addition, the real-time requirement for viewing and listening to personal content with a small amount of data is relatively strong. Therefore, although it also depends on the bandwidth, the decoding device first receives, decodes, and reproduces the base layer with the highest priority. The decoding device can also receive the enhancement layer during this period. In the case where the reproduction is looped and reproduced more than twice, for example, a high-quality image is reproduced including the enhancement layer. In this way, if it is a scalable-encoded stream, an experience can be provided where the moving image is rough at the stage when it is not selected or just started to be viewed, but the stream gradually becomes smooth and the image quality improves. In addition to scalable encoding, the same experience can also be provided when the first rough stream and the second stream encoded with reference to the moving image of the first time are configured as one stream.

[0403] [Other usage examples]

[0404] In addition, these encoding or decoding processes are usually processed in the LSIex500 possessed by each terminal. The LSIex500 can be either a single chip or a structure composed of multiple chips. In addition, software for moving image encoding or decoding can also be loaded into a certain recording medium (such as CD-ROM, floppy disk, hard disk, etc.) that can be read by a computer ex111, etc., and the encoding process and decoding process can be performed using this software. Furthermore, when the smart phone ex115 is equipped with a camera, it is also possible to transmit the moving image data obtained by this camera. The moving image data at this time is the data after being encoded by the LSIex500 possessed by the smart phone ex115.

[0405] In addition, the LSIex500 can also be a structure that downloads and activates application software. In this case, the terminal first determines whether the terminal corresponds to the encoding method of the content or has the ability to execute a specific service. When the terminal does not correspond to the encoding method of the content or does not have the ability to execute a specific service, the terminal downloads the codec or application software, and then performs content acquisition and reproduction.

[0406] In addition, it is not limited to the content supply system ex100 via the Internet ex101. It is also possible to incorporate at least one of the moving image encoding device (image encoding device) or the moving image decoding device (image decoding device) of the above-described embodiments in a digital broadcast system. Since the broadcast radio wave uses a satellite, etc. to carry multiplexed data that multiplexes video and audio and is transmitted and received, there is a difference suitable for multicast compared to the structure of the content supply system ex100 that is easy for unicast, but the same application can be performed for the encoding process and the decoding process.

[0407] [Hardware structure]

[0408] Figure 38 This is a diagram showing the smart phone ex115. In addition,Figure 39 This is a diagram showing the structural example of the smart phone ex115. The smart phone ex115 has an antenna ex450 for transmitting and receiving radio waves with the base station ex110, a camera unit ex465 capable of shooting images and still pictures, and a display unit ex458 for displaying the images shot by the camera unit ex465 and decoding the data such as the images received by the antenna ex450. The smart phone ex115 also has an operation unit ex466 such as a touch panel, a sound output unit ex457 such as a speaker for outputting sound or audio, a sound input unit ex456 such as a microphone for inputting sound, a memory unit ex467 capable of storing the shot images or still pictures, the recorded sound, the received images or still pictures, the encoded or decoded data such as emails, or a slot unit ex464 as an interface unit with the SIM ex468, and the SIM ex468 is used to identify the user and perform authentication for accessing various data represented by the network. In addition, an external memory can be used instead of the memory unit ex467.

[0409] In addition, the main control unit ex460 that comprehensively controls the display unit ex458, the operation unit ex466, etc. is interconnected with the power supply circuit unit ex461, the operation input control unit ex462, the video signal processing unit ex455, the camera interface unit ex463, the display control unit ex459, the modulation / demodulation unit ex452, the multiplexing / demultiplexing unit ex453, the audio signal processing unit ex454, the slot unit ex464, and the memory unit ex467 via the bus ex470.

[0410] If the power key is turned on by the user's operation, the power supply circuit unit ex461 starts the smart phone ex115 to an operable state by supplying power to each unit from the battery pack.

[0411] The intelligent phone ex115 performs processes such as calls and data communication under the control of the main control unit ex460 having a CPU, ROM, RAM, etc. During a call, the voice signal collection unit ex456 collects a voice signal, which is then converted into a digital voice signal by the voice signal processing unit ex454. The digital voice signal is subjected to spread spectrum processing by the modulation / demodulation unit ex452. After the digital-to-analog conversion processing and frequency conversion processing are performed by the transmission / reception unit ex451, the signal is transmitted via the antenna ex450. In addition, the received data is amplified and subjected to frequency conversion processing and analog-to-digital conversion processing. The modulation / demodulation unit ex452 performs inverse spread spectrum processing, and the voice signal processing unit ex454 converts it into an analog voice signal, which is then output from the voice output unit ex457. During data communication, text, still image, or video data is sent to the main control unit ex460 via the operation input control unit ex462 by operating the operation unit ex466 of the main body, and the sending and receiving processes are performed in the same way. In the data communication mode, when sending video, still images, or video and voice, the video signal processing unit ex455 compresses and encodes the video signal stored in the memory unit ex467 or the video signal input from the camera unit ex465 using the moving image encoding method shown in the above embodiments, and sends the encoded video data to the multiplexing / demultiplexing unit ex453. In addition, the voice signal processing unit ex454 encodes the voice signal collected by the voice input unit ex456 during the process of shooting video, still images, etc. by the camera unit ex465, and sends the encoded voice data to the multiplexing / demultiplexing unit ex453. The multiplexing / demultiplexing unit ex453 multiplexes the encoded video data and the encoded voice data in a specified manner, and the modulation / demodulation unit (modulation / demodulation circuit unit) ex452 and the transmission / reception unit ex451 perform modulation processing and conversion processing, and then transmit the signal via the antenna ex450.

[0412] When receiving an image attached to an email or a chat tool, or an image linked on a web page, etc., in order to decode the multiplexed data received via the antenna ex450, the multiplexing / demultiplexing unit ex453 separates the multiplexed data into a bitstream of video data and a bitstream of audio data by demultiplexing the multiplexed data, supplies the encoded video data to the video signal processing unit ex455 via the synchronization bus ex470, and supplies the encoded audio data to the audio signal processing unit ex454. The video signal processing unit ex455 decodes the video signal by a video decoding method corresponding to the moving image encoding method described in the above embodiments, and displays the video or still image included in the linked moving image file from the display unit ex458 via the display control unit ex459. In addition, the audio signal processing unit ex454 decodes the audio signal and outputs the sound from the audio output unit ex457. Also, since real-time streaming media is becoming popular, depending on the user's situation, there may be occasions where the reproduction of sound is inappropriate in society. Therefore, as an initial value, a structure that does not reproduce the audio signal but only reproduces the video data is preferred. It is also possible to reproduce the sound synchronously only when the user performs an operation such as clicking on the video data.

[0413] In addition, here, the smartphone ex115 is taken as an example for explanation, but as the terminal, three installation forms can be considered, namely, a transmitting terminal having only an encoder, a receiving terminal having only a decoder, in addition to the transceiver terminal having both an encoder and a decoder. Further, in the digital broadcast system, it is assumed that multiplexed data in which audio data, etc. are multiplexed in video data is received and transmitted for explanation, but in the multiplexed data, character data associated with the video, etc. can also be multiplexed in addition to the audio data, and it is also possible to receive or transmit the video data itself instead of the multiplexed data.

[0414] Also, it is assumed that the main control unit ex460 including the CPU controls the encoding or decoding process for explanation, but in many cases, the terminal has a GPU. Therefore, it is also possible to configure a structure in which the performance of the GPU is utilized to process a larger area together by a memory shared by the CPU and the GPU, or a memory that manages addresses in a shared manner. Thereby, the encoding time can be shortened, real-time performance can be ensured, and low latency can be achieved. In particular, it is more effective if the processes of motion estimation, deblocking filtering, SAO (Sample Adaptive Offset), and transform / quantization are performed together by the GPU in units of pictures instead of by the CPU.

[0415] The encoding device according to an embodiment of the present disclosure may also be an encoding device that encodes a picture, and includes a processor and a memory; the above-mentioned processor has: a block division determination unit that divides the above-mentioned picture read from the above-mentioned memory into a plurality of blocks by using a set of block division patterns obtained by combining one or more block division patterns, and the above-mentioned block division pattern defines a division type; and an encoding unit that encodes the above-mentioned plurality of blocks; the above-mentioned set of block division patterns is composed of a first block division pattern and a second block division pattern, the above-mentioned first block division pattern defines a division direction and a division number for dividing a first block, and the above-mentioned second block division pattern defines a division direction and a division number for dividing a second block, which is one of the blocks obtained after dividing the above-mentioned first block; in the above-mentioned block division determination unit, when the above-mentioned division number of the above-mentioned first block division pattern is 3, the above-mentioned second block is the central block among the blocks obtained after dividing the above-mentioned first block, and the above-mentioned division direction of the above-mentioned second block division pattern is the same as the above-mentioned division direction of the above-mentioned first block division pattern, the above-mentioned second block division pattern only includes a block division pattern with a division number of 3.

[0416] The parameter for identifying the above-mentioned second block division pattern in the encoding device according to an embodiment of the present disclosure may also include a first flag indicating in which direction, the horizontal direction or the vertical direction, the above-mentioned block is divided, and does not include a second flag indicating the division number for dividing the above-mentioned block.

[0417] The encoding device according to an embodiment of the present disclosure may also be an encoding device that encodes a picture, and includes a processor and a memory; the above-mentioned processor has: a block division determination unit that divides the above-mentioned picture read from the above-mentioned memory into a plurality of blocks by using a set of block division patterns obtained by combining one or more block division patterns, and the above-mentioned block division pattern defines a division type; and an encoding unit that encodes the above-mentioned plurality of blocks; the above-mentioned set of block division patterns is composed of a first block division pattern and a second block division pattern, the above-mentioned first block division pattern defines a division direction and a division number for dividing a first block, and the above-mentioned second block division pattern defines a division direction and a division number for dividing a second block, which is one of the blocks obtained after dividing the above-mentioned first block; when the above-mentioned division number of the above-mentioned first block division pattern is 3, the above-mentioned second block is the central block among the blocks obtained after dividing the above-mentioned first block, and the above-mentioned division direction of the above-mentioned second block division pattern is the same as the above-mentioned division direction of the above-mentioned first block division pattern, the above-mentioned block division determination unit does not use the above-mentioned second block division pattern with a division number of 2.

[0418] The encoding device according to an embodiment of the present disclosure may also be an encoding device that encodes an image, and includes a processor and a memory; the processor has: a block division determination unit that divides the image read from the memory into a plurality of blocks using a set of block division patterns obtained by combining one or more block division patterns, where the block division patterns define division types; and an encoding unit that encodes the plurality of blocks; the set of block division patterns includes a first block division pattern and a second block division pattern that respectively define a division direction and a division number; the block division determination unit restricts the use of the second block division pattern with the division number of 2.

[0419] The parameter for identifying the second block division pattern in the encoding device according to an embodiment of the present disclosure may also include a first flag indicating in which direction, horizontal or vertical, the block is divided, and a second flag indicating whether the block is divided into two or more.

[0420] The above parameter in the encoding device according to an embodiment of the present disclosure may also be configured in slice data.

[0421] The encoding device according to an embodiment of the present disclosure may also be an encoding device that encodes an image, and includes a processor and a memory; the processor has: a block division determination unit that divides the image read from the memory into a block set composed of a plurality of blocks using a set of block division patterns obtained by combining one or more block division patterns, where the block division patterns define division types; and an encoding unit that encodes the plurality of blocks; when the first block set obtained using the first set of block division patterns is the same as the second block set obtained using the second set of block division patterns, the block division determination unit divides using only one of the first set of block division patterns and the second set of block division patterns.

[0422] The block division determination unit in the encoding device according to an embodiment of the present disclosure may also divide using the block division pattern set with the smaller of the first code amount of the first set of block division patterns and the second code amount of the second set of block division patterns based on the first code amount of the first set of block division patterns and the second code amount of the second set of block division patterns.

[0423] The block division determination unit in the encoding device according to an embodiment of the present disclosure may also divide using the block division pattern set that appears first in a preset order in the first set of block division patterns and the second set of block division patterns when the first code amount is equal to the second code amount based on the first code amount of the first set of block division patterns and the second code amount of the second set of block division patterns.

[0424] The decoding device according to an embodiment of the present disclosure may also be a decoding device that decodes an encoded signal, and includes a processor and a memory; the above-mentioned processor has: a block division determination unit that divides the above-mentioned encoded signal read from the above-mentioned memory into a plurality of blocks by using a set of block division patterns obtained by combining one or more block division patterns, and the above-mentioned block division pattern defines a division type; and a decoding unit that decodes the above-mentioned plurality of blocks; the above-mentioned set of block division patterns is composed of a first block division pattern and a second block division pattern, the above-mentioned first block division pattern defines a division direction and a division number for dividing a first block, and the above-mentioned second block division pattern defines a division direction and a division number for dividing a second block, which is one of the blocks obtained after the division of the above-mentioned first block; in the above-mentioned block division determination unit, when the above-mentioned division number of the above-mentioned first block division pattern is 3, the above-mentioned second block is the central block among the blocks obtained after the division of the above-mentioned first block, and the above-mentioned division direction of the above-mentioned second block division pattern is the same as the above-mentioned division direction of the above-mentioned first block division pattern, the above-mentioned second block division pattern only includes a block division pattern with a division number of 3.

[0425] The parameter for identifying the above-mentioned second block division pattern in the decoding device according to an embodiment of the present disclosure may also include a first flag indicating in which direction, the horizontal direction or the vertical direction, the above-mentioned block is divided, and does not include a second flag indicating the division number for dividing the above-mentioned block.

[0426] The decoding device according to an embodiment of the present disclosure may also be a decoding device that decodes an encoded signal, and includes a processor and a memory; the above-mentioned processor has: a block division determination unit that divides the above-mentioned encoded signal read from the above-mentioned memory into a plurality of blocks by using a set of block division patterns obtained by combining one or more block division patterns, and the above-mentioned block division pattern defines a division type; and a decoding unit that decodes the above-mentioned plurality of blocks; the above-mentioned set of block division patterns is composed of a first block division pattern and a second block division pattern, the above-mentioned first block division pattern defines a division direction and a division number for dividing a first block, and the above-mentioned second block division pattern defines a division direction and a division number for dividing a second block, which is one of the blocks obtained after the division of the above-mentioned first block; the above-mentioned block division determination unit does not use the above-mentioned second block division pattern with a division number of 2 when the above-mentioned division number of the above-mentioned first block division pattern is 3, the above-mentioned second block is the central block among the blocks obtained after the division of the above-mentioned first block, and the above-mentioned division direction of the above-mentioned second block division pattern is the same as the above-mentioned division direction of the above-mentioned first block division pattern.

[0427] The decoding device according to an embodiment of the present disclosure may also be a decoding device that decodes an encoded signal, and includes a processor and a memory; the processor has: a block division determination unit that divides the encoded signal read from the memory into a plurality of blocks using a set of block division patterns obtained by combining one or more block division patterns, where the block division patterns define division types; and a decoding unit that decodes the plurality of blocks; the set of block division patterns includes a first block division pattern and a second block division pattern that respectively define a division direction and a division quantity; the block division determination unit restricts the use of the second block division pattern with the division quantity of 2.

[0428] The parameter for identifying the second block division pattern in the decoding device according to an embodiment of the present disclosure may also include a first flag indicating in which direction, horizontal or vertical, the block is divided, and a second flag indicating whether the block is divided into two or more.

[0429] The above parameter in the decoding device according to an embodiment of the present disclosure may also be configured in slice data.

[0430] The decoding device according to an embodiment of the present disclosure may also be a decoding device that decodes an encoded signal, and includes a processor and a memory; the processor has: a block division determination unit that divides the encoded signal read from the memory into a block set composed of a plurality of blocks using a set of block division patterns obtained by combining one or more block division patterns, where the block division patterns define division types; and a decoding unit that decodes the plurality of blocks; when the first block set obtained using the first set of block division patterns is the same as the second block set obtained using the second set of block division patterns, the block division determination unit divides using only one of the first set of block division patterns and the second set of block division patterns.

[0431] The block division determination unit in the decoding device according to an embodiment of the present disclosure may also divide using the block division pattern set with the smaller one of the first code quantity of the first set of block division patterns and the second code quantity of the second set of block division patterns based on the first code quantity of the first set of block division patterns and the second code quantity of the second set of block division patterns.

[0432] The block division determination unit in the decoding device according to an embodiment of the present disclosure may also divide using the block division pattern set that appears first in a preset order in the first set of block division patterns and the second set of block division patterns when the first code quantity is equal to the second code quantity based on the first code quantity of the first set of block division patterns and the second code quantity of the second set of block division patterns.

[0433] The encoding method according to an embodiment of the present disclosure may also be to use a set of block segmentation patterns obtained by combining one or more block segmentation patterns to divide a picture read from a memory into a plurality of blocks, where the block segmentation patterns define the segmentation type; encode the plurality of blocks; the set of block segmentation patterns consists of a first block segmentation pattern and a second block segmentation pattern, the first block segmentation pattern defines the segmentation direction and the number of segments for dividing the first block, and the second block segmentation pattern defines the segmentation direction and the number of segments for dividing a second block, which is one of the blocks obtained after the segmentation of the first block; in the above segmentation, when the number of segments in the first block segmentation pattern is 3, the second block is the central block among the blocks obtained after the segmentation of the first block, and the segmentation direction of the second block segmentation pattern is the same as the segmentation direction of the first block segmentation pattern, the second block segmentation pattern only includes a block segmentation pattern with the number of segments being 3.

[0434] The parameter for identifying the second block segmentation pattern in the encoding method according to an embodiment of the present disclosure may also include a first flag indicating in which direction, the horizontal direction or the vertical direction, the block is segmented, and does not include a second flag indicating the number of segments for dividing the block.

[0435] The encoding method according to an embodiment of the present disclosure may also be to have: a step of using a set of block segmentation patterns obtained by combining one or more block segmentation patterns to divide a picture read from a memory into a plurality of blocks, where the block segmentation patterns define the segmentation type; and a step of encoding the plurality of blocks; the set of block segmentation patterns consists of a first block segmentation pattern and a second block segmentation pattern, the first block segmentation pattern defines the segmentation direction and the number of segments for dividing the first block, and the second block segmentation pattern defines the segmentation direction and the number of segments for dividing a second block, which is one of the blocks obtained after the segmentation of the first block; in the step of performing the above segmentation, when the number of segments in the first block segmentation pattern is 3, the second block is the central block among the blocks obtained after the segmentation of the first block, and the segmentation direction of the second block segmentation pattern is the same as the segmentation direction of the first block segmentation pattern, the second block segmentation pattern with the number of segments being 2 is not used.

[0436] The encoding method according to an embodiment of the present disclosure may also be to have: a step of using a set of block segmentation patterns obtained by combining one or more block segmentation patterns to divide a picture read from a memory into a plurality of blocks, where the block segmentation patterns define the segmentation type; and a step of encoding the plurality of blocks; the set of block segmentation patterns includes a first block segmentation pattern and a second block segmentation pattern that respectively define the segmentation direction and the number of segments; in the step of performing the above segmentation, the use of the second block segmentation pattern with the number of segments being 2 is restricted.

[0437] The parameter for identifying the second block splitting pattern in the encoding method according to an embodiment of the present disclosure may also include a first flag indicating in which direction (horizontal or vertical) the block is to be split, and a second flag indicating whether the block is to be split into two or more parts.

[0438] The above parameter in the encoding method according to an embodiment of the present disclosure may also be configured in the slice data.

[0439] The encoding method according to an embodiment of the present disclosure may also include: a step of using a set of block splitting patterns obtained by combining one or more block splitting patterns to split a picture read from a memory into a set of blocks composed of multiple blocks, where the block splitting pattern defines a splitting type; and a step of encoding the multiple blocks; in the step of performing the above splitting, when the first set of blocks obtained by using the first set of block splitting patterns is the same as the second set of blocks obtained by using the second set of block splitting patterns, only one of the first set of block splitting patterns and the second set of block splitting patterns is used for splitting.

[0440] In the step of performing the above splitting in the encoding method according to an embodiment of the present disclosure, it may also be based on the first code amount of the first set of block splitting patterns and the second code amount of the second set of block splitting patterns, and the set of block splitting patterns with the smaller one of the first code amount and the second code amount is used for splitting.

[0441] In the step of performing the above splitting in the encoding method according to an embodiment of the present disclosure, it may also be based on the first code amount of the first set of block splitting patterns and the second code amount of the second set of block splitting patterns. When the first code amount is equal to the second code amount, the set of block splitting patterns that appears first in a preset order in the first set of block splitting patterns and the second set of block splitting patterns is used for splitting.

[0442] The decoding method according to an embodiment of the present disclosure may also be to use a set of block segmentation patterns obtained by combining one or more block segmentation patterns to divide an encoded signal read from a memory into a plurality of blocks, where the block segmentation pattern defines a segmentation type; decode the plurality of blocks; the set of block segmentation patterns includes a first block segmentation pattern and a second block segmentation pattern, the first block segmentation pattern defines a segmentation direction and a segmentation number for dividing a first block, and the second block segmentation pattern defines a segmentation direction and a segmentation number for dividing a second block, which is one of the blocks obtained after the division of the first block; in the above division, when the segmentation number of the first block segmentation pattern is 3, the second block is the central block among the blocks obtained after the division of the first block, and the segmentation direction of the second block segmentation pattern is the same as the segmentation direction of the first block segmentation pattern, the second block segmentation pattern only includes a block segmentation pattern with a segmentation number of 3.

[0443] The parameter for identifying the second block segmentation pattern in the decoding method according to an embodiment of the present disclosure may also include a first flag indicating in which direction, the horizontal direction or the vertical direction, the block is divided, and does not include a second flag indicating the segmentation number for dividing the block.

[0444] The decoding method according to an embodiment of the present disclosure may also be to have: a step of using a set of block segmentation patterns obtained by combining one or more block segmentation patterns to divide an encoded signal read from a memory into a plurality of blocks, where the block segmentation pattern defines a segmentation type; and a step of decoding the plurality of blocks; the set of block segmentation patterns includes a first block segmentation pattern and a second block segmentation pattern, the first block segmentation pattern defines a segmentation direction and a segmentation number for dividing a first block, and the second block segmentation pattern defines a segmentation direction and a segmentation number for dividing a second block, which is one of the blocks obtained after the division of the first block; in the step of performing the above division, when the segmentation number of the first block segmentation pattern is 3, the second block is the central block among the blocks obtained after the division of the first block, and the segmentation direction of the second block segmentation pattern is the same as the segmentation direction of the first block segmentation pattern, the second block segmentation pattern with a segmentation number of 2 is not used.

[0445] The decoding method according to an embodiment of the present disclosure may also be to have: a step of using a set of block segmentation patterns obtained by combining one or more block segmentation patterns to divide an encoded signal read from a memory into a plurality of blocks, where the block segmentation pattern defines a segmentation type; and a step of decoding the plurality of blocks; the set of block segmentation patterns includes a first block segmentation pattern and a second block segmentation pattern that respectively define a segmentation direction and a segmentation number; in the step of performing the above division, the use of the second block segmentation pattern with a segmentation number of 2 is restricted.

[0446] The decoding method according to an embodiment of the present disclosure may also include: a step of using a set of block segmentation patterns obtained by combining one or more block segmentation patterns to divide an encoded signal read from a memory into a set of blocks composed of a plurality of blocks, where the block segmentation pattern defines a segmentation type; and a step of decoding the plurality of blocks; in the step of performing the division, when the first set of blocks obtained by using the first set of block segmentation patterns is the same as the second set of blocks obtained by using the second set of block segmentation patterns, only one of the first set of block segmentation patterns or the second set of block segmentation patterns is used for the division.

[0447] The picture compression program according to an embodiment of the present disclosure may also include: using a set of block segmentation patterns obtained by combining one or more block segmentation patterns to divide a picture read from a memory into a plurality of blocks, where the block segmentation pattern defines a segmentation type; decoding the plurality of blocks; the set of block segmentation patterns is composed of a first block segmentation pattern and a second block segmentation pattern, the first block segmentation pattern defines a segmentation direction and a segmentation number for dividing a first block, and the second block segmentation pattern defines a segmentation direction and a segmentation number for dividing a second block, which is one of the blocks obtained after the division of the first block; in the above division, when the segmentation number of the first block segmentation pattern is 3, the second block is the central block among the blocks obtained after the division of the first block, and the segmentation direction of the second block segmentation pattern is the same as the segmentation direction of the first block segmentation pattern, the second block segmentation pattern only includes a block segmentation pattern with a segmentation number of 3.

[0448] The picture compression program according to an embodiment of the present disclosure may also include: a step of using a set of block segmentation patterns obtained by combining one or more block segmentation patterns to divide a picture read from a memory into a plurality of blocks, where the block segmentation pattern defines a segmentation type; and a step of encoding the plurality of blocks; the set of block segmentation patterns is composed of a first block segmentation pattern and a second block segmentation pattern, the first block segmentation pattern defines a segmentation direction and a segmentation number for dividing a first block, and the second block segmentation pattern defines a segmentation direction and a segmentation number for dividing a second block, which is one of the blocks obtained after the division of the first block; in the step of performing the division, when the segmentation number of the first block segmentation pattern is 3, the second block is the central block among the blocks obtained after the division of the first block, and the segmentation direction of the second block segmentation pattern is the same as the segmentation direction of the first block segmentation pattern, the second block segmentation pattern with a segmentation number of 2 is not used.

[0449] The picture compression program according to an embodiment of the present disclosure may also include: a step of dividing a picture read from a memory into a plurality of blocks by using a set of block division patterns obtained by combining one or more block division patterns, where the block division patterns define division types; and a step of encoding the plurality of blocks; the set of block division patterns includes a first block division pattern and a second block division pattern that respectively define a division direction and a division quantity; in the step of performing the division, the use of the second block division pattern with the division quantity of 2 is restricted.

[0450] The picture compression program according to an embodiment of the present disclosure may also include: a step of dividing a picture read from a memory into a block set composed of a plurality of blocks by using a set of block division patterns obtained by combining one or more block division patterns, where the block division patterns define division types; and a step of encoding the plurality of blocks; in the step of performing the division, when the first block set obtained by using the first set of block division patterns is the same as the second block set obtained by using the second set of block division patterns, only one of the first set of block division patterns or the second set of block division patterns is used for division.

[0451] Industrial applicability

[0452] It can be used for encoding / decoding of multimedia data, especially image and video encoding / decoding devices using block encoding / decoding.

[0453] Reference numeral description

[0454] 100 Encoding device

[0455] 102 Division unit

[0456] 104 Subtraction unit

[0457] 106, 5001 Transformation unit

[0458] 108, 5002 Quantization unit

[0459] 110, 5009 Entropy encoding unit

[0460] 112, 5003, 6002 Inverse quantization unit

[0461] 114, 5004, 6003 Inverse transformation unit

[0462] 116 Addition unit

[0463] 118, 5005, 6004 Block memory

[0464] 120 Loop filtering unit

[0465] 122, 5006, 6005 Frame memory

[0466] 124, 5007, 6006 Intra prediction unit

[0467] 126, 5008, 6007 Inter prediction unit

[0468] 128 Prediction control unit

[0469] 200 Decoding device

[0470] 202, 6001 Entropy decoding unit

[0471] 204 Inverse quantization unit

[0472] 206 Inverse transformation unit

[0473] 208 Addition unit

[0474] 210 Block memory

[0475] 212 Loop filtering unit

[0476] 214 Frame memory

[0477] 216 Intra prediction unit

[0478] 218 Inter prediction unit

[0479] 220 Prediction control unit

[0480] 5000 Video encoding device

[0481] 5010, 6008 Block segmentation decision unit

[0482] 6000 Video decoding device

Claims

1. An encoding device that encodes pictures, wherein, Comprising: a processor; and a memory; The above-mentioned processor performs the following processing: Using a set of block splitting patterns obtained by combining one or more block splitting patterns, splitting the above-mentioned picture read from the above-mentioned memory into a plurality of blocks, and the above-mentioned block splitting pattern defines the splitting type; and Encoding the above-mentioned plurality of blocks; The above-mentioned set of block splitting patterns consists of a first block splitting pattern and a second block splitting pattern. The above-mentioned first block splitting pattern defines the splitting direction and the number of splits for splitting the first block, and the above-mentioned second block splitting pattern defines the splitting direction and the number of splits for splitting the second block, which is one of the blocks obtained after splitting the first block; The parameter for identifying the above-mentioned second block splitting pattern includes a first flag indicating in which direction, among the horizontal direction and the vertical direction, the above-mentioned second block is split; When the above-mentioned number of splits in the above-mentioned first block splitting pattern is 3, the above-mentioned second block is the central block among the blocks obtained after splitting the first block, and the above-mentioned splitting direction of the above-mentioned second block splitting pattern indicated by the above-mentioned first flag is the same as the above-mentioned splitting direction of the above-mentioned first block splitting pattern, the above-mentioned second block splitting pattern only includes the block splitting pattern with the number of splits being 3; When the above-mentioned number of splits in the above-mentioned first block splitting pattern is 3, the above-mentioned second block is the central block among the blocks obtained after splitting the first block, and the above-mentioned splitting direction of the above-mentioned second block splitting pattern indicated by the above-mentioned first flag is different from the above-mentioned splitting direction of the above-mentioned first block splitting pattern, the above-mentioned second block splitting pattern includes the block splitting pattern with the number of splits being 2.

2. A decoding device decodes an encoded signal, wherein, Comprising: a processor; and a memory; The above-mentioned processor performs the following processing: Using a set of block splitting patterns obtained by combining one or more block splitting patterns, splitting the above-mentioned encoded signal read from the above-mentioned memory into a plurality of blocks, and the above-mentioned block splitting pattern defines the splitting type; and Decoding the above-mentioned plurality of blocks; The above-mentioned set of block splitting patterns consists of a first block splitting pattern and a second block splitting pattern. The above-mentioned first block splitting pattern defines the splitting direction and the number of splits for splitting the first block, and the above-mentioned second block splitting pattern defines the splitting direction and the number of splits for splitting the second block, which is one of the blocks obtained after splitting the first block; The parameter for identifying the above-mentioned second block splitting pattern includes a first flag indicating in which direction, among the horizontal direction and the vertical direction, the above-mentioned second block is split; When the above-mentioned number of splits in the above-mentioned first block splitting pattern is 3, the above-mentioned second block is the central block among the blocks obtained after splitting the first block, and the above-mentioned splitting direction of the above-mentioned second block splitting pattern indicated by the above-mentioned first flag is the same as the above-mentioned splitting direction of the above-mentioned first block splitting pattern, the above-mentioned second block splitting pattern only includes the block splitting pattern with the number of splits being 3; When the number of divisions in the above-described first block division pattern is 3, the second block is the central block among the blocks obtained after the division of the first block, and the division direction of the second block division pattern indicated by the first flag is different from the division direction of the first block division pattern, the second block division pattern includes a block division pattern with the number of divisions being 2.

3. A non-transitory storage medium that stores a bit stream and is readable by a computer, wherein the bit stream includes information for causing a computer that receives the bit stream to perform a decoding process; the information is for causing the computer to perform the following processes: using a set of block division patterns obtained by combining one or more block division patterns, dividing an encoded signal read from a memory into a plurality of blocks, the block division pattern defining a division type; and decoding the plurality of blocks; the set of block division patterns is composed of a first block division pattern and a second block division pattern, the first block division pattern defining a division direction and a number of divisions for dividing a first block, and the second block division pattern defining a division direction and a number of divisions for dividing a second block, which is one of the blocks obtained after the division of the first block; a parameter for identifying the second block division pattern includes a first flag indicating in which direction, either the horizontal direction or the vertical direction, the second block is divided; when the number of divisions in the first block division pattern is 3, the second block is the central block among the blocks obtained after the division of the first block, and the division direction of the second block division pattern indicated by the first flag is the same as the division direction of the first block division pattern, the second block division pattern only includes a block division pattern with the number of divisions being 3; when the number of divisions in the first block division pattern is 3, the second block is the central block among the blocks obtained after the division of the first block, and the division direction of the second block division pattern indicated by the first flag is different from the division direction of the first block division pattern, the second block division pattern includes a block division pattern with the number of divisions being 2.

Images