Image encoding device, image decoding device and non-transitory memory medium
Patent Information
- Application Number
- TW115112931
- Authority / Receiving Office
- TW · TW
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2017-03-31
- Filing Date
- 2018-03-28
- Publication Date
- 2026-11-11
- Estimated Expiration
- 2038-03-27
AI Technical Summary
Conventional image and video coding methods face inefficiencies in compression due to increased overhead in signaling block partitioning patterns as the partitioning depth increases, particularly when using block sizes ranging from 4x4 to 256x256.
An encoding device that improves compression efficiency by selecting and encoding block segmentation patterns, determining whether to use specific segmentation modes, and encoding blocks into sub-blocks of varying shapes or sizes, reducing unnecessary signaling overhead.
Enhances compression efficiency by optimizing block segmentation information encoding, thereby improving image encoding performance.
Smart Images

Figure TWG2TB001921132_001 
Figure TWG2TB001921132_002 
Figure TWG2TB001921132_003
Abstract
Description
Technical Field
[0001] Invention Field This disclosure relates to a method and apparatus for encoding and decoding images and graphics using block segmentation. Prior Technology
[0002] Background of the Invention In conventional image and video coding methods, images are typically divided into blocks, and encoding and decoding are performed at the block level. Recent developments in image specifications have enabled encoding and decoding in various block sizes, in addition to typical 8x8 or 16x16 sizes. For image encoding and decoding, sizes ranging from 4x4 to 256x256 can be used.
[0003] Prior technology documents Non-patent literature Non-patent literature 1: H.265 (ISO / IEC 23008-2 HEVC (High Efficiency Video Coding)) Summary of the Invention
[0004] Invention Summary The problem the invention aims to solve To represent a range of sizes from 4x4 to 256x256, block partitioning patterns (such as quadtrees, binary trees, and ternary trees) and block partitioning information, such as partition flags (e.g., split flags), are used to determine and signal blocks. As the partitioning depth increases, the overhead of this signaling increases. This increased overhead then reduces image compression efficiency.
[0005] Here, the present invention discloses an encoding device that provides an encoding device that can improve compression efficiency in the encoding of block segmentation information. The means to solve the problem
[0006] This disclosure discloses an image encoding apparatus that encodes blocks of an image, comprising a processor and a memory. The processor uses the memory to perform the following processing: writing a first parameter into a bit stream, wherein the first parameter is a segmentation pattern identified from a plurality of segmentation patterns for dividing a first block into sub-blocks; determining whether the first parameter has identified the first segmentation pattern; at least based on the determination that the first parameter has identified the first segmentation pattern, determining whether to not select a second segmentation pattern as a candidate for dividing a second block different from the first block; when it is determined that the second segmentation pattern has been selected, as a candidate for dividing the first block... When selecting candidates for segmenting the aforementioned second block, a segmentation mode is selected from a plurality of segmentation modes, including the aforementioned second segmentation mode, and a second parameter displaying the selection result is written into a bit stream; when it has been determined that the aforementioned second segmentation mode is not selected as a candidate for segmenting the aforementioned second block, a segmentation mode different from the aforementioned second segmentation mode is selected, and this segmentation mode divides the block into sub-blocks, which are sub-blocks of different shapes or sizes compared with the sub-blocks generated by the aforementioned second segmentation mode; the aforementioned second block is divided into sub-blocks using the selected segmentation mode; and the segmented aforementioned block is encoded.
[0007] Furthermore, such general or specific forms can be realized through systems, methods, integrated circuits, computer programs, or recording media such as computer-readable CD-ROMs, or through any combination of systems, methods, integrated circuits, computer programs, and recording media. The effects of the invention
[0008] According to this study, compression efficiency can be improved in the encoding of block segmentation information. Simple Explanation of the Diagram
[0009] Figure 1 is a block diagram showing the functional configuration of the encoding device in Embodiment 1.
[0010] Figure 2 is a diagram showing an example of block partitioning in implementation form 1.
[0011] Figure 3 is a table showing the transformation basis functions corresponding to each transformation type.
[0012] Figure 4A is a diagram showing an example of the shape of the filter used by ALF.
[0013] Figure 4B is another example of the shape of the filter used by ALF.
[0014] Figure 4C is another example of the shape of the filter used by ALF.
[0015] Figure 5A shows 67 internal prediction patterns in the internal prediction.
[0016] Figure 5B is a flowchart illustrating the general outline of the predictive image correction process using OBMC.
[0017] Figure 5C is a conceptual diagram illustrating the outline of the predictive image correction process using OBMC.
[0018] Figure 5D shows an example of FRUC.
[0019] Figure 6 is a diagram used to illustrate pattern matching (bidirectional matching) between two blocks along a movement trajectory.
[0020] Figure 7 is a diagram used to illustrate pattern matching (template matching) between the template in the current image and the block in the reference image.
[0021] Figure 8 is a diagram used to illustrate the model assuming constant velocity linear motion.
[0022] Figure 9A is a diagram illustrating the derivation of the movement vector of a sub-block unit, which is based on the movement vectors of a plurality of adjacent blocks.
[0023] Figure 9B is a diagram illustrating the outline of the move vector derivation process in the merge mode.
[0024] Figure 9C is a conceptual diagram used to illustrate the outline of DMVR processing.
[0025] Figure 9D is a diagram illustrating the general outline of a predictive image generation method that uses luminance correction processing with LIC processing.
[0026] Figure 10 is a block diagram showing the functional configuration of the decoding device in Embodiment 1.
[0027] Figure 11 is a flowchart showing the image encoding process of embodiment 2.
[0028] Figure 12 is a flowchart showing the image decoding process of embodiment 2.
[0029] Figure 13 is a flowchart showing the image encoding process of embodiment 3.
[0030] Figure 14 is a flowchart showing the image decoding process of embodiment 3.
[0031] Figure 15 is a block diagram showing the structure of the image / image decoding device in embodiment 2 or 3.
[0032] Figure 16 is a block diagram showing the structure of the image / image decoding device in embodiment 2 or 3.
[0033] Figure 17 is an example of a conceivable location of the first parameter within the compressed image stream in embodiment 2 or 3.
[0034] Figure 18 is a diagram showing an example of the conceivable location of the second parameter within the compressed image stream in embodiment 2 or 3.
[0035] Figure 19 is an example of a second parameter following the first parameter in embodiment 2 or 3.
[0036] Figure 20 is an example of not selecting one of the second segmentation modes in embodiment 2, as shown in step (2c), for the segmentation of a 2NxN pixel block.
[0037] Figure 21 is an example of not selecting one of the second segmentation modes in embodiment 2, as shown in step (2c), for the segmentation of Nx2N pixel blocks.
[0038] Figure 22 is an example of not selecting one of the second segmentation modes in embodiment 2, as shown in step (2c), for the segmentation of NxN pixel blocks.
[0039] Figure 23 is an example of not selecting one of the second segmentation modes in embodiment 2, as shown in step (2c), for the segmentation of NxN pixel blocks.
[0040] Figure 24 is an example of dividing a 2NxN pixel block using the selected segmentation mode when the second segmentation mode is not selected, as shown in step (3) in embodiment 2.
[0041] Figure 25 is an example of dividing an Nx2N pixel block using the selected segmentation mode when the second segmentation mode is not selected, as shown in step (3) in embodiment 2.
[0042] Figure 26 is an example of dividing an NxN pixel block using the selected segmentation mode when the second segmentation mode is not selected, as shown in step (3) in embodiment 2.
[0043] Figure 27 is an example of dividing an NxN pixel block using the selected segmentation mode when the second segmentation mode is not selected, as shown in step (3) in embodiment 2.
[0044] Figure 28 is an example of a segmentation pattern used to divide an NxN pixel block in Embodiment 2. (a) to (h) are diagrams showing different segmentation patterns.
[0045] Figure 29 is an example of the segmentation type and segmentation direction used to divide an NxN pixel block in Embodiment 3. (1), (2), (3) and (4) are different segmentation types, (1a), (2a), (3a) and (4a) are segmentation modes with different segmentation types in the vertical direction, and (1b), (2b), (3b) and (4b) are segmentation modes with different segmentation types in the horizontal direction.
[0046] Figure 30 shows the advantages of encoding the segmentation pattern before the segmentation pattern in Embodiment 3, compared to encoding the segmentation direction before the segmentation pattern.
[0047] Figure 31A is a diagram showing an example of using a smaller set of partitioning patterns to divide a block into one of its sub-blocks in the encoding of partitioning patterns.
[0048] Figure 31B is a diagram showing an example of using a smaller set of partitioning patterns to divide a block into one of its sub-blocks in the encoding of partitioning patterns.
[0049] Figure 32A is a diagram showing an example of dividing a block into one of its sub-blocks using the first segmentation pattern set that appears in a predetermined order from a plurality of segmentation pattern sets.
[0050] Figure 32B is a diagram showing an example of dividing a block into one of its sub-blocks using the first segmentation pattern set that appears in a predetermined order from a plurality of segmentation pattern sets.
[0051] Figure 32C is a diagram showing an example of dividing a block into one of its sub-blocks using the first set of partitioning patterns that appears in a predetermined order from a plurality of partitioning pattern sets.
[0052] Figure 33 is a diagram of the overall structure of the content supply system for realizing content distribution services.
[0053] Figure 34 is a diagram showing an example of the encoding construction in adjustable encoding.
[0054] Figure 35 is a diagram showing an example of the encoding construction in the case of adjustable encoding.
[0055] Figure 36 is an example of a webpage display screen.
[0056] Figure 37 is an example of a webpage display screen.
[0057] Figure 38 shows an example of a smartphone.
[0058] Figure 39 shows an example of a smartphone. Implementation
[0059] Detailed Description of Preferred Embodiments The following is a detailed explanation of the implementation methods, referring to the diagrams.
[0060] Furthermore, each of the embodiments described below is a general or specific example. The values, shapes, materials, constituent elements, their arrangement and connection, steps, and order of steps shown in the following embodiments are merely illustrative and are not intended to limit the scope of the claim. Also, among the constituent elements in the following embodiments, those constituent elements not described in the independent claim item displaying the highest-level concept are described as arbitrary constituent elements. (Implementation Form 1)
[0061] First, an overview of Embodiment 1 will be described with reference to one example of the encoding and decoding apparatus and the processing and / or configuration described in the various embodiments of this disclosure, which are applicable to the following description. However, Embodiment 1 is merely one example of the encoding and decoding apparatus and the processing and / or configuration described in the various embodiments of this disclosure, and the processing and / or configuration described in this disclosure may also be implemented in encoding and decoding apparatuses different from Embodiment 1.
[0062] For implementation form 1, when applying the processing and / or configuration described in the various forms disclosed herein, for example, any of the following methods may also be used. (1) For the encoding or decoding device of embodiment 1, among the plurality of constituent elements constituting the encoding or decoding device, the constituent element corresponding to the constituent element described in each embodiment of the present disclosure is replaced with the constituent element described in each embodiment of the present disclosure; (2) For the encoding or decoding device of embodiment 1, for a portion of the plurality of constituent elements constituting the encoding or decoding device, after any changes such as addition, replacement, or deletion of functions or desired processing are applied to the constituent elements constituting a portion of the constituent elements constituting the encoding or decoding device, the constituent elements corresponding to the constituent elements described in the various embodiments of this disclosure are replaced with the constituent elements described in the various embodiments of this disclosure; (3) For the method to be implemented by the encoding or decoding device of Embodiment 1, for the addition of processing and / or a part of the processing contained in the method, any changes such as replacement or deletion are first applied, and then the processing corresponding to the processing described in each embodiment of this disclosure is replaced with the processing described in each embodiment of this disclosure; (4) The constituent elements constituting a portion of the plurality of constituent elements constituting the encoding or decoding device of Embodiment 1, and the constituent elements described in the various embodiments of this disclosure, the constituent elements having one of the functions of the constituent elements described in the various embodiments of this disclosure, or the constituent elements constituting one of the processes to be performed to implement the constituent elements described in the various embodiments of this disclosure, are combined and implemented; (5) The constituent elements that possess one part of the functions of a constituent element that constitutes a plurality of constituent elements constituting the encoding or decoding device of Embodiment 1, or the constituent elements that perform one part of the processing performed by a constituent element that constitutes a plurality of constituent elements constituting the encoding or decoding device of Embodiment 1, are combined with the constituent elements described in the various embodiments of this disclosure, the constituent elements that possess one part of the functions of the constituent elements described in the various embodiments of this disclosure, or the constituent elements that perform one part of the processing performed by the constituent elements described in the various embodiments of this disclosure to implement the device; (6) For the method to be implemented by the encoding or decoding apparatus of embodiment 1, among the plurality of processes contained in the method, the process corresponding to the process described in each embodiment of this disclosure is replaced with the process described in each embodiment of this disclosure; (7) A portion of the plurality of processes contained in the method to be implemented by the encoding or decoding device of Embodiment 1 is combined with the processes described in the various embodiments of this disclosure. Furthermore, the implementation methods of the processing and / or configurations described in the various embodiments disclosed herein are not limited to the examples described above. For example, they may be implemented in devices used for different purposes than the motion picture / image encoding device or motion picture / image decoding device disclosed in Embodiment 1, or the processing and / or configurations described in each embodiment may be implemented individually. Additionally, the processing and / or configurations described in different embodiments may be combined for implementation. [Overview of the encoding device]
[0063] First, an overview of the encoding device in Embodiment 1 will be described. Figure 1 is a block diagram showing the functional configuration of the encoding device 100 in Embodiment 1. The encoding device 100 is a motion image / image encoding device that encodes motion images / images in block units.
[0064] As shown in Figure 1, the encoding device 100 is an apparatus for encoding images in block units, including: a segmentation unit 102, a subtraction unit 104, a conversion unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse conversion unit 114, an addition unit 116, a block memory 118, a loop filter unit 120, a frame memory 122, an intra-prediction unit 124, an inter-prediction unit 126, and a prediction control unit 128.
[0065] The encoding device 100 is implemented, for example, using a general-purpose processor and memory. When the software program stored in memory is executed by the processor, the processor functions as a segmentation unit 102, a subtraction unit 104, a conversion unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse conversion unit 114, an addition unit 116, a loop filter unit 120, an internal prediction unit 124, an inter-prediction unit 126, and a prediction control unit 128. Alternatively, the encoding device 100 can be implemented as one or more dedicated electronic circuits corresponding to the segmentation unit 102, the subtraction unit 104, the conversion unit 106, the quantization unit 108, the entropy encoding unit 110, the inverse quantization unit 112, the inverse conversion unit 114, the addition unit 116, the loop filter unit 120, the internal prediction unit 124, the inter-prediction unit 126, and the prediction control unit 128.
[0066] The following describes the constituent elements of the encoding device 100. [Divider]
[0067] The segmentation unit 102 divides the images contained in the input dynamic image into a plurality of blocks and outputs each block to the subtraction unit 104. For example, the segmentation unit 102 first segments the image into blocks of a fixed size (e.g., 128×128). These fixed-size blocks are sometimes referred to as coding tree units (CTUs). Next, the segmentation unit 102 divides each of the fixed-size blocks into variable-size blocks (e.g., 64×64 or less) according to recursive quadtree and / or binary tree block segmentation. These variable-size blocks are sometimes referred to as coding units (CUs), prediction units (PUs), or transformation units (TUs). Alternatively, in this embodiment, there is no need to distinguish between CUs, PUs, and TUs; some or all of the blocks within the image can be processed as CUs, PUs, or TUs.
[0068] Figure 2 is a diagram showing an example of block partitioning in Implementation 1. In Figure 2, solid lines represent block boundaries obtained through quaternion tree block partitioning, and dashed lines represent block boundaries obtained through binary tree block partitioning.
[0069] Here, block 10 is a square block of 128×128 pixels (128×128 block). This 128×128 block 10 is first divided into 4 square blocks of 64×64 (quadtree block partitioning).
[0070] The top-left 64×64 block is further vertically divided into two 32×64 blocks, and the left 32×64 block is further vertically divided into two 16×64 blocks (binary tree block partitioning). As a result, the top-left 64×64 block is divided into two 16×64 blocks (11 and 12) and a 32×64 block (13).
[0071] The 64×64 block in the upper right corner is horizontally divided into two 64×32 blocks, 14 and 15 (binary tree block partitioning).
[0072] The bottom left 64×64 block is divided into four 32×32 square blocks (quadtree block partitioning). Of these four 32×32 blocks, the top left and bottom right blocks are further divided. The top left 32×32 block is vertically divided into two 16×32 rectangular blocks, and the right 16×32 block is further horizontally divided into two 16×16 blocks (binary tree block partitioning). The bottom right 32×32 block is horizontally divided into two 32×16 blocks (binary tree block partitioning). As a result, the bottom left 64×64 block is divided into one 16×32 block (16), two 16×16 blocks (17 and 18), two 32×32 blocks (19 and 20), and two 32×16 blocks (21 and 22).
[0073] The 64x64 block 23 in the lower right corner is not divided.
[0074] As shown above, in Figure 2, block 10 is divided into 13 variable-size blocks 11 to 23 based on the recursive quad-tree and binary tree block partitioning. This partitioning is sometimes referred to as QTBT (quad-tree plus binary tree) partitioning.
[0075] Furthermore, in Figure 2, a block is divided into four or two blocks (quadtree or binary tree block partitioning), but the partitioning is not limited to these. For example, a block can also be divided into three blocks (ternary tree block partitioning). Such partitioning, including ternary tree block partitioning, is sometimes referred to as MBT (multi-type tree) partitioning. [Subtraction Section]
[0076] The subtraction unit 104 subtracts the prediction signal (prediction sample) from the original signal (original sample) in block units divided by the segmentation unit 102. That is, the subtraction unit 104 calculates the prediction error (also called residual) of the encoded target block (hereinafter referred to as the current block). Then, the subtraction unit 104 outputs the calculated prediction error to the conversion unit 106.
[0077] The original signal is the input signal of the encoding device 100, which represents the image signals (e.g., luminance signal and two chroma signals) of each picture that constitutes the dynamic image. In the following text, the signals representing the images will also be referred to as samples. [Transition Section]
[0078] The conversion unit 106 converts the prediction error in the spatial region into conversion coefficients in the frequency region and outputs the conversion coefficients to the quantization unit 108. Specifically, the conversion unit 106 performs a predetermined discrete cosine transform (DCT) or discrete sine transform (DST) on the prediction error in the spatial region, for example.
[0079] Alternatively, the transformation unit 106 can adaptively select a transformation type from a plurality of transformation types and use the transformation basis function corresponding to the selected transformation type to convert the prediction error into transformation coefficients. Such a transformation is sometimes referred to as EMT (explicit multiple core transform) or AMT (adaptive multiple transform).
[0080] Several transformation types are available, including, for example, DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII. Figure 3 is a table showing the transformation basis functions corresponding to each transformation type. In Figure 3, N represents the number of input pixels. The selection of a transformation type from these multiple transformation types can be based on, for example, the type of prediction (intra-prediction and inter-prediction) or the intra-prediction mode.
[0081] Information indicating whether such EMT or AMT is applicable (e.g., referred to as the AMT flag) and information indicating the selected conversion type are signaled at the CU level. Furthermore, the signaling of this information is not limited to the CU level; it can also be at other levels (e.g., sequence level, picture level, slice level, tile level, or CTU level).
[0082] Furthermore, the transformation unit 106 can also further transform the transformation coefficients (transformation results). This further transformation is sometimes referred to as AST (adaptive secondary transform) or NSST (non-separable secondary transform). For example, the transformation unit 106 performs a further transformation based on the individual sub-blocks (e.g., 4×4 sub-blocks) contained in the block corresponding to the transformation coefficients of the internal prediction error. Information displaying whether NSST is applicable and information about the transformation matrix used in NSST are signaled at the CU level. However, the signaling of this information is not necessarily limited to the CU level; it can also be at other levels (e.g., sequence level, image level, slice level, block level, or CTU level).
[0083] Here, a separable transformation refers to the method of separating the dimensions of the input in a direction and performing several transformations, while a non-separable transformation refers to the method of combining more than 2 dimensions into one dimension when the input is multi-dimensional, and then performing the transformation together.
[0084] For example, taking an inseparable transformation as an example, one could exemplify the following: when the input is a 4×4 block, the block is treated as an array with 16 elements, and the array is transformed using a 16×16 transformation matrix.
[0085] Similarly, treating a 4×4 input block as an array with one of 16 elements and then performing several Givens rotations on that array (Hypercube Givens Transform) is also an example of a non-separable transformation. [Quantitative Department]
[0086] The quantization unit 108 quantizes the conversion coefficients output from the conversion unit 106. Specifically, the quantization unit 108 scans the conversion coefficients of the current block in a predetermined scan order and quantizes the conversion coefficients according to the quantization parameters (QP) corresponding to the scanned conversion coefficients. Then, the quantization unit 108 outputs the quantized conversion coefficients (hereinafter referred to as quantization coefficients) of the current block to the entropy encoding unit 110 and the dequantization unit 112.
[0087] The predetermined order is the order used for quantization / dequantization of the conversion coefficients. For example, the predetermined scan order is defined by ascending power order (from low frequency to high frequency) or descending power order (from high frequency to low frequency).
[0088] Quantization parameters define the quantization step size (quantization width). For example, increasing the value of the quantization parameter increases the quantization step size. In other words, increasing the value of the quantization parameter increases the quantization error. [Entropy Coding Department]
[0089] The entropy coding unit 110 performs variable-length encoding on the quantization coefficients input from the quantization unit 108, thereby generating an encoded signal (encoded bit stream). Specifically, the entropy coding unit 110 performs, for example, binarization on the quantization coefficients and arithmetic encoding on the binarized signal. [Anti-Quantitative Department]
[0090] The dequantization unit 112 dequantizes the quantization coefficients, which are inputs from the quantization unit 108. Specifically, the dequantization unit 112 dequantizes the quantization coefficients of the current block in a predetermined scan order. Then, the dequantization unit 112 outputs the dequantized conversion coefficients of the current block to the inversion conversion unit 114. [Reverse Conversion Section]
[0091] The inverse conversion unit 114 performs an inverse conversion on the input conversion coefficients from the inverse quantization unit 112, thereby restoring the prediction error. Specifically, the inverse conversion unit 114 performs an inverse conversion on the conversion coefficients corresponding to the conversion performed by the conversion unit 106, thereby restoring the prediction error of the current block. Then, the inverse conversion unit 114 outputs the restored prediction error to the addition unit 116.
[0092] Furthermore, the recovered prediction error is inconsistent with the prediction error calculated by the subtraction unit 104 because information was lost during quantization. That is, the recovered prediction error contains quantization error. [Addition Department]
[0093] The adder 116 adds the prediction error input from the inverse conversion unit 114 to the prediction sample input from the prediction control unit 128, thereby reconstructing the current block. Then, the adder 116 outputs the reconstructed block to the block memory 118 and the loop filter unit 120. The reconstructed block is sometimes also referred to as a local decoding block. [Block Memory]
[0094] Block memory 118 is a memory used to store blocks, wherein the blocks are the blocks that are predicted and referenced within the image being encoded (hereinafter referred to as the current image). Specifically, block memory 118 stores the reconstructed blocks output from the addition unit 116. [Loop Filtering Section]
[0095] The loop filter unit 120 applies loop filtering to the block reconstructed by the adder unit 116 and outputs the filtered reconstructed block to the frame memory 122. Loop filtering refers to filters used within the encoding loop (in-loop filters), such as deblocking filters (DF), sample adaptive offset (SAO), and adaptive loop filters (ALF).
[0096] In ALF, a least squares error filter is used to remove coding distortions. For example, for each 2×2 sub-block within the current block, a filter is selected from a plurality of filters based on the direction and activity of the local gradient.
[0097] Specifically, firstly, sub-blocks (e.g., 2×2 sub-blocks) are classified into multiple categories (e.g., 15 or 25 categories). The classification of sub-blocks is based on the direction and activity of the gradient. For example, using the gradient direction value D (e.g., 0 to 2 or 0 to 4) and the gradient activity value A (e.g., 0 to 4), a classification value C (e.g., C = 5D + A) is calculated. Then, based on the classification value C, the sub-blocks are classified into multiple categories (e.g., 15 or 25 categories).
[0098] The gradient direction value D is derived, for example, by comparing the gradients in multiple directions (e.g., horizontal, vertical, and two diagonal directions). Similarly, the gradient activity value A is derived, for example, by summing the gradients in multiple directions and quantizing the result of the summation.
[0099] Based on the results of this classification, the filter to be used for a sub-block is determined from among a plurality of filters.
[0100] Regarding the shape of the filters used in ALF, for example, a circularly symmetrical shape can be used. Figures 4A to 4C show multiple examples of the shapes of filters used in ALF. Figure 4A shows a 5×5 diamond-shaped filter, Figure 4B shows a 7×7 diamond-shaped filter, and Figure 4C shows a 9×9 diamond-shaped filter. The information showing the filter shape is signaled at the image level. However, the signaling of the filter shape information is not limited to the image level; it can also be at other levels (e.g., sequence level, slice level, block level, CTU level, or CU level).
[0101] The on / off state of ALF (Alternating Current Level) is determined, for example, by picture level or CU (Computer Unit) level. For instance, for luminance, the CU level determines whether ALF is applicable, while for chroma, the picture level determines whether ALF is applicable. The information indicating whether ALF is on / off is displayed is signaled at picture level or CU level. However, the display of ALF on / off information is not limited to picture level or CU level; other levels can also be used (e.g., sequence level, slice level, block level, or CTU level).
[0102] The coefficient set of a selectable plurality of filters (e.g., up to 15 or 25 filters) is signaled at the picture level. Alternatively, the signaling of the coefficient set is not limited to the picture level; it can also be at other levels (e.g., sequence level, slice level, block level, CTU level, CU level, or sub-block level). [Frame Memory]
[0103] Frame memory 122 is a memory unit used to store reference images used in in-betweening prediction, sometimes referred to as a frame buffer. Specifically, frame memory 122 stores reconstructed blocks that have been filtered by loop filter unit 120. [Internal Forecasting Department]
[0104] The intra-prediction unit 124 performs intra-prediction (also known as intra-image prediction) of the current image by referencing the blocks stored in the reference block memory 118, thereby generating a prediction signal (intra-prediction signal). Specifically, the intra-prediction unit 124 performs intra-prediction by referencing samples (e.g., brightness values, color difference values) of blocks adjacent to the current block, thereby generating an intra-prediction signal, and outputs the intra-prediction signal to the prediction control unit 128.
[0105] For example, the internal prediction unit 124 performs internal prediction using one of a plurality of pre-defined internal prediction patterns. The plurality of internal prediction patterns includes one or more non-directional prediction patterns and a plurality of directional prediction patterns.
[0106] One or more non-directional prediction modes, including, for example, the planar prediction mode and the DC prediction mode as specified in the H.265 / HEVC (High-Efficiency Video Coding) specification (Non-Patent Document 1).
[0107] Multiple directional prediction modes, for example, include the 33 directional prediction modes specified in the H.265 / HEVC specification. Alternatively, in addition to the 33 directional prediction modes, multiple directional prediction modes may further include 32 additional directional prediction modes (a total of 65 directional prediction modes). Figure 5A shows the 67 internal prediction modes (2 non-directional prediction modes and 65 directional prediction modes) in the internal prediction. Solid arrows represent the 33 directions specified in the H.265 / HEVC specification, and dashed arrows represent the additional 32 directions.
[0108] Furthermore, in the intraprediction of chromatic difference blocks, luminance blocks can also be referenced. That is, based on the luminance component of the current block, the chromatic difference component of the current block can also be predicted. Such intraprediction is sometimes called CCLM (cross-component linear model) prediction. This kind of intraprediction mode that references luminance blocks to predict chromatic difference blocks (e.g., called CCLM mode) can also be added as an intraprediction mode for chromatic difference blocks.
[0109] The intra-prediction unit 124 can also correct the pixel values after intra-prediction based on the gradient of reference pixels in the horizontal / vertical directions. Intra-prediction accompanied by such correction is sometimes referred to as PDPC (position-dependent intra-prediction combination). Information indicating the presence or absence of PDPC (e.g., a PDPC flag) is displayed, for example, signaled at the CU level. However, the signaling of this information is not limited to the CU level and can also be at other levels (e.g., sequence level, image level, slice level, block level, or CTU level). [Inter-departmental Forecasting]
[0110] Inter-prediction unit 126 performs inter-prediction (also called inter-frame prediction) of the current block by referring to a reference image, thereby generating a prediction signal (inter-prediction signal). The reference image is a reference image stored in frame memory 122, and is different from the current image. Inter-prediction is performed on a unit of the current block or sub-blocks within the current block (e.g., 4×4 blocks). For example, inter-prediction unit 126 performs motion estimation within the reference image for the current block or sub-block. Next, inter-prediction unit 126 uses motion information (e.g., motion vectors) obtained through motion estimation to perform motion compensation, thereby generating an inter-prediction signal for the current block or sub-block. Then, inter-prediction unit 126 outputs the generated inter-prediction signal to prediction control unit 128.
[0111] Motion information used for motion compensation is signaled. For the signaling of motion vectors, motion vector predictors can also be used. That is, the difference between the motion vector and the motion vector predictor can also be signaled.
[0112] Furthermore, in addition to using motion information from the current block obtained through motion estimation, motion information from adjacent blocks can also be used to generate inter-block prediction signals. Specifically, the prediction signal based on motion information obtained through motion estimation and the prediction signal based on motion information from adjacent blocks can be weighted and summed to generate inter-block prediction signals using sub-block units within the current block. This inter-block prediction (motion compensation) is sometimes referred to as OBMC (overlapped block motion compensation).
[0113] In this OBMC mode, information displaying the size of sub-blocks used in OBMC (e.g., referred to as OBMC block size) is signaled at the sequence level. Furthermore, information indicating whether OBMC mode is applicable (e.g., referred to as the OBMC flag) is signaled at the CU level. Additionally, the signaling level of this information is not limited to the sequence level and CU level; it can also be other levels (e.g., image level, slice level, block level, CTU level, or sub-block level).
[0114] The OBMC mode will be explained in more detail. Figures 5B and 5C are flowcharts and conceptual diagrams illustrating the general outline of the predictive image correction processing performed in OBMC.
[0115] First, the motion vector (MV) assigned to the encoded object block is used to obtain the predicted image (Pred) obtained by the usual motion compensation.
[0116] Secondly, the movement vector (MV_L) of the encoded left adjacent block is applied to the encoded target block to obtain the prediction image (Pred_L). The aforementioned prediction image is weighted and superimposed with Pred_L to perform the first correction of the prediction image.
[0117] In the same way, the movement vector (MV_U) of the adjacent blocks that have been encoded is applied to the target block to obtain the prediction image (Pred_U). The prediction image that has undergone the first correction is weighted and superimposed with Pred_U to perform the second correction of the prediction image, and this is used as the final prediction image.
[0118] Furthermore, while this paper describes a two-stage correction method using left and top adjacent blocks, it is also possible to create a configuration that uses right or bottom adjacent blocks to perform more corrections than a two-stage method.
[0119] Alternatively, the overlapping area can be only a portion of the area near the block boundary, rather than the entire pixel area of the block.
[0120] Furthermore, although this explanation focuses on the correction process for a predicted image from a single reference image, the same approach applies when correcting a predicted image from multiple reference images. After obtaining the corrected predicted images from each reference image, the resulting predicted images are further superimposed to form the final predicted image.
[0121] Alternatively, the aforementioned processing target block can be a predicted block unit or a sub-block unit that further divides the predicted block.
[0122] One method for determining whether OBMC processing is applicable is using an `obmc_flag`, which is a signal indicating whether OBMC processing is applicable. For example, in an encoding device, it is determined whether the area to be encoded belongs to a motion-complex region. If it does, the value is set to 1 as the `obmc_flag`, and OBMC processing is applied for encoding. If it does not belong to a motion-complex region, the value is set to 0 as the `obmc_flag`, and OBMC processing is not applied for encoding. On the other hand, in a decoding device, the `obmc_flag` recorded in the stream is decoded, and the application of OBMC processing is switched accordingly for decoding.
[0123] Alternatively, motion information can be exported from the decoding device without being signaled. For example, the merge mode specified in the H.265 / HEVC standard can be used. Also, motion estimation can be performed on the decoding device side to export motion information. In this case, motion estimation is performed without using the pixel values of the current block.
[0124] Here, we will explain the mode in which motion estimation is performed on the decoding device side. This mode of motion estimation on the decoding device side is sometimes called PMMVD (pattern matched motion vector derivation) mode or FRUC (frame rate up-conversion) mode.
[0125] One example of FRUC processing is shown in Figure 5D. First, referencing the movement vectors of spatially or temporally adjacent encoded blocks, a plurality of candidate lists (which can also be common to the merge list) are generated, each with a predicted movement vector. Second, the best candidate MV is selected from the plurality of candidate MVs already registered in the candidate lists. For example, the evaluation value of each candidate in the candidate list is calculated, and one candidate is selected based on the evaluation value.
[0126] Next, based on the selected candidate movement vectors, the movement vector for the current block is derived. Specifically, for example, the movement vector of the selected candidate (the best candidate MV) is derived as the movement vector for the current block. Alternatively, for example, pattern matching can be performed in the region surrounding a location within a reference image, where the reference image corresponds to the movement vector of the selected candidate. That is, the region surrounding the best candidate MV is searched in the same way, and when an MV with a good evaluation value is found, the best candidate MV is updated to the aforementioned MV, and this MV can be used as the final MV for the current block. Alternatively, this process can be omitted.
[0127] When processing at the sub-block level, the process can be exactly the same.
[0128] Furthermore, the evaluation value can be calculated by obtaining the difference value of the reconstructed image through pattern matching between the region within the reference image corresponding to the motion vector and the predetermined region. In addition to the difference value, other information can also be used to calculate the evaluation value.
[0129] For pattern matching, either first pattern matching or second pattern matching is used. First pattern matching and second pattern matching are sometimes referred to as bilateral matching and template matching, respectively.
[0130] In the first pattern matching, pattern matching is performed between two blocks that are two different blocks within two different reference images, and along the motion trajectory of the current block. Therefore, in the first pattern matching, regions within other reference images along the motion trajectory of the current block are used as predetermined regions for calculating the evaluation values of the aforementioned candidates.
[0131] Figure 6 illustrates an example of pattern matching (bidirectional matching) between two blocks along a movement trajectory. As shown in Figure 6, in the first pattern matching, among the pairs of two blocks along the movement trajectory of the current block (Cur block) and within two different reference images (Ref0, Ref1), the best matching pair is searched, thereby deriving two movement vectors (MV0, MV1). Specifically, for the current block, the difference between the reconstructed image at a specified position within the first encoded reference image (Ref0) specified by the candidate MV and the reconstructed image at a specified position within the second encoded reference image (Ref1) specified by the symmetrical MV that has been scaled by display time intervals is derived. The obtained difference value is used to calculate an evaluation value. Among the multiple candidate MVs, the candidate MV with the best evaluation value is selected as the final MV.
[0132] Under the assumption of continuous movement trajectories, the movement vectors (MV0, MV1) indicating two reference blocks are proportional to the temporal distances (TD0, TD1) between the current image (Cur Pic) and the two reference images (Ref0, Ref1). For example, if the current image is located temporally between the two reference images, and the temporal distances from the current image to the two reference images are equal, a mirror-symmetric bidirectional movement vector can be derived in the first pattern matching.
[0133] In the second pattern matching, pattern matching is performed between the template in the current image (the block adjacent to the current block in the current image (e.g., the upper and / or left adjacent block)) and the block in the reference image. Therefore, in the second pattern matching, the block adjacent to the current block in the current image is used as the predetermined area for calculating the aforementioned candidate evaluation value.
[0134] Figure 7 illustrates an example of pattern matching (template matching) between a template in the current image and a block in the reference image. As shown in Figure 7, in the second pattern matching, the block in the reference image (Ref0) that best matches the block in the current image (Cur Pic) and adjacent to the current block (Cur block) is searched, thereby deriving the movement vector of the current block. Specifically, for the current block, the difference between the reconstructed image of the encoded regions on the left and top adjacent sides or either side, and the reconstructed image at the same position in the encoded reference image (Ref0) specified by the candidate MV is derived. Using the obtained difference value, an evaluation value is calculated, and the candidate MV with the best evaluation value among the multiple candidate MVs is selected as the best candidate MV.
[0135] Information indicating whether FRUC mode is applicable (e.g., referred to as the FRUC flag) is signaled at the CU level. Furthermore, when FRUC mode is applicable (e.g., when the FRUC flag is true), information indicating the pattern matching method (first pattern match or second pattern match) (e.g., referred to as the FRUC mode flag) is signaled at the CU level. Additionally, the signaling of this information is not limited to the CU level and can also be at other levels (e.g., sequence level, image level, slice level, block level, CTU level, or sub-block level).
[0136] Here, we will explain the mode for deriving the movement vector from a model that assumes uniform linear motion. This mode is sometimes referred to as the BIO (bi-directional optical flow) mode.
[0137] Figure 8 illustrates the model assuming uniform linear motion. In Figure 8, (vx, vy) represents the velocity vector, and τ0 and τ1 represent the time distances between the current image (Cur Pic) and two reference images (Ref0, Ref1). (MVx0, MVy0) represents the movement vector corresponding to reference image Ref0, and (MVx1, MVy1) represent the movement vectors corresponding to reference image Ref1.
[0138] At this time, under the assumption of constant linear motion, the velocity vectors (vx, vy) are expressed as (MVx0, MVy0) and (MVx1, MVy1) as (vxτ0, vyτ0) and (-vxτ1, -vyτ1), respectively, so that the following optical flow equation (1) holds. (Number 1)
[0139] Here, I(k) represents the brightness value of the reference image k (k=0,1) after motion compensation. The optical flow equation is one in which the sum of (i) the temporal derivative of the brightness value, (ii) the product of the horizontal velocity and the horizontal component of the spatial gradient of the reference image, and (iii) the product of the vertical velocity and the vertical component of the spatial gradient of the reference image equals zero. Based on this optical flow equation combined with Hermite interpolation, the motion vector of the block unit obtained from the merge list, etc., is corrected in pixels.
[0140] Alternatively, the movement vector can be derived on the decoding device side using a method different from that used to derive the movement vector based on a model assuming constant linear motion. For example, the movement vector can be derived on a sub-block basis based on the movement vectors of a plurality of adjacent blocks.
[0141] Here, we will explain the mode of deriving movement vectors on a sub-block basis based on the movement vectors of a plurality of adjacent blocks. This mode is sometimes referred to as the affine motion compensation prediction mode.
[0142] Figure 9A illustrates the derivation of the movement vectors for a sub-block unit, which is based on the movement vectors of multiple adjacent blocks. In Figure 9A, the current block contains 16 4×4 sub-blocks. Here, based on the movement vectors of the adjacent blocks, the movement vector v0 of the upper left control point of the current block is derived, and based on the movement vectors of the adjacent sub-blocks, the movement vector v1 of the upper right control point of the current block is derived. Then, using the two movement vectors v0 and v1, the movement vectors (vx, vy) of each sub-block within the current block are derived via the following equation (2). (Number 2)
[0143] Here, x and y represent the horizontal and vertical positions of the sub-block, respectively, and w represents the pre-defined weighting coefficient.
[0144] Such an affine motion compensation prediction pattern may also include several patterns with different methods for deriving the movement vectors of the top-left and top-right control points. The information displaying such an affine motion compensation prediction pattern (e.g., referred to as an affine flag) is signaled at the CU level. However, the signaling of this information displaying the affine motion compensation prediction pattern is not limited to the CU level and can also be at other levels (e.g., sequence level, image level, slice level, block level, CTU level, or sub-block level). [Forecasting and Control Department]
[0145] The prediction control unit 128 selects either an internal prediction signal or an inter-prediction signal, and outputs the selected signal as the prediction signal to the subtraction unit 104 and the addition unit 116.
[0146] Here, an example of deriving the motion vector of an encoded object image via the merge mode is illustrated. Figure 9B is a diagram illustrating the outline of the motion vector deriving process performed using the merge mode.
[0147] First, a list of candidate predicted MVs that have been registered is generated. Candidate predicted MVs include: spatial adjacency predicted MVs, which are the MVs of multiple encoded blocks spatially adjacent to the encoded object block; temporal adjacency predicted MVs, which are the MVs of blocks near the location of the encoded object block projected onto the encoded reference image; combined predicted MVs, which are MVs generated by combining the MV values of spatial adjacency predicted MVs and temporal adjacency predicted MVs; and zero predicted MVs, which are MVs with a value of zero, etc.
[0148] Next, select one predicted MV from the multiple predicted MVs already registered in the predicted MV list, and use it as the MV of the encoded object block.
[0149] Furthermore, in the variable-length encoding section, merge_idx is recorded in the stream and encoded, where merge_idx is a signal indicating which prediction MV has been selected.
[0150] Additionally, the predicted MVs listed in the predicted MV list illustrated in Figure 9B are just one example. They may be a different number than those shown in the figure, or a composition that does not include any of the predicted MVs shown in the figure, or a composition that includes predicted MVs other than those shown in the figure.
[0151] Alternatively, the MV of the encoded object block derived from the merge mode can be used for the DMVR processing described later to determine the final MV.
[0152] Here, we will explain an example of using DMVR processing to determine MV.
[0153] Figure 9C is a conceptual diagram used to illustrate the outline of DMVR processing.
[0154] First, taking the most suitable MVP set in the processing object block as the candidate MV, according to the aforementioned candidate MV, the processed image in the L0 direction, i.e. the first reference image, and the processed image in the L1 direction, i.e. the second reference image, are obtained respectively. The average of each reference pixel is taken to generate a template.
[0155] Next, using the aforementioned template, the surrounding areas of the candidate MVs for the first and second reference images are searched respectively, and the MV with the lowest cost is selected as the final MV. Furthermore, the cost value is calculated using the differences between the pixel values of the template and the pixel values of the search area, as well as the MV value.
[0156] Furthermore, the general outline of the processing described herein is essentially the same in both the encoding and decoding devices.
[0157] Alternatively, even if the processing is not described here, other processing methods can be used as long as they can search for the surrounding elements of candidate music videos and export the final music video.
[0158] Here, we will explain the mode of using LIC processing to generate predicted images.
[0159] Figure 9D is a diagram illustrating the general outline of a predictive image generation method using luminance correction processing based on LIC.
[0160] First, the MV is exported from the reference image, which is the encoded image. The MV is used to obtain the reference image corresponding to the encoded object block.
[0161] Secondly, for the encoded object block, the brightness pixel values of the left and top adjacent encoded surrounding reference areas, and the brightness pixel values of the corresponding positions within the reference image specified by MV, are used to extract information on how the display brightness value changes between the reference image and the encoded object image, and then the brightness correction parameters are calculated.
[0162] For the reference image within the reference image specified by MV, the aforementioned brightness correction parameters are used to perform brightness correction processing, thereby generating a predicted image relative to the encoded object block.
[0163] Furthermore, the shape of the aforementioned surrounding reference area in Figure 9D is just one example; other shapes may also be used.
[0164] Furthermore, the process of generating a prediction image from a single reference image has already been explained here. However, the same applies to generating a prediction image from multiple reference images. First, the reference images obtained from each reference image are subjected to brightness correction processing in the same way, and then the prediction image is generated.
[0165] Regarding methods for determining whether LIC processing is applicable, one approach is to use `lic_flag`, which is a signal indicating whether LIC processing is applicable. Specifically, in the encoding device, it is determined whether the area to be encoded belongs to a region experiencing brightness changes. If it does, `lic_flag` is set to 1, and LIC processing is applied for encoding. If it does not belong to a region experiencing brightness changes, `lic_flag` is set to 0, and LIC processing is not applied for encoding. On the other hand, in the decoding device, the `lic_flag` recorded in the stream is decoded, and the application of LIC processing is switched according to this value during decoding.
[0166] Another method for determining whether LIC processing is applicable is based on whether LIC processing has been applied to surrounding blocks. For example, when the encoded target block is in merge mode, the method determines whether the surrounding encoded blocks selected during the MV export in merge mode processing have been encoded using LIC processing. Based on the result, it switches whether LIC processing is applied and proceeds with encoding accordingly. Furthermore, the decoding process is exactly the same in this example. [Overview of the Decoding Device]
[0167] Next, an overview of the decoding device will be provided. This decoding device can decode the encoded signal (encoded bit stream) output from the encoding device 100 described above. Figure 10 is a block diagram showing the functional configuration of the decoding device 200 in Embodiment 1. The decoding device 200 is a motion image / image decoding device that decodes motion images / images in block units.
[0168] As shown in Figure 10, the decoding device 200 includes: an entropy decoding unit 202, an inverse quantization unit 204, an inverse conversion unit 206, an addition unit 208, a block memory 210, a loop filter unit 212, a frame memory 214, an internal prediction unit 216, an inter-prediction unit 218, and a prediction control unit 220.
[0169] The decoding device 200 can be implemented, for example, using a general-purpose processor and memory. When the software program stored in memory is executed by the processor, the processor operates as an entropy decoding unit 202, an inverse quantization unit 204, an inverse conversion unit 206, an adder 208, a loop filter 212, an internal prediction unit 216, an inter-prediction unit 218, and a prediction control unit 220. Alternatively, the decoding device 200 can be implemented as one or more dedicated electronic circuits corresponding to the entropy decoding unit 202, the inverse quantization unit 204, the inverse conversion unit 206, the adder 208, the loop filter 212, the internal prediction unit 216, the inter-prediction unit 218, and the prediction control unit 220.
[0170] The following describes the components of the decoding device 200. [Entropy Decoding Department]
[0171] The entropy decoding unit 202 performs entropy decoding on the encoded bitstream. Specifically, the entropy decoding unit 202 performs, for example, arithmetic decoding to convert the encoded bitstream into a binary signal. Next, the entropy decoding unit 202 debinarizes the binary signal. In this way, the entropy decoding unit 202 outputs the quantization coefficients to the dequantization unit 204 in block units. [Anti-Quantitative Department]
[0172] The dequantization unit 204 dequantizes the quantization coefficients of the decoded target block (hereinafter referred to as the current block), where the decoded target block is the input from the entropy decoding unit 202. Specifically, the dequantization unit 204 dequantizes each quantization coefficient of the current block according to the quantization parameter corresponding to that quantization coefficient. Then, the dequantization unit 204 outputs the dequantized quantization coefficients (i.e., conversion coefficients) of the current block to the inverse conversion unit 206. [Reverse Conversion Section]
[0173] The inverse conversion unit 206 performs an inverse conversion on the conversion coefficients to restore the prediction error, wherein the conversion coefficients are input from the inverse quantization unit 204.
[0174] For example, when the information interpreted from the encoded bit stream indicates that EMT or AMT is applicable (e.g., the AMT flag is true), the inverse conversion unit 206 performs inverse conversion on the conversion coefficients of the current block based on the information of the conversion type interpreted from the display.
[0175] Furthermore, for example, when the information already interpreted from the encoded bit stream is applicable to NSST, the inverse conversion unit 206 applies an inverse reconversion to the conversion coefficients. [Addition Department]
[0176] The addition unit 208 adds the prediction error to the prediction sample to reconstruct the current block, wherein the prediction error is the input from the inversion unit 206 and the prediction sample is the input from the prediction control unit 220. Then, the addition unit 208 outputs the reconstructed block to the block memory 210 and the loop filter unit 212. [Block Memory]
[0177] Block memory 210 is a memory unit used to store blocks referenced in the internal prediction and within the decoded object image (hereinafter referred to as the current image). Specifically, block memory 210 stores the reconstructed blocks output from the adder 208. [Loop Filtering Section]
[0178] The loop filter unit 212 performs loop filtering on the block that has been reconstructed by the adder unit 208, and outputs the filtered reconstructed block to the frame memory 214 and the display device, etc.
[0179] The information displayed shows whether the ALF is on or off as interpreted from the encoded bitstream. When the ALF is on, based on the direction and activity of a portion of the gradient, one filter is selected from a plurality of filters, and the selected filter is applied to reconstruct the block. [Frame Memory]
[0180] Frame memory 214 is a memory unit used to store reference images used in in-between prediction, and is sometimes referred to as a frame buffer. Specifically, frame memory 214 stores the reconstructed blocks filtered by loop filter 212. [Internal Forecasting Department]
[0181] The internal prediction unit 216 performs internal prediction based on the internal prediction pattern already interpreted from the encoded bit stream, and references the blocks within the current image stored in the block memory 210 to generate a prediction signal (internal prediction signal). Specifically, the internal prediction unit 216 performs internal prediction by referencing samples (e.g., luminance values, chrominance values) of blocks adjacent to the current block, thereby generating an internal prediction signal, and outputs the internal prediction signal to the prediction control unit 220.
[0182] In addition, when the internal prediction mode of the reference luminance block is selected in the internal prediction of the color difference block, the internal prediction unit 216 can also predict the color difference component of the current block based on the luminance component of the current block.
[0183] Furthermore, when the information interpreted from the encoded bitstream shows the application of PDPC, the internal prediction unit 216 corrects the pixel value after internal prediction based on the gradient of the reference pixel in the horizontal / vertical direction. [Inter-departmental Forecasting]
[0184] Inter-prediction unit 218 uses reference images stored in reference frame memory 214 to predict the current block. Prediction is performed on a unit of the current block or sub-blocks within the current block (e.g., 4×4 blocks). For example, inter-prediction unit 218 uses motion information (e.g., motion vectors) already interpreted from the coded bit stream to perform motion compensation, thereby generating an inter-prediction signal for the current block or sub-block, and outputs the inter-prediction signal to prediction control unit 220.
[0185] Furthermore, when the information interpreted from the encoded bit stream indicates that OBMC mode is applicable, the interpretation unit 218 not only uses the movement information of the current block obtained through movement estimation, but also uses the movement information of adjacent blocks to generate an interpretation signal.
[0186] Furthermore, when the information interpreted from the encoded bitstream indicates that FRUC mode is applicable, the intermediate prediction unit 218 performs motion estimation based on the pattern matching method (bidirectional matching or template matching) interpreted from the encoded stream, thereby deriving motion information. Then, the intermediate prediction unit 218 uses the derived motion information to perform motion compensation.
[0187] Furthermore, when the BIO mode is applied, the inter-prediction unit 218 derives the movement vector based on a model assuming constant velocity linear motion. Also, when the information interpreted from the encoded bitstream indicates that the affine motion compensation prediction mode is applicable, the inter-prediction unit 218 derives the movement vector in sub-block units based on the movement vectors of a plurality of adjacent blocks. [Forecasting and Control Department]
[0188] The prediction control unit 220 selects either the internal prediction signal or the inter-prediction signal, and outputs the selected signal as the prediction signal to the adder 208. (Implementation Form 2)
[0189] The encoding and decoding processes of Embodiment 2 will be specifically described with reference to Figures 11 and 12, and the encoding and decoding devices of Embodiment 2 will be specifically described with reference to Figures 15 and 16. [Encoding Processing]
[0190] Figure 11 shows the image encoding process of embodiment 2.
[0191] First, in step S1001, a first parameter is written into the bit stream, wherein the first parameter is a segmentation pattern identified from a plurality of segmentation patterns to divide the first block into a plurality of sub-blocks. If a segmentation pattern is used, the block is divided into a plurality of sub-blocks. If different segmentation patterns are used, the block is divided into a plurality of sub-blocks with different shapes, different heights, or different widths.
[0192] Figure 28 shows an example of a segmentation mode used to divide an NxN pixel block in Embodiment 2. In Figure 28, (a) to (h) show different segmentation modes. As shown in Figure 28, when segmentation mode (a) is used, an NxN pixel block (e.g., 16x16 pixels, where "N" can be any multiple of 4 between 8 and 128) is divided into two N / 2xN pixel (e.g., 8x16 pixel) sub-blocks. When segmentation mode (b) is used, the NxN pixel block is divided into an N / 4xN pixel (e.g., 4x16 pixel) sub-block and a 3N / 4xN pixel (e.g., 12x16 pixel) sub-block. When segmentation mode (c) is used, the NxN pixel block is divided into a 3N / 4xN pixel (e.g., 12x16 pixel) sub-block and an N / 4xN pixel (e.g., 4x16 pixel) sub-block. When using segmentation mode (d), an NxN pixel block is divided into (N / 4)xN pixel (e.g., 4x16 pixel) sub-blocks, N / 2xN pixel (e.g., 8x16 pixel) sub-blocks, and N / 4xN pixel (e.g., 4x16 pixel) sub-blocks. When using segmentation mode (e), an NxN pixel block is divided into two NxN / 2 pixel (e.g., 16x8 pixel) sub-blocks. When using segmentation mode (f), an NxN pixel block is divided into NxN / 4 pixel (e.g., 16x4 pixel) sub-blocks and Nx3N / 4 pixel (e.g., 16x12 pixel) sub-blocks. When using segmentation mode (g), an NxN pixel block is divided into Nx3N / 4 pixel (e.g., 16x12 pixel) sub-blocks and NxN / 4 pixel (e.g., 16x4 pixel) sub-blocks. When using the segmentation mode (h), an NxN pixel block will be segmented into NxN / 4 pixel (e.g., 16x4 pixel) sub-blocks, NxN / 2 pixel (e.g., 16x8 pixel) sub-blocks, and NxN / 4 pixel (e.g., 16x4 pixel) sub-blocks.
[0193] Secondly, in step S1002, it is determined whether the first parameter has identified the first segmentation pattern.
[0194] Secondly, in step S1003, at least based on whether the first segmentation mode has been identified by the first parameter, it is determined whether the second segmentation mode should not be selected as a candidate for segmenting the second block.
[0195] Two different sets of partitioning patterns may divide a block into sub-blocks of the same shape and size. For example, as shown in Figure 31A, the sub-blocks in (1b) and (2c) have the same shape and size. A set of partitioning patterns may include at least two partitioning patterns. For example, as shown in (1a) and (1b) of Figure 31A, a set of partitioning patterns may include, in addition to ternary tree vertical partitioning, binary tree vertical partitioning of the middle sub-block and non-partitioning of other sub-blocks. As another example, as shown in (2a), (2b) and (2c) of Figure 31A, other sets of partitioning patterns may include, in addition to binary tree vertical partitioning, binary tree vertical partitioning of the two outer sub-blocks. Any set of partitioning patterns results in sub-blocks of the same shape and size.
[0196] When selecting from two partitioning pattern sets, the partitioning pattern set with fewer bits or fewer bytes is chosen. These two partitioning pattern sets divide the block into sub-blocks of the same shape and size, and are encoded with different numbers of bits or different numbers of bytes in the bitstream. Note that the number of bits and bytes are equivalent to the amount of code.
[0197] When selecting from two segmentation pattern sets, the first segmentation pattern set to appear in a predetermined order is chosen. These two segmentation pattern sets divide the block into sub-blocks of the same shape and size, and are encoded with the same number of bits or binary representations in the bitstream. The predetermined order can also be, for example, based on the number of segmentation patterns within each segmentation pattern set.
[0198] Figures 31A and 31B illustrate an example of using a smaller set of segmentation patterns (with fewer bits) to divide a block into sub-blocks in the encoding of segmentation patterns. In this example, when the left NxN pixel block has been vertically divided into two sub-blocks, the second segmentation pattern for the right NxN pixel block is not selected in step (2c). This is because, in the segmentation pattern encoding method of Figure 31B, the second segmentation pattern set (2a, 2b, 2c) requires more bits for encoding the segmentation patterns compared to the first segmentation pattern set (1a, 1b).
[0199] Figures 32A to 32C illustrate an example of dividing a block into sub-blocks using the first segmentation pattern set appearing in a predetermined order from a plurality of segmentation pattern sets. In this example, when a 2NxN / 2 pixel block is vertically divided into three sub-blocks, in step (2c), the second segmentation pattern used for the lower 2NxN / 2 pixel block is not selected. This is because in the segmentation pattern encoding method of Figure 32B, the second segmentation pattern set (2a, 2b, 2c) has the same binary number as the first segmentation pattern set (1a, 1b, 1c, 1d), and the segmentation pattern set shown in Figure 32C appears after the first segmentation pattern set (1a, 1b, 1c, 1d) in the predetermined order. The predetermined order of the plurality of segmentation pattern sets can also be fixed or signaled within the bitstream.
[0200] Figure 20 shows an example of how, in Embodiment 2, a 2NxN pixel block is divided without selecting the second segmentation mode, as shown in step (2c). As shown in Figure 20, using the first segmentation method (i), as in step (1a), a 2Nx2N pixel (e.g., 16x16 pixel) block can be divided into four NxN pixel (e.g., 8x8 pixel) sub-blocks. Similarly, using the second segmentation method (ii), as in step (2a), a 2Nx2N pixel block can also be horizontally divided into two 2NxN pixel (e.g., 16x8 pixel) sub-blocks. Here, in the second segmentation method (ii), as in step (2b), when the upper 2NxN pixel block (first block) is vertically divided into two NxN pixel sub-blocks using the first segmentation mode, in step (2c), the second segmentation mode is not selected as a candidate segmentation mode, where the second segmentation mode is the mode that vertically divides the lower 2NxN pixel block (second block) into two NxN pixel sub-blocks. This is because it would result in a sub-block size that is the same as the sub-block size obtained through the four-segmentation of the first segmentation method (i).
[0201] As shown above, in Figure 20, if the first segmentation mode is used, the first block is equally divided into two sub-blocks in the vertical direction. If the second segmentation mode is used, the second segmentation mode will not be selected as a candidate when the second block, which is vertically adjacent to the first block, is equally divided into two sub-blocks in the vertical direction.
[0202] Figure 21 shows an example of how, in Embodiment 2, the Nx2N pixel block is divided without selecting the second segmentation mode, as shown in step (2c). As shown in Figure 21, the first segmentation method (i), as in step (1a), can be used to divide the 2Nx2N pixel block into four NxN pixel sub-blocks. Alternatively, the second segmentation method (ii), as in step (2a), can be used to vertically divide the 2Nx2N pixel block into two 2NxN pixel (e.g., 8x16 pixel) sub-blocks. In the second segmentation method (ii), as in step (2b), when the left Nx2N pixel block (the first block) is horizontally divided into two NxN pixel sub-blocks using the first segmentation mode, in step (2c), the second segmentation mode, which is to horizontally divide the right Nx2N pixel block (the second block) into two NxN pixel sub-blocks, will not be selected as a candidate segmentation mode. This is because it would result in sub-blocks of the same size as those obtained through the four-segmentation of the first segmentation method (i).
[0203] As shown above, in Figure 21, if the first segmentation mode is used, the first block is equally divided into two sub-blocks in the horizontal direction. If the second segmentation mode is used, the second segmentation mode is not selected as a candidate when the second block, which is horizontally adjacent to the first block, is equally divided into two sub-blocks in the horizontal direction.
[0204] Figure 22 shows an example in Embodiment 2 where, as shown in step (2c), the second segmentation mode is not selected for the segmentation of an NxN pixel block. As shown in Figure 22, the first segmentation method (i), as in step (1a), can be used to vertically segment a 2NxN pixel block (e.g., 16x8 pixels, where "N" can be any multiple of 4 from 8 to 128) into N / 2xN pixel sub-blocks, NxN pixel sub-blocks, and N / 2xN pixel sub-blocks (e.g., 4x8 pixel sub-blocks, 8x8 pixel sub-blocks, and 4x8 pixel sub-blocks). Alternatively, the second segmentation method (ii), as in step (2a), can be used to segment a 2NxN pixel block into two NxN pixel sub-blocks. In the first segmentation method (i), in step (1b), the central NxN pixel block can be vertically divided into two N / 2xN pixel (e.g., 4x8 pixel) sub-blocks. In the second segmentation method (ii), as in step (2b), when the left NxN pixel block (the first block) is vertically divided into two N / 2xN pixel sub-blocks, in step (2c), the segmentation pattern of vertically dividing the right NxN pixel block (the second block) into two N / 2xN pixel sub-blocks is not selected as a candidate for possible segmentation patterns. This is because it produces sub-blocks with the same size as those obtained through the first segmentation method (i), i.e., four N / 2xN pixel sub-blocks.
[0205] As shown above, in Figure 22, if the first segmentation mode is used, the first block is equally divided into two sub-blocks in the vertical direction. If the second segmentation mode is used, the second segmentation mode is not selected as a candidate when the second block, which is adjacent to the first block in the horizontal direction, is equally divided into two sub-blocks in the vertical direction.
[0206] Figure 23 shows an example in Embodiment 2 where, as shown in step (2c), the second segmentation mode is not selected for the segmentation of an NxN pixel block. As shown in Figure 23, the first segmentation method (i), as in step (1a), can be used to segment an Nx2N pixel block (e.g., 8x16 pixels, where "N" can be any multiple of 4 from 8 to 128) into NxN / 2 pixel sub-blocks, NxN pixel sub-blocks, and NxN / 2 pixel sub-blocks (e.g., 8x4 pixel sub-blocks, 8x8 pixel sub-blocks, and 8x4 pixel sub-blocks). Alternatively, the second segmentation method, as in step (2a), can be used to segment into two NxN pixel sub-blocks. In the first segmentation method (i), as in step (1b), the central NxN pixel block can be segmented into two NxN / 2 pixel sub-blocks. In the second segmentation method (ii), as in step (2b), when the upper NxN pixel block (the first block) is horizontally segmented into two NxN / 2 pixel sub-blocks, in step (2c), the segmentation pattern of horizontally segmenting the lower NxN pixel block (the second block) into two NxN / 2 pixel sub-blocks is not selected as a candidate for possible segmentation patterns. This is because it produces sub-blocks with the same size as those obtained through the first segmentation method (i), namely, sub-blocks of 4 NxN / 2 pixels.
[0207] As shown above, in Figure 23, if the first segmentation mode is used, the first block is equally divided into two sub-blocks in the horizontal direction. If the second segmentation mode is used, the second segmentation mode is not selected as a candidate when the second block, which is vertically adjacent to the first block, is equally divided into two sub-blocks in the horizontal direction.
[0208] If the second segmentation pattern set has been determined to be selected as a candidate for segmenting the second block (N in S1003), in step S1004, a segmentation pattern is selected from the plurality of segmentation patterns, including the second segmentation pattern as a candidate. In step S1005, the second parameter, which will display the selection result, is written to the bit stream.
[0209] If it has been determined that the second segmentation mode is not selected as a candidate for segmenting the second block (as in S1003), in step S1006, a segmentation mode different from the second segmentation mode is selected to segment the second block. The selected segmentation mode divides the block into sub-blocks with different shapes or sizes compared to the sub-blocks generated by the second segmentation mode.
[0210] Figure 24 shows an example of segmenting a 2NxN pixel block using the selected segmentation mode in Embodiment 2, as in step (3), when the second segmentation mode is not selected. As shown in Figure 24, the selected segmentation mode, as shown in Figures 24(c) and (f), divides the current 2NxN pixel block (the lower block in this example) into three sub-blocks. The sizes of the three sub-blocks can also be different. For example, in the three sub-blocks, the larger sub-block can also have twice the width / height of the smaller sub-block. As another example, the selected segmentation mode, as shown in Figures 24(a), (b), (d), and (e), can also divide the current block into two sub-blocks of different sizes (asymmetric binary tree). For example, when using an asymmetric binary tree, the larger sub-block can have three times the width / height of the smaller sub-block.
[0211] Figure 25 shows an example of segmenting an Nx2N pixel block using the selected segmentation mode in Embodiment 2, as shown in step (3), when the second segmentation mode is not selected. As shown in Figure 25, the selected segmentation mode can segment the current Nx2N pixel block (the right block in this example), as shown in Figures 25(c) and (f), into three sub-blocks. The sizes of the three sub-blocks can also be different. For example, in the three sub-blocks, the larger sub-block can also have twice the width / height of the smaller sub-block. As another example, the selected segmentation mode can also segment the current block, as shown in Figures 25(a), (b), (d), and (e), into two sub-blocks of different sizes (asymmetric binary tree). For example, when using an asymmetric binary tree, the larger sub-block can have three times the width / height of the smaller sub-block.
[0212] Figure 26 shows an example of dividing an NxN pixel block using the selected segmentation mode in Embodiment 2, as shown in step (3), when the second segmentation mode is not selected. As shown in Figure 26, in step (1), the 2NxN pixel block is vertically divided into two NxN pixel sub-blocks. In step (2), the left NxN pixel block is vertically divided into two N / 2xN pixel sub-blocks. In step (3), using the selected segmentation mode for the current NxN pixel block (the left block in this example), as shown in (c) and (f) of Figure 26, the current block can be divided into three sub-blocks. The sizes of the three sub-blocks can also be different. For example, in the three sub-blocks, the larger sub-block can also have twice the width / height of the smaller sub-block. For example, the selected partitioning pattern can also divide the current block, as shown in Figures 26(a), (b), (d), and (e), into two sub-blocks of different sizes (asymmetric binary tree). For instance, when using an asymmetric binary tree, the larger sub-block can have three times the width / height of the smaller sub-block.
[0213] Figure 27 shows an example of dividing an NxN pixel block using the selected segmentation mode in Embodiment 2, as shown in step (3), when the second segmentation mode is not selected. As shown in Figure 27, in step (1), the Nx2N pixel block is horizontally divided into two NxN pixel sub-blocks. In step (2), the upper NxN pixel block is horizontally divided into two NxN / 2 pixel sub-blocks. In step (3), using the selected segmentation mode for the current NxN pixel block (the lower block in this example), as shown in (c) and (f) of Figure 27, the current block can be divided into three sub-blocks. The sizes of the three sub-blocks can also be different. For example, in the three sub-blocks, the larger sub-block can also have twice the width / height of the smaller sub-block. For example, the selected partitioning pattern can also divide the current block, as shown in Figures 27(a), (b), (d), and (e), into two sub-blocks of different sizes (asymmetric binary tree). For instance, when using an asymmetric binary tree, the larger sub-block can have a width / height three times that of the smaller sub-block.
[0214] Figure 17 shows the conceivable locations of the first parameter within a compressed video stream. As shown in Figure 17, the first parameter can be configured within a video parameter set, sequence parameter set, picture parameter set, slice header, or coding tree unit. The first parameter can indicate the method for dividing a block into multiple sub-blocks. For example, the first parameter can contain a flag indicating whether the block is divided horizontally or vertically. The first parameter can also include parameters indicating whether to divide the block into two or more sub-blocks.
[0215] Figure 18 shows the conceivable locations of the second parameter within the compressed video stream. As shown in Figure 18, the second parameter can be configured within a video parameter set, sequence parameter set, picture parameter set, slice header, or coding tree unit. The second parameter can indicate the method of dividing a block into multiple sub-blocks. For example, the second parameter can contain a flag indicating whether the block is divided horizontally or vertically. The second parameter can also include parameters indicating whether the block is divided into two or more sub-blocks. As shown in Figure 19, the second parameter is configured within the bit stream following the first parameter.
[0216] Block 1 and Block 2 are different blocks. Block 1 and Block 2 may also be contained in the same frame. For example, Block 1 may be the block above Block 2. Also, for example, Block 1 may be the block to the left of Block 2.
[0217] In step S1007, the second block is divided into sub-blocks using the selected segmentation mode. In step S1008, the segmented blocks are encoded. [Encoding device]
[0218] Figure 15 is a block diagram showing the configuration of the image / image encoding device in embodiment 2 or 3.
[0219] The image encoding apparatus 5000 is an apparatus for encoding the input image / image according to each block to generate an encoded output bitstream. As shown in FIG15, the image encoding apparatus 5000 includes: a conversion unit 5001, a quantization unit 5002, an inverse quantization unit 5003, an inverse conversion unit 5004, a block memory 5005, a frame memory 5006, an intra-prediction unit 5007, an inter-prediction unit 5008, an entropy encoding unit 5009, and a block segmentation determination unit 5010.
[0220] The input image is input to the adder, and the added value is input to the conversion unit 5001. The conversion unit 5001 converts the added value into frequency coefficients based on the block segmentation pattern derived by the block segmentation determination unit 5010, and outputs the frequency coefficients to the quantization unit 5002. The block segmentation pattern can be associated with a block segmentation mode, block segmentation type, or block segmentation direction. The quantization unit 5002 quantizes the input quantization coefficients and inputs the quantized value to the inverse quantization unit 5003 and the entropy encoding unit 5009.
[0221] The dequantization unit 5003 dequantizes the quantized value output from the quantization unit 5002 and outputs the frequency coefficient to the inverse conversion unit 5004. The inverse conversion unit 5004 performs inverse frequency conversion on the frequency coefficient according to the block partitioning mode derived by the block partitioning determination unit 5010, converts the frequency coefficient into sample values of the bit stream, and outputs the sample values to the adder.
[0222] The adder adds the sample values of the bitstream output from the inverse conversion unit 5004 to the predicted image / image values output from the intermediate / intermediate prediction units 5007 and 5008, and outputs the added value to the block memory 5005 or the frame memory 5006 for further prediction. The block segmentation determination unit 5010 collects block information from the block memory 5005 or the frame memory 5006 and derives parameters related to the block segmentation mode. If the derived block segmentation mode is used, the block will be segmented into multiple sub-blocks. The intermediate / intermediate prediction units 5007 and 5008 explore the images / images stored in the block memory 5005, or the images / images in the frame memory 5006 reconstructed using the block segmentation mode derived by the block segmentation determination unit 5010, and estimate, for example, the image / image region most similar to the input image of the prediction target.
[0223] The entropy encoding unit 5009 encodes the quantized value output from the quantization unit 5002, encodes the parameters from the block segmentation determination unit 5010, and outputs a bit stream. [Decoding Processing]
[0224] Figure 12 shows the image decoding process of embodiment 2.
[0225] First, in step S2001, a first parameter is interpreted from the bit stream. This first parameter is the segmentation pattern used to divide the first block into sub-blocks from a plurality of segmentation patterns. If a segmentation pattern is used, the block is divided into sub-blocks. If different segmentation patterns are used, the block is divided into sub-blocks with different shapes, different heights, or different widths.
[0226] Figure 28 shows an example of a segmentation mode for dividing an NxN pixel block in Embodiment 2. In Figure 28, (a) to (h) show different segmentation modes. As shown in Figure 28, if segmentation mode (a) is used, an NxN pixel block (e.g., 16x16 pixels, where the value of "N" can be any multiple of 4 from 8 to 128) is divided into two N / 2xN pixel (e.g., 8x16 pixel) sub-blocks. If segmentation mode (b) is used, the NxN pixel block is divided into an N / 4xN pixel (e.g., 4x16 pixel) sub-block and a 3N / 4xN pixel (e.g., 12x16 pixel) sub-block. If segmentation mode (c) is used, the NxN pixel block is divided into a 3N / 4xN pixel (e.g., 12x16 pixel) sub-block and an N / 4xN pixel (e.g., 4x16 pixel) sub-block. If segmentation mode (d) is used, an NxN pixel block will be divided into (N / 4)xN pixel (e.g., 4x16 pixel) sub-blocks, N / 2xN pixel (e.g., 8x16 pixel) sub-blocks, and N / 4xN pixel (e.g., 4x16 pixel) sub-blocks. If segmentation mode (e) is used, an NxN pixel block will be divided into two NxN / 2 pixel (e.g., 16x8 pixel) sub-blocks. If segmentation mode (f) is used, an NxN pixel block will be divided into NxN / 4 pixel (e.g., 16x4 pixel) sub-blocks and Nx3N / 4 pixel (e.g., 16x12 pixel) sub-blocks. If segmentation mode (g) is used, an NxN pixel block will be divided into Nx3N / 4 pixel (e.g., 16x12 pixel) sub-blocks and NxN / 4 pixel (e.g., 16x4 pixel) sub-blocks. If the segmentation mode (h) is used, the NxN pixel block will be segmented into NxN / 4 pixel (e.g., 16x4 pixel) sub-blocks, NxN / 2 pixel (e.g., 16x8 pixel) sub-blocks, and NxN / 4 pixel (e.g., 16x4 pixel) sub-blocks.
[0227] Secondly, in step S2002, it is determined whether the first parameter has identified the first segmentation pattern.
[0228] Secondly, in step S2003, at least based on the determination of whether the first segmentation mode has been identified by the first parameter, it is determined whether the second segmentation mode should not be selected as a candidate for segmenting the second block.
[0229] Two different sets of partitioning patterns may divide a block into sub-blocks of the same shape and size. For example, as shown in Figure 31A, the sub-blocks in (1b) and (2c) have the same shape and size. A set of partitioning patterns may contain at least two partitioning patterns. For example, as shown in (1a) and (1b) of Figure 31A, a set of partitioning patterns, following a ternary tree vertical partition, may also include a binary tree vertical partition of the middle sub-block and non-partitioning of other blocks. As another example, as shown in (2a), (2b) and (2c) of Figure 31A, other sets of partitioning patterns, following a binary tree vertical partition, may also include binary tree vertical partitions of the two outer sub-blocks. Any set of partitioning patterns will also result in sub-blocks of the same shape and size.
[0230] When selecting from two partitioning pattern sets, the partitioning pattern set with fewer binary bits or fewer bits is selected, wherein the two partitioning pattern sets divide the block into sub-blocks of the same shape and size, and are encoded in the bit stream with different binary bits or different bit numbers.
[0231] When selecting from two segmentation pattern sets, the first segmentation pattern set that appears in a predetermined order is chosen, wherein the two segmentation pattern sets divide the block into sub-blocks of the same shape and size, and are encoded in the bit stream with the same number of bits or the same number of bytes. The predetermined order may be, for example, based on the number of segmentation patterns in each segmentation pattern set.
[0232] Figures 31A and 31B illustrate an example of dividing a block into sub-blocks using a set of segmentation patterns with fewer bits in the encoding of segmentation patterns. In this example, when the NxN pixel block on the left is vertically divided into two sub-blocks, in step (2c), the second segmentation pattern used for the NxN pixel block on the right is not selected. This is because, in the encoding method of the segmentation patterns in Figure 31B, the second segmentation pattern set (2a, 2b, 2c) requires more bits for encoding the segmentation patterns compared to the first segmentation pattern set (1a, 1b).
[0233] Figure 32A shows an example of dividing a block into sub-blocks using the first segmentation pattern set appearing in a predetermined order among a plurality of segmentation pattern sets. In this example, when a 2NxN / 2 pixel block is vertically divided into 3 sub-blocks, in step (2c), the second segmentation pattern used for the lower 2NxN / 2 pixel block is not selected. This is because in the segmentation pattern encoding method of Figure 32B, the second segmentation pattern set (2a, 2b, 2c) has the same binary number as the first segmentation pattern set (1a, 1b, 1c, 1d), and appears after the first segmentation pattern set (1a, 1b, 1c, 1d) in the predetermined order of the segmentation pattern sets shown in Figure 32C. The predetermined order of the plurality of segmentation pattern sets can also be fixed, or it can be signaled within the bitstream.
[0234] Figure 20 shows an example in Embodiment 2 where, as shown in step (2c), the second segmentation mode is not selected for the segmentation of a 2NxN pixel block. As shown in Figure 20, using the first segmentation method (i), as in step (1a), a 2Nx2N pixel (e.g., 16x16 pixel) block can be divided into four NxN pixel (e.g., 8x8 pixel) sub-blocks. Furthermore, using the second segmentation method (ii), as in step (2a), a 2Nx2N pixel block can also be horizontally divided into two 2NxN pixel (e.g., 16x8 pixel) sub-blocks. In the second segmentation method (ii), as in step (2b), when the upper 2NxN pixel block (first block) is vertically divided into two NxN pixel sub-blocks using the first segmentation mode, in step (2c), the second segmentation mode is not selected as a candidate for a possible segmentation mode, wherein the second segmentation mode is the mode that vertically divides the lower 2NxN pixel block (second block) into two NxN pixel sub-blocks. This is because it would result in sub-blocks with the same size as those obtained through the four-segmentation of the first segmentation method (i).
[0235] As shown above, in Figure 20, if the first segmentation mode is used, the first block is equally divided into two sub-blocks in the vertical direction. If the second segmentation mode is used, the second segmentation mode will not be selected as a candidate when the second block, which is vertically adjacent to the first block, is equally divided into two sub-blocks in the vertical direction.
[0236] Figure 21 shows an example of implementing embodiment 2, where, as shown in step (2c), the Nx2N pixel block is divided without selecting the second segmentation mode. As shown in Figure 21, using the first segmentation method (i), as in step (1a), the 2Nx2N pixel block can be equally divided into four NxN pixel sub-blocks. Furthermore, using the second segmentation method (ii), as in step (2a), the 2Nx2N pixel block can also be vertically and equally divided into two 2NxN pixel (e.g., 8x16 pixel) sub-blocks. In the second segmentation method (ii), as in step (2b), when the left Nx2N pixel block (first block) is horizontally divided into two NxN pixel sub-blocks using the first segmentation mode, in step (2c), the second segmentation mode is not selected as a candidate segmentation mode, whereby the second segmentation mode is to horizontally divide the right Nx2N pixel block (second block) into two NxN pixel sub-blocks. This is because it would result in sub-blocks with the same size as those obtained through the four-segmentation of the first segmentation method (i).
[0237] As shown above, in Figure 21, if the first segmentation mode is used, the first block is equally divided into two sub-blocks in the horizontal direction. If the second segmentation mode is used, the second segmentation mode is not selected as a candidate when the second block, which is horizontally adjacent to the first block, is equally divided into two sub-blocks in the horizontal direction.
[0238] Figure 22 shows an example in Embodiment 2 where, as shown in step (2c), the second segmentation mode is not selected for the segmentation of an NxN pixel block. As shown in Figure 22, using the first segmentation method (i), as in step (1a), a 2NxN pixel block (e.g., 16x8 pixels, where "N" can be any multiple of 4 from 8 to 128) can be vertically segmented into N / 2xN pixel sub-blocks, NxN pixel sub-blocks, and N / 2xN pixel sub-blocks (e.g., 4x8 pixel sub-blocks, 8x8 pixel sub-blocks, and 4x8 pixel sub-blocks). Furthermore, using the second segmentation method (ii), as in step (2a), a 2NxN pixel block can also be segmented into two NxN pixel sub-blocks. In the first segmentation method (i), in step (1b), the central NxN pixel block can be vertically divided into two N / 2xN pixel (e.g., 4x8 pixel) sub-blocks. In the second segmentation method (ii), as in step (2b), when the left NxN pixel block (the first block) is vertically divided into two N / 2xN pixel sub-blocks, in step (2c), the segmentation mode of vertically dividing the right NxN pixel block (the second block) into two N / 2xN pixel sub-blocks is not selected as a candidate segmentation mode. This is because it produces a sub-block size that is the same as the sub-block size obtained through the first segmentation method (i), i.e., a sub-block of 4 N / 2xN pixels.
[0239] As shown above, in Figure 22, if the first segmentation mode is used, the first block is equally divided into two sub-blocks in the vertical direction. If the second segmentation mode is used, the second segmentation mode is not selected as a candidate when the second block, which is adjacent to the first block in the horizontal direction, is equally divided into two sub-blocks in the vertical direction.
[0240] Figure 23 shows an example in Embodiment 2 where, as shown in step (2c), the second segmentation mode is not selected for the segmentation of an NxN pixel block. As shown in Figure 23, using the first segmentation method (i), as in step (1a), an Nx2N pixel block (e.g., 8x16 pixels, where "N" can be any multiple of 4 from 8 to 128) can be segmented into NxN / 2 pixel sub-blocks, NxN pixel sub-blocks, and NxN / 2 pixel sub-blocks (e.g., 8x4 pixel sub-blocks, 8x8 pixel sub-blocks, and 8x4 pixel sub-blocks). Furthermore, using the second segmentation method, as in step (2a), it can also be segmented into two NxN pixel sub-blocks. In the first segmentation method (i), as in step (1b), the centrally located NxN pixel block can be segmented into two NxN / 2 pixel sub-blocks. In the second segmentation method (ii), as in step (2b), when the upper NxN pixel block (the first block) is horizontally segmented into two NxN / 2 pixel sub-blocks, in step (2c), the segmentation mode of horizontally segmenting the lower NxN pixel block (the second block) into two NxN / 2 pixel sub-blocks is not selected as a candidate segmentation mode. This is because it produces sub-blocks with the same size as those obtained through the first segmentation method (i), namely, sub-blocks of 4 NxN / 2 pixels.
[0241] As shown above, in Figure 23, if the first segmentation mode is used, the first block is equally divided into two sub-blocks in the horizontal direction. If the second segmentation mode is used, the second segmentation mode is not selected as a candidate when the second block, which is vertically adjacent to the first block, is equally divided into two sub-blocks in the horizontal direction.
[0242] If it has been determined that the second segmentation mode set has been selected as a candidate for segmenting the second block (N in S2003), in step S2004, the second parameter is interpreted from the bit stream, and a segmentation mode is selected from a plurality of segmentation modes including the second segmentation mode as a candidate.
[0243] If it has been determined that the second segmentation mode is not selected as a candidate for segmenting the second block (as in S2003), in step S2005, a segmentation mode different from the second segmentation mode is selected to segment the second block. The selected segmentation mode divides the block into sub-blocks with different shapes or sizes compared to the sub-blocks generated by the second segmentation mode.
[0244] Figure 24 shows an example of segmenting a 2NxN pixel block using the selected segmentation mode in Embodiment 2, as shown in step (3), when the second segmentation mode is not selected. As shown in Figure 24, the selected segmentation mode, as shown in Figures 24(c) and (f), can segment the current 2NxN pixel block (the lower block in this example) into 3 sub-blocks. The sizes of the 3 sub-blocks can also be different. For example, in the 3 sub-blocks, the larger sub-block can also have twice the width / height of the smaller sub-block. As another example, the selected segmentation mode, as shown in Figures 24(a), (b), (d), and (e), can also segment the current block into 2 sub-blocks of different sizes (asymmetric binary tree). For example, when using an asymmetric binary tree, the larger sub-block can have three times the width / height of the smaller sub-block.
[0245] Figure 25 shows an example of segmenting an Nx2N pixel block using the selected segmentation mode in Embodiment 2, as shown in step (3), when the second segmentation mode is not selected. As shown in Figure 25, the selected segmentation mode can segment the current Nx2N pixel block (the right block in this example), as shown in Figures 25(c) and (f), into three sub-blocks. The sizes of the three sub-blocks can also be different. For example, in the three sub-blocks, the larger sub-block can also have twice the width / height of the smaller sub-block. As another example, the selected segmentation mode can also segment the current block, as shown in Figures 25(a), (b), (d), and (e), into two sub-blocks of different sizes (asymmetric binary tree). For example, when using an asymmetric binary tree, the larger sub-block can have three times the width / height of the smaller sub-block.
[0246] Figure 26 shows an example of segmenting an NxN pixel block using the selected segmentation mode in Embodiment 2, as shown in step (3), when the second segmentation mode is not selected. As shown in Figure 26, in step (1), the 2NxN pixel block is vertically segmented into two NxN pixel sub-blocks. In step (2), the left NxN pixel block is vertically segmented into two N / 2xN pixel sub-blocks. In step (3), using the selected segmentation mode for the current NxN pixel block (the left block in this example), as shown in (c) and (f) of Figure 26, the current block can be segmented into three sub-blocks. The sizes of the three sub-blocks can also be different. For example, in the three sub-blocks, the larger sub-block can also have twice the width / height of the smaller sub-block. For example, the selected partitioning pattern can also divide the current block, as shown in Figures 26(a), (b), (d), and (e), into two sub-blocks of different sizes (asymmetric binary tree). For instance, when using an asymmetric binary tree, the larger sub-block can have a width / height three times that of the smaller sub-block.
[0247] Figure 27 shows an example of segmenting an NxN pixel block using the selected segmentation mode in Embodiment 2, as shown in step (3), when the second segmentation mode is not selected. As shown in Figure 27, in step (1), the Nx2N pixel block is horizontally segmented into two NxN pixel sub-blocks. In step (2), the upper NxN pixel block is horizontally segmented into two NxN / 2 pixel sub-blocks. In step (3), using the selected segmentation mode for the current NxN pixel block (the lower block in this example), as shown in (c) and (f) of Figure 27, the current block can be segmented into three sub-blocks. The sizes of the three sub-blocks can also be different. For example, in the three sub-blocks, the larger sub-block can also have twice the width / height of the smaller sub-block. For example, the selected partitioning pattern can also divide the current block, as shown in Figures 27(a), (b), (d), and (e), into two sub-blocks of different sizes (asymmetric binary tree). For instance, when using an asymmetric binary tree, the larger sub-block can have three times the width / height of the smaller sub-block.
[0248] Figure 17 shows the conceivable locations of the first parameter within a compressed video stream. As shown in Figure 17, the first parameter can be configured within a video parameter set, sequence parameter set, picture parameter set, slice header, or coding tree unit. The first parameter can indicate the method of dividing a block into multiple sub-blocks. For example, the first parameter may contain a flag indicating whether the block is divided horizontally or vertically. The first parameter may also include parameters indicating whether to divide the block into two or more sub-blocks.
[0249] Figure 18 shows the conceivable locations of the second parameter within the compressed video stream. As shown in Figure 18, the second parameter can be configured within a video parameter set, sequence parameter set, picture parameter set, slice header, or coding tree unit. The second parameter can indicate the method of dividing the block into multiple sub-blocks. For example, the second parameter may contain a flag indicating whether the block is divided horizontally or vertically. The second parameter may also include parameters indicating whether the block is divided into two or more sub-blocks. As shown in Figure 19, the second parameter is configured within the bit stream following the first parameter.
[0250] Block 1 and Block 2 are different blocks. Block 1 and Block 2 may also be contained in the same frame. For example, Block 1 may be the block above Block 2. Also, for example, Block 1 may be the block to the left of Block 2.
[0251] In step S2006, the second block is divided into sub-blocks using the selected segmentation mode. In step S2007, the segmented blocks are decoded. [Decoding device]
[0252] Figure 16 is a block diagram showing the structure of the image / image decoding device in embodiment 2 or 3.
[0253] The image decoding apparatus 6000 is a device used to decode the input encoded bitstream block by block and output the image / picture. As shown in Figure 16, the image decoding apparatus 6000 includes: an entropy decoding unit 6001, an inverse quantization unit 6002, an inverse conversion unit 6003, a block memory 6004, a frame memory 6005, an intra-prediction unit 6006, an inter-prediction unit 6007, and a block segmentation determination unit 6008.
[0254] The input encoded bitstream is input to the entropy decoding unit 6001. After the input encoded bitstream is input to the entropy decoding unit 6001, the entropy decoding unit 6001 decodes the input encoded bitstream, outputs the parameters to the block segmentation determination unit 6008, and outputs the decoded value to the dequantization unit 6002.
[0255] The inverse quantization unit 6002 inverse-quantizes the decoded value and outputs the frequency coefficients to the inverse conversion unit 6003. The inverse conversion unit 6003 performs inverse frequency conversion on the frequency coefficients based on the block segmentation pattern derived by the block segmentation determination unit 6008, converts the frequency coefficients into sample values, and outputs the sample values to the adder. The block segmentation pattern can be associated with a block segmentation mode, block segmentation type, or block segmentation direction. The adder adds the sample values to the predicted image / image values output from the intermediate / intermediate prediction units 6006 and 6007, outputs the added value to the display, and for further prediction, outputs the added value to the block memory 6004 or the frame memory 6005. The block segmentation determination unit 6008 collects block information from the block memory 6004 or the frame memory 6005 and derives the block segmentation pattern using the parameters decoded by the entropy decoding unit 6001. If the exported block segmentation pattern is used, the block will be divided into multiple sub-blocks. Then, the internal / intermediate prediction units 6006 and 6007 predict the image / image region of the target block from the image / image stored in the block memory 6004, or from the image / image in the frame memory 6005 which is reconstructed by the block segmentation pattern exported by the block segmentation determination unit 6008. [Effects, etc.]
[0256] As described above, the image encoding apparatus 5000 of this embodiment is an image encoding apparatus that encodes blocks of an image. It includes a processor and a memory. The processor uses the memory to perform the following steps: writing a first parameter into a bit stream, wherein the first parameter is a segmentation pattern identified from a plurality of segmentation patterns for dividing a first block into sub-blocks; determining whether the first parameter has identified the first segmentation pattern; at least based on the determination that the first parameter has identified the first segmentation pattern, determining whether to not select a second segmentation pattern as a candidate for dividing a second block different from the first block; and when it is determined that the second segmentation pattern has been selected... When a candidate segmentation mode is selected for dividing the aforementioned second block, a segmentation mode is chosen from a plurality of segmentation modes that include the aforementioned second segmentation mode as a candidate, and a second parameter displaying the selection result is written into a bit stream; when it has been determined that the aforementioned second segmentation mode is not selected as a candidate segmentation mode for dividing the aforementioned second block, a segmentation mode different from the aforementioned second segmentation mode is selected, and the segmentation mode divides the block into sub-blocks, the sub-blocks being sub-blocks of different shapes or sizes compared with the sub-blocks generated by the aforementioned second segmentation mode; the aforementioned second block is divided into sub-blocks using the selected segmentation mode; and the previously segmented aforementioned block is encoded.
[0257] Therefore, if it has been determined that the second segmentation mode should not be selected as a candidate for segmenting the second block, a different segmentation mode can be chosen. This reduces the number of selectable segmentation modes, decreases the amount of code required for each segmentation mode, and improves compression efficiency.
[0258] Furthermore, in the image encoding apparatus 5000 of this embodiment, the selected segmentation mode, which is different from the aforementioned second segmentation mode, can also divide the block into more sub-blocks than the aforementioned second segmentation mode, or divide the block into the same number of sub-blocks but different sizes as the aforementioned second segmentation mode.
[0259] Therefore, when the second segmentation mode is not selected as a candidate, it can be segmented into sub-blocks that are different from the sub-blocks obtained by using the second segmentation mode.
[0260] Furthermore, in the image encoding apparatus 5000 of this embodiment, in determining whether to not select the aforementioned second segmentation mode, when the aforementioned first parameter has identified the first segmentation mode, it is determined whether to not select the aforementioned second segmentation mode as the aforementioned candidate based on the first encoding amount of the first segmentation mode set including the aforementioned first segmentation mode and the aforementioned second segmentation mode, and the second encoding amount of the second segmentation mode set different from the aforementioned first segmentation mode set. The aforementioned second segmentation mode set may also contain at least one segmentation mode, and the aforementioned at least one segmentation mode is used to divide the block into sub-blocks with the same shape and size as the sub-blocks obtained by dividing the aforementioned first block and the aforementioned second block using the aforementioned first segmentation mode and the aforementioned second segmentation mode.
[0261] Therefore, based on the coding amount of the segmentation pattern set, it can be determined whether to exclude the second segmentation pattern as a candidate, thus reducing the coding amount of the segmentation pattern and improving compression efficiency.
[0262] Furthermore, in the image encoding apparatus 5000 of this embodiment, when determining whether to not select the aforementioned second segmentation mode, it can be determined that the aforementioned second segmentation mode is not selected when the aforementioned first encoding amount is greater than the aforementioned second encoding amount.
[0263] This prevents the selection of segmentation pattern sets with excessive coding, reduces the coding amount of segmentation patterns, and improves compression efficiency.
[0264] Furthermore, in the image encoding apparatus 5000 of this embodiment, when determining whether to not select the aforementioned second segmentation mode, if the aforementioned first encoding amount is equal to the aforementioned second encoding amount, and if the aforementioned second segmentation mode set appears before the aforementioned first segmentation mode set in a predetermined order of a plurality of segmentation mode sets including the aforementioned first segmentation mode set and the aforementioned second segmentation mode set, it is determined that the aforementioned second segmentation mode is not selected.
[0265] Therefore, when the coding amounts of the two segmentation pattern sets are equal, the segmentation pattern set can be selected according to a predetermined order.
[0266] As described above, the image decoding apparatus 6000 of this embodiment is an image decoding apparatus that decodes image blocks, including a processor and a memory. The processor uses the memory to perform the following steps: interpreting a first parameter from a bitstream, wherein the first parameter is a segmentation pattern identified from a plurality of segmentation patterns for dividing a first block into sub-blocks; determining whether the first parameter has identified the first segmentation pattern; at least based on the determination of whether the first parameter has identified the first segmentation pattern, determining whether to not select a second segmentation pattern as a candidate for dividing a second block different from the first block; if it has been determined that the second segmentation pattern has been selected, then... When dividing the aforementioned second block into candidates, a second parameter is interpreted from the bit stream, wherein the second parameter is used to select a segmentation mode for dividing the aforementioned second block from a plurality of segmentation modes including the aforementioned second segmentation mode as candidates; when it has been determined that the aforementioned second segmentation mode is not selected as a candidate for dividing the aforementioned second block, a segmentation mode different from the aforementioned second segmentation mode is selected, and this segmentation mode divides the block into sub-blocks of different shapes or sizes compared with the sub-blocks generated by the aforementioned second segmentation mode; the aforementioned second block is divided into sub-blocks using the selected aforementioned segmentation mode; and the previously segmented aforementioned block is decoded.
[0267] Therefore, when it has been determined that the second segmentation mode is not a candidate for segmenting the second block, a different segmentation mode can be selected. This reduces the number of selectable segmentation modes, decreases the amount of encoding for each segmentation mode, and improves compression efficiency.
[0268] Furthermore, in the image decoding apparatus 6000 of this embodiment, the selected segmentation mode, which is different from the aforementioned second segmentation mode, can also divide the block into more sub-blocks than the aforementioned second segmentation mode, or divide the block into the same number of sub-blocks but different sizes as the aforementioned second segmentation mode.
[0269] Therefore, when the second partitioning pattern is not selected as a candidate, it can be partitioned into sub-blocks that are different from the sub-blocks obtained by using the second partitioning pattern.
[0270] Furthermore, in the image decoding apparatus 6000 of this embodiment, when determining whether to not select the aforementioned second segmentation mode, if the aforementioned first parameter has identified the first segmentation mode, it is determined whether to not select the aforementioned second segmentation mode as the aforementioned candidate based on the first encoding amount of the first segmentation mode set including the aforementioned first segmentation mode and the aforementioned second segmentation mode, and the second encoding amount of the second segmentation mode set different from the aforementioned first segmentation mode set. The aforementioned second segmentation mode set may also include at least one segmentation mode, which is used to divide the block into sub-blocks with the same shape and size as the sub-blocks obtained by dividing the aforementioned first block and the aforementioned second block using the aforementioned first segmentation mode and the aforementioned second segmentation mode.
[0271] Therefore, based on the amount of coding in the segmentation pattern set, it can be determined whether to exclude the second segmentation pattern as a candidate, which can reduce the amount of coding in the segmentation pattern and improve compression efficiency.
[0272] Furthermore, in the image decoding device 6000 of this embodiment, when determining whether to not select the aforementioned second segmentation mode, it can be determined that the aforementioned second segmentation mode is not selected when the aforementioned first encoding amount is greater than the aforementioned second encoding amount.
[0273] This prevents the selection of a large set of segmentation patterns, reduces the amount of code in each segmentation pattern, and improves compression efficiency.
[0274] Furthermore, in the image decoding apparatus 6000 of this embodiment, when determining whether to not select the aforementioned second segmentation mode, if the aforementioned first encoding amount is equal to the aforementioned second encoding amount, and if the aforementioned second segmentation mode set appears before the aforementioned first segmentation mode set in a predetermined order of a plurality of segmentation mode sets including the aforementioned first segmentation mode set and the aforementioned second segmentation mode set, it is determined that the aforementioned second segmentation mode is not selected.
[0275] Therefore, when the coding amounts of the two segmentation pattern sets are equal, the segmentation pattern set can be selected according to a predetermined order.
[0276] This sample may also be implemented in combination with at least a portion of other samples disclosed herein. Furthermore, a portion of the processing described in the flowchart of this sample, a portion of the apparatus configuration, a portion of the syntax, etc., may be implemented in combination with other samples. (Implementation Form 3)
[0277] The encoding and decoding processes of Embodiment 3 will be described in detail with reference to Figures 13 and 14. The encoding and decoding devices of Embodiment 3 will be described in detail with reference to Figures 15 and 16. [Encoding Processing]
[0278] Figure 13 shows the image encoding process of embodiment 3.
[0279] First, in step S3001, the first parameter is written into the bit stream, wherein the first parameter is the segmentation pattern used to divide the first block into sub-blocks from a plurality of segmentation patterns.
[0280] In the next step S3002, the second parameter of the display segmentation direction is written into the bit stream. The second parameter is configured within the bit stream after the first parameter. The segmentation type can also be combined with the segmentation direction to form a segmentation mode. The segmentation type displays the number of sub-blocks used to divide the block and the segmentation ratio.
[0281] Figure 29 is an example of the segmentation pattern and segmentation direction used to segment NxN pixel blocks in Embodiment 3. In Figure 29, (1), (2), (3), and (4) are different segmentation patterns; (1a), (2a), (3a), and (4a) are segmentation patterns with different segmentation directions in the vertical direction; and (1b), (2b), (3b), and (4b) are segmentation patterns with different segmentation directions in the horizontal direction. As shown in Figure 29, when the segmentation ratio is 1:1 and the block is cut in the vertical direction using a symmetrical binary tree (i.e., two sub-blocks), the NxN pixel block is segmented using segmentation pattern (1a). When the segmentation ratio is 1:1 and the block is cut in the horizontal direction using a symmetrical binary tree (i.e., two sub-blocks), the NxN pixel block is segmented using segmentation pattern (1b). When cutting with a 1:3 ratio and using an asymmetric binary tree (i.e., 2 sub-blocks) in the vertical direction, an NxN pixel block is segmented using segmentation mode (2a). When cutting with a 1:3 ratio and using an asymmetric binary tree (i.e., 2 sub-blocks) in the horizontal direction, an NxN pixel block is segmented using segmentation mode (2b). When cutting with a 3:1 ratio and using an asymmetric binary tree (i.e., 2 sub-blocks) in the vertical direction, an NxN pixel block is segmented using segmentation mode (3a). When cutting with a 3:1 ratio and using an asymmetric binary tree (i.e., 2 sub-blocks) in the horizontal direction, an NxN pixel block is segmented using segmentation mode (3b). When cutting with a 1:2:1 ratio and using a ternary tree (i.e., 3 sub-blocks) in the vertical direction, an NxN pixel block is segmented using segmentation mode (4a). When the segmentation ratio is 1:2:1 and the horizontal direction is cut using a ternary tree (i.e., 3 sub-blocks), the NxN pixel block is segmented using the segmentation mode (4b).
[0282] Figure 17 shows the conceivable locations of the first parameter within a compressed video stream. As shown in Figure 17, the first parameter can be configured within a video parameter set, sequence parameter set, picture parameter set, slice header, or coding tree unit. The first parameter can indicate the method of dividing a block into multiple sub-blocks. For example, the first parameter may contain a flag indicating whether the block is divided horizontally or vertically. The first parameter may also include parameters indicating whether to divide the block into two or more sub-blocks.
[0283] Figure 18 shows the conceivable locations of the second parameter within the compressed video stream. As shown in Figure 18, the second parameter can be configured within a video parameter set, sequence parameter set, picture parameter set, slice header, or coding tree unit. The second parameter can indicate the method of dividing the block into multiple sub-blocks. For example, the second parameter may contain a flag indicating whether the block is divided horizontally or vertically. The second parameter may also include parameters indicating whether the block is divided into two or more sub-blocks. As shown in Figure 19, the second parameter is configured within the bit stream following the first parameter.
[0284] Figure 30 shows the advantages of encoding the segmentation pattern before the segmentation pattern, compared to the case where the segmentation direction is encoded before the segmentation pattern. In this example, for an unsupported size (16x2 pixels), it is not necessary to encode the segmentation direction when the horizontal segmentation direction is invalid. In this example, the segmentation direction is determined to be the vertical segmentation direction, and the horizontal segmentation direction is invalid. Encoding the segmentation pattern before the segmentation direction, compared to encoding the segmentation direction before the segmentation pattern, is better at suppressing the encoded bits obtained by encoding the segmentation direction.
[0285] Thus, based on pre-defined conditions indicating whether block partitioning is feasible or not, it can be determined whether to partition the block in both the horizontal and vertical directions. Then, if it is determined that only one of the horizontal or vertical directions is feasible for partitioning, the writing of the bitstream for that partitioning direction can be skipped. Furthermore, if it is determined that neither the horizontal nor vertical directions are feasible for partitioning, the writing of the bitstream for the partitioning type can be skipped, except for the partitioning direction.
[0286] Pre-defined conditions for the feasibility or infeasibility of block segmentation can be established, such as by defining the size (number of pixels) or the number of segmentations. These conditions can also be defined in advance within a standard specification. Furthermore, the conditions for the feasibility or infeasibility of block segmentation can be included in video parameter sets, sequence parameter sets, image parameter sets, slice headers, or coding tree units. The conditions for the feasibility or infeasibility of block segmentation can be fixed for all blocks, or they can be dynamically switched based on the characteristics of the blocks (e.g., luminance and chrominance blocks) or the characteristics of the images (e.g., I, P, B images).
[0287] In step S3003, the block is divided into sub-blocks using the identified segmentation pattern and the displayed segmentation direction. In step S3004, the segmented blocks are encoded. [Encoding device]
[0288] Figure 15 is a block diagram showing the structure of the image / image encoding device in embodiment 2 or 3.
[0289] The image encoding apparatus 5000 is an apparatus used to encode an input image / picture by block and generate an encoded output bitstream. As shown in FIG15, the image encoding apparatus 5000 includes: a conversion unit 5001, a quantization unit 5002, an inverse quantization unit 5003, an inverse conversion unit 5004, a block memory 5005, a frame memory 5006, an intra-prediction unit 5007, an inter-prediction unit 5008, an entropy encoding unit 5009, and a block segmentation determination unit 5010.
[0290] The input image is input to the adder, and the added value is output to the conversion unit 5001. The conversion unit 5001 converts the added value into frequency coefficients based on the block segmentation type and direction derived by the block segmentation determination unit 5010, and outputs the frequency coefficients to the quantization unit 5002. The block segmentation type and direction can be associated with the block segmentation mode, block segmentation type, or block segmentation direction. The quantization unit 5002 quantizes the input quantization coefficients and outputs the quantized value to the inverse quantization unit 5003 and the entropy encoding unit 5009.
[0291] The dequantization unit 5003 dequantizes the quantized value output from the quantization unit 5002 and outputs the frequency coefficient to the inverse conversion unit 5004. The inverse conversion unit 5004 performs inverse frequency conversion on the frequency coefficient according to the block partitioning pattern derived by the block partitioning determination unit 5010, converts the frequency coefficient into sample values of the bit stream, and outputs the sample values to the adder.
[0292] The adder adds the sample values of the bitstream output from the inversion unit 5004 to the predicted image / image values output from the intermediate / intermediate prediction units 5007 and 5008, and outputs the added value to the block memory 5005 or the frame memory 5006 for further prediction. The block segmentation determination unit 5010 collects block information from the block memory 5005 or the frame memory 5006 and derives parameters regarding the block segmentation type and direction. If the derived block segmentation type and direction are used, the block will be divided into multiple sub-blocks. The internal / intermediate prediction units 5007 and 5008 explore the images / images stored in the block memory 5005, or the images / images in the frame memory 5006 reconstructed by the block segmentation pattern and direction derived by the block segmentation determination unit 5010, and estimate, for example, the image / image region most similar to the input image / image of the prediction target.
[0293] The entropy encoding unit 5009 encodes the quantized value output from the quantization unit 5002, encodes the parameters from the block segmentation determination unit 5010, and outputs a bit stream. [Decoding Processing]
[0294] Figure 14 shows the image decoding process of embodiment 3.
[0295] First, in step S4001, a first parameter is interpreted from the bit stream. This first parameter is the segmentation pattern used to divide the first block into sub-blocks from among a plurality of segmentation patterns.
[0296] In the next step S4002, the second parameter of the segmentation direction is interpreted and displayed from the bit stream. The second parameter is appended to the first parameter in the bit stream. The segmentation type can also be combined with the segmentation direction to form a segmentation pattern. The segmentation type displays the number of sub-blocks used to divide the block and the segmentation ratio.
[0297] Figure 29 shows an example of the segmentation pattern and direction used to divide an NxN pixel block in Embodiment 3. In Figure 29, (1), (2), (3), and (4) are different segmentation patterns; (1a), (2a), (3a), and (4a) are segmentation patterns with different vertical segmentation directions and different segmentation patterns; and (1b), (2b), (3b), and (4b) are segmentation patterns with different horizontal segmentation directions and different segmentation patterns. As shown in Figure 29, when the segmentation ratio is 1:1 and the block is cut vertically using a symmetrical binary tree (i.e., two sub-blocks), the NxN pixel block is divided using segmentation pattern (1a). When the segmentation ratio is 1:1 and the block is cut horizontally using a symmetrical binary tree (i.e., two sub-blocks), the NxN pixel block is divided using segmentation pattern (1b). When cutting with a 1:3 ratio and using an asymmetric binary tree (i.e., 2 sub-blocks) in the vertical direction, an NxN pixel block is segmented using segmentation mode (2a). When cutting with a 1:3 ratio and using an asymmetric binary tree (i.e., 2 sub-blocks) in the horizontal direction, an NxN pixel block is segmented using segmentation mode (2b). When cutting with a 3:1 ratio and using an asymmetric binary tree (i.e., 2 sub-blocks) in the vertical direction, an NxN pixel block is segmented using segmentation mode (3a). When cutting with a 3:1 ratio and using an asymmetric binary tree (i.e., 2 sub-blocks) in the horizontal direction, an NxN pixel block is segmented using segmentation mode (3b). When cutting with a 1:2:1 ratio and using a ternary tree (i.e., 3 sub-blocks) in the vertical direction, an NxN pixel block is segmented using segmentation mode (4a). When the segmentation ratio is 1:2:1 and the horizontal direction is cut using a ternary tree (i.e., 3 sub-blocks), the NxN pixel block is segmented using the segmentation mode (4b).
[0298] Figure 17 shows conceivable locations of the first parameter within a compressed video stream. As shown in Figure 17, the first parameter can be configured within a video parameter set, sequence parameter set, picture parameter set, slice header, or coding tree unit. The first parameter can indicate the method of dividing a block into multiple sub-blocks. For example, the first parameter may include an identifier of the segmentation type described above. For example, the first parameter may contain a flag indicating whether the block is divided horizontally or vertically. The first parameter may also include parameters indicating whether the block is divided into two or more sub-blocks.
[0299] Figure 18 shows the conceivable locations of the second parameter within the compressed video stream. As shown in Figure 18, the second parameter can be configured within a video parameter set, sequence parameter set, picture parameter set, slice header, or coding tree unit. The second parameter can indicate the method of dividing a block into multiple sub-blocks. For example, the second parameter can contain a flag indicating whether the block is divided horizontally or vertically. That is, the second parameter can include parameters indicating the division direction. The second parameter can also include parameters indicating whether the block is divided into two or more sub-blocks. As shown in Figure 19, the second parameter is configured within the bit stream following the first parameter.
[0300] Figure 30 shows the advantages of encoding the segmentation pattern before the segmentation pattern, compared to the case where the segmentation direction is encoded before the segmentation pattern. In this example, for an unsupported size (16x2 pixels), it is not necessary to encode the segmentation direction when the horizontal segmentation direction is invalid. In this example, the segmentation direction is determined to be the vertical segmentation direction, and the horizontal segmentation direction is invalid. Encoding the segmentation pattern before the segmentation direction, compared to encoding the segmentation direction before the segmentation pattern, is better at suppressing the encoded bits obtained by encoding the segmentation direction.
[0301] Thus, based on pre-defined conditions indicating whether block segmentation is feasible or not, it can be determined whether to segment the block in both the horizontal and vertical directions. Then, if it is determined that only one of the horizontal or vertical directions is feasible, the interpretation of the segmentation direction from the bitstream can be skipped. Furthermore, if it is determined that neither the horizontal nor vertical directions are feasible, the interpretation of the segmentation pattern from the bitstream can be skipped, in addition to the segmentation direction.
[0302] Pre-defined conditions for the feasibility or infeasibility of block segmentation can be established, such as by defining the size (number of pixels) or the number of segmentations. These conditions can also be defined in advance within a standard specification. Furthermore, the conditions for the feasibility or infeasibility of block segmentation can be included in video parameter sets, sequence parameter sets, image parameter sets, slice headers, or coding tree units. The conditions for the feasibility or infeasibility of block segmentation can be fixed for all blocks, or they can be dynamically switched based on the characteristics of the blocks (e.g., luminance and chrominance blocks) or the characteristics of the images (e.g., I, P, B images).
[0303] In step S4003, the block is divided into sub-blocks using the identified segmentation pattern and the displayed segmentation direction. In step S4004, the segmented blocks are decoded. [Decoding device]
[0304] Figure 16 is a block diagram showing the structure of the image / image decoding device in embodiment 2 or 3.
[0305] The image decoding apparatus 6000 is a device used to decode the input encoded bitstream on a block-by-block basis and output the image / picture. As shown in FIG16, the image decoding apparatus 6000 includes: an entropy decoding unit 6001, an inverse quantization unit 6002, an inverse conversion unit 6003, a block memory 6004, a frame memory 6005, an intra-prediction unit 6006, an inter-prediction unit 6007, and a block segmentation determination unit 6008.
[0306] The input encoded bitstream is input to the entropy decoding unit 6001. After the input encoded bitstream is input to the entropy decoding unit 6001, the entropy decoding unit 6001 decodes the input encoded bitstream, outputs the parameters to the block segmentation determination unit 6008, and outputs the decoded value to the dequantization unit 6002.
[0307] The dequantization unit 6002 dequantizes the decoded value and outputs the frequency coefficients to the inverse conversion unit 6003. The inverse conversion unit 6003 performs inverse frequency conversion on the frequency coefficients based on the block segmentation pattern and direction derived from the block segmentation determination unit 6008, converting the frequency coefficients into sample values and outputting the sample values to the adder. The block segmentation pattern and direction can be associated with the block segmentation mode, block segmentation pattern, or block segmentation direction. The adder adds the sample values to the predicted image / image values output from the intermediate / intermediate prediction units 6006 and 6007, outputs the added value to the display, and for further prediction, outputs the added value to the block memory 6004 or the frame memory 6005. The block segmentation determination unit 6008 collects block information from the block memory 6004 or the frame memory 6005 and derives the block segmentation pattern and direction using the parameters decoded by the entropy decoding unit 6001. If the derived block segmentation pattern and direction are used, the block will be divided into multiple sub-blocks. Then, the internal / intermediate prediction units 6006 and 6007 predict the image / image region of the target block from the image / image stored in the block memory 6004, or from the image / image in the frame memory 6005 reconstructed by the block segmentation pattern and direction derived by the block segmentation determination unit 6008. [Effects, etc.]
[0308] As described above, the image encoding apparatus 5000 of this embodiment is an image encoding apparatus that encodes blocks of an image. It includes a processor and a memory. The processor uses the memory to perform the following processing: writing a first parameter into a bit stream, wherein the first parameter is a segmentation pattern identified from a plurality of segmentation patterns, the segmentation pattern being used to divide a block into two or more sub-blocks; writing a second parameter into the bit stream, wherein the second parameter is a display segmentation direction and is appended to the first parameter in the bit stream; dividing the block into sub-blocks using the identified segmentation pattern and the displayed segmentation direction; and encoding the segmented blocks.
[0309] Therefore, a second parameter indicating the segmentation direction can be written after the first parameter that identifies the segmentation pattern. Thus, when it is not necessary to indicate the segmentation direction, the second parameter can be omitted, reducing the amount of encoding for the segmentation pattern and improving compression efficiency.
[0310] Furthermore, in the image encoding apparatus 5000 of this embodiment, if a segmentation type is used together with an identified segmentation direction, the block will be segmented into sub-blocks in the identified segmentation direction; if different segmentation types are used together with segmentation directions, the block will be segmented into sub-blocks of different shapes, different heights, or different widths.
[0311] Therefore, by using the segmentation type and direction, the shape, height, or width of the sub-blocks can be controlled.
[0312] Furthermore, in the image encoding apparatus 5000 of this embodiment, the aforementioned processor can further utilize the segmentation pattern identified by the aforementioned first parameter to determine whether the aforementioned block can be segmented in both the horizontal and vertical directions. When it has been determined that only one of the aforementioned horizontal and vertical directions can be segmented, the writing of the aforementioned second parameter to the bit stream is skipped, and the aforementioned segmentation pattern identified and the other of the aforementioned horizontal and vertical directions are used to segment the aforementioned block into sub-blocks.
[0313] Therefore, when block segmentation cannot be performed in either the horizontal or vertical direction, the writing of the second parameter indicating the segmentation direction can be omitted. That is, the second parameter can be omitted in the bitstream, which can reduce the amount of coding in the block segmentation method and improve compression efficiency.
[0314] As described above, the image decoding apparatus 6000 of this embodiment is an image decoding apparatus that decodes blocks of an image. It includes a processor and a memory. The processor uses the memory to perform the following processing: interpreting a first parameter from a bit stream, wherein the first parameter is a segmentation pattern identified from a plurality of segmentation patterns, the segmentation pattern being used to divide a block into two or more sub-blocks; interpreting a second parameter from a bit stream, wherein the second parameter is a display segmentation direction and is appended to the first parameter in the bit stream; dividing the block into sub-blocks using the identified segmentation pattern and the displayed segmentation direction; and decoding the segmented block.
[0315] Therefore, a second parameter indicating the segmentation direction can be written after the first parameter that identifies the segmentation pattern. Thus, when it is not necessary to indicate the segmentation direction, the second parameter can be omitted, reducing the amount of encoding for the segmentation pattern and improving compression efficiency.
[0316] Furthermore, in the image decoding device 6000 of this embodiment, if the segmentation type is used together with the identified segmentation direction, the block will be segmented into sub-blocks in the identified segmentation direction; if different segmentation types are used together with the segmentation direction, the block will be segmented into sub-blocks of different shapes, different heights, or different widths.
[0317] Therefore, by using the segmentation type and direction, the shape, height, or width of the sub-blocks can be controlled.
[0318] Furthermore, in the image decoding apparatus 6000 of this embodiment, the aforementioned processor can further utilize the segmentation pattern identified by the aforementioned first parameter to determine whether the aforementioned block can be segmented in both the horizontal and vertical directions. When it has been determined that segmentation can be performed in either the aforementioned horizontal or the aforementioned vertical direction, the interpretation of the aforementioned second parameter from the bit stream is skipped, and the aforementioned block is segmented into sub-blocks using the identified segmentation pattern and the other of the aforementioned horizontal and the aforementioned vertical directions as the aforementioned segmentation direction.
[0319] Therefore, when block segmentation cannot be performed in either the horizontal or vertical direction, the interpretation of the second parameter indicating the segmentation direction can be omitted. That is, the second parameter can be omitted within the bitstream, reducing the amount of coding in the block segmentation method and improving compression efficiency.
[0320] This sample may also be implemented in combination with at least a portion of other samples disclosed herein. Furthermore, a portion of the processing described in the flowchart of this sample, a portion of the apparatus configuration, a portion of the syntax, etc., may be implemented in combination with other samples. [Implementation Mode 4]
[0321] In the above embodiments, each functional block can typically be implemented using an MPU and memory. Furthermore, the processing performed by each functional block can typically be implemented by reading software (programs) recorded on a recording medium such as ROM from a program execution unit such as a processor. This software can also be distributed via download or by recording on a recording medium such as semiconductor memory. Alternatively, each functional block can, of course, be implemented using hardware (dedicated circuitry).
[0322] Furthermore, the processing described in each embodiment can be implemented using a single device (system) for centralized processing, or it can be implemented using multiple devices for distributed processing. Also, the number of processors executing the above program can be singular or multiple. That is, centralized processing or distributed processing can be performed.
[0323] The embodiments disclosed herein are not limited to the above-described embodiments, and various modifications can be made, which are also included within the scope of the embodiments disclosed herein.
[0324] Furthermore, examples of applications of the motion image encoding method (image encoding method) or motion image decoding method (image decoding method) shown in the above embodiments and systems using such methods will be described. This system is characterized by having an image encoding device using the image encoding method, an image decoding device using the image decoding method, and an image encoding / decoding device having both. Other components of the system can be appropriately modified according to the needs of the situation. [Usage Example]
[0325] Figure 33 is an overall structural diagram of the content delivery system ex100 that implements content distribution services. The communication service provision area is divided into desired sizes, and each cell is equipped with a base station ex106, ex107, ex108, ex109, and ex110, which are fixed wireless stations.
[0326] In this content delivery system ex100, various devices such as computers ex111, game consoles ex112, cameras ex113, home appliances ex114, and smartphones ex115 are connected to the Internet ex101 via Internet service provider ex102 or communication network ex104, and base stations ex106 to ex110. This content delivery system ex100 can be configured to combine any of the above elements for connection. Alternatively, it can connect the devices directly or indirectly via telephone networks or short-range wireless connections without using base stations ex106 to ex110, which are fixed wireless stations. Furthermore, the streaming server ex103 is connected to the computers ex111, game consoles ex112, cameras ex113, home appliances ex114, and smartphones ex115 via the Internet ex101, etc. Furthermore, the streaming server ex103 is connected to terminals and other devices in the hotspot within the aircraft ex117 via satellite ex116.
[0327] Alternatively, wireless access points or hotspots can be used to replace base stations ex106 to ex110. Furthermore, the streaming server ex103 can connect directly to the communication network ex104 without going through the Internet ex101 or the Internet service provider ex102, or it can connect directly to the aircraft ex117 without going through the satellite ex116.
[0328] The EX113 camera is a digital camera or similar device capable of capturing both still and moving images. The EX115 smartphone generally refers to a smartphone, mobile phone, or PHS (Personal Handyphone System) compatible with 2G, 3G, 3.9G, 4G, and future 5G mobile communication systems.
[0329] Home appliances (ex118) include refrigerators, or machines in home fuel cell cogeneration systems, etc.
[0330] In the content supply system ex100, a terminal with a camera function is connected to the streaming server ex103 via a base station ex106, etc., to perform live streaming. During live streaming, the terminal (computer ex111, game console ex112, camera ex113, home appliance ex114, smartphone ex115, and terminal inside an airplane ex117, etc.) sends data obtained as described above to the streaming server ex103. This data is obtained by encoding still or moving images captured by the user using the terminal as described in the above embodiments, and by multiplexing the encoded image data and the corresponding audio data after audio encoding. In other words, each terminal functions as an image encoding device as disclosed in this disclosure.
[0331] On the other hand, the streaming server ex103 distributes the content data to be sent to the requesting client. The client refers to a terminal such as a computer ex111, game console ex112, camera ex113, home appliance ex114, smartphone ex115, or airplane ex117 that can decode the encoded data. Each machine that has received the distributed data decodes it and then plays it. That is, each machine functions as an image decoding device as described in this disclosure. [Distributed processing]
[0332] Furthermore, the streaming server ex103 can also be multiple servers or multiple computers, distributing data processing or recording across distributors. For example, the streaming server ex103 can also be implemented using a CDN (Contents Delivery Network), which distributes content by connecting numerous edge servers scattered around the world. In a CDN, physically close edge servers are dynamically allocated based on the client. The content is then cached and distributed by that edge server, thereby reducing latency. Moreover, in the event of any errors or changes in communication status due to increased traffic, multiple edge servers can distribute processing, or the distribution can be switched to other edge servers, thus bypassing the affected network portion and continuing distribution, thereby achieving high-speed and stable distribution.
[0333] Furthermore, it's not just about distributing the data in a decentralized manner; the encoding of the captured data can also be performed on each terminal, on the server side, or distributed among them. For example, the encoding process typically involves two processing loops. The first loop checks the complexity of the image, or the amount of encoding, in frames or scenes. The second loop performs processing to maintain image quality and improve encoding efficiency. For instance, the terminal performs the first encoding, while the server that has received the content performs the second encoding. This reduces the processing burden on each terminal while improving content quality and efficiency. If there's a request for near-real-time decoding, the data already encoded on one terminal can be received and played on another terminal, thus achieving more flexible real-time distribution.
[0334] For another example, cameras like the ex113 extract features from images, compress the data related to these features as meta-data, and send it to a server. The server uses these features to determine the importance of the target and adjusts the quantization precision accordingly to compress the image. The feature data is particularly effective in improving the accuracy and efficiency of motion vector prediction during subsequent compression on the server. Alternatively, simple encoding methods such as VLC (Variable Length Coding) can be performed on the terminal, while more demanding encoding methods such as CABAC (Context-Adaptive Binary Arithmetic Coding) can be performed on the server.
[0335] Furthermore, in other examples, such as stadiums, shopping malls, or factories, there may be multiple images of almost identical scenes captured by multiple terminals. In such cases, the data from the multiple terminals that have taken the images, as well as other terminals and servers that have not taken images as needed, are distributed and encoded separately using units such as GOP (Group of Pictures), image units, or blocks of images. This reduces latency and achieves better real-time performance.
[0336] Furthermore, since multiple video data sets depict nearly identical scenes, management and / or instructions can be implemented on the server to cross-reference video data captured by each terminal. Alternatively, the server can receive encoded data from each terminal, change the reference relationships between the multiple data sets, or correct or replace the images themselves for re-encoding. This allows for streaming with improved quality and efficiency for each individual data set.
[0337] Furthermore, the server can also transcode the video data by changing its encoding method before distributing the video data. For example, the server can convert MPEG encoding to VP encoding, or H.264 to H.265.
[0338] Thus, encoding processing can be performed through a terminal or one or more servers. Therefore, in the following text, the entity performing the processing is referred to as "server" or "terminal," but some or all of the processing performed on the server can also be performed on the terminal, or some or all of the processing performed on the terminal can be performed on the server. Furthermore, the same applies to decoding processing. [3D, Multiple Perspectives]
[0339] In recent years, there has been an increasing trend of integrating and utilizing images or videos captured by multiple cameras such as the EX113 and / or smartphones such as the EX115, which are almost synchronized with each other, of different scenes or of the same scene captured from different perspectives. The images captured by each terminal are integrated based on the relative positional relationship between the terminals or the regions of consistent feature points contained in the images.
[0340] The server not only encodes 2D moving images, but also automatically or at user-specified times encodes still images based on scene analysis of the moving images, and then sends them to the receiving terminal. Furthermore, when the server can obtain the relative positional relationships between the camera terminals, it can generate 3D shapes of the scene, not only from 2D moving images, but also from images of the same scene captured from different perspectives. Additionally, the server can encode 3D data generated through point clouds, and can select from or reconstruct images captured by multiple terminals based on the results of using 3D data to identify or track people or targets, in order to generate the image to be sent to the receiving terminal.
[0341] After this process, users can choose any image corresponding to each camera terminal to view the scene, or they can view content from any viewpoint cut out from 3D data reconstructed using multiple images or videos. Furthermore, similar to the images, sound can be collected from multiple different perspectives, allowing the server to work with the images, multiplexing sound and images from specific perspectives or spaces before sending them.
[0342] Furthermore, in recent years, content that establishes a correspondence between the real world and the virtual world, such as Virtual Reality (VR) and Augmented Reality (AR), has gradually become more widespread. In the case of VR images, the server can create separate viewpoint images for the right and left eyes. Through Multi-View Coding (MVC), encoding can be performed that allows reference between the images from different viewpoints, or they can be encoded as different streams without mutual reference. When decoding different streams, it is advisable to reproduce the virtual 3D space according to the user's viewpoint, ensuring synchronization and playback.
[0343] In AR image processing, the server overlays virtual object information in virtual space onto camera information in real space based on the 3D position or the user's viewpoint movement. The decoding device can also acquire or retain virtual object information and 3D data, and generate 2D images in response to the user's viewpoint movement, thus creating overlay data. Alternatively, the decoding device can send the user's viewpoint movement to the server in addition to virtual object information requests. The server then uses the received viewpoint movement to create overlay data from the 3D data stored on the server, encodes the overlay data, and distributes it to the decoding device. Another option is that the overlay data, in addition to RGB, also has an α value indicating transparency. The server sets the α value of the portion outside the target area created from the 3D data to 0, and encodes this portion as transparent. Alternatively, the server can generate data using chroma keying, where predetermined RGB values are set as the background, and the portion outside the target area is set as the background color.
[0344] Similarly, the decoding of distributed data can be performed on each client terminal, on the server side, or distributed among them. For example, a terminal can first send a request to the server, allowing other terminals to receive the requested content, decode it, and then send the decoded signal to a device with a display. This allows for distributed processing and selection of suitable content without relying on the performance of the communicating terminals themselves, thus enabling the playback of high-quality data. As another example, while receiving large image data on a TV, a portion of the image, such as squares, can be decoded and displayed on the viewer's personal terminal. This allows for sharing the overall image and enabling users to identify their assigned area or areas requiring more detailed examination.
[0345] Furthermore, in the future, whether indoors or outdoors, under conditions where short-range, medium-range, or long-range wireless communication is available, it is foreseeable that distribution system specifications such as MPEG-DASH will be used to seamlessly receive content while switching appropriate data for the communication during the connection, regardless of whether it is indoors or outdoors. This allows users to freely choose decoding or display devices, such as indoor or outdoor displays, and switch between them in real time, not only using their own terminals. Also, based on their location information, they can switch between decoding and display terminals while decoding is in progress. This makes it feasible to display map information on the walls or ground of buildings near display devices while moving towards a destination. Furthermore, the bit rate of the received data can be switched based on the ease of access to encoded data on the network, such as whether the encoded data can be cached from the receiving terminal's server in a short time or copied to an edge server in a Content Delivery Service. [Adjustable Encoding]
[0346] The switching of content is illustrated using the adjustable stream shown in Figure 34. This stream is a compressed and encoded stream using the motion graphics encoding method described in the above embodiments. While the server can have multiple streams with the same content but different qualities, it can also be configured to flexibly utilize the characteristics of time-based / spatial adjustable streams to switch content. This time-based / spatial adjustable stream is implemented through layered encoding as shown in the figure. That is, the decoding side determines which layer to decode based on internal factors such as performance and the status of the communication band. This allows the decoding side to freely switch between low-resolution and high-resolution content during decoding. For example, if you want to watch the remainder of a video viewed on a smartphone (ex115) while on the go at home using a device like a web TV, the device only needs to decode the same stream to a different layer, thus reducing the burden on the server side.
[0347] Furthermore, as mentioned above, in addition to encoding the image at each layer and establishing scalability through an enhancement layer above the base layer, the enhancement layer can also contain interpretive information such as image-based statistical information. The decoder then uses this interpretive information to perform hyper-image analysis on the base layer image, thereby producing high-quality content. Hyper-image analysis can refer to either improving the signal-to-noise ratio (SN ratio) at the same resolution or increasing the resolution. Interpretive information includes information about the linear or nonlinear filter coefficients used in a specific hyper-image analysis process, or information about the parameter values in filtering, machine learning, or least squares operations used in a specific hyper-image analysis process.
[0348] Alternatively, the image can be structured as follows: Based on the meaning of the object within the image, the image is divided into blocks, and the decoding side selects the blocks to be decoded, thus decoding only a portion of the area. Furthermore, the attributes of the object (people, vehicles, balls, etc.) and its position within the image (coordinates within the same image, etc.) are stored as interpretation information. This allows the decoding side to specify the desired object's position based on the interpretation information and determine the block containing that object. For example, as shown in Figure 35, the interpretation information is stored using a data storage structure different from pixel data, such as SEI information in HEVC. This interpretation information may include, for example, the position, size, or color of the main object.
[0349] Alternatively, interpretive information can be stored in units consisting of multiple images, such as streams, sequences, or random access units. This allows the decoder to obtain information such as the time a specific person appears in the image, and by combining this with the image unit information, it can identify the image containing a specific target and the target's location within that image. [Webpage optimization]
[0350] Figure 36 shows an example of a webpage display on a computer such as ex111. Figure 37 shows an example of a webpage display on a smartphone such as ex115. As shown in Figures 36 and 37, a webpage may include multiple linked images, where these linked images are links to the image content. The way these linked images are viewed varies depending on the viewing device. When multiple linked images are visible on the screen, until the user explicitly selects a linked image, or until the linked image is near the center of the screen or the entire linked image enters the screen, the display device (decoding device) displays static images or I-images of each content as linked images, or displays images like GIF animations using multiple static images or I-images, or only receives the basic layer and decodes and displays the images.
[0351] When a user selects a linked image, the display device prioritizes the base layer for decoding. Additionally, if the HTML of the webpage contains information about adjustable content, the display device can decode up to the enhancement layer. Furthermore, to ensure real-time performance, before selection or when the communication bandwidth is extremely narrow, the display device only decodes and displays images referenced in front (I-images, P-images, and only B-images referenced in front), thereby reducing the delay between decoding and displaying the preceding images (the delay from the start of content decoding to the start of display). Alternatively, the display device can disregard image references and use all B-images and P-images as references, performing a rough initial decoding, and then, after a period of time, performing normal decoding as more images are received. [Autonomous Driving]
[0352] Furthermore, when sending and receiving static images or video data such as 2D or 3D map information for the purpose of autonomous driving or supporting driving, the receiving terminal can receive not only image data belonging to a layer or higher, but also weather or construction information as interpretive information, and establish corresponding relationships for decoding. Additionally, interpretive information can also belong to a layer, or it can simply perform multiplexing with image data.
[0353] At this time, since the vehicle, drone, or airplane containing the receiving terminal is moving, the receiving terminal will send its location information when requesting reception. This allows for seamless reception and decoding while switching base stations from EX106 to EX110. Furthermore, the receiving terminal can dynamically switch the interpretation of the received information level or the update level of map information based on the user's selection, user status, or communication frequency band status.
[0354] As described above, in the content supply system ex100, the client can receive, decode, and play the encoded information sent by the user in real time. Distribution of personal content
[0355] Furthermore, the content delivery system ex100 not only distributes long-duration, high-quality content from video distribution providers, but also short-duration, low-quality content from individuals through unicast or multicast. It is also anticipated that such personal content will increase in the future. To further enhance personal content, the server can perform editing and encoding processes. This can be achieved, for example, through the following configuration.
[0356] During or after shooting, the server performs image processing, including real-time error detection, scene searching, meaning analysis, and target detection, based on the original image or encoded data. Then, based on the identification results, the server performs manual or automatic editing, such as correcting out-of-focus or camera shake, deleting less important scenes (e.g., scenes with lower brightness or out-of-focus images), emphasizing target edges, or adjusting color tones. The server then encodes the edited data. Furthermore, knowing that longer shooting times lead to lower viewership, the server can automatically edit not only less important scenes as mentioned above but also scenes with less movement, based on image processing results, to create content within a specific time frame to fit the shooting time. Alternatively, the server can generate a digest based on the meaning analysis of the scene and encode it.
[0357] Furthermore, in personal content, leaving it as is can lead to instances where copyright, moral rights, or portrait rights infringements are included, or the scope of sharing exceeds the intended scope, which is undesirable for the individual. Therefore, for example, servers can intentionally defocus images of faces or homes at the periphery of the image for encoding. Additionally, servers can identify whether the image contains a face different from a pre-registered person, and if so, blur or pixelate the face. Alternatively, in pre- or post-processing of encoding, from a copyright perspective, users can specify the people or background areas they want to process, and the server can replace the specified areas with other images or apply blurring or other effects. In the case of people, in moving images, the face can be replaced while tracking the person.
[0358] Furthermore, due to the high real-time requirements for viewing personal content with small data volumes, although bandwidth varies, the decoding device prioritizes receiving the basic layer for decoding and playback. During this time, the decoding device can also receive the enhancement layer, and in cases of looping or playback more than twice, it will play high-definition video along with the enhancement layer. If adjustable encoding is used for streaming, the following experience can be provided: although the animation is initially rough or coarse, it gradually becomes more refined and the image quality improves. In addition to adjustable encoding, the same experience can be provided by constructing a single stream using the rough stream from the first playback and the second stream encoded based on the first animation. [Other Use Cases]
[0359] Furthermore, such encoding or decoding processing is generally performed within the LSIex500 chip found in each terminal device. The LSIex500 can be a single chip or composed of multiple chips. Alternatively, software for encoding or decoding motion images can be installed on certain recording media (CD-ROM, floppy disk, or hard disk, etc.) that can be read by a computer such as the ex111, and the software can be used for encoding or decoding. Furthermore, when the smartphone ex115 is equipped with a camera, it can also send video data captured by that camera. This video data has already been encoded using the LSIex500 chip found in the smartphone ex115.
[0360] Alternatively, the LSIex500 can also be configured to launch by downloading application software. In this case, the terminal first needs to determine whether it supports the content encoding method or whether it has the capability to execute specific services. If the terminal does not support the content encoding method or does not have the capability to execute specific services, it needs to download the codec or application software before acquiring and playing the content.
[0361] Furthermore, not limited to the content delivery system ex100 via the Internet ex101, at least one of the aforementioned embodiments of a motion graphics encoding device (image encoding device) or a motion graphics decoding device (image decoding device) can also be installed in a digital broadcasting system. Since it uses multiplexed data that has undergone multiplexing processing of images and sound to carry on radio waves used for broadcasting via satellites, etc., for transmission and reception, although the digital broadcasting system is advantageous for multi-point broadcasting compared to the content delivery system ex100 which is easy to perform single-point transmission, the same application of encoding and decoding processing can still be performed. [Hardware Configuration]
[0362] Figure 38 shows a diagram of the smartphone ex115. Figure 39 shows a configuration example of the smartphone ex115. The smartphone ex115 includes: an antenna ex450 for transmitting and receiving radio waves with the base station ex110; a camera unit ex465 for capturing images and still images; and a display unit ex458 for displaying decoded data, including images captured by the camera unit ex465 and images received by the antenna ex450. The smartphone ex115 further includes: an operation unit ex466, which is a touch panel; a sound output unit ex457, which is a speaker for outputting sound or audio; a sound input unit ex456, which is a microphone for inputting sound; a memory unit ex467, which can store encoded or decoded data such as captured images or still images, recorded audio, received images or still images, and emails; and a slot unit ex464, which is the interface between the SIMex468 and the user-specific authentication system, primarily for accessing various data via the network. Alternatively, an external memory module can be used instead of the memory unit ex467.
[0363] Furthermore, the main control unit ex460, which integrates the control of the display unit ex458 and the operation unit ex466, is connected to the power circuit unit ex461, the operation input control unit ex462, the image signal processing unit ex455, the camera interface unit ex463, the display control unit ex459, the modulation / demodulation unit ex452, the multiplexing / splitting unit ex453, the audio signal processing unit ex454, the slot unit ex464, and the memory unit ex467 via a bus ex470.
[0364] The power circuit section ex461 supplies power from the battery pack to various parts when the power switch is turned on by the user, thereby enabling the smartphone ex115 to start and become operational.
[0365] The smartphone ex115 is controlled by a main control unit ex460 containing a CPU, ROM, and RAM, for call and data communication processing. During a call, the audio signal received by the audio input unit ex456 is converted into a digital audio signal by the audio signal processing unit ex454. This signal undergoes spectrum spreading processing in the modulation / demodulation unit ex452, digital-to-analog conversion and frequency conversion processing in the transmit / receive unit ex451, and is then transmitted via the antenna ex450. Conversely, the received data is amplified, undergoes frequency conversion and analog-to-digital conversion processing, undergoes spectrum despreading processing in the modulation / demodulation unit ex452, is converted into an analog audio signal in the audio signal processing unit ex454, and is then output from the audio output unit ex457. In data communication mode, text, still images, or video data are sent to the main control unit ex460 via the operation unit ex466 of the main unit and are similarly processed for transmission and reception. In data communication mode, when sending images, still images, or images and sound, the image signal processing unit ex455 compresses and encodes the image signal stored in the memory unit ex467 or the image signal input from the camera unit ex465 using the motion picture encoding method shown in the above embodiments, and sends the encoded image data to the multiplexing / splitting unit ex453. Furthermore, the sound signal processing unit ex454 encodes the sound signal received by the sound input unit ex456 during the capture of images or still images by the camera unit ex465, and sends the encoded sound data to the multiplexing / splitting unit ex453. The multiplexing / demultiplexing unit ex453 multiplexes the encoded video data and the encoded audio data in a predetermined manner, and performs modulation and conversion processing in the modulation / demodulation unit (modulation / demodulation circuit unit) ex452 and the transmitting / receiving unit ex451, and transmits them via the antenna ex450.
[0366] When receiving images attached to emails or chats, or images linked to web pages, the multiplexing / demultiplexing unit ex453 separates the multiplexing data received via antenna ex450 into bitstreams of image data and bitstreams of audio data in order to decode the multiplexing data. The encoded image data is then supplied to the image signal processing unit ex455 via synchronization bus ex470, and the encoded audio data is supplied to the audio signal processing unit ex454. The image signal processing unit ex455 decodes the image signal using a motion picture decoding method corresponding to the motion picture encoding method described in the above embodiments, and displays the image or still image contained in the linked motion picture file from the display unit ex458 via the display control unit ex459. The audio signal processing unit ex454 decodes the audio signal and outputs sound from the audio output unit ex457. Furthermore, given the widespread adoption of real-time streaming, depending on the user's situation, audio playback might lead to situations that are inappropriate for society. Therefore, as an initial consideration, it is preferable to play only the video data and not the audio signal. Alternatively, audio could only be played synchronously when the user performs an action, such as selecting video data.
[0367] Furthermore, this explanation uses the smartphone EX115 as an example. Regarding the terminal, three installation methods can be considered: a signal transceiver terminal with both an encoder and a decoder, a transceiver terminal with only an encoder, and a transceiver terminal with only a decoder. Moreover, in a digital broadcasting system, the explanation focuses on multiplexed data that includes audio data on top of video data. However, in addition to audio data, the multiplexed data can also include text data related to the video data, and can also receive or transmit the video data itself, rather than multiplexed data.
[0368] Furthermore, this discussion focuses on the case where the main control unit (ex460) with a CPU controls encoding or decoding processing. However, most terminals also have GPUs. Therefore, the configuration described later is also possible. This involves flexibly utilizing the GPU's performance by using shared memory between the CPU and GPU, or memory with managed addresses that allows for shared access, and processing large areas together. This shortens encoding time, ensures real-time performance, and achieves low latency. In particular, it is efficient to perform motion estimation, deblocking filtering, SAO (Sample Adaptive Offset), conversion, and quantization processing on a per-image basis, rather than using the CPU, via the GPU. Industrial utilization
[0369] It can be used for encoding / decoding of multi-media data, especially image and video encoding / decoding using block encoding / decoding.
[0370] 100: Encoding device 102: Segmentation 104: Subtraction Section 106,5001: Conversion Section 108,5002: Quantitative Department 110,5009: Entropy Coding Department 112,5003,6002: Anti-quantitative department 114,5004,6003: Reverse conversion section 116: Addition Department 118,5005,6004: Block memory 120: Loop Filtering Section 122,5006,6005: Frame memory 124,5007,6006: Internal Forecasting Department 126,5008,6007: Interim Prediction Department 128: Predictive Control Department 200: Decoding device 202,6001: Entropy Decoding Department 204: Anti-Quantitative Department 206: Reverse Conversion Section 208: Addition Department 210: Block Memory 212: Loop Filtering Section 214: Frame Memory 216: Internal Forecasting Department 218: Interim Prediction Department 220: Predictive Control Department 5000: Image Encoding Device 5010, 6008: Block partitioning decision-making department 6000: Image Decoding Device ex100: Content Supply System ex101: Internet ex102: Internet Service Provider ex103: Streaming Server ex104: Communication Network EX106 to EX110: Base Station ex111: Computer ex112: Game console ex113: Camera ex114: Home Appliances ex115: Smartphone ex116: Satellite ex117: Airplane ex450: Antenna ex451: Transmitter / Receiver Unit ex452: Modulation / Demodulation Unit ex453: Multiplexing / Separation Section ex454: Audio Signal Processing Unit EX455: Image Signal Processing Unit ex456: Audio Input Section ex457: Audio Output Section ex458: Display Section ex459: Display Control Unit EX460: Main Control Unit ex461: Power Supply Circuit Section ex462: Operation Input Control Unit ex463: Camera Interface ex464: Slot section ex465: Camera Department ex466: Operations Department ex467: Memory Department ex468:SIM ex470: Bus
Claims
1. An image encoding apparatus for encoding blocks of an image, comprising: a circuit; and a memory associated with the circuit, wherein, during operation, the circuit obtains a block from a coding tree unit (CTU), divides the block into a plurality of sub-blocks in a first direction using a first segmentation mode, and encodes the plurality of sub-blocks, wherein, when the size of the block is Nx2N pixels, and the first direction is along 2N pixels, and N is an integer, the first segmentation mode includes dividing the block into a plurality of sub-blocks comprising at least one sub-block of size N / 4x2N pixels, and excludes dividing the block into sub-blocks of size 2N / 2x2N pixels, wherein the first direction is a vertical direction or a horizontal direction.
2. An image decoding apparatus for decoding blocks of an image, comprising: a circuit; and a memory associated with the circuit, wherein, during operation, the circuit obtains a block from a coding tree unit (CTU), divides the block into a plurality of sub-blocks in a first direction using a first segmentation mode, and decodes the plurality of sub-blocks, wherein, when the size of the block is Nx2N pixels, the first direction is along 2N pixels, and N is an integer, the first segmentation mode includes dividing the block into a plurality of sub-blocks comprising at least one sub-block of size N / 4x2N pixels, and excludes dividing the block into sub-blocks of size 2N / 2x2N pixels, wherein the first direction is a vertical direction or a horizontal direction.
3. A non-transitory memory medium storing a bitstream, the bitstream containing information for causing a computer receiving the bitstream to perform decoding processing, the information containing at least information regarding a first segmentation mode for causing the computer to perform the decoding processing, the decoding processing being the computer performing the following processing: obtaining a block from a coding tree unit (CTU); segmenting the block into a plurality of sub-blocks in a first direction using the first segmentation mode; decoding the plurality of sub-blocks; and, in the case where the size of the block is Nx2N pixels, the first direction is along 2N pixels, and N is an integer, the first segmentation mode includes segmenting the block into a plurality of sub-blocks containing at least one sub-block of size N / 4x2N pixels, and excluding segmenting the block into sub-blocks of size 2N / 2x2N pixels, the first direction being a vertical or horizontal direction.
Citation Information
Patent Citations
Picture coding supporting block merging and skip mode
TWI559740B
Intra prediction mode coding with directional partitions
US9532058B2
