Encoding devices, decoding devices, and computer-readable non-transitory media

Abstract

The encoding device is an encoding device that performs quantization to encode a moving image, and has circuitry and memory. The circuitry uses the memory to transform a first quantization matrix for a plurality of transformation coefficients of a square block, thereby generating a second quantization matrix for a plurality of transformation coefficients of a rectangular block from the first quantization matrix, and quantizing the plurality of transformation coefficients of the rectangular block using the second quantization matrix.

Description

Technical Field

[0001] Invention Field This invention relates to an encoding device for encoding dynamic images, etc. Prior Technology

[0002] Background of the Invention Previously, H.265 was used as a specification for encoding moving images, and the aforementioned H.265 is also known as HEVC (High Efficiency Video Coding) (Non-Patent Document 1). Prior technology documents

[0003] Non-patent literature Non-patent literature 1: H.265 (ISO / IEC 23008-2 HEVC) / HEVC (High Efficiency Video Coding) Summary of the Invention

[0004] Invention Summary The problem the invention aims to solve However, coding efficiency will decrease if the various shapes of rectangular blocks contained in a dynamic image are not quantized efficiently.

[0005] Therefore, this disclosure provides an encoding device or the like that can efficiently quantize rectangular blocks of various shapes contained in a moving image.

[0006] The means to solve the problem The disclosed single-state encoding device is an encoding device for encoding dynamic images by quantization, and includes circuitry and memory. The circuitry uses the memory to transform a first quantization matrix for a plurality of transformation coefficients of a square block, thereby generating a second quantization matrix for a plurality of transformation coefficients of a rectangular block from the first quantization matrix, and quantizing the plurality of transformation coefficients of the rectangular block using the second quantization matrix.

[0007] Furthermore, such general or specific forms can be implemented by systems, devices, methods, integrated circuits, computer programs, or non-transitory recording media such as computer-readable CD-ROMs, or by any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0008] Invention Effects The single-state encoding device disclosed herein can efficiently quantize rectangular blocks of various shapes contained in a moving image. Simple Explanation of the Diagram

[0009] Figure 1 is a block diagram showing the functional configuration of the encoding device in Embodiment 1.

[0010] Figure 2 is a diagram illustrating an example of block partitioning in implementation mode 1.

[0011] Figure 3 is a table showing the transformation basis functions corresponding to each transformation type.

[0012] Figure 4A is a diagram showing an example of the shape of the filter used in ALF.

[0013] Figure 4B is another example of the shape of the filter used in ALF.

[0014] Figure 4C is another example of the shape of the filter used in ALF.

[0015] Figure 5A is a diagram showing the 67 intra-prediction modes of intra-frame prediction.

[0016] Figure 5B is a flowchart illustrating the outline of the predictive image correction process using OBMC.

[0017] Figure 5C is a conceptual diagram illustrating the outline of the predictive image correction process using OBMC.

[0018] Figure 5D is a diagram illustrating an example of FRUC.

[0019] Figure 6 is a diagram used to illustrate pattern matching (bidirectional matching) between two blocks along the moving track.

[0020] Figure 7 is a diagram used to illustrate the style matching (template matching) between the template in the current image and the block in the reference image.

[0021] Figure 8 is a diagram used to illustrate the model that assumes uniform linear motion.

[0022] Figure 9A is a diagram illustrating how to derive the movement vector of a sub-block unit from the movement vectors of a complex number of adjacent blocks.

[0023] Figure 9B is a diagram illustrating the outline of the process derived using the move vector in the merge mode.

[0024] Figure 9C is a conceptual diagram illustrating the outline of DMVR processing.

[0025] Figure 9D is a diagram illustrating the general outline of a predictive image generation method that uses brightness correction processing with LIC processing.

[0026] Figure 10 is a block diagram showing the functional configuration of the decoding device in Embodiment 1.

[0027] Figure 11 is a diagram illustrating the first example of the encoding process using a quantization matrix in an encoding device.

[0028] Figure 12 is a diagram showing an example of the decoding process using a quantization matrix in the decoding device corresponding to the encoding device described in Figure 11.

[0029] Figure 13 is a diagram illustrating the first example of generating a quantization matrix for a rectangular block from a quantization matrix used for a square block in steps S102 of Figure 11 and S202 of Figure 12.

[0030] Figure 14 illustrates the method of generating the quantization matrix for the rectangular block illustrated in Figure 13 by down-transformation from the quantization matrix used for the corresponding square block.

[0031] Figure 15 is a diagram illustrating the second example of generating a quantization matrix for a rectangular block from a quantization matrix used for a square block in steps S102 of Figure 11 and S202 of Figure 12.

[0032] Figure 16 illustrates a method for generating the quantization matrix for the rectangular block illustrated in Figure 15 by upconversion from the quantization matrix used for the corresponding square block.

[0033] Figure 17 is a diagram illustrating the second example of the encoding process using a quantization matrix in an encoding device.

[0034] Figure 18 is a diagram illustrating an example of the decoding process using a quantization matrix in a decoding device corresponding to the encoding device described in Figure 17.

[0035] Figure 19 is a diagram illustrating the quantization matrix example of the size of the effective conversion coefficient region corresponding to each block size, which is used to explain steps S701 in Figure 17 and S801 in Figure 18.

[0036] Figure 20 is a modified example of the second example of the encoding process using a quantization matrix in an encoding device.

[0037] Figure 21 is a diagram illustrating an example of the decoding process using a quantization matrix in a decoding device corresponding to the encoding device described in Figure 20.

[0038] Figure 22 is a diagram illustrating the first example of generating a quantization matrix for a rectangular block from a quantization matrix used for a square block, as shown in steps S1002 of Figure 20 and S1102 of Figure 21.

[0039] Figure 23 illustrates the method of generating the quantization matrix for the rectangular block illustrated in Figure 22 from the quantization matrix used for the corresponding square block by down-transformation.

[0040] Figure 24 is a diagram illustrating the second example of generating a quantization matrix for a rectangular block from a quantization matrix used for a square block in steps S1002 of Figure 20 and S1102 of Figure 21.

[0041] Figure 25 illustrates a method for generating the quantization matrix for the rectangular block illustrated in Figure 24 by upconversion from the quantization matrix used for the corresponding square block.

[0042] Figure 26 is a diagram illustrating the third example of the encoding process using a quantization matrix in an encoding device.

[0043] Figure 27 is a diagram illustrating an example of the decoding process using a quantization matrix in a decoding device corresponding to the encoding device described in Figure 26.

[0044] Figure 28 is a diagram illustrating an example of a method for generating a quantization matrix using a common method, based on the quantization coefficient values ​​of the quantization matrix of only diagonal components for each block size, in steps S1601 of Figure 26 and S1701 of Figure 27.

[0045] Figure 29 is a diagram illustrating steps S1601 in Figure 26 and S1701 in Figure 27, showing another example of a method for generating a quantization matrix from the quantization coefficient values ​​of the quantization matrix of only diagonal components for processing object blocks of each block size using a common method.

[0046] Figure 30 is a block diagram showing an example of the installation of the encoding device.

[0047] Figure 31 is a flowchart illustrating an example of the operation of the encoding device shown in Figure 30.

[0048] Figure 32 is a block diagram showing an example of the installation of the decoding device.

[0049] Figure 33 is a flowchart illustrating an example of the operation of the decoding device shown in Figure 32.

[0050] Figure 34 is a diagram showing the overall structure of the content supply system that enables content publishing services.

[0051] Figure 35 is a diagram illustrating an example of coding construction in adaptable coding.

[0052] Figure 36 is a diagram illustrating an example of coding construction in adaptability coding.

[0053] Figure 37 is an example of a webpage display.

[0054] Figure 38 is an example of a webpage display.

[0055] Figure 39 is a diagram showing an example of a smartphone.

[0056] Figure 40 is a block diagram showing an example of the structure of a smartphone. Implementation

[0057] Forms used to implement inventions The following diagrams illustrate the specific implementation details.

[0058] Furthermore, the embodiments described below are all general or specific examples. The numerical values, shapes, materials, constituent elements, the arrangement and connection of constituent elements, steps, and the order of steps shown in the following embodiments are examples, and their purpose is not to limit the scope of the patent application. Also, among the constituent elements of the following embodiments, those constituent elements not described in the independent claim representing the highest-level concept are described as arbitrary constituent elements. (Implementation Form 1)

[0059] First, an overview of Embodiment 1 is provided as an example of an encoding and decoding apparatus applicable to the processing and / or configuration described in the various embodiments of this disclosure. However, Embodiment 1 is only one example of an encoding and decoding apparatus applicable to the processing and / or configuration described in the various embodiments of this disclosure, and the processing and / or configuration described in the various embodiments of this disclosure may also be implemented in encoding and decoding apparatuses different from Embodiment 1.

[0060] When applying the processing and / or configuration described in the various embodiments of this disclosure to Embodiment 1, any of the following may also be performed, for example. (1) For the encoding or decoding device of embodiment 1, the constituent elements of the plurality of constituent elements constituting the encoding or decoding device that correspond to the constituent elements described in the various embodiments of this disclosure are replaced with the constituent elements described in the various embodiments of this disclosure; (2) For the encoding or decoding device of embodiment 1, after any changes such as addition, replacement, or deletion of the constituent elements of the plurality of constituent elements constituting the encoding or decoding device, the constituent elements corresponding to the constituent elements described in each embodiment of this disclosure are replaced with the constituent elements described in each embodiment of this disclosure; (3) The method implemented by the encoding or decoding device of embodiment 1, after applying arbitrary changes such as substitution or deletion to the addition of processing and / or a part of the processing contained in the method, replaces the processing corresponding to the processing described in each embodiment of this disclosure with the processing described in each embodiment of this disclosure; (4) The encoding or decoding device of embodiment 1 is implemented by combining a portion of the constituent elements of the plurality of constituent elements with the constituent elements described in the various embodiments of this disclosure, a constituent element having a portion of the functions of the constituent elements described in the various embodiments of this disclosure, or a constituent element implementing a portion of the processing performed by the constituent elements described in the various embodiments of this disclosure; (5) A component having a portion of the functions of a portion of the components constituting the encoding or decoding device of Embodiment 1, or a component implementing a portion of the processing performed by a portion of the components constituting the encoding or decoding device of Embodiment 1, is combined with the components described in the various embodiments of this disclosure, the component having a portion of the functions of the components described in the various embodiments of this disclosure, or the component implementing a portion of the processing performed by the components described in the various embodiments of this disclosure; (6) The method implemented by the encoding or decoding apparatus of embodiment 1, wherein the processing of the plurality of processing contained in the method that corresponds to the processing described in each of the embodiments disclosed herein is replaced with the processing described in each of the embodiments disclosed herein; (7) A portion of the processing contained in the method implemented by the encoding or decoding device of embodiment 1 is combined with the processing described in the various embodiments disclosed herein.

[0061] Furthermore, the implementation methods of the processing and / or configurations described in the various embodiments disclosed herein are not limited to the above examples. For example, they can be implemented in devices used for different purposes than the motion picture / image encoding device or motion picture / image decoding device disclosed in Embodiment 1, or the processing and / or configurations described in each embodiment can be implemented individually. Also, they can be implemented by combining the processing and / or configurations described in different embodiments. [Overview of the encoding device]

[0062] First, an overview of the encoding device in Embodiment 1 will be described. Figure 1 is a block diagram showing the functional configuration of the encoding device 100 in Embodiment 1. The encoding device 100 is a motion picture / image encoding device that encodes motion pictures / images in blocks.

[0063] As shown in Figure 1, the encoding device 100 is an image encoding device that encodes images in blocks, and includes a segmentation unit 102, a subtraction unit 104, a conversion unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse conversion unit 114, an addition unit 116, a block memory 118, a loop filter unit 120, a frame memory 122, an intra-frame prediction unit 124, an inter-frame prediction unit 126, and a prediction control unit 128.

[0064] The encoding device 100 is implemented using, for example, a general-purpose processor and memory. When the processor executes the software program stored in memory, it functions as a segmentation unit 102, a subtraction unit 104, a conversion unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse conversion unit 114, an addition unit 116, a loop filter unit 120, an intra-frame prediction unit 124, an inter-frame prediction unit 126, and a prediction control unit 128. Alternatively, the encoding device 100 may be implemented using one or more dedicated electronic circuits corresponding to the segmentation unit 102, the subtraction unit 104, the conversion unit 106, the quantization unit 108, the entropy encoding unit 110, the inverse quantization unit 112, the inverse conversion unit 114, the addition unit 116, the loop filter unit 120, the intra-frame prediction unit 124, the inter-frame prediction unit 126, and the prediction control unit 128.

[0065] The following describes the constituent components of the encoding device 100. [Divider]

[0066] The segmentation unit 102 divides the images contained in the input dynamic image into a plurality of blocks and outputs each block to the subtraction unit 104. For example, the segmentation unit 102 first segments the image into blocks of fixed size (e.g., 128×128). These fixed-size blocks are sometimes called coding tree units (CTUs). Then, the segmentation unit 102 divides each fixed-size block into blocks of variable size (e.g., 64×64 or less) according to the recursive quadtree and / or binary tree block segmentation. These variable-size blocks are sometimes called coding units (CUs), prediction units (PUs), or transformation units (TUs). Furthermore, in this embodiment, it is not necessary to distinguish between CUs, PUs, and TUs, and some or all blocks in the image can be processed as CUs, PUs, or TUs.

[0067] Figure 2 is a diagram illustrating an example of block partitioning in implementation form 1. In Figure 2, solid lines represent block boundaries in quaternion tree block partitioning, and dashed lines represent block boundaries in binary tree block partitioning.

[0068] Here, block 10 is a square block of 128×128 pixels (128×128 block). This 128×128 block 10 is first divided into 4 square blocks of 64×64 (quadtree block partitioning).

[0069] The top-left 64×64 block is further vertically divided into two rectangular 32×64 blocks, and the left 32×64 block is further vertically divided into two rectangular 16×64 blocks (binary tree block partitioning). As a result, the top-left 64×64 block is divided into two 16×64 blocks 11 and 12 and a 32×64 block 13.

[0070] The upper right 64×64 block is horizontally divided into two rectangular 64×32 blocks, 14 and 15 (binary tree block division).

[0071] The bottom left 64×64 block is divided into four 32×32 square blocks (quadtree block partitioning). Of these four 32×32 blocks, the top left and bottom right blocks are further partitioned. The top left 32×32 block is vertically partitioned into two 16×32 rectangular blocks, and the right 16×32 block is further horizontally partitioned into two 16×16 blocks (binary tree block partitioning). The bottom right 32×32 block is horizontally partitioned into two 32×16 blocks (binary tree block partitioning). As a result, the bottom left 64×64 block is divided into 16×32 block 16, two 16×16 blocks 17 and 18, two 32×32 blocks 19 and 20, and two 32×16 blocks 21 and 22.

[0072] The bottom right 64x64 block 23 is not divided.

[0073] As shown above, in Figure 2, block 10 is divided into 13 variable-size blocks 11 to 23 based on the recursive quad-tree and binary tree block partitioning. This type of partitioning is sometimes called QTBT (quad-tree plus binary tree) partitioning.

[0074] Furthermore, as shown in Figure 2, although a block can be divided into 4 or 2 blocks (quadtree or binary tree block partitioning), the partitioning is not limited to these. For example, a block can also be divided into 3 blocks (ternary tree block partitioning). This type of partitioning, which includes ternary tree block partitioning, is sometimes called MBT (multi-type tree) partitioning. [Subtraction Department]

[0075] The subtraction unit 104 subtracts the prediction signal (prediction sample) from the original signal (original sample) in units of blocks divided by the segmentation unit 102. That is, the subtraction unit 104 calculates the prediction error (also called residual) of the encoding target block (hereinafter referred to as the current block). Then, the subtraction unit 104 outputs the calculated prediction error to the conversion unit 106.

[0076] The original signal is the input signal of the encoding device 100, which is a signal representing the images of each picture that constitutes the moving image (e.g., luminance signal and two chroma signals). Hereinafter, the signal representing the image is sometimes also referred to as a sample. [Transition Section]

[0077] The conversion unit 106 converts the prediction error in the spatial domain into conversion coefficients in the frequency domain and outputs the conversion coefficients to the quantization unit 108. Specifically, the conversion unit 106 performs a predetermined discrete cosine transform (DCT) or discrete sine transform (DST) on the prediction error in the spatial domain, for example.

[0078] Furthermore, the transformation unit 106 can also adaptively select a transformation type from a plurality of transformation types and use the transformation basis function corresponding to the selected transformation type to convert the prediction error into transformation coefficients. This type of transformation is sometimes called EMT (explicit multiple core transform) or AMT (adaptive multiple transform).

[0079] The complex transformation types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII. Figure 3 is a table showing the basis functions corresponding to each transformation type. In Figure 3, N represents the number of input pixels. When selecting a transformation type from these complex transformation types, it may depend on, for example, the type of prediction (intra-frame prediction and inter-frame prediction) or the intra-frame prediction mode.

[0080] Information indicating whether EMT or AMT is applicable (e.g., referred to as the AMT flag) and information indicating the selected conversion type are signaled at the CU level. Furthermore, the signaling of such information is not limited to the CU level and can also be at other levels (e.g., sequence level, picture level, slice level, tile level, or CTU level).

[0081] Furthermore, the transformation unit 106 can also perform a further transformation on the transformation coefficients (transformation results). This type of further transformation is sometimes called AST (adaptive secondary transform) or NSST (non-separable secondary transform). For example, the transformation unit 106 performs a further transformation on each sub-block (e.g., a 4×4 sub-block) contained in the block of transformation coefficients corresponding to the intra-frame prediction error. The information indicating whether NSST is applicable and the related information of the transformation matrix used for NSST are signaled at the CU level. Moreover, the signaling of such information is not limited to the CU level, and can also be at other levels (e.g., sequence level, image level, slice level, tile level, or CTU level).

[0082] Separable transformation refers to the method of performing multiple transformations by separating the input in each direction according to the number of dimensions, while non-separable transformation refers to the method of treating two or more dimensions as a single dimension and performing a transformation when the input is multidimensional.

[0083] For example, the following can be used as an example of a non-separable transformation: when the input is a 4×4 block, it is treated as an array with 16 elements, and the array is transformed using a 16×16 transformation matrix.

[0084] Similarly, if we consider a 4×4 input block as an array with 16 elements, performing a complex number of Givens rotations (Hypercube Givens Transform) on this array is also an example of a Non-Separable transformation. [Quantitative Department]

[0085] The quantization unit 108 quantizes the conversion coefficients output from the conversion unit 106. Specifically, the quantization unit 108 scans the conversion coefficients of the current block in a predetermined scan order and quantizes the conversion coefficients according to the quantization parameters (QP) corresponding to the scanned conversion coefficients. Then, the quantization unit 108 outputs the quantized conversion coefficients (hereinafter referred to as quantization coefficients) of the current block to the entropy encoding unit 110 and the dequantization unit 112.

[0086] The specified order is the order in which the conversion coefficients are quantized / dequantized. For example, the specified scan order is defined by ascending frequency (from low frequency to high frequency) or descending frequency (from high frequency to low frequency).

[0087] The quantization parameter defines the quantization step size (quantization width). For example, increasing the quantization parameter value increases the quantization step size. In other words, increasing the quantization parameter value increases the quantization error. [Entropy Coding Department]

[0088] The entropy coding unit 110 generates a coded signal (coded bit stream) by performing variable-length encoding on the quantization coefficients input from the quantization unit 108. Specifically, the entropy coding unit 110, for example, binarizes the quantization coefficients and performs arithmetic encoding on the binary signal. [Anti-Quantitative Department]

[0089] The dequantization unit 112 dequantizes the quantization coefficients input from the quantization unit 108. Specifically, the dequantization unit 112 dequantizes the quantization coefficients of the current block in a predetermined scanning order. Then, the dequantization unit 112 outputs the dequantized conversion coefficients of the current block to the deconversion unit 114. [Reverse Conversion Section]

[0090] The inverse conversion unit 114 restores the prediction error by inversely converting the conversion coefficients input from the inverse quantization unit 112. Specifically, the inverse conversion unit 114 restores the prediction error of the current block by performing an inverse conversion on the conversion coefficients corresponding to the conversion of the conversion unit 106. Then, the inverse conversion unit 114 outputs the restored prediction error to the addition unit 116.

[0091] Furthermore, since the prediction error of the restoration is due to the loss of information caused by quantization, it will not be consistent with the prediction error calculated by the subtraction unit 104. That is, the prediction error of the restoration includes quantization error. [Adding Department]

[0092] The addition unit 116 reconstructs the current block by adding the prediction error input from the inverse conversion unit 114 and the prediction sample input from the prediction control unit 128. Then, the addition unit 116 outputs the reconstructed block to the block memory 118 and the loop filter unit 120. The reconstructed block is sometimes also called a local decoding block. [Block Memory]

[0093] Block memory 118 is a memory unit used to store blocks within the encoded object image (hereinafter referred to as the current image) that are referenced for intra-frame prediction. Specifically, block memory 118 stores the reconstructed blocks output from the arithmetic unit 116. [Loop Filtering Section]

[0094] The loop filter unit 120 applies loop filtering to the blocks reconstructed by the arithmetic unit 116 and outputs the filtered reconstructed blocks to the frame memory 122. Loop filtering refers to filters used within the coding loop, including, for example, deblocking filters (DF), sample adaptive offset (SAO), and adaptive loop filters (ALF).

[0095] In ALF, the least square error filter is used to remove coding distortion. For example, for each of the 2×2 sub-blocks in the current block, a filter is selected from a plurality of filters based on the direction and activity of the local gradient.

[0096] Specifically, firstly, sub-blocks (e.g., 2×2 sub-blocks) are classified into multiple groups (e.g., 15 or 25 groups). Sub-block classification is based on the direction and activity of the gradient. For example, the classification value C (e.g., C = 5D + A) is calculated using the gradient direction value D (e.g., 0~2 or 0~4) and the gradient activity value A (e.g., 0~4). Then, based on the classification value C, the sub-blocks are classified into multiple groups (e.g., 15 or 25 groups).

[0097] The gradient direction value D is derived, for example, by comparing the gradients in multiple directions (e.g., horizontal, vertical, and two diagonal directions). Similarly, the gradient activity value A is derived, for example, by summing the gradients in multiple directions and quantizing the summation result.

[0098] Based on the results of this classification, the filter to be used for a sub-block is determined from a plurality of filters.

[0099] The shape of the filter used in an ALF can utilize, for example, a circularly symmetrical shape. Figures 4A-4C are diagrams showing multiple examples of the shapes of filters used in an ALF. Figure 4A shows a 5×5 rhombus-shaped filter, Figure 4B shows a 7×7 rhombus-shaped filter, and Figure 4C shows a 9×9 rhombus-shaped filter. The information representing the filter shape is signaled at the image level. Furthermore, the signaling of the filter shape information is not limited to the image level; it can also be at other levels (e.g., sequence level, slice level, tile level, CTU level, or CU level).

[0100] The on / off state of ALF is determined at, for example, the picture level or the CU level. For instance, regarding luminance, the CU level determines whether ALF is applicable; regarding chromatic aberration, the picture level determines whether ALF is applicable. The information indicating whether ALF is on / off is signaled at the picture level or the CU level. Furthermore, the signaling of ALF on / off information is not limited to the picture level or the CU level; it can also be at other levels (such as sequence level, slice level, tile level, or CTU level).

[0101] The coefficient set of selectable multiple filters (e.g., up to 15 or 25 filters) is signaled at the picture level. Furthermore, the signaling of the coefficient set is not limited to the picture level, but can also be at other levels (e.g., sequence level, slice level, tile level, CTU level, CU level, or sub-block level). [Frame Memory]

[0102] Frame memory 122 is a memory unit used to store reference images used for inter-frame prediction, and is sometimes called a frame buffer. Specifically, frame memory 122 stores reconstructed blocks filtered by loop filter unit 120. Intra-frame prediction unit

[0103] The intra-frame prediction unit 124 references blocks within the current image stored in the block memory 118 to perform intra-frame prediction (also known as intra-frame prediction) of the current block, thereby generating a prediction signal (intra-frame prediction signal). Specifically, the intra-frame prediction unit 124 references samples (e.g., luminance values, chrominance values) of blocks adjacent to the current block to perform intra-frame prediction, thereby generating an intra-frame prediction signal, and outputs the intra-frame prediction signal to the prediction control unit 128.

[0104] For example, the intra-prediction unit 124 performs intra-prediction using one of a predefined plurality of intra-prediction modes. The plurality of intra-prediction modes includes one or more non-directional prediction modes and a plurality of directional prediction modes.

[0105] One or more non-directional prediction modes include, for example, the Planar prediction mode and the DC prediction mode as specified in the H.265 / HEVC (High-Efficiency Video Coding) specification (Non-Patent Document 1).

[0106] Multiple directional prediction modes include, for example, the 33-directional prediction modes specified in the H.265 / HEVC specification. Furthermore, in addition to the 33 directions, multiple directional prediction modes may further include 32-directional prediction modes (a total of 65 directional prediction modes). Figure 5A is a diagram showing 67 intra-frame prediction modes (2 non-directional prediction modes and 65 directional prediction modes). Solid arrows represent the 33 directions specified in the H.265 / HEVC specification, and dashed arrows represent the additional 32 directions.

[0107] Furthermore, in intra-frame prediction of chromatic aberration blocks, luma blocks can also be referenced. That is, the chromatic aberration component of the current block can be predicted based on its luma component. This type of intra-frame prediction is sometimes called CCLM (cross-component linear model) prediction. This type of intra-frame prediction mode for chromatic aberration blocks that references luma blocks (e.g., called CCLM mode) can also be included as one of the intra-frame prediction modes for chromatic aberration blocks.

[0108] The intra-prediction unit 124 can also correct the intra-predicted pixel values ​​based on the gradients of reference pixels in the horizontal / vertical directions. Intra-prediction accompanied by such corrections is sometimes called PDPC (position-dependent intra-prediction combination). Information indicating whether PDPC is applicable (e.g., a PDPC flag) is signaled at, for example, the CU level. Furthermore, the signaling of this information is not limited to the CU level; it can also be at other levels (e.g., sequence level, image level, tile level, or CTU level). [Inter-frame prediction department]

[0109] The inter-frame prediction unit 126 references a reference image stored in the frame memory 122, which is different from the current image, to perform inter-frame prediction (also called inter-frame prediction) for the current block, thereby generating a prediction signal (inter-frame prediction signal). Inter-frame prediction is performed on a unit of the current block or sub-blocks within the current block (e.g., 4×4 blocks). For example, the inter-frame prediction unit 126 performs motion estimation within the reference image for the current block or sub-block. Then, the inter-frame prediction unit 126 uses the motion information (e.g., motion vector) obtained through motion estimation to perform motion compensation, thereby generating the inter-frame prediction signal for the current block or sub-block. Then, the inter-frame prediction unit 126 outputs the generated inter-frame prediction signal to the prediction control unit 128.

[0110] Motion information used for motion compensation is signaled. The signaling of the motion vector can also be achieved using a motion vector predictor. In other words, the difference between the motion vector and the motion vector predictor can also be signaled.

[0111] Furthermore, inter-frame prediction signals can be generated not only using motion information of the current block obtained through motion estimation, but also using motion information of adjacent blocks. Specifically, the prediction signal based on motion information obtained through motion estimation and the prediction signal based on motion information of adjacent blocks can be weighted and calculated to generate inter-frame prediction signals on a per-sub-block basis within the current block. This type of inter-frame prediction (motion compensation) is sometimes called OBMC (overlapped block motion compensation).

[0112] In this type of OBMC mode, information indicating the size of OBMC sub-blocks (e.g., OBMC block size) is signaled at the sequence level. Furthermore, information indicating whether or not the OBMC mode is applied (e.g., OBMC flag) is signaled at the CU level. Moreover, the signaling level of this information is not limited to the sequence level and CU level; it can also be other levels (e.g., image level, slice level, tile level, CTU level, or sub-block level).

[0113] To illustrate the OBMC model in more detail, Figures 5B and 5C are flowcharts and conceptual diagrams illustrating the overview of the predicted image correction process using OBMC.

[0114] First, using the motion vectors (MV) assigned to the encoded object blocks, a general motion-compensated prediction image (Pred) is obtained.

[0115] Next, the movement vector (MV_L) of the encoded left adjacent block is applied to the encoded target block to obtain the prediction image (Pred_L). The prediction image and Pred_L are then weighted and overlapped to perform the first correction of the prediction image.

[0116] Similarly, the movement vector (MV_U) of the encoded upper adjacent block is applied to the encoded target block to obtain the prediction image (Pred_U). The prediction image that has undergone the first correction mentioned above is weighted and overlapped with Pred_U to perform the second correction of the prediction image, which is then used as the final prediction image.

[0117] Furthermore, although the method described here is a two-stage correction using the left adjacent block and the top adjacent block, it is also possible to use the right adjacent block or the bottom adjacent block to perform more than two stages of correction.

[0118] Furthermore, the overlapping area may not be the entire pixel area of ​​the block, but only a part of the area near the block boundary.

[0119] Furthermore, although the process of correcting the predicted image from a single reference image has been described here, the process of correcting the predicted image from multiple reference images is also the same: after obtaining the predicted images corrected from each reference image, the predicted images are further superimposed to obtain the final predicted image.

[0120] Furthermore, the aforementioned processing target blocks can be in units of predicted blocks or sub-blocks formed by further dividing the predicted blocks.

[0121] Methods for determining whether OBMC processing is applicable include, for example, using the signal `obmc_flag`, which indicates whether OBMC processing is applicable. Specifically, in an encoding device, it is determined whether the block to be encoded belongs to a motion-complex region. If it does, `obmc_flag` is set to 1, and OBMC processing is applied for encoding; if it does not belong to a motion-complex region, `obmc_flag` is set to 0, and OBMC processing is not applied for encoding. Furthermore, in a decoding device, the value of `obmc_flag` recorded in the decoding stream is used to switch whether OBMC processing is applied for decoding.

[0122] Furthermore, motion information can also be derived from the decoding device without signaling. For example, the merging mode specified in the H.265 / HEVC standard can be used. Alternatively, motion information can be derived by performing motion estimation on the decoding device side. In this case, motion estimation is performed without using the pixel values ​​of the current block.

[0123] Here, we will explain the mode of motion estimation performed on the decoding device side. This mode of motion estimation performed on the decoding device side is sometimes called PMMVD (pattern matched motion vector derivation) mode or FRUC (frame rate up-conversion) mode.

[0124] Figure 5D illustrates an example of FRUC processing. First, referencing the movement vectors of coded blocks spatially or temporally adjacent to the current block, multiple candidate lists (which can also be shared with the merge list) are generated, each containing a movement vector predictor. Next, the best candidate MV is selected from the multiple candidate MVs registered in the candidate lists. For example, the evaluation value of each candidate in the candidate list is calculated, and one candidate is selected based on the evaluation value.

[0125] Then, based on the selected candidate motion vectors, the motion vector used for the current block is derived. Specifically, for example, the selected candidate motion vector (best candidate MV) is directly derived as the motion vector used for the current block. Alternatively, for example, style matching can be performed on the surrounding area of ​​the position in the reference image corresponding to the selected candidate motion vector to derive the motion vector used for the current block. That is, the surrounding area of ​​the best candidate MV can also be estimated in the same way, and if there is an MV with a better evaluation value, the best candidate MV is updated to the aforementioned MV and used as the final MV of the current block. Furthermore, a configuration that does not perform this processing can also be adopted.

[0126] When processing at the sub-block level, the exact same processing method can be used.

[0127] Furthermore, the evaluation value is calculated by determining the difference between the reconstructed image and the specified region through style matching within the reference image corresponding to the motion vector. In addition to the difference value, other information can also be used to calculate the evaluation value.

[0128] Style matching utilizes first-style matching and second-style matching. First-style matching and second-style matching are sometimes referred to as bilateral matching and template matching, respectively.

[0129] The first style matching is performed between two blocks in two different reference images, specifically between two blocks that follow the motion trajectory of the current block. Therefore, the first style matching uses regions in other reference images along the motion trajectory of the current block as the specified regions used to calculate the aforementioned candidate evaluation values.

[0130] Figure 6 illustrates an example of pattern matching (bidirectional matching) between two blocks along a movement track. As shown in Figure 6, in the first pattern matching, two movement vectors (MV0, MV1) are derived by estimating the best matching pair among two blocks in two different reference images (Ref0, Ref1) along the movement track of the current block. Specifically, for the current block, the difference between the reconstructed image at a specified position in the first encoded reference image (Ref0) specified by the candidate MV and the reconstructed image at a specified position in the second encoded reference image (Ref1) specified by the symmetrical MV is derived. An evaluation value is calculated using the obtained difference value, where the symmetrical MV is the MV scaled by the candidate MV over a display time interval. The candidate MV with the best evaluation value among the plurality of candidate MVs is selected as the final MV.

[0131] Assuming a continuous movement trajectory, the movement vectors (MV0, MV1) indicating two reference blocks are proportional to the temporal distances (TD0, TD1) between the current image (Cur Pic) and the two reference images (Ref0, Ref1). For example, if the current image is located between two reference images in time, and the temporal distances from the current image to the two reference images are equal, then matching in style 1 will result in a reflectively symmetric bidirectional movement vector.

[0132] The second style matching involves matching the template within the current image (blocks adjacent to the current block within the current image (e.g., top and / or left adjacent blocks)) with blocks in the reference image. Therefore, the second style matching utilizes blocks adjacent to the current block within the current image as the defined region used to calculate the aforementioned candidate evaluation value.

[0133] Figure 7 illustrates an example of style matching (template matching) between a template in the current image and a block in the reference image. As shown in Figure 7, in the second style matching, the movement vector of the current block is derived by estimating the block in the reference image (Ref0) that best matches the block adjacent to the current block in the current image (Cur block). Specifically, for the current block, the difference between the reconstructed images of the encoded regions of its left and top neighbors (or one of them) and the reconstructed image at the same position in the encoded reference image (Ref0) specified by the candidate MV is derived. An evaluation value is calculated using the obtained difference value, and the candidate MV with the best evaluation value among the plurality of MV candidates is selected as the best candidate MV.

[0134] Information indicating whether or not FRUC mode is applied (e.g., referred to as the FRUC flag) is signaled at the CU level. Furthermore, when FRUC mode is applied (e.g., when the FRUC flag is true), information indicating the style matching method (first style matching or second style matching) (e.g., referred to as the FRUC mode flag) is signaled at the CU level. Moreover, the signaling of this information is not limited to the CU level; it can also be at other levels (e.g., sequence level, image level, slice level, tile level, CTU level, or sub-block level).

[0135] This section explains the mode for deriving the translation vector based on a model that assumes uniform linear motion. This mode is sometimes referred to as the BIO (bi-directional optical flow) mode.

[0136] Figure 8 is a diagram illustrating the model assuming constant linear motion. In Figure 8, (vx, vy) represents the velocity vector, and τ0 and τ1 represent the time distances between the current image (Cur Pic) and the two reference images (Ref 0, Ref 1), respectively. (MVx 0, MVy 0) represents the movement vector corresponding to reference image Ref 0, and (MVx 1, MVy 1) represents the movement vector corresponding to reference image Ref 1.

[0137] At this point, under the assumption of constant linear motion of velocity vector (vx,vy), (MVx 0,MVy 0) and (MVx 1,MVy 1) are expressed as (v xτ 0,vy τ 0) and (-v xτ 1,-vy τ 1) respectively, and the following optical flow equation (1) holds. [Number 1]

[0138] Here, I(k) represents the brightness value of the reference image k (k=0,1) after motion compensation. The aforementioned optical flow equation states that the sum of (i), (ii), and (iii) equals zero, where (i) is the temporal derivative of the brightness value, (ii) is the product of the horizontal velocity and the horizontal component of the spatial gradient of the reference image, and (iii) is the product of the vertical velocity and the vertical component of the spatial gradient of the reference image. Based on the combination of the aforementioned optical flow equation and Hermite interpolation, the motion vector of the block unit obtained from the merge list, etc., is corrected in pixels.

[0139] Furthermore, a different method can be used to derive the movement vector on the decoding device side, instead of deriving it based on a model that assumes uniform linear motion. For example, the movement vector can be derived on a sub-block basis based on the movement vectors of a plurality of adjacent blocks.

[0140] This section explains a pattern for deriving movement vectors based on the movement vectors of a plurality of adjacent blocks, on a sub-block basis. This pattern is sometimes referred to as the affine motion compensation prediction pattern.

[0141] Figure 9A illustrates how to derive the movement vector of a sub-block unit based on the movement vectors of multiple adjacent blocks. In Figure 9A, the current block contains 16 4×4 sub-blocks. Here, based on the movement vectors of the adjacent blocks, the movement vector v0 of the upper left control point of the current block is derived, and based on the movement vectors of the adjacent sub-blocks, the movement vector v1 of the upper right control point of the current block is derived. Then, using the two movement vectors v0 and v1, the movement vectors (vx, vy) of each sub-block within the current block are derived using the following equation (2). [Number 2]

[0142] Here, x and y represent the horizontal and vertical positions of the sub-block, respectively, and w represents the pre-determined weighting coefficient.

[0143] This type of affine motion compensation prediction pattern may also include several patterns with different methods for deriving the motion vectors of the top-left and top-right control points. The information representing this type of affine motion compensation prediction pattern (e.g., affine flags) is signaled at the CU level. Furthermore, the signaling of the information representing this affine motion compensation prediction pattern is not limited to the CU level; it can also be at other levels (e.g., sequence level, image level, slice level, tile level, CTU level, or sub-block level). [Forecasting and Control Department]

[0144] The prediction control unit 128 selects either the intra-frame prediction signal or the inter-frame prediction signal, and outputs the selected signal as the prediction signal to the subtraction unit 104 and the addition unit 116.

[0145] Here, an example of deriving the motion vector of an encoded object image using a merge mode is explained. Figure 9B is a diagram illustrating the outline of the motion vector deriving process using a merge mode.

[0146] First, a list of candidate predicted MVs is generated. Candidate predicted MVs include: spatial adjacency predicted MVs, which are the MVs of multiple encoded blocks spatially surrounding the encoded object block; temporal adjacency predicted MVs, which are the MVs of blocks near the location in the encoded reference image that projects the encoded object block; combined predicted MVs, which are MVs generated by combining the MV values ​​of spatial and temporal adjacency predicted MVs; and zero predicted MVs, i.e., MVs with a value of zero, etc.

[0147] Next, the MV for the encoded object block is determined by selecting one MV from the multiple MVs registered in the MV list.

[0148] Furthermore, in the variable-length encoding section, the signal merge_idx, which indicates the selected prediction MV, is recorded in the stream and encoded.

[0149] Furthermore, the predicted MVs listed in the predicted MV list illustrated in Figure 9B are an example. Their number may differ from the number shown in the figure. They may also consist of a portion of the predicted MVs not included in the figure, or they may consist of predicted MVs not included in the figure.

[0150] Furthermore, the MV of the encoded object block derived by the merge mode can also be used for the DMVR process described later, thereby determining the final MV.

[0151] This illustrates an example of using DMVR processing to determine the MV.

[0152] Figure 9C is a conceptual diagram illustrating the outline of DMVR processing.

[0153] First, the best MVP set in the processing object block is taken as the candidate MV. According to the aforementioned candidate MV, reference pixels are obtained from the processed image in the L0 direction, i.e., the first reference image, and the processed image in the L1 direction, i.e., the second reference image. The template is generated by averaging the values ​​of each reference pixel.

[0154] Next, using the aforementioned template, the surrounding areas of the candidate MVs for the first and second reference images are estimated respectively, and the MV with the lowest cost is selected as the final MV. Furthermore, the cost value is calculated using the differences between the pixel values ​​of the template and the pixel values ​​of the estimated area, as well as the MV value.

[0155] Furthermore, the processing principles described herein are essentially the same for both the encoding and decoding devices.

[0156] Furthermore, instead of using the processing described herein, other processing methods can be used, as long as they can estimate the periphery of the candidate MV and derive the final MV.

[0157] This section explains the pattern of generating predicted images using LIC processing.

[0158] Figure 9D is a diagram illustrating the general outline of a predictive image generation method using luminance correction processing with LIC.

[0159] First, export the MV (Image Manager of the Reference Image) used to obtain the reference image corresponding to the encoded object block from the encoded image, i.e., the reference image.

[0160] Next, for the encoded object block, information representing how the brightness value changes in the reference image and the encoded object image is extracted using the brightness pixel values ​​of the left and top adjacent encoded surrounding reference areas, and the brightness pixel values ​​at the same position in the reference image specified by MV, and the brightness correction parameters are calculated.

[0161] The reference image within the reference image specified by MV is subjected to brightness correction processing using the aforementioned brightness correction parameters, thereby generating a predicted image for the encoded object block.

[0162] Furthermore, the shape of the aforementioned surrounding reference area in Figure 9D is one example; other shapes may also be used.

[0163] Furthermore, this describes the process of generating a predicted image from a single reference image. However, the process of generating a predicted image from multiple reference images is the same: the reference images obtained from each reference image are processed in the same way to correct their brightness before generating the predicted image.

[0164] One method for determining whether LIC processing is applicable includes, for example, using a signal `lic_flag` that indicates whether LIC processing is applicable. Specifically, in an encoding device, it is determined whether the area to be encoded belongs to a region where brightness changes. If it belongs to a region where brightness changes, `lic_flag` is set to 1, and LIC processing is applied for encoding. If it does not belong to a region where brightness changes, `lic_flag` is set to 0, and LIC processing is not applied for encoding. Furthermore, in a decoding device, the value of `lic_flag` recorded in the decoding stream is used to switch whether LIC processing is applied for decoding.

[0165] Other methods for determining whether LIC processing is applicable include, for example, determining whether LIC processing is applied to surrounding blocks. As a specific example, when the encoded target block is in merge mode, it is determined whether the surrounding encoded blocks selected during MV export in the merge process have been encoded using LIC processing, and the application of LIC processing is switched accordingly. Furthermore, in this case, the decoding process is exactly the same. [Overview of the Decoding Device]

[0166] Next, an outline of a decoding device capable of decoding the encoded signal (encoded bit stream) output from the aforementioned encoding device 100 will be described. Figure 10 is a block diagram showing the functional configuration of the decoding device 200 in Embodiment 1. The decoding device 200 is a motion picture / image decoding device that decodes motion pictures / images in blocks.

[0167] As shown in Figure 10, the decoding device 200 includes an entropy decoding unit 202, an inverse quantization unit 204, an inverse conversion unit 206, an addition unit 208, a block memory 210, a loop filter unit 212, a frame memory 214, an intra-frame prediction unit 216, an inter-frame prediction unit 218, and a prediction control unit 220.

[0168] The decoding device 200 is implemented using, for example, a general-purpose processor and memory. When the processor executes the software program stored in memory, it functions as an entropy decoding unit 202, an inverse quantization unit 204, an inverse conversion unit 206, an adder 208, a loop filter 212, an intra-frame prediction unit 216, an inter-frame prediction unit 218, and a prediction control unit 220. Alternatively, the decoding device 200 can be implemented using one or more dedicated electronic circuits corresponding to the entropy decoding unit 202, the inverse quantization unit 204, the inverse conversion unit 206, the adder 208, the loop filter 212, the intra-frame prediction unit 216, the inter-frame prediction unit 218, and the prediction control unit 220.

[0169] The following describes the components of the decoding device 200. [Entropy Decoding Department]

[0170] The entropy decoding unit 202 performs entropy decoding on the encoded bitstream. Specifically, the entropy decoding unit 202, for example, arithmetically decodes the encoded bitstream into a binary signal. Then, the entropy decoding unit 202 debinarizes the binary signal. In this way, the entropy decoding unit 202 outputs the quantization coefficients to the dequantization unit 204 in blocks. [Anti-Quantitative Department]

[0171] The dequantization unit 204 dequantizes the quantization coefficients of the decoded target block (hereinafter referred to as the current block) input from the entropy decoding unit 202. Specifically, the dequantization unit 204 dequantizes each quantization coefficient of the current block according to the quantization parameter corresponding to that quantization coefficient. Then, the dequantization unit 204 outputs the dequantized quantization coefficients (i.e., conversion coefficients) of the current block to the inverse conversion unit 206. [Reverse Conversion Section]

[0172] The inverse conversion unit 206 restores the prediction error by inversely converting the conversion coefficients input from the inverse quantization unit 204.

[0173] For example, when the representation information interpreted from the encoded bit stream is applicable to EMT or AMT (e.g., the AMT flag is true), the inverse conversion unit 206 performs inverse conversion on the conversion coefficients of the current block based on the interpreted representation conversion type information.

[0174] For example, when the information interpreted from the encoded bit stream is represented by NSST, the inverse conversion unit 206 applies inverse reconversion to the conversion coefficients. [Adding Department]

[0175] The addition unit 208 reconstructs the current block by adding the prediction error input from the inverse conversion unit 206 and the prediction sample input from the prediction control unit 220. Then, the addition unit 208 outputs the reconstructed block to the block memory 210 and the loop filter unit 212. [Block Memory]

[0176] Block memory 210 is a memory unit used to store blocks referenced for intra-frame prediction and blocks within the decoded target image (hereinafter referred to as the current image). Specifically, block memory 210 stores reconstructed blocks output from the arithmetic unit 208. [Loop Filtering Section]

[0177] The loop filter unit 212 applies loop filtering to the blocks reconstructed by the arithmetic unit 208 and outputs the filtered reconstructed blocks to the frame memory 214 and the display device, etc.

[0178] When the information indicating the on / off state of ALF is interpreted from the encoded bitstream, indicating that ALF is on, one filter is selected from a plurality of filters based on the direction and activity of the local gradient, and the selected filter is applied to the reconstructed block. [Frame Memory]

[0179] Frame memory 214 is a memory unit used to store reference images used for inter-frame prediction, and is sometimes called a frame buffer. Specifically, frame memory 214 stores the reconstructed blocks filtered by loop filter unit 212. Intra-frame prediction unit

[0180] The intra-prediction unit 216 performs intra-prediction based on the intra-prediction pattern interpreted from the coded bitstream, and references blocks within the current image stored in the block memory 210 to generate a prediction signal (intra-prediction signal). Specifically, the intra-prediction unit 216 performs intra-prediction by referencing samples (e.g., luminance values, chrominance values) of blocks adjacent to the current block, thereby generating an intra-prediction signal, and outputs the intra-prediction signal to the prediction control unit 220.

[0181] Furthermore, when selecting the intra-prediction mode of the reference luminance block in the intra-prediction of the chromatic difference block, the intra-prediction unit 216 can also predict the chromatic difference component of the current block based on the luminance component of the current block.

[0182] Furthermore, when the information interpreted from the encoded bitstream is used to represent PDPC, the intra-prediction unit 216 corrects the intra-predicted pixel values ​​based on the gradients of reference pixels in the horizontal / vertical directions. [Inter-frame prediction department]

[0183] The inter-frame prediction unit 218 predicts the current block by referring to a reference image stored in the frame memory 214. The prediction is performed on a unit of the current block or sub-blocks within the current block (e.g., 4×4 blocks). For example, the inter-frame prediction unit 218 uses motion information (e.g., motion vectors) interpreted from the coded bit stream to perform motion compensation, thereby generating an inter-frame prediction signal for the current block or sub-block, and outputs the inter-frame prediction signal to the prediction control unit 220.

[0184] Furthermore, when the information interpreted from the coded bit stream is applicable to OBMC mode, the inter-frame prediction unit 218 can not only use the motion information of the current block obtained by motion estimation, but also the motion information of adjacent blocks to generate inter-frame prediction signals.

[0185] Furthermore, when the information interpreted from the coded bitstream is in FRUC mode, the inter-frame prediction unit 218 performs motion estimation according to the pattern matching method (bidirectional matching or template matching) interpreted from the coded bitstream, thereby deriving motion information. Then, the inter-frame prediction unit 218 uses the derived motion information to perform motion compensation.

[0186] Furthermore, when using BIO mode, the inter-frame prediction unit 218 derives the motion vector based on a model assuming constant-speed linear motion. Also, when the information representation interpreted from the encoded bitstream is used in affine motion compensation prediction mode, the inter-frame prediction unit 218 derives the motion vector on a sub-block basis based on the motion vectors of a plurality of adjacent blocks. [Forecasting and Control Department]

[0187] The prediction control unit 220 selects either the intra-frame prediction signal or the inter-frame prediction signal, and outputs the selected signal as the prediction signal to the addition unit 208. [First state sample]

[0188] The following describes the encoding device 100, decoding device 200, encoding method, and decoding method of the first state pattern disclosed herein. [Example 1 of encoding and decoding using quantization matrices]

[0189] Figure 11 is a diagram illustrating the first example of the encoding process using a quantization matrix (QM) in the encoding apparatus 100. Furthermore, the encoding apparatus 100 described here performs encoding processing on square or rectangular blocks of images (hereinafter also referred to as frames) contained in a moving image.

[0190] First, in step S101, the quantization unit 108 generates a QM for square blocks. The QM for square blocks is a quantization matrix of multiple transformation coefficients for a square block. Hereinafter, the QM for square blocks will also be referred to as the first quantization matrix. Furthermore, the quantization unit 108 can generate the QM for square blocks from a value defined and set by the user in the encoding device 100, or it can adaptively generate the QM for square blocks using the encoding information of an already encoded image. Also, the entropy encoding unit 110 can record the signal related to the QM for square blocks generated by the quantization unit 108 in the bitstream. At this time, the QM for square blocks can also be encoded in the sequence header area, image header area, slice header area, auxiliary information area, or area storing other parameters in the stream. Alternatively, the QM for square blocks may not be recorded in the stream. In this case, the quantization unit 108 can also use a preset QM value for square blocks predefined in the specification. Furthermore, the entropy coding unit 110 may not record the coefficients (i.e., quantization coefficients) of all matrices of the square block using QM in the stream, but only record a portion of the quantization coefficients required to generate the QM in the stream. In this way, the amount of information to be encoded can be reduced.

[0191] Next, in step S102, the quantization unit 108 uses the square block QM generated in step S101 to generate a rectangular block QM. The rectangular block QM is a quantization matrix of a plurality of conversion coefficients for the rectangular block. Hereinafter, the rectangular block QM will also be referred to as the second quantization matrix. Furthermore, the entropy encoding unit 110 does not record the rectangular block QM signal in the stream.

[0192] Furthermore, the processing in steps S101 and S102 can be performed simultaneously at the beginning of sequence processing, the beginning of image processing, or the beginning of slice processing, or it can be performed in a portion of the block unit processing at a time. Also, the QM generated by the quantization unit 108 in steps S101 and S102 can be configured as follows: multiple types of QMs are generated for blocks of the same size, depending on conditions such as luminance block / chrominance block, intra-frame prediction block / inter-frame prediction block, and others.

[0193] Next, the block-based loop begins. First, in step S103, the intra-frame prediction unit 124 or the inter-frame prediction unit 126 performs prediction processing on a block-by-block basis, using intra-frame prediction or inter-frame prediction, etc. In step S104, the conversion unit 106 performs conversion processing on the generated prediction residual image using discrete cosine transform (DCT), etc. In step S105, the quantization unit 108 performs quantization processing on the generated conversion coefficients using the outputs of steps S101 and S102, namely, QM for square blocks and QM for rectangular blocks. Furthermore, for inter-frame prediction, the pattern of blocks in different images of the reference and processing target block can be used, as well as the pattern of blocks in the image of the reference processing target block. At this time, the QM for inter-frame prediction can be used for both patterns, or the QM for intra-frame prediction can be used for the pattern of blocks in the image of the reference processing target block. Furthermore, in step S106, the dequantization unit 112 performs dequantization processing on the quantized conversion coefficients using the outputs of steps S101 and S102, namely the QM for square blocks and the QM for rectangular blocks. In step S107, the inverse conversion unit 114 performs inverse conversion processing on the dequantized conversion coefficients, thereby generating a residual (prediction error) image. Next, in step S108, the addition unit 116 generates a reconstructed image by adding the residual image and the prediction image. This series of processing flows is repeated to end the loop of the block unit.

[0194] Therefore, even in encoding methods with rectangular blocks of various shapes, it is not necessary to record the QM corresponding to each rectangular block shape in the stream; only the QM corresponding to the square block needs to be recorded in the stream for encoding processing. In other words, if the encoding apparatus 100 according to the first state sample of this disclosure does not record the QM corresponding to the rectangular block in the stream, the amount of code in the header region can be reduced. Furthermore, if the encoding apparatus 100 according to the first state sample of this disclosure can generate the QM corresponding to the rectangular block from the QM corresponding to the square block, the amount of code in the header region will not increase, and an appropriate QM can be used for the rectangular block. Therefore, if the encoding apparatus 100 according to the first state sample of this disclosure can efficiently quantize rectangular blocks of various shapes, the possibility of improving encoding efficiency is increased. Moreover, it is also possible not to record the QM for the square block in the stream, or to use a preset QM value for the square block defined in the specification.

[0195] Furthermore, this processing flow is just one example; the recorded processing order can also be changed, a portion of the recorded processing can be deleted, or unrecorded processing can be added.

[0196] Figure 12 is a diagram illustrating an example of the decoding process using a quantization matrix (QM) in the decoding apparatus 200 corresponding to the encoding apparatus 100 described in Figure 11. Furthermore, the decoding apparatus 200 described here performs decoding processing on each square or rectangular block after the screen is divided.

[0197] First, in step S201, the entropy decoding unit 202 decodes the signal related to the QM for the square block from the streaming stream, and generates the QM for the square block using the decoded signal. Furthermore, the QM for the square block can also be decoded from the sequence header area, image header area, slice header area, auxiliary information area, or area storing other parameters of the streaming stream. Alternatively, the QM for the square block can be generated without decoding from the streaming stream. In this case, preset values ​​defined in the specification can also be used as the QM for the square block. Also, the entropy decoding unit 202 can generate the QM by decoding only a portion of the quantization coefficients required to generate the QM from the streaming stream, without decoding all the quantization coefficients of the matrix used to generate the QM.

[0198] Next, in step S202, the entropy decoding unit 202 uses the square block QM generated in step S201 to generate a rectangular block QM. Furthermore, the entropy decoding unit 202 does not decode the rectangular block QM signal from the streaming stream.

[0199] Furthermore, the processing in steps S201 and S202 can be performed simultaneously at the beginning of sequence processing, the beginning of image processing, or the beginning of slice processing, or it can be performed in block unit processing, with a portion of the processing done each time. Also, the QM generated by the entropy decoding unit 202 in steps S201 and S202 can be configured as follows: for blocks of the same size, multiple types of QM are generated based on conditions such as luminance block / chrominance block, intra-frame prediction block / inter-frame prediction block, and other factors.

[0200] Next, the block-level loop begins. First, in step S203, the intra-frame prediction unit 216 or the inter-frame prediction unit 218 performs prediction processing on a block-by-block basis, using intra-frame prediction or inter-frame prediction, etc. In step S204, the inverse conversion unit 204 performs inverse quantization processing on the quantized conversion coefficients (i.e., quantization coefficients) decoded from the stream, using the outputs of steps S201 and S202, namely, QM for square blocks and QM for rectangular blocks. Furthermore, for inter-frame prediction, the pattern of blocks in different images of the image to which the target block belongs can be used, as well as the pattern of blocks in the image to which the target block belongs. At this time, the QM for inter-frame prediction can be used for both patterns, or the QM for intra-frame prediction can be used for the pattern of blocks in the image to which the target block belongs. Next, in step S205, the inverse conversion unit 206 performs inverse conversion processing on the inverse quantized conversion coefficients, thereby generating a residual (prediction error) image. Next, in step S206, the addition unit 208 generates a reconstructed image by adding the residual image and the predicted image. This series of processing steps is repeated to end the loop of the block unit.

[0201] Therefore, even in encoding methods with rectangular blocks of various shapes, it is not necessary to record the QM corresponding to each rectangular block shape in the stream; only the QM corresponding to the square block needs to be recorded in the stream for decoding. In other words, with the decoding apparatus 200 of the first state according to this disclosure, since the QM corresponding to the rectangular block is not recorded in the stream, the amount of code in the header region can be reduced. Furthermore, with the decoding apparatus 200 of the first state according to this disclosure, since the QM corresponding to the square block can be generated for the rectangular block, the amount of code in the header region will not increase, and an appropriate QM can be used for the rectangular block. Therefore, with the decoding apparatus 200 of the first state according to this disclosure, since it can efficiently quantize rectangular blocks of various shapes, the possibility of improving encoding efficiency is increased. Moreover, it is also possible not to record the QM for the square block in the stream, or to use a preset QM value for the square block defined in the specification.

[0202] Furthermore, this processing flow is just one example; the recorded processing order can also be changed, a portion of the recorded processing can be deleted, or unrecorded processing can be added. [The first example of using QM's generation method to create a rectangular block]

[0203] Figure 13 is a diagram illustrating step S102 in Figure 11 and step S202 in Figure 12, showing a first example of generating a rectangular block from a QM using a QM. Furthermore, the process described here is a common process for both the encoding device 100 and the decoding device 200.

[0204] In Figure 13, for square blocks with dimensions ranging from 2×2 to 256×256, the dimensions of the rectangular blocks QM generated from the square blocks of each size are correspondingly recorded. In the example shown in Figure 13, the characteristic is that the length of the long side of each rectangular block is the same as the length of one side of the corresponding square block. In other words, in this example, the characteristic is that the size of the processing target block, i.e., the rectangular block, is smaller than the size of the square block. That is, the encoding device 100 and decoding device 200 of the first state disclosed herein generate the rectangular block QM by down-converting the square block QM with one side having the same length as the long side of the processing target block, i.e., the rectangular block.

[0205] Furthermore, Figure 13 shows the correspondence between square block QMs of various block sizes, without distinguishing between luminance and chrominance blocks, and rectangular block QMs generated from each square block QM. A suitable correspondence between square block QMs and rectangular block QMs for practical use can also be derived. For example, in the 4:2:0 format, the luminance block is twice the size of the chrominance block. Therefore, when referencing the luminance block in the process of generating rectangular block QMs from square block QMs, the usable square block QMs correspond to square blocks of sizes from 4×4 to 256×256. In this case, among the rectangular block QMs generated from square block QMs, only QMs corresponding to rectangular blocks with a short side length of 4 or more and a long side length of 256 or less are used. Furthermore, when referencing color difference blocks in the process of generating a rectangular block QM from a square block QM, the square block QM that can be used corresponds to square blocks with dimensions from 2×2 to 128×128. At this time, in the rectangular block QM generated from the square block QM, only QMs corresponding to rectangular blocks with dimensions of 2 or more for the shorter side and 128 or less for the longer side are used.

[0206] Furthermore, in the case of a 4:4:4 format, the luminance block and the chrominance block are blocks of the same size. Therefore, when referencing the chrominance block in the process of generating rectangular blocks using QM, the same applies as when referencing the luminance block; the square blocks that can be used using QM correspond to square blocks of sizes from 4×4 to 256×256.

[0207] In this way, the correspondence between QM for square blocks and QM for rectangular blocks can be appropriately derived according to the actual format used.

[0208] Furthermore, the block size shown in Figure 13 is an example, and is not limited to this. For example, QM can be used for block sizes other than those shown in Figure 13, or for square blocks using only a portion of the block sizes shown in Figure 13.

[0209] Figure 14 illustrates the method of generating the rectangular block QM shown in Figure 13 from the corresponding square block QM by down-conversion.

[0210] In the example of Figure 14, an 8×4 rectangular block is generated from an 8×8 square block using QM.

[0211] In the following transformation process, the multiple matrix elements of the QM used for the square block can also be divided into groups with the same number of multiple matrix elements as the QM used for the rectangular block. For each of the multiple groups, the multiple matrix elements contained in the group are continuously arranged in the horizontal or vertical direction of the square block. For each of the multiple groups, the matrix element located on the lowest frequency side of the multiple matrix elements contained in the group is determined as the matrix element in the QM used for the rectangular block corresponding to that group.

[0212] For example, in Figure 14, the multiple matrix elements of the QM for an 8×8 square block are each enclosed by a specified number of thick circles. These specified number of thick-circled matrix elements form a group. In the downconversion process illustrated in Figure 14, the 8×8 square block QM is divided in such a way that the number of these groups is the same as the number of matrix elements (also called quantization coefficients) of the rectangular block QM generated from the 8×8 square block QM. In the example of Figure 14, two quantization coefficients adjacent in the vertical direction form a group. Then, in the 8×8 square block QM, the quantization coefficients located on the lowest frequency side (the upper side in the example of Figure 14) in each group are selected as the values ​​for the 8×4 rectangular block QM.

[0213] Furthermore, the method of selecting one quantization coefficient from each group as the QM value for the rectangular block is not limited to the example above; other methods can also be used. For example, the quantization coefficient located at the highest frequency side within the group can be used as the QM value for the rectangular block, or the quantization coefficient located in the mid-frequency range can be used as the QM value for the rectangular block. Alternatively, the average, minimum, maximum, or median value of all or a subset of quantization coefficients within the group can be used. Moreover, when the calculation result of this equivalent value produces a decimal point, unconditional rounding, unconditional rounding, or rounding can be used to obtain an integer.

[0214] Furthermore, the method of selecting one quantization coefficient from each group of the square block QM can also be switched according to the frequency domain of each group in the square block QM. For example, in a low-frequency group, the quantization coefficient located at the lowest frequency side of the group can be selected; in a high-frequency group, the quantization coefficient located at the highest frequency side of the group can be selected; and in a mid-frequency group, the quantization coefficient located in the mid-frequency range of the group can be selected.

[0215] Furthermore, instead of deriving QM from the square blocks, the lowest frequency component of the generated rectangular blocks (the top left quantization coefficient in the example of Figure 14) can be recorded in the stream, and the configuration can be directly set from the stream. In this case, since the amount of information recorded in the stream increases, the amount of code in the header area increases. However, since the quantization coefficient of the lowest frequency component of QM, which has the greatest impact on image quality, can be directly controlled, the possibility of improving image quality is increased.

[0216] Furthermore, although this example illustrates how to convert a square block using QM vertically downwards to generate a rectangular block using QM, the same method can be used to convert a square block using QM horizontally downwards to generate a rectangular block using QM. [The second example of the rectangular block generation method used in the first example]

[0217] Figure 15 is a diagram illustrating step S102 in Figure 11 and step S202 in Figure 12, showing a second example of generating a rectangular block using QM from a square block. Furthermore, the process described here is a common process for both the encoding device 100 and the decoding device 200.

[0218] In Figure 15, for square blocks with dimensions ranging from 2×2 to 256×256, the dimensions of the rectangular block QMs generated from the square blocks of each size are correspondingly recorded. In the example shown in Figure 15, the feature is that the length of the longer side of each rectangular block is the same as the length of one side of the corresponding square block. In other words, in this example, the feature is that the size of the processing target block, i.e., the rectangular block, is larger than the size of the square block. That is, the encoding device 100 and decoding device 200 of the first state disclosed herein generate the rectangular block QM by up-converting the square block QM, which has one side of the same length as the shorter side of the processing target block, i.e., the rectangular block.

[0219] Furthermore, Figure 15 shows the correspondence between square block QMs of various block sizes and rectangular block QMs generated from each square block QM, without distinguishing between luminance and chrominance blocks. A suitable correspondence between square block QMs and rectangular block QMs for practical use can also be derived. For example, in the 4:2:0 format, when referencing luminance blocks in the process of generating rectangular block QMs from square block QMs, only QMs corresponding to rectangular blocks with a short side length of 4 or more and a long side length of 256 or less are used in the rectangular block QMs generated from square block QMs. Similarly, when referencing chrominance blocks in the process of generating rectangular block QMs from square block QMs, only QMs corresponding to rectangular blocks with a short side length of 2 or more and a long side length of 128 or less are used in the rectangular block QMs generated from square block QMs. Furthermore, in the case of the 4:4:4 format, since the content is the same as that illustrated in Figure 13, the explanation here is omitted.

[0220] In this way, the correspondence between QM for square blocks and QM for rectangular blocks can be appropriately derived according to the actual format used.

[0221] Furthermore, the block size shown in Figure 15 is an example, and is not limited to this. For example, a square block QM can be used with block sizes other than those shown in Figure 15, or a square block QM can be used with only a portion of the block sizes shown in Figure 15.

[0222] Figure 16 illustrates the method of generating the rectangular block QM shown in Figure 15 from the corresponding square block QM by upconversion.

[0223] In the example of Figure 16, QM is used to generate an 8×4 rectangular block from a 4×4 square block.

[0224] In the upconversion process, (i) the rectangular block is divided into multiple matrix elements of QM into the same number of groups as the square block is divided into multiple matrix elements of QM, and the multiple matrix elements contained in each group are repeated to determine the matrix elements in the rectangular block QM corresponding to that group; or (ii) the multiple matrix elements of QM are determined by performing linear interpolation between adjacent matrix elements in the multiple matrix elements of QM.

[0225] For example, in Figure 16, the multiple matrix elements of the QM for an 8×4 rectangular block are each enclosed by a thick circle a predetermined number of times. These predetermined number of thick-circled matrix elements form a group. In the upconversion process illustrated in Figure 16, the 8×4 rectangular block QM is divided in such a way that the number of these groups is the same as the number of matrix elements (also called quantization coefficients) of the corresponding 4×4 square block QM. In the example of Figure 16, two quantization coefficients adjacent in the left-right direction form a group. Next, for the 8×4 rectangular block QM, the quantization coefficient values ​​corresponding to the square block QM of that group are selected as the quantization coefficients constituting each group, and the group is filled to obtain the 8×4 rectangular block QM value.

[0226] Furthermore, the method for deriving the quantization coefficients within each group of QM for the rectangular block is not limited to the example above; other methods can also be used. For instance, linear interpolation can be performed by referring to the quantization coefficient values ​​in the adjacent frequency domain to make the quantization coefficient values ​​within each group continuous and then derive them. Moreover, when the calculation result of this equivalent value produces a decimal point, unconditional rounding, unconditional rounding, or rounding can be used to make it an integer.

[0227] Furthermore, the method for deriving the quantization coefficients within each group of a rectangular block using QM can be switched according to the frequency domain in which each group of the rectangular block is located in QM. For example, for groups located at low frequencies, the quantization coefficient values ​​within the group can be derived by making the values ​​as small as possible, while for groups located at high frequencies, the quantization coefficient values ​​within the group can be derived by making the values ​​as large as possible.

[0228] Furthermore, instead of deriving QM from the square blocks, the lowest frequency component of the generated rectangular blocks (the top left quantization coefficient in the example of Figure 16) can be recorded in the stream, and the configuration can be directly set from the stream. In this case, since the amount of information recorded in the stream increases, the amount of code in the header area increases. However, since the quantization coefficient of the lowest frequency component of QM, which has the greatest impact on image quality, can be directly controlled, the possibility of improving image quality is increased.

[0229] Furthermore, although this example illustrates how to convert a square block to a horizontal direction using QM to generate a rectangular block using QM, the same method can be used to convert a square block to a vertical direction using QM to generate a rectangular block using QM. [Other variations of the first example using quantization matrix encoding and decoding]

[0230] Alternatively, depending on the size of the generated rectangular block, the method for generating a QM for rectangular blocks can be switched between the first example of the method described in Figures 13 and 14, and the second example of the method for generating a QM for rectangular blocks described in Figures 15 and 16, to generate a QM for rectangular blocks from a QM for square blocks. For example, one method is to compare the ratio of the length and width of the rectangular block (the down-conversion or up-conversion ratio) with a threshold; if it is greater than the threshold, the first example is used; if it is less than the threshold, the second example is used. Another method is to indicate in the streaming description which method of the first and second examples is used for each size of the rectangular block and switch accordingly. Therefore, since the down-conversion and up-conversion processing can be switched according to the size of the rectangular block, a more suitable QM for rectangular blocks can be generated.

[0231] Furthermore, up-conversion and down-conversion can be applied to a single rectangular block combination without switching between up-conversion and down-conversion for each size of the rectangular block. For example, a 32×32 square block can be up-converted horizontally using QM to generate a 64×32 rectangular block using QM. Then, the 64×32 rectangular block can be down-converted vertically using QM to generate a 64×16 rectangular block using QM.

[0232] Alternatively, a rectangular block can be converted upwards in two directions. For example, a 16×16 square block can be converted upwards vertically using QM to generate a 16×32 rectangular block using QM. Then, the 16×32 rectangular block can be converted upwards horizontally using QM to generate a 64×32 rectangular block using QM.

[0233] Furthermore, a rectangular block can also be down-converted in two directions. For example, a 64×64 square block can be down-converted horizontally using QM to generate a 32×64 rectangular block using QM. Then, the 32×64 rectangular block can be down-converted vertically using QM to generate a 32×16 rectangular block using QM. [The effect of the first example of encoding and decoding using quantization matrices]

[0234] According to the encoding apparatus 100 and decoding apparatus 200 of the first state pattern disclosed herein, based on the configuration described using Figures 11 and 12, even in encoding methods with rectangular blocks of various shapes, it is not necessary to record the QM corresponding to each shape of the rectangular block in the stream. Instead, the encoding and decoding processing of the rectangular block is performed by recording only the QM corresponding to the square block in the stream. In other words, according to the encoding apparatus 100 and decoding apparatus 200 of the first state pattern disclosed herein, since the QM corresponding to the rectangular block is not recorded in the stream, the amount of code in the header region can be reduced. Furthermore, according to the encoding apparatus 100 and decoding apparatus 200 of the first state pattern disclosed herein, since the QM corresponding to the square block can be generated, the amount of code in the header region can be reduced without increasing the amount of code, and an appropriate QM can be used for the rectangular block. Therefore, if the encoding device 100 and decoding device 200 according to the first state pattern disclosed herein can efficiently quantize rectangular blocks of various shapes, the possibility of improving encoding efficiency increases.

[0235] For example, encoding device 100 is an encoding device that performs quantization to encode dynamic images, and includes circuitry and memory; the aforementioned circuitry uses the aforementioned memory to transform a first quantization matrix of a plurality of transformation coefficients for a square block, thereby generating a second quantization matrix of a plurality of transformation coefficients for a rectangular block from the aforementioned first quantization matrix, and quantizing the plurality of transformation coefficients of the aforementioned rectangular block using the aforementioned second quantization matrix.

[0236] Therefore, since a quantization matrix corresponding to a rectangular block is generated from a quantization matrix corresponding to a square block, it is possible to omit encoding the quantization matrix corresponding to the rectangular block. Thus, the amount of code is reduced, and processing efficiency is improved. Therefore, according to the encoding device 100, rectangular blocks can be quantized efficiently.

[0237] For example, in the encoding device 100, the aforementioned circuit may also encode only the first quantization matrix in the first quantization matrix and the second quantization matrix into a bit stream.

[0238] This reduces the amount of code. Therefore, based on encoding device 100, processing efficiency is improved.

[0239] For example, in the encoding device 100, the aforementioned circuit can also perform downconversion processing on the aforementioned first quantization matrix of the plurality of conversion coefficients of the aforementioned square block to generate the aforementioned second quantization matrix of the plurality of conversion coefficients of the aforementioned rectangular block, wherein the aforementioned square block has one side of the same length as the long side of the processing object block, i.e., the aforementioned rectangular block.

[0240] In this way, the encoding device 100 can efficiently generate a quantization matrix corresponding to a square block having one side of the same length as the long side of the rectangular block.

[0241] For example, in the encoding device 100, the aforementioned circuit can also, during the aforementioned down-conversion process, divide the plurality of matrix elements of the aforementioned first quantization matrix into groups of the same number as the plurality of matrix elements of the aforementioned second quantization matrix. For each of the plurality of groups, the plurality of matrix elements contained in the group are continuously arranged in the horizontal or vertical direction of the aforementioned square block. For each of the plurality of groups, the matrix element located on the lowest frequency side of the plurality of matrix elements contained in the group, the matrix element located on the highest frequency side of the plurality of matrix elements contained in the group, or the average value of the plurality of matrix elements contained in the group is determined as the matrix element in the aforementioned second quantization matrix corresponding to that group.

[0242] In this way, the encoding device 100 can generate a quantization matrix corresponding to a rectangular block more efficiently from the quantization matrix corresponding to the square block.

[0243] For example, in the encoding device 100, the aforementioned circuit can also perform upconversion processing on the aforementioned first quantization matrix of the plurality of conversion coefficients of the aforementioned square block to generate the aforementioned second quantization matrix of the plurality of conversion coefficients of the aforementioned rectangular block, wherein the aforementioned square block has one side of the same length as the short side of the processing object block, i.e., the aforementioned rectangular block.

[0244] In this way, the encoding device 100 can efficiently generate a quantization matrix corresponding to a square block having one side of the same length as the short side of the rectangular block.

[0245] For example, in the encoding device 100, the aforementioned circuit may also perform the aforementioned upconversion process by (i) dividing the plurality of matrix elements of the aforementioned second quantization matrix into groups of the same number as the plurality of matrix elements of the aforementioned first quantization matrix, and repeating the plurality of matrix elements contained in each of the plurality of groups to determine the matrix elements in the aforementioned second quantization matrix corresponding to that group, or (ii) determining the plurality of matrix elements of the aforementioned second quantization matrix by performing linear interpolation between adjacent matrix elements in the plurality of matrix elements of the aforementioned second quantization matrix.

[0246] In this way, the encoding device 100 can generate a quantization matrix corresponding to a rectangular block more efficiently from the quantization matrix corresponding to the square block.

[0247] For example, in the encoding device 100, the aforementioned circuit may also generate the aforementioned second quantization matrix of a plurality of conversion coefficients for the aforementioned rectangular block by switching the following methods according to the ratio of the short side length to the long side length of the processing object block, i.e., the aforementioned rectangular block. The switching methods include: generating the matrix by performing the aforementioned downconversion process on the aforementioned first quantization matrix of a plurality of conversion coefficients for the aforementioned square block, wherein the aforementioned square block has one side with the same length as the long side of the aforementioned rectangular block; and generating the matrix by performing the aforementioned upconversion process on the aforementioned first quantization matrix of a plurality of conversion coefficients for the aforementioned square block, wherein the aforementioned square block has one side with the same length as the short side of the aforementioned rectangular block.

[0248] In this way, the encoding device 100 can switch between downconversion and upconversion according to the block size of the processing object block, and generate a more appropriate quantization matrix corresponding to a rectangular block from the quantization matrix corresponding to a square block.

[0249] Furthermore, the decoding device 200 is a decoding device that performs inverse quantization to decode dynamic images, and includes circuitry and memory. The aforementioned circuitry uses the aforementioned memory to convert a first quantization matrix for a plurality of conversion coefficients of a square block, thereby generating a second quantization matrix for a plurality of conversion coefficients of a rectangular block from the aforementioned first quantization matrix, and performing inverse quantization on the plurality of quantization coefficients of the aforementioned rectangular block using the aforementioned second quantization matrix.

[0250] Therefore, since a quantization matrix corresponding to a rectangular block is generated from a quantization matrix corresponding to a square block, it is not necessary to decode the quantization matrix corresponding to the rectangular block. Thus, the amount of code can be reduced, and processing efficiency is improved. Therefore, according to the decoding device 200, the rectangular block can be efficiently dequantized.

[0251] For example, in the decoding device 200, the aforementioned circuit may also decode only the first quantization matrix from the bit stream, which is the first quantization matrix among the first and second quantization matrices.

[0252] This allows for a reduction in the amount of code. Therefore, based on the decoding device 200, processing efficiency is improved.

[0253] For example, in the decoding device 200, the aforementioned circuit can also perform downconversion processing on the aforementioned first quantization matrix of the plurality of conversion coefficients of the aforementioned square block to generate the aforementioned second quantization matrix of the plurality of conversion coefficients of the aforementioned rectangular block, wherein the aforementioned square block has one side of the same length as the long side of the processing object block, i.e., the aforementioned rectangular block.

[0254] In this way, the decoding device 200 can efficiently generate a quantization matrix corresponding to a rectangular block from a quantization matrix corresponding to a square block having one side of the same length as the long side of the rectangular block.

[0255] For example, in the decoding device 200, the aforementioned circuit can also, in the aforementioned down-conversion process, divide the plurality of matrix elements of the aforementioned first quantization matrix into the same number of groups as the plurality of matrix elements of the aforementioned second quantization matrix. For each of the plurality of groups, the matrix element located on the lowest frequency side of the plurality of matrix elements contained in the group, the conversion coefficient located on the highest frequency side of the plurality of matrix elements contained in the group, or the average value of the plurality of matrix elements contained in the group, is determined as the matrix element in the aforementioned second quantization matrix corresponding to that group.

[0256] In this way, the decoding device 200 can generate a quantization matrix corresponding to a rectangular block more efficiently from the quantization matrix corresponding to the square block.

[0257] For example, in the decoding device 200, the aforementioned circuit can also perform upconversion processing on the aforementioned first quantization matrix of the plurality of conversion coefficients of the aforementioned square block to generate the aforementioned second quantization matrix of the plurality of conversion coefficients of the aforementioned rectangular block, wherein the aforementioned square block has one side of the same length as the short side of the processing object block, i.e., the aforementioned rectangular block.

[0258] In this way, the decoding device 200 can efficiently generate a quantization matrix corresponding to a rectangular block from a quantization matrix corresponding to a square block having one side of the same length as the short side of the rectangular block.

[0259] For example, in the decoding device 200, the aforementioned circuit can also perform the aforementioned upconversion process by (i) dividing the plurality of matrix elements of the aforementioned second quantization matrix into groups of the same number as the plurality of matrix elements of the aforementioned first quantization matrix, and repeating the plurality of matrix elements contained in each of the plurality of groups to determine the matrix elements in the aforementioned second quantization matrix corresponding to that group, or (ii) determining the plurality of matrix elements of the aforementioned second quantization matrix by performing linear interpolation between adjacent matrix elements in the plurality of matrix elements of the aforementioned second quantization matrix.

[0260] In this way, the decoding device 200 can generate a quantization matrix corresponding to a rectangular block more efficiently from the quantization matrix corresponding to the square block.

[0261] For example, in the decoding device 200, the aforementioned circuit may also generate the aforementioned second quantization matrix of the conversion coefficients for the aforementioned rectangle by switching the following methods according to the ratio of the short side length to the long side length of the processing object block, i.e., the aforementioned rectangular block. The switching methods include: generating the matrix by performing the aforementioned downconversion process on the aforementioned first quantization matrix of the plurality of conversion coefficients for the aforementioned square block, wherein the aforementioned square block has one side with the same length as the long side of the aforementioned rectangular block; and generating the matrix by performing the aforementioned upconversion process on the aforementioned first quantization matrix of the plurality of conversion coefficients for the aforementioned square block, wherein the aforementioned square block has one side with the same length as the short side of the aforementioned rectangular block.

[0262] In this way, the decoding device 200 can switch between downconversion and upconversion according to the block size of the processing target block, and generate a more appropriate quantization matrix corresponding to a rectangular block from the quantization matrix corresponding to a square block.

[0263] Furthermore, the encoding method is a method of encoding dynamic images by performing quantization. It transforms the first quantization matrix of the complex transformation coefficients for square blocks, thereby generating the second quantization matrix of the complex transformation coefficients for rectangular blocks from the aforementioned first quantization matrix, and quantizing the complex transformation coefficients of the aforementioned rectangular blocks using the aforementioned second quantization matrix.

[0264] Therefore, since a quantization matrix corresponding to a rectangular block is generated from a quantization matrix corresponding to a square block, it is possible to omit encoding the quantization matrix corresponding to the rectangular block. This reduces the amount of code and improves processing efficiency. Thus, based on the encoding method, rectangular blocks can be quantized efficiently.

[0265] Furthermore, the decoding method is a decoding method for decoding dynamic images by performing inverse quantization. It transforms the first quantization matrix of the complex transformation coefficients for square blocks, thereby generating a second quantization matrix of the complex transformation coefficients for rectangular blocks from the aforementioned first quantization matrix, and inverse quantization is performed on the complex quantization coefficients of the aforementioned rectangular blocks using the aforementioned second quantization coefficients.

[0266] Therefore, since a quantization matrix corresponding to a rectangular block is generated from the quantization matrix corresponding to the square block, it is not necessary to decode the quantization matrix corresponding to the rectangular block. Thus, the amount of code can be reduced, improving processing efficiency. Therefore, based on the decoding method, the rectangular block can be efficiently dequantized.

[0267] This sample may also be implemented in combination with at least a portion of other samples disclosed herein. Furthermore, a portion of the processing described in the flowchart of this sample, a portion of the device configuration, a portion of the syntax, etc., may also be implemented in combination with other samples. [Second State Sample]

[0268] The following describes the encoding device 100, decoding device 200, encoding method, and decoding method of the second state pattern disclosed herein. [Example 2 of encoding and decoding using quantization matrices]

[0269] Figure 17 is a diagram illustrating a second example of the encoding process using a quantization matrix (QM) in the encoding device 100. Furthermore, the encoding device 100 described here performs encoding processing on each square or rectangular block after the screen is divided.

[0270] First, in step S701, the quantization unit 108 generates the QM of the effective conversion coefficient region corresponding to the size of each block size of the square block and the rectangular block. In other words, the quantization unit 108 only quantizes the multiple conversion coefficients in the low-frequency region of the multiple conversion coefficients contained in the processing target block using the quantization matrix.

[0271] The entropy encoding unit 110 records the signal corresponding to the effective conversion coefficient region generated in step S701 in the stream. In other words, the entropy encoding unit 110 encodes the signal of the quantization matrix corresponding only to a plurality of conversion coefficients within a specified range on the low-frequency side in the bit stream. Furthermore, the quantization unit 108 can generate the QM corresponding to the effective conversion coefficient region from the value defined and set by the user in the encoding device 100, or adaptively generate the QM corresponding to the effective conversion coefficient region using the encoding information of the already encoded image. Also, the QM corresponding to the effective conversion coefficient region can be encoded in the sequence header region, image header region, slice header region, auxiliary information region, or region storing other parameters in the stream. Furthermore, it is also possible not to record the QM corresponding to the effective conversion coefficient region in the stream. In this case, the quantization unit 108 can also use a preset value predefined in the specification as the QM value corresponding to the effective conversion coefficient region.

[0272] Furthermore, the processing in step S701 can be performed simultaneously at the beginning of sequence processing, the beginning of image processing, or the beginning of slice processing, or it can be performed in blocks at a time during block unit processing. Also, the QM generated in step S701 can be configured as follows: for blocks of the same size, multiple types of QMs are generated based on conditions such as luminance blocks / chrominance blocks, intra-frame prediction blocks / inter-frame prediction blocks, and other factors.

[0273] Furthermore, in the processing flow shown in Figure 17, the processing steps other than step S701 are loop processing of block units, which is the same as the processing of the first example described using Figure 11.

[0274] Therefore, when the block size of the processing target block only includes a portion of the low-frequency region among the multiple conversion coefficients contained in the processing target block as the region containing the effective conversion coefficients, it is not necessary to redundantly record the QM signals of the invalid regions in the stream for encoding processing. Thus, since the amount of code in the header region can be reduced, the possibility of improving encoding efficiency increases.

[0275] Furthermore, this processing flow is just one example; the recorded processing order can also be changed, a portion of the recorded processing can be deleted, or unrecorded processing can be added.

[0276] Figure 18 is a diagram illustrating an example of the decoding process using a quantization matrix (QM) in the decoding apparatus 200 corresponding to the encoding apparatus 100 described in Figure 17. Furthermore, the decoding apparatus 200 described here performs decoding processing on each square or rectangular block after the screen is divided.

[0277] First, in step S801, the entropy decoding unit 202 decodes the signal corresponding to the QM of the effective conversion coefficient region from the stream, and generates the QM corresponding to the effective conversion coefficient region using the decoded signal. The QM corresponding to the effective conversion coefficient region is the QM of the effective conversion coefficient region corresponding to the size of each block of the processing object block. Furthermore, the QM corresponding to the effective conversion coefficient region can also be decoded from the sequence header region, image header region, slice header region, auxiliary information region, or region storing other parameters of the stream. Alternatively, the QM corresponding to the effective conversion coefficient region can be decoded without decoding from the stream. In this case, for example, a preset value predefined in the specification can be used as the QM corresponding to the effective conversion coefficient region.

[0278] Furthermore, the processing in step S801 can be performed simultaneously at the beginning of sequence processing, the beginning of image processing, or the beginning of slice processing, or it can be performed in blocks at a time during block unit processing. Also, the QM generated by the entropy decoding unit 202 in step S801 can be configured to generate multiple types of QMs for blocks of the same size, depending on conditions such as luminance block / chrominance block, intra-frame prediction block / inter-frame prediction block, and others.

[0279] Furthermore, in the processing flow shown in FIG18, the processing flow other than step S801 is a loop processing of block units, which is the same as the processing flow of the first example described using FIG12.

[0280] Therefore, when the block size of the processing target block only includes a portion of the low-frequency region among the multiple conversion coefficients contained in the processing target block as the region containing valid conversion coefficients, decoding can be performed without redundantly recording the QM signals of the invalid regions in the stream. Thus, by reducing the amount of code in the header region, the potential for improving coding efficiency increases.

[0281] Furthermore, this processing flow is just one example; the recorded processing order can also be changed, a portion of the recorded processing can be deleted, or unrecorded processing can be added.

[0282] Figure 19 is a diagram illustrating steps S701 in Figure 17 and S801 in Figure 18, showing an example of the size QM of the effective conversion coefficient region for each block size. Furthermore, the process described here is a common process for both the encoding device 100 and the decoding device 200.

[0283] Figure 19(a) shows an example when the processing target block is a square block with a size of 64×64. Only the 32×32 region on the low-frequency side, indicated by the diagonal line, contains the effective conversion coefficients. In the processing target block, the conversion coefficients outside this effective conversion coefficient region are forcibly set to 0, meaning the conversion coefficients are invalid, and therefore no quantization or dequantization processing is required. In other words, the encoding device 100 and decoding device 200 of the second state disclosed herein only generate a 32×32 QM, which corresponds to the 32×32 region on the low-frequency side indicated by the diagonal line in the figure.

[0284] Next, Figure 19(b) shows an example when the block size of the processed object block is a rectangular block of 64×32. Similar to the example in Figure 19(a), the encoding device 100 and the decoding device 200 generate only a 32×32 QM, which corresponds to a 32×32 region on the low-frequency side.

[0285] Next, Figure 19(c) shows an example when the block size of the processed object block is a rectangular block of 64×16. Unlike the example in Figure 19(a), since the vertical block size is only 16, the encoding device 100 and the decoding device 200 only generate a 32×16 QM, which corresponds to the 32×16 region on the low-frequency side.

[0286] Thus, when either the length or width of the block being processed is greater than 32, the conversion coefficients of the region greater than 32 are invalidated, and only the region below 32 is used as the valid conversion coefficient region and as the processing object for quantization and dequantization, in order to generate the quantization coefficients of QM, as well as the encoding and decoding of the QM signal to the stream.

[0287] This eliminates the need to redundantly record the QM signals of invalid regions into the stream for encoding and decoding, thus reducing the amount of code in the header region. Consequently, the potential for improved encoding efficiency increases.

[0288] Furthermore, the size of the effective conversion coefficient area illustrated in Figure 19 is one example; other sizes can also be used. For instance, when processing a luminance block, an area up to 32×32 can be used as the effective conversion coefficient area; when processing a chromatic aberration block, an area up to 16×16 can be used as the effective conversion coefficient area. Also, when the long side of the processing object is 64, an area up to 32×32 can be used as the effective conversion coefficient area; when the long side of the processing object is 128 or 256, an area up to 62×62 can be used as the effective conversion coefficient area.

[0289] Furthermore, the following configuration can also be adopted: using the same processing as described in the first state sample, the coefficients of the quantization matrix corresponding to all frequency components of the square and rectangle are first generated once, and then the QM corresponding only to the effective conversion coefficient region described using Figure 19 is generated. In this case, the signal quantity related to the QM recorded in the stream is no different from that described in the first state sample, but the method described in the first state sample can be directly used to maintain the state of being able to generate all square and rectangle QMs, omitting the quantization processing outside the effective conversion coefficient region. In this way, the possibility of reducing the amount of processing related to quantization is increased. [A variation of the second example of encoding and decoding using quantization matrices]

[0290] Figure 20 is a modified example of the second example of the encoding process using a quantization matrix (QM) in the encoding device 100. Furthermore, the encoding device 100 described here performs encoding processing on each square or rectangular block after the screen is divided.

[0291] In this modified example, the configuration of the second example shown in FIG17 is combined with the configuration of the first example shown in FIG11, and the processing of steps S1001 and S1002 is performed to replace step S701 of FIG17.

[0292] First, in step S1001, the quantization unit 108 generates a QM for the square block. At this time, the QM for the square block is the QM corresponding to the size of the effective conversion coefficient region of the square block. Then, the entropy encoding unit 110 records the signal related to the QM for the square block generated in step S1001 in the stream. At this time, the signal related to the QM recorded in the stream is only the signal related to the quantization coefficients corresponding to the effective conversion coefficient region.

[0293] Next, in step S1002, the quantization unit 108 uses the square block QM generated in step S1001 to generate a rectangular block QM. Furthermore, at this time, the entropy encoding unit 110 does not record the signal related to the rectangular block QM in the streaming stream.

[0294] Furthermore, in the processing flow shown in FIG20, the processing other than steps S1001 and S1002 is a loop processing of block units, which is the same as the processing of the first example described using FIG11.

[0295] Therefore, even in encoding rectangular blocks of various shapes, it is unnecessary to record the QM signals corresponding to each rectangular block shape in the stream. Encoding can be performed by only recording the QM signals corresponding to square blocks. Furthermore, when only a portion of the transformation coefficients within the block containing multiple transformation coefficients is considered as the valid region (i.e., the valid transformation coefficient region), it is unnecessary to redundantly record the QM signals for invalid regions in the stream. Thus, by reducing the amount of code in the header region and using QM for rectangular blocks, the potential for improved encoding efficiency increases.

[0296] Furthermore, this processing flow is just one example; the recorded processing order can also be changed, a portion of the recorded processing can be deleted, or unrecorded processing can be added.

[0297] Figure 21 is a diagram illustrating an example of the decoding process using a quantization matrix (QM) in the decoding apparatus 200 corresponding to the encoding apparatus 100 described in Figure 20. Furthermore, the decoding apparatus 200 described here performs decoding processing on each square or rectangular block after the screen is divided.

[0298] In this modified example, the configuration of the second example shown in FIG18 is combined with the configuration of the first example shown in FIG12, and the processing of steps S1101 and S1102 is performed to replace step S801 in FIG18.

[0299] First, in step S1101, the entropy decoding unit 202 decodes the signal related to the square block QM from the stream, and generates the square block QM using the decoded signal. At this time, the signal related to the square block QM decoded from the stream is only related to the quantization coefficients corresponding to the effective conversion coefficient region. Therefore, the square block QM generated by the entropy decoding unit 202 is the QM corresponding to the size of the effective conversion coefficient region.

[0300] Next, in step S1102, the entropy decoding unit 202 uses the square block QM generated in step S1101 to generate a rectangular block QM. Furthermore, at this time, the entropy decoding unit 202 does not decode the signal related to the rectangular block QM from the stream.

[0301] Furthermore, in the processing flow shown in Figure 21, the processing other than steps S1101 and S1102 is a loop processing of block units, which is the same as the processing of the first example described using Figure 12.

[0302] Therefore, even in encoding rectangular blocks of various shapes, it is unnecessary to record the QM signals corresponding to each rectangular block shape in the stream. Decoding can be performed by only recording the QM signals corresponding to square blocks. Furthermore, when only a portion of the transformation coefficients within the block containing multiple transformation coefficients is considered as the valid region (i.e., the valid transformation coefficient region) for the block size of the target block, decoding can be performed without redundantly recording the QM signals for invalid regions in the stream. Thus, since the amount of code in the header region can be reduced while using QM for rectangular blocks, the possibility of improving encoding efficiency is increased.

[0303] Furthermore, this processing flow is just one example; the recorded processing order can also be changed, a portion of the recorded processing can be deleted, or unrecorded processing can be added. [The second variation of the rectangular block uses the first example of the QM generation method]

[0304] Figure 22 is a diagram illustrating step S1002 in Figure 20 and step S1102 in Figure 21, showing a first example of generating a rectangular block from a square block using a QM. Furthermore, the process described here is a common process for both the encoding device 100 and the decoding device 200.

[0305] In Figure 22, for square blocks with dimensions ranging from 2×2 to 256×256, the dimensions of the rectangular blocks generated by QM from the square blocks of each size are correspondingly recorded. In the example shown in Figure 22, the dimensions of the processing target block and the dimensions of the effective conversion coefficient region within the processing target block are displayed. The values ​​recorded in parentheses represent the dimensions of the effective conversion coefficient region within the processing target block. Furthermore, for rectangular blocks whose dimensions are equal to the dimensions of the processing target block and the effective conversion coefficient region within the processing target block, since the processing is the same as in the first example illustrated in Figure 13, their values ​​are omitted from the correspondence table shown in Figure 22.

[0306] Herein lies the characteristic that the length of the long side of each rectangular block is the same as the length of one side of the corresponding square block, and the rectangular block is smaller than the square block. In other words, the rectangular block is generated by down-converting the square block using QM.

[0307] Furthermore, Figure 22 shows the correspondence between square block QMs of various block sizes, without distinguishing between luminance and chrominance blocks, and rectangular block QMs generated from each square block QM. A suitable correspondence between square block QMs and rectangular block QMs for practical use can also be derived. For example, in the 4:2:0 format, the luminance block is twice the size of the chrominance block. Therefore, when referencing the luminance block in the process of generating rectangular block QMs from square block QMs, the usable square block QMs correspond to square blocks of sizes from 4×4 to 256×256. In this case, among the rectangular block QMs generated from square block QMs, only QMs corresponding to rectangular blocks with a short side length of 4 or more and a long side length of 256 or less are used. Furthermore, when referencing color difference blocks in the process of generating a rectangular block QM from a square block QM, only the QM corresponding to a rectangular block with a short side length of 2 or more and a long side length of 128 or less is used in the rectangular block QM generated from the square block QM. Moreover, regarding the 4:4:4 format, the same applies as illustrated in Figure 13.

[0308] In this way, the correspondence between QM for square blocks and QM for rectangular blocks can be appropriately derived according to the actual format used.

[0309] Furthermore, the size of the effective conversion coefficient region shown in Figure 22 is one example, and other sizes of the effective conversion coefficient region other than those shown in Figure 22 can also be used.

[0310] Furthermore, the block size shown in Figure 22 is an example, and is not limited to this. For example, block sizes other than those shown in Figure 22 can be used, or only a portion of the block sizes shown in Figure 22 can be used.

[0311] Figure 23 illustrates the method of generating the rectangular block QM illustrated in Figure 22 from the corresponding square block QM by down-conversion.

[0312] In the example of Figure 23, the effective conversion coefficient region QM corresponding to a rectangular block of 64×32 is generated from the effective conversion coefficient region QM of 32×32 in the square block corresponding to 64×64.

[0313] First, as shown in Figure 23(a), a 64×64 square block QM with a 32×64 effective area is generated by extending the quantization coefficients of the QM corresponding to the 32×32 effective conversion coefficient area in a vertical direction. The method of extending the slope mentioned above includes, for example, extending so that the difference between the quantization coefficients of the 31st column and the quantization coefficients of the 32nd column becomes the difference between its subsequent adjacent coefficients; or deriving the change in the difference between the quantization coefficients of the 30th column and the quantization coefficients of the 31st column, and the difference between the quantization coefficients of the 31st column and the quantization coefficients of the 32nd column, and extending the difference between its subsequent adjacent quantization coefficients while correcting with the change mentioned above.

[0314] Next, as shown in Figure 23(b), using the same method as described in Figure 14, the 64×64 square block in the middle of the 32×64 effective area is down-converted using QM to generate a 64×32 rectangular block using QM. In this case, the effective area is the 32×32 region shown by the diagonal line in the 64×32 rectangular block QM of Figure 23.

[0315] Furthermore, although this example illustrates how to convert a square block with a valid area using QM vertically downwards to generate a rectangular block using QM, the same method as in the example of Figure 23 can also be used when converting a square block with a valid area using QM horizontally downwards to generate a rectangular block using QM.

[0316] Furthermore, although this example illustrates the generation of a rectangular block QM using a two-stage process via a central square block QM, it is also possible to generate a rectangular block QM directly from a square block QM with an effective area without using a central square block QM, by using a conversion formula that can derive the same processing result as in the example of Figure 23. [The second example of the rectangular block generated using the QM method]

[0317] Figure 24 is a diagram illustrating a second example of generating a rectangular block from a square block using QM, as shown in steps S1002 of Figure 20 and S1102 of Figure 21. Furthermore, the process described here is a common process for both the encoding device 100 and the decoding device 200.

[0318] In Figure 24, for square blocks with dimensions ranging from 2×2 to 256×256, the dimensions of the rectangular blocks generated by QM from the square blocks of each size are correspondingly recorded. In the example shown in Figure 24, the dimensions of the processing target block and the dimensions of the effective conversion coefficient region within the processing target block are displayed. The values ​​recorded in parentheses represent the dimensions of the effective conversion coefficient region within the processing target block. Furthermore, for rectangular blocks where the dimensions of the processing target block and the effective conversion coefficient region are equal, since the processing is the same as in the first example described in Figure 15, the values ​​are omitted from the correspondence table shown in Figure 24.

[0319] Herein lies the characteristic that the shorter side length of each rectangular block is the same as the shorter side length of the corresponding square block, and the rectangular block is larger than the square block. In other words, the rectangular block is generated by upconverting the square block using QM.

[0320] Furthermore, Figure 24 shows the correspondence between square block QMs of various block sizes, without distinguishing between luminance and chrominance blocks, and rectangular block QMs generated from each square block QM. A suitable correspondence between square block QMs and rectangular block QMs for practical use can also be derived. For example, in the 4:2:0 format, the luminance block is twice the size of the chrominance block. Therefore, when referencing the luminance block in the process of generating rectangular block QMs from square block QMs, only QMs corresponding to rectangular blocks with a short side length of 4 or more and a long side length of 256 or less are used in the rectangular block QMs generated from the square block QMs. Furthermore, when referencing color difference blocks in the process of generating a rectangular block QM from a square block QM, only the QM corresponding to a rectangular block with a short side length of 2 or more and a long side length of 128 or less is used in the rectangular block QM generated from the square block QM. Moreover, regarding the 4:4:4 format, the same applies as illustrated in Figure 13.

[0321] In this way, the correspondence between QM for square blocks and QM for rectangular blocks can be appropriately derived according to the actual format used.

[0322] Furthermore, the size of the effective conversion coefficient region shown in Figure 24 is one example; other sizes of the effective conversion coefficient region besides those shown in Figure 24 can also be used.

[0323] Furthermore, the block size shown in Figure 24 is an example, and is not limited to this. For example, block sizes other than those shown in Figure 24 can be used, or only a portion of the block sizes shown in Figure 24 can be used.

[0324] Figure 25 illustrates the method of generating the rectangular block QM illustrated in Figure 24 from the corresponding square block QM by upconversion.

[0325] In the example of Figure 25, the effective conversion coefficient region QM corresponding to a rectangular block of 64×32 is generated from the effective conversion coefficient region QM of the 32×32 effective conversion coefficient region in the square block corresponding to 32×32.

[0326] First, as shown in Figure 25(a), using the same method as described in Figure 16, the 32×32 square block is upconverted using QM to generate the central 64×32 rectangular block using QM. At this time, the effective area is also upconverted to 64×32.

[0327] Next, as shown in Figure 25(b), a 64×32 rectangular block with an effective area of ​​32×32 is generated by cropping only the 32×32 portion of the low-frequency side of the 64×32 effective area.

[0328] Furthermore, although this example illustrates how to convert a square block to a horizontal direction using QM to generate a rectangular block using QM, the same method can be used to convert a square block to a vertical direction using QM to generate a rectangular block using QM.

[0329] Furthermore, although this example illustrates generating a rectangular block QM in two stages via the intermediate rectangular block QM, it is also possible to generate a rectangular block QM directly from a square block QM without going through the intermediate rectangular block QM, using a conversion formula that can derive the same processing result as the example in Figure 25. [Other variations of the second example of encoding and decoding using quantization matrices]

[0330] Alternatively, depending on the size of the generated rectangular block, the method for generating a QM for rectangular blocks can be switched between the first example of the method described in Figures 22 and 23, and the second example of the method for generating a QM for rectangular blocks described in Figures 24 and 25, to generate a QM for rectangular blocks from a QM for square blocks. For example, one method is to compare the ratio of the length and width of the rectangular block (the down-conversion or up-conversion ratio) with a threshold; if it is greater than the threshold, the first example is used; if it is less than the threshold, the second example is used. Another method is to indicate in the streaming description which method of the first and second examples is used for each size of the rectangular block and switch accordingly. Therefore, since the down-conversion and up-conversion processing can be switched according to the size of the rectangular block, a more suitable QM for rectangular blocks can be generated. [The effect of the second example and its variation on the encoding and decoding of quantization matrices]

[0331] If the encoding apparatus 100 and decoding apparatus 200 according to the second state pattern disclosed herein, by utilizing the configurations illustrated in Figures 17 and 18, only the region containing a portion of the multiple conversion coefficients within the processing target block is considered as the valid region for the block size. Therefore, it is unnecessary to redundantly record the QM signals of the invalid regions in the stream, and encoding and decoding of rectangular blocks can be performed. Thus, since the amount of code in the header region can be reduced, the possibility of improving encoding efficiency increases.

[0332] Furthermore, according to the second-state variation of the encoding apparatus 100 and decoding apparatus 200 of this disclosure, by utilizing the configuration described in Figures 20 and 21, even in encoding methods with rectangular blocks of various shapes, it is not necessary to record the QM corresponding to each shape of the rectangular block in the stream. Instead, the encoding and decoding of the rectangular blocks can be performed by recording only the QM corresponding to the square block in the stream. In other words, according to the second-state variation of the encoding apparatus 100 and decoding apparatus 200 of this disclosure, since the QM corresponding to the square block can be generated, the amount of code in the header region can be reduced, and an appropriate QM can also be used for the rectangular block. Therefore, according to the second-state variation of the encoding apparatus 100 and decoding apparatus 200 of this disclosure, since the rectangular blocks of various shapes can be efficiently quantized, the possibility of improving encoding efficiency is increased.

[0333] For example, the encoding device 100 is an encoding device that performs quantization to encode a moving image, and includes circuitry and memory; the aforementioned circuitry uses the aforementioned memory to quantize only a plurality of conversion coefficients within a specified range on the low-frequency region side of the plurality of conversion coefficients contained in the processing object block using a quantization matrix.

[0334] Therefore, since the processing target block is quantized using only a quantization matrix corresponding to a specified range of low-frequency regions that have a significant visual impact, the image quality of the moving image is less likely to degrade. Furthermore, since only the quantization matrix corresponding to this specified range is encoded, the amount of code is reduced. Therefore, according to the encoding device 100, the image quality of the moving image is less likely to degrade, and processing efficiency is improved.

[0335] For example, in the encoding device 100, the aforementioned circuit may also encode only the signal of the aforementioned quantization matrix corresponding to the aforementioned multiple conversion coefficients within the aforementioned specified range of the aforementioned low-frequency region into a bit stream.

[0336] This reduces the amount of code. Therefore, based on encoding device 100, processing efficiency is improved.

[0337] For example, in the encoding device 100, the aforementioned processing target block can also be a square block or a rectangular block. The aforementioned circuit can also generate a second quantization matrix of a plurality of conversion coefficients within the aforementioned specified range for the aforementioned square block by converting the first quantization matrix of a plurality of conversion coefficients within the aforementioned specified range for the aforementioned rectangular block, and use it as the aforementioned quantization matrix. Only the aforementioned first quantization matrix among the aforementioned first quantization matrix and the aforementioned second quantization matrix is ​​encoded as a signal related to the aforementioned quantization matrix in the bit stream.

[0338] Therefore, since the encoding device 100 generates a first quantization matrix corresponding to a predetermined range of the low-frequency region side of the square block for processing object blocks containing many shapes including rectangular blocks, and only encodes the first quantization matrix, the amount of code is reduced. Furthermore, since the encoding device 100 generates a second quantization matrix corresponding to a predetermined range of the low-frequency region side of the rectangular block from the first quantization matrix, the image quality of the moving image is less likely to degrade. Therefore, according to the encoding device 100, the image quality of the moving image is less likely to degrade, and the processing efficiency is improved.

[0339] For example, in the encoding device 100, the aforementioned circuit can also perform down-conversion processing on the aforementioned first quantization matrix of a plurality of conversion coefficients within a predetermined range for the aforementioned square block to generate the aforementioned second quantization matrix of a plurality of conversion coefficients within the aforementioned predetermined range for the aforementioned rectangular block, wherein the aforementioned square block has one side of the same length as the long side of the aforementioned processing object block, i.e., the aforementioned rectangular block.

[0340] In this way, the encoding device 100 can efficiently quantize rectangular blocks.

[0341] For example, in the encoding device 100, the aforementioned circuit can also extend the plurality of matrix elements of the aforementioned first quantization matrix outward in a predetermined direction during the aforementioned down-conversion process, and divide the extended plurality of matrix elements of the aforementioned first quantization matrix into a number of groups equal to the number of matrix elements of the aforementioned second quantization matrix. For each of the plurality of groups, the matrix element located on the lowest frequency side of the plurality of matrix elements contained in the group, the matrix element located on the highest frequency side of the plurality of matrix elements contained in the group, or the average value of the plurality of matrix elements contained in the group is determined as the matrix element in the aforementioned second quantization matrix corresponding to that group.

[0342] In this way, the encoding device 100 can quantize rectangular blocks more efficiently.

[0343] For example, in the encoding device 100, the aforementioned circuit can also perform upconversion processing on the first quantization matrix of the plurality of conversion coefficients within the aforementioned specified range for the aforementioned square block to generate the aforementioned second quantization matrix of the plurality of conversion coefficients within the aforementioned specified range for the aforementioned rectangular block, wherein the aforementioned square block has one side of the same length as the short side of the aforementioned processing object block, i.e., the aforementioned rectangular block.

[0344] In this way, the encoding device 100 can efficiently quantize rectangular blocks.

[0345] For example, in the encoding device 100, the aforementioned circuit can also perform the aforementioned upconversion process, (i) dividing the plurality of matrix elements of the aforementioned second quantization matrix into groups of the same number as the plurality of matrix elements of the aforementioned first quantization matrix, and repeating the plurality of matrix elements contained in each of the plurality of groups in a predetermined direction, thereby extending the aforementioned first quantization matrix in the predetermined direction, and extracting the same number of matrix elements as the plurality of matrix elements of the aforementioned second quantization matrix from the extended aforementioned first quantization matrix, or (ii) extending the aforementioned first quantization matrix in the predetermined direction by performing linear interpolation between the matrix elements adjacent to the matrix elements in the plurality of matrix elements of the aforementioned second quantization matrix, and extracting the same number of matrix elements as the plurality of matrix elements of the aforementioned second quantization matrix from the extended aforementioned first quantization matrix.

[0346] In this way, the encoding device 100 can quantize rectangular blocks more efficiently.

[0347] For example, in the encoding device 100, the aforementioned circuit may also generate the aforementioned second quantization matrix for a plurality of conversion coefficients within the aforementioned specified range of the aforementioned rectangular block by switching the following methods in accordance with the ratio of the short side length to the long side length of the aforementioned processing object block, i.e., the aforementioned rectangular block. The switching methods include: generating the matrix by performing the aforementioned downconversion process on the aforementioned first quantization matrix for a plurality of conversion coefficients within the aforementioned specified range of the aforementioned square block, wherein the aforementioned square block has one side with the same length as the long side of the aforementioned rectangular block; and generating the matrix by performing the aforementioned upconversion process on the aforementioned first quantization matrix for a plurality of conversion coefficients within the aforementioned specified range of the aforementioned square block, wherein the aforementioned square block has one side with the same length as the short side of the aforementioned rectangular block.

[0348] In this way, the encoding device 100 can switch between downconversion and upconversion according to the block size of the processing object block, and generate a more appropriate quantization matrix corresponding to a rectangular block from the quantization matrix corresponding to a square block.

[0349] Furthermore, the decoding device 200 is a decoding device that performs inverse quantization to decode dynamic images, and it includes circuitry and memory. The aforementioned circuitry utilizes the aforementioned memory to perform inverse quantization using a quantization matrix only for a plurality of quantization coefficients within a specified range on the low-frequency region side of the plurality of quantization coefficients contained in the processing object block.

[0350] Therefore, since the processing target block is quantized using only a quantization matrix corresponding to a specified range of low-frequency regions that have a significant visual impact, the image quality of the moving image is less likely to degrade. Furthermore, since only the quantization matrix corresponding to this specified range is encoded, the amount of code can be reduced. Therefore, according to the decoding device 200, the image quality of the moving image is less likely to degrade, and processing efficiency is improved.

[0351] For example, in the decoding device 200, the aforementioned circuit may also encode only the signal of the aforementioned quantization matrix corresponding to the aforementioned multiple conversion coefficients within the aforementioned specified range of the aforementioned low-frequency region into a bit stream.

[0352] This allows for a reduction in the amount of code. Therefore, based on the decoding device 200, processing efficiency is improved.

[0353] For example, in the decoding device 200, the aforementioned processing target block can also be a square block or a rectangular block. The aforementioned circuit can also generate a second quantization matrix of multiple conversion coefficients within the aforementioned specified range for the aforementioned rectangular block from the first quantization matrix by converting the first quantization matrix of multiple conversion coefficients within the aforementioned specified range for the aforementioned square block, and use it as the aforementioned quantization matrix. Only the first quantization matrix in the aforementioned first quantization matrix and the aforementioned second quantization matrix is ​​decoded from the bit stream to serve as the signal related to the aforementioned quantization matrix.

[0354] Therefore, since the decoding device 200 generates a first quantization matrix corresponding to a predetermined range of the low-frequency region side of the square block for processing object blocks containing many shapes including rectangular blocks, and only encodes the first quantization matrix, the amount of code can be reduced. Furthermore, since the decoding device 200 generates a second quantization matrix corresponding to a predetermined range of the low-frequency region side of the rectangular block from the first quantization matrix, the image quality of the moving image is less likely to degrade. Therefore, according to the decoding device 200, the image quality of the moving image is less likely to degrade, and the processing efficiency is improved.

[0355] For example, in the decoding device 200, the aforementioned circuit can also perform down-conversion processing on the aforementioned first quantization matrix of a plurality of conversion coefficients within a specified range for the aforementioned square block to generate the aforementioned second quantization matrix of a plurality of conversion coefficients within the specified range for the aforementioned rectangular block, wherein the aforementioned square block has one side of the same length as the long side of the aforementioned processing object block, i.e., the aforementioned rectangular block.

[0356] In this way, the decoding device 200 can efficiently dequantize the rectangular block.

[0357] For example, in the decoding device 200, the aforementioned circuit can also extend the plurality of matrix elements of the aforementioned first quantization matrix outward in a predetermined direction during the aforementioned down-conversion process, and divide the extended plurality of matrix elements of the aforementioned first quantization matrix into a number of groups equal to the number of matrix elements of the aforementioned second quantization matrix. For each of the plurality of groups, the matrix element located on the lowest frequency side of the plurality of matrix elements contained in the group, the matrix element located on the highest frequency side of the plurality of matrix elements contained in the group, or the average value of the plurality of matrix elements contained in the group is determined as the matrix element in the aforementioned second quantization matrix corresponding to that group.

[0358] In this way, the decoding device 200 can perform dequantization more efficiently on rectangular blocks.

[0359] For example, in the decoding device 200, the aforementioned circuit can also perform upconversion processing on the aforementioned first quantization matrix of the plurality of conversion coefficients within the aforementioned specified range for the aforementioned square block to generate the aforementioned second quantization matrix of the plurality of conversion coefficients within the aforementioned specified range for the aforementioned rectangular block, wherein the aforementioned square block has one side of the same length as the short side of the aforementioned processing object block, i.e., the aforementioned rectangular block.

[0360] In this way, the decoding device 200 can efficiently dequantize the rectangular block.

[0361] For example, in the decoding device 200, the aforementioned circuit can also perform the aforementioned upconversion process, (i) dividing the plurality of matrix elements of the aforementioned second quantization matrix into groups of the same number as the plurality of matrix elements of the aforementioned first quantization matrix, and repeating the plurality of matrix elements contained in each of the plurality of groups in a predetermined direction, thereby extending the aforementioned first quantization matrix in the predetermined direction, and extracting the same number of matrix elements as the plurality of matrix elements of the aforementioned second quantization matrix from the extended aforementioned first quantization matrix, or (ii) extending the aforementioned first quantization matrix in the predetermined direction by performing linear interpolation between the matrix elements adjacent to the matrix elements in the plurality of matrix elements of the aforementioned second quantization matrix, and extracting the same number of matrix elements as the plurality of matrix elements of the aforementioned second quantization matrix from the extended aforementioned first quantization matrix.

[0362] In this way, the decoding device 200 can perform dequantization more efficiently on rectangular blocks.

[0363] For example, in the decoding device 200, the aforementioned circuit may also generate the aforementioned second quantization matrix for a plurality of conversion coefficients within the aforementioned specified range of the aforementioned rectangular block by switching the following methods in accordance with the ratio of the short side length to the long side length of the aforementioned processing object block, i.e., the aforementioned rectangular block. The switching methods include: generating the matrix by performing the aforementioned downconversion process on the aforementioned first quantization matrix for a plurality of conversion coefficients within the aforementioned specified range of the aforementioned square block, wherein the aforementioned square block has one side with the same length as the long side of the aforementioned rectangular block; and generating the matrix by performing the aforementioned upconversion process on the aforementioned first quantization matrix for a plurality of conversion coefficients within the aforementioned specified range of the aforementioned square block, wherein the aforementioned square block has one side with the same length as the short side of the aforementioned rectangular block.

[0364] In this way, the decoding device 200 can switch between downconversion and upconversion according to the block size of the processing target block, and generate a more appropriate quantization matrix corresponding to a rectangular block from the quantization matrix corresponding to a square block.

[0365] Furthermore, the encoding method is a method of encoding dynamic images by quantization. It uses a quantization matrix to quantize only a number of conversion coefficients within a specified range of the low-frequency region among the number of conversion coefficients contained in the processing object block.

[0366] Therefore, since the processing block is quantized using only a quantization matrix covering a specified range corresponding to the low-frequency regions within the processing block that have a significant visual impact, the image quality of moving images is less likely to degrade. Furthermore, since only the quantization matrix corresponding to this specified range is encoded, the amount of code is reduced. Thus, based on this encoding method, the image quality of moving images is less likely to degrade, and processing efficiency is improved.

[0367] Furthermore, the decoding method is a decoding method for decoding dynamic images by performing inverse quantization. It only performs inverse quantization on a specified range of multiple conversion coefficients in the low-frequency region of the multiple conversion coefficients contained in the processing object block using a quantization matrix.

[0368] Therefore, since the processing block is quantized using only a quantization matrix covering a specified range corresponding to the low-frequency region within the processing block that has a significant visual impact, the image quality of moving images is less likely to degrade. Furthermore, since only the quantization matrix corresponding to this specified range is encoded, the amount of code can be reduced. Thus, based on the decoding method, the image quality of moving images is less likely to degrade, and processing efficiency is improved.

[0369] This sample may also be implemented in combination with at least a portion of other samples disclosed herein. Furthermore, a portion of the processing described in the flowchart of this sample, a portion of the device configuration, a portion of the syntax, etc., may also be implemented in combination with other samples. [Third-state sample]

[0370] The following describes the encoding device 100, decoding device 200, encoding method, and decoding method of the third state pattern disclosed herein. [Example 3 of encoding and decoding using quantization matrices]

[0371] Figure 26 is a diagram illustrating a third example of the encoding process using a quantization matrix (QM) in the encoding device 100. Furthermore, the encoding device 100 described here performs encoding processing on each square or rectangular block after the screen is divided.

[0372] First, in step S1601, the quantization unit 108 generates a QM (hereinafter also referred to as the diagonal component QM) corresponding to the diagonal component of the processing target block. For each block size of the processing target block, i.e., a square block or a rectangular block of various shapes, the quantization coefficient values ​​of the diagonal component QM are used to generate the QM corresponding to the processing target block using the common method described below. In other words, the quantization unit 108 generates a quantization matrix for the processing target block from the diagonal component of the quantization matrix for a plurality of transformation coefficients continuously arranged in the diagonal direction of the processing target block among the plurality of transformation coefficients contained in the processing target block. Furthermore, using the common method means that the common method is used for all processing target blocks regardless of the shape and size of the block. Also, the diagonal component refers to, for example, a plurality of coefficients along the diagonal from the low-frequency side to the high-frequency side of the processing target block.

[0373] The entropy encoding unit 110 records the signal of the diagonal component of the QM generated in step S1601 in the stream. In other words, the entropy encoding unit 110 encodes the signal of the diagonal component of the quantization matrix in the bit stream.

[0374] Furthermore, the quantization unit 108 can generate quantization coefficient values ​​for the diagonal component QM from values ​​defined and set by the user in the encoding device 100, or adaptively generate quantization coefficient values ​​for the diagonal component QM using the encoding information of already encoded images. Also, the diagonal component QM can be encoded in the sequence header area, image header area, slice header area, auxiliary information area, or area storing other parameters of the stream. Furthermore, the diagonal component QM may not be recorded in the stream. In this case, the quantization unit 108 can also use preset values ​​predefined in the specification as the diagonal component QM.

[0375] Furthermore, the processing in step S1601 can be performed simultaneously at the beginning of sequence processing, the beginning of image processing, or the beginning of slice processing, or it can be performed in blocks at a time during block unit processing. Also, the QM generated in step S1601 can be configured as follows: for blocks of the same size, multiple types of QMs are generated based on conditions such as luminance blocks / chrominance blocks, intra-frame prediction blocks / inter-frame prediction blocks, and other factors.

[0376] Furthermore, in the processing flow shown in Figure 26, the processing steps other than step S1601 are loop processing of block units, which is the same as the processing of the first example described using Figure 11.

[0377] Therefore, by recording only the quantization coefficients of the QM for the diagonal components in the stream, the encoding of the target block can be performed, without having to record all the quantization coefficients of the QM for the target blocks of each block size in the stream. Thus, even in encoding methods using blocks of many shapes containing rectangular blocks, the amount of code in the header region can be generated and used to correspond to the QM of the target block without significantly increasing the amount of code, thereby increasing the potential for improved encoding efficiency.

[0378] Furthermore, this processing flow is just one example; the recorded processing order can also be changed, a portion of the recorded processing can be deleted, or unrecorded processing can be added.

[0379] Figure 27 is a diagram illustrating an example of the decoding process using a quantization matrix (QM) in the decoding apparatus 200 corresponding to the encoding apparatus 100 described in Figure 26. Furthermore, the decoding apparatus 200 described here performs decoding processing on each square or rectangular block after the screen is divided.

[0380] First, in step S1701, the entropy decoding unit 202 decodes the signal related to the QM of the diagonal component only from the stream. Using the decoded signal related to the QM of the diagonal component only, and employing the common method described below, it generates the QM for each block size corresponding to various shapes of processing object blocks, such as square blocks and rectangular blocks. Furthermore, the QM of the diagonal component only can also be decoded from the sequence header area, image header area, slice header area, auxiliary information area, or area storing other parameters of the stream. Alternatively, the QM of the diagonal component only may not be decoded from the stream. In this case, for example, a preset value predefined in the specification can be used as the QM of the diagonal component only.

[0381] Furthermore, the processing in step S1701 can be performed simultaneously at the beginning of sequence processing, the beginning of image processing, or the beginning of slice processing, or it can be performed in blocks at a time during block unit processing. Also, the QM generated by the entropy decoding unit 202 in step S1701 can be configured as follows: for blocks of the same size, multiple types of QMs are generated based on conditions such as luminance block / chrominance block, intra-frame prediction block / inter-frame prediction block, and other factors.

[0382] Furthermore, in the processing flow shown in Figure 27, the processing flow other than step S1701 is a loop processing of block units, which is the same as the processing of the first example described using Figure 12.

[0383] Therefore, decoding of the target block can be performed by only recording the quantization coefficients of the QM of the diagonal components in the stream, without having to record the quantization coefficients of the QM of each block size in the stream. This reduces the amount of code in the header region, thus increasing the potential for improved coding efficiency.

[0384] Furthermore, this processing flow is just one example; the recorded processing order can also be changed, a portion of the recorded processing can be deleted, or unrecorded processing can be added.

[0385] Figure 28 is a diagram illustrating an example of a method for generating the QM of the target block from the quantization coefficient values ​​of the QM of only the diagonal components, based on each block size, using the common method described below, in steps S1601 of Figure 26 and S1701 of Figure 27. Furthermore, the process described here is a common process for both the encoding device 100 and the decoding device 200.

[0386] The encoding device 100 and decoding device 200 of the third state generate the quantization matrix (QM) of the processing object block by repeating each element of a plurality of matrix elements of the diagonal component of the processing object block in the horizontal and vertical directions. More specifically, the encoding device 100 and decoding device 200 generate the QM of the processing object block by extending the quantization coefficient values ​​of the QM of the diagonal component directly upwards and to the left, that is, by continuously configuring the same values.

[0387] Furthermore, although this example illustrates the method for generating the QM of the processing object block when the processing object block is a square block, when the processing object block is a rectangular block, the QM of the processing object block can also be generated from the quantization coefficient values ​​of the QM of the diagonal components, just like in the example of Figure 28.

[0388] Figure 29 is a diagram illustrating steps S1601 in Figure 26 and S1701 in Figure 27, which describes another example of a method for generating the QM of the processing target block from the quantization coefficient values ​​of the QM of only the diagonal components, based on the block size of each block of the processing target block, using the common method described below. Furthermore, the processing described here is common to both the encoding device 100 and the decoding device 200.

[0389] The encoding device 100 and decoding device 200 of the third state can also generate the quantization matrix of the processing object block by repeating each element of the multiple matrix elements of the diagonal component of the processing object block diagonally. More specifically, the encoding device 100 and decoding device 200 generate the QM of the processing object block by extending the quantization coefficient values ​​of the QM of the diagonal component directly to the lower left and upper right, that is, by continuously configuring the same value.

[0390] At this time, in addition to the quantization coefficients of the diagonal components, the encoding device 100 and the decoding device 200 can also utilize the quantization coefficients of the nearby components of the diagonal components to generate the QM of the processing target block by repeating each quantization coefficient diagonally. In other words, the quantization matrix (QM) of the processing target block can also be generated from multiple matrix elements of the diagonal components and matrix elements located near the diagonal components. Therefore, when it is difficult to fill all the quantization coefficients of the processing target block with only the diagonal components, all the quantization coefficients can still be filled by using the quantization coefficients of the nearby components. Furthermore, the nearby components of the diagonal components are, for example, any of the multiple coefficients adjacent to the diagonal from the low-frequency side to the high-frequency side of the processing target block.

[0391] For example, the quantization coefficient of the component near the diagonal component is the quantization coefficient at the position shown in Figure 29. The encoding device 100 and the decoding device 200 can set the signal of the quantization coefficient of the component near the diagonal component during streaming encoding or decoding, or they can derive it without streaming encoding or decoding by interpolating from the adjacent quantization coefficient values ​​of the diagonal component using linear interpolation or the like.

[0392] Furthermore, although this example illustrates the method for generating the QM of the processing object block when the processing object block is a square block, when the processing object block is a rectangular block, the QM of the processing object block can also be generated from the quantization coefficient values ​​of the QM of the diagonal components, just like in the example in Figure 29. [Other variations of the third example using quantization matrix encoding and decoding]

[0393] The examples in Figures 28 and 29 illustrate a method for generating the quantization coefficients of the entire QM of the processing target block from the quantization coefficients of the diagonal component QM. However, it is also possible to generate the coefficients of only a portion of the QM of the processing target block from the quantization coefficients of the diagonal component QM. For example, regarding the QM corresponding to the low-frequency region of the processing target block, all the quantization coefficients contained in that QM can be encoded and decoded in the stream, and the quantization coefficients of the QM corresponding to the mid-frequency and high-frequency regions of the processing target block can be generated only from the quantization coefficients of the diagonal component QM of the processing target block.

[0394] Furthermore, the method for generating a rectangular block QM from a square block QM can also employ a combination of the third example illustrated in Figures 26 and 27 and the first example illustrated in Figures 11 and 12. For example, the square block QM is generated as described in the third example, using either of the two common methods described above from the quantization coefficient values ​​of the QM for only the diagonal components of the square block; the rectangular block QM is generated using the square block QM generated as described in the first example.

[0395] Furthermore, the method for generating a rectangular block QM from a square block QM can also employ a combination of the third example illustrated in Figures 26 and 27 and the second example illustrated in Figures 17 and 18. For example, from the quantization coefficient values ​​of the QM of only the diagonal components of the effective conversion coefficient region, any of the above-described common methods can be used to generate a QM for the size of the effective conversion coefficient region corresponding to the block size of the processed object block. [The effect of the third example of encoding and decoding using quantization matrices]

[0396] If the encoding device 100 and decoding device 200 according to the third state pattern disclosed herein are used, by utilizing the configuration described in Figures 26 and 27, even if not all the quantization coefficients of the QM of each block size of the target block are recorded in the stream, as long as only the quantization coefficients of the QM of the diagonal components of the target block are recorded in the stream, the encoding and decoding of the target block can be performed. Therefore, since the amount of code in the header region can be reduced, the possibility of improving encoding efficiency is increased.

[0397] For example, encoding device 100 is an encoding device that performs quantization to encode dynamic images, and includes circuitry and memory; the aforementioned circuitry uses the aforementioned memory to generate the aforementioned quantization matrix for the aforementioned processing object block from the diagonal components of the quantization matrix for the aforementioned processing object block, for the plurality of conversion coefficients consecutively arranged in the diagonal direction of the aforementioned processing object block, and encodes the signals of the aforementioned diagonal components in a bitstream.

[0398] Therefore, even in encoding methods that utilize blocks of many shapes, including rectangular blocks, the amount of code is reduced because only the diagonal components of the target block are encoded. Thus, processing efficiency is improved according to the encoding device 100.

[0399] For example, in the encoding device 100, the aforementioned circuit can also generate the aforementioned quantization matrix by repeating each element of the plurality of matrix elements of the aforementioned diagonal component in the horizontal and vertical directions.

[0400] Therefore, since the quantization coefficients of the processing object block are generated from the quantization coefficients of the diagonal components of the processing object block, it is possible to not encode all the quantization matrices of the processing object block. Thus, due to the reduction in code size, processing efficiency is improved. Therefore, according to the encoding device 100, the processing object block can be quantized efficiently.

[0401] For example, in the encoding device 100, the aforementioned circuit can also generate the aforementioned quantization matrix by repeating each element of the plurality of matrix elements of the aforementioned diagonal component diagonally.

[0402] Therefore, since the quantization coefficients of the processing object block are generated from the quantization coefficients of the diagonal components of the processing object block, it is possible to not encode all the quantization matrices of the processing object block. Thus, due to the reduction in code size, processing efficiency is improved. Therefore, according to the encoding device 100, the processing object block can be quantized efficiently.

[0403] For example, in the encoding device 100, the aforementioned circuit can also generate the aforementioned quantization matrix from a plurality of matrix elements of the aforementioned diagonal components and a plurality of matrix elements located near the aforementioned diagonal components.

[0404] In this way, since the encoding device 100 generates the quantization coefficients contained in the processing object block from the quantization coefficients of the diagonal components and the nearby components of the processing object block, a more appropriate quantization matrix corresponding to the processing object block can be generated.

[0405] For example, in the encoding device 100, the aforementioned circuit may not encode the signals of the plurality of matrix elements located near the aforementioned diagonal component into the streaming stream, but instead generate them by interpolating the plurality of matrix elements located near the aforementioned diagonal component from the plurality of adjacent matrix elements among the plurality of matrix elements of the aforementioned diagonal component.

[0406] Therefore, processing efficiency is improved due to the reduction in code size. As a result, the encoding device 100 can efficiently quantize the block to be processed.

[0407] For example, in the encoding device 100, the aforementioned circuit can also quantize the multiple conversion coefficients within a specified range of the low-frequency region side among the multiple conversion coefficients contained in the aforementioned processing object block using a quantization matrix, and only encode the signal of the aforementioned quantization matrix corresponding to the multiple conversion coefficients within the specified range of the aforementioned low-frequency region side in the streaming stream. For the multiple conversion coefficients outside the aforementioned specified range of the low-frequency region side among the multiple conversion coefficients contained in the aforementioned processing object block, quantization is performed using the aforementioned multiple matrix elements of the diagonal components.

[0408] Therefore, in the processing target block, since all quantization coefficients within the specified range of the low-frequency region side, which has a significant impact on visual perception, are encoded, the image quality of the moving image is not easily degraded. Furthermore, in the processing target block, since the multiple conversion coefficients outside the specified range of the low-frequency region side are quantized using the quantization matrix of the diagonal component of the processing target block, the amount of code is reduced, and processing efficiency is improved. Therefore, according to the encoding device 100, the image quality of the moving image is not easily degraded, and processing efficiency is improved.

[0409] For example, in the encoding device 100, the aforementioned quantization matrix may also be a quantization matrix that corresponds only to the multiple conversion coefficients within the aforementioned specified range of the multiple conversion coefficients contained in the aforementioned processing object block, specifically the multiple conversion coefficients in the aforementioned low-frequency region.

[0410] Therefore, according to the encoding device 100, in the processing object block, since a quantization matrix corresponding to a specified range of low-frequency regions that have a large impact on vision is generated, the image quality of the dynamic image is not easily degraded.

[0411] For example, in the encoding device 100, the aforementioned processing target block can also be a square block or a rectangular block. The aforementioned circuit can also generate a second quantization matrix of a plurality of conversion coefficients within the aforementioned specified range for the aforementioned square block by converting the first quantization matrix of a plurality of conversion coefficients within the aforementioned specified range for the aforementioned rectangular block, and use it as the aforementioned quantization matrix. Only the aforementioned first quantization matrix among the aforementioned first quantization matrix and the aforementioned second quantization matrix is ​​encoded as a signal related to the aforementioned quantization matrix in the bit stream.

[0412] Therefore, since only the first quantization matrix corresponding to a specified range of square blocks is encoded, the amount of code is reduced. Furthermore, since a second quantization matrix corresponding to a specified range of rectangular blocks is generated from the first quantization matrix, the image quality of the moving image is less likely to degrade. Moreover, the specified range is located in the low-frequency region that has a significant impact on visual perception. Therefore, according to the encoding device 100, the image quality of the moving image is less likely to degrade, and processing efficiency is improved.

[0413] For example, in the encoding device 100, the aforementioned circuit can also perform down-conversion processing on the aforementioned first quantization matrix of a plurality of conversion coefficients within the aforementioned specified range for the aforementioned square block to generate the aforementioned second quantization matrix of a plurality of conversion coefficients within the aforementioned specified range for the aforementioned rectangular block, wherein the aforementioned square block has one side of the same length as the long side of the aforementioned processing object block, i.e., the aforementioned rectangular block.

[0414] In this way, the encoding device 100 can efficiently generate a second quantization matrix corresponding to a specified range of rectangular blocks.

[0415] For example, in the encoding device 100, the aforementioned circuit can also perform upconversion processing on the aforementioned first quantization matrix of the plurality of conversion coefficients within the aforementioned specified range for the aforementioned square block to generate the aforementioned second quantization matrix of the plurality of conversion coefficients within the aforementioned specified range for the aforementioned rectangular block, wherein the aforementioned square block has one side of the same length as the short side of the aforementioned processing object block, i.e., the aforementioned rectangular block.

[0416] In this way, the encoding device 100 can efficiently generate a second quantization matrix corresponding to a specified range of rectangular blocks.

[0417] For example, in the encoding device 100, the aforementioned circuit may also generate the aforementioned second quantization matrix for a plurality of conversion coefficients within the aforementioned specified range of the aforementioned rectangular block by switching the following methods in accordance with the ratio of the short side length to the long side length of the aforementioned processing object block, i.e., the aforementioned rectangular block. The switching methods include: generating the matrix by performing the aforementioned downconversion process on the aforementioned first quantization matrix for a plurality of conversion coefficients within the aforementioned specified range of the aforementioned square block, wherein the aforementioned square block has one side with the same length as the long side of the aforementioned rectangular block; and generating the matrix by performing the aforementioned upconversion process on the aforementioned first quantization matrix for a plurality of conversion coefficients within the aforementioned specified range of the aforementioned square block, wherein the aforementioned square block has one side with the same length as the short side of the aforementioned rectangular block.

[0418] In this way, the encoding device 100 can generate a more appropriate second quantization matrix corresponding to a rectangular block from a first quantization matrix corresponding to a specified range of a square block by switching down-conversion and up-conversion.

[0419] Furthermore, the decoding device 200 is a decoding device that performs inverse quantization to decode dynamic images, and includes circuitry and memory. The aforementioned circuitry uses the aforementioned memory to generate the aforementioned quantization matrix for the aforementioned processing object block from the diagonal components of the quantization matrix for the aforementioned processing object block, for the plurality of quantization coefficients that are consecutively arranged in the diagonal direction of the aforementioned processing object block among the plurality of conversion coefficients contained in the processing object block, and decodes the signal related to the aforementioned diagonal components from the bit stream.

[0420] Therefore, even in decoding methods that utilize blocks of many shapes, including rectangular blocks, the amount of code can be reduced since only the diagonal components of the target block are decoded. Thus, processing efficiency is improved according to the decoding device 200.

[0421] For example, in the decoding device 200, the aforementioned circuit can also generate the aforementioned quantization matrix by repeating each element of the aforementioned diagonal component's plurality of matrix elements in the horizontal and vertical directions.

[0422] Therefore, since the quantization coefficients of the processing object block are generated from the quantization coefficients of the diagonal components of the processing object block, it is possible to decode not all the quantization matrices of the processing object block. Thus, processing efficiency is improved because the amount of code can be reduced. Therefore, according to the decoding device 200, the processing object block can be efficiently dequantized.

[0423] For example, in the decoding device 200, the aforementioned circuit can also generate the aforementioned quantization matrix by repeating each element of the aforementioned diagonal component in a diagonal direction.

[0424] Therefore, since the quantization coefficients of the processing object block are generated from the quantization coefficients of the diagonal components of the processing object block, it is possible to decode not all the quantization matrices of the processing object block. Thus, processing efficiency is improved because the amount of code can be reduced. Therefore, according to the decoding device 200, the processing object block can be efficiently dequantized.

[0425] For example, in the decoding device 200, the aforementioned circuit can also generate the aforementioned quantization matrix from a plurality of matrix elements of the aforementioned diagonal components and a plurality of matrix elements located near the aforementioned diagonal components.

[0426] In this way, since the decoding device 200 generates the quantization coefficients contained in the processing object block from the quantization coefficients of the diagonal components and the nearby components of the processing object block, a more appropriate quantization matrix corresponding to the processing object block can be generated.

[0427] For example, in the decoding device 200, the aforementioned circuit may not decode the signals of the plurality of matrix elements located near the aforementioned diagonal component from the bit stream, but instead generate them by interpolating the plurality of matrix elements located near the aforementioned diagonal component from the plurality of adjacent matrix elements among the plurality of matrix elements of the aforementioned diagonal component.

[0428] This improves processing efficiency by reducing the amount of code. Therefore, the decoding device 200 can efficiently dequantize the target block.

[0429] For example, in the decoding device 200, the aforementioned circuit can also dequantize a plurality of quantization coefficients within a specified range of the low-frequency region side among the plurality of conversion coefficients contained in the aforementioned processing object block using a quantization matrix, and only decode the signal of the aforementioned quantization matrix corresponding to the plurality of conversion coefficients within the aforementioned specified range of the aforementioned low-frequency region side from the bit stream. For a plurality of quantization coefficients outside the aforementioned specified range of the aforementioned low-frequency region side among the plurality of quantization coefficients contained in the aforementioned processing object block, dequantize using the aforementioned multiple matrix elements of the diagonal components.

[0430] Therefore, in the processing target block, since decoding all quantization coefficients within a specified range in the low-frequency region that has a significant impact on visual quality, the image quality of the moving image is not easily degraded. Furthermore, in the processing target block, since dequantization is performed using the quantization matrix of the diagonal component of the processing target block for the multiple quantization coefficients outside the specified range in the low-frequency region, the amount of code can be reduced, improving processing efficiency. Therefore, according to the decoding device 200, the image quality of the moving image is not easily degraded, and the processing efficiency is improved.

[0431] For example, in the decoding device 200, the aforementioned quantization matrix may also be a quantization matrix that corresponds only to the plurality of conversion coefficients within the aforementioned specified range on the low-frequency region side of the plurality of conversion coefficients contained in the aforementioned processing object block.

[0432] Therefore, according to the decoding device 200, in the processing object block, since a quantization matrix corresponding to a specified range of low-frequency regions that have a large impact on vision is generated, the image quality of the dynamic image is not easily degraded.

[0433] For example, in the decoding device 200, the aforementioned processing target block can also be a square block or a rectangular block. The aforementioned circuit can also generate a second quantization matrix of multiple conversion coefficients within the aforementioned specified range for the aforementioned rectangular block from the first quantization matrix by converting the first quantization matrix of multiple conversion coefficients within the aforementioned specified range for the aforementioned square block, and use it as the aforementioned quantization matrix. Only the first quantization matrix in the aforementioned first quantization matrix and the aforementioned second quantization matrix is ​​decoded from the bit stream to serve as the signal related to the aforementioned quantization matrix.

[0434] Therefore, since only the first quantization matrix corresponding to a specified range of square blocks is decoded, the amount of code can be reduced. Furthermore, since a second quantization matrix corresponding to a specified range of rectangular blocks is generated from the first quantization matrix, the image quality of the moving image is less likely to degrade. Moreover, the specified range is located in the low-frequency region, which has a significant impact on visual perception. Therefore, according to the decoding device 200, the image quality of the moving image is less likely to degrade, and processing efficiency is improved.

[0435] For example, in the decoding device 200, the aforementioned circuit can also perform down-conversion processing on the aforementioned first quantization matrix of a plurality of conversion coefficients within the aforementioned specified range for the aforementioned square block to generate the aforementioned second quantization matrix of a plurality of conversion coefficients within the aforementioned specified range for the aforementioned rectangular block, wherein the aforementioned square block has one side of the same length as the long side of the aforementioned processing object block, i.e., the aforementioned rectangular block.

[0436] In this way, the decoding device 200 can efficiently generate a second quantization matrix corresponding to a specified range of rectangular blocks.

[0437] For example, in the decoding device 200, the aforementioned circuit can also perform upconversion processing on the aforementioned first quantization matrix of the plurality of conversion coefficients within the aforementioned specified range for the aforementioned square block to generate the aforementioned second quantization matrix of the plurality of conversion coefficients within the aforementioned specified range for the aforementioned rectangular block, wherein the aforementioned square block has one side of the same length as the short side of the aforementioned processing object block, i.e., the aforementioned rectangular block.

[0438] In this way, the decoding device 200 can efficiently generate a second quantization matrix corresponding to a specified range of rectangular blocks.

[0439] For example, in the decoding device 200, the aforementioned circuit may also generate the aforementioned second quantization matrix for a plurality of conversion coefficients within the aforementioned specified range of the aforementioned rectangular block by switching the following methods in accordance with the ratio of the short side length to the long side length of the aforementioned processing object block, i.e., the aforementioned rectangular block. The switching methods include: generating the matrix by performing the aforementioned downconversion process on the aforementioned first quantization matrix for a plurality of conversion coefficients within the aforementioned specified range of the aforementioned square block, wherein the aforementioned square block has one side with the same length as the long side of the aforementioned rectangular block; and generating the matrix by performing the aforementioned upconversion process on the aforementioned first quantization matrix for a plurality of conversion coefficients within the aforementioned specified range of the aforementioned square block, wherein the aforementioned square block has one side with the same length as the short side of the aforementioned rectangular block.

[0440] In this way, the decoding device 200 can generate a more appropriate second quantization matrix corresponding to a rectangular block from the first quantization matrix corresponding to a specified range of a square block by switching down-conversion and up-conversion.

[0441] Furthermore, the encoding method is an encoding method for encoding dynamic images by performing quantization. For a plurality of transformation coefficients contained in the processing object block, a plurality of transformation coefficients are continuously arranged in the diagonal direction of the aforementioned processing object block. The aforementioned quantization matrix for the aforementioned processing object block is generated from the diagonal component of the quantization matrix, and the signals of the aforementioned diagonal component are encoded in the bit stream.

[0442] Therefore, even in encoding methods that utilize blocks of many shapes containing rectangular blocks, the amount of code is reduced because only the diagonal components of the target block are encoded. Thus, processing efficiency is improved depending on the encoding method.

[0443] Furthermore, the decoding method is a decoding method for decoding dynamic images by performing inverse quantization. For a plurality of quantization coefficients contained in the processing object block, a plurality of quantization coefficients continuously arranged in the diagonal direction of the aforementioned processing block are generated from the diagonal components of the quantization matrix for the aforementioned processing object block, and the signal related to the aforementioned diagonal components is decoded from the bit stream.

[0444] Therefore, even in decoding methods that utilize blocks of many shapes containing rectangular blocks, the amount of information is reduced because only the diagonal components of the target block are decoded. Thus, processing efficiency is improved by adjusting the decoding method.

[0445] This sample may also be implemented in combination with at least a portion of other samples disclosed herein. Furthermore, a portion of the processing described in the flowchart of this sample, a portion of the device configuration, a portion of the syntax, etc., may also be implemented in combination with other samples. [Installation Example]

[0446] Figure 30 is a block diagram showing an example of the installation of the encoding device 100. The encoding device 100 includes a circuit 160 and a memory 162. For example, a plurality of components of the encoding device 100 shown in Figure 1 are installed by means of the circuit 160 and memory 162 shown in Figure 30.

[0447] Circuit 160 is an electronic circuit that performs information processing and can be accessed in memory 162. For example, circuit 160 is a dedicated or general-purpose electronic circuit that uses memory 162 to encode moving images. Circuit 160 can be a processor like a CPU. Alternatively, circuit 160 can be an assembly of multiple electronic circuits.

[0448] Furthermore, for example, circuit 160 can also function as a plurality of constituent elements of the encoding device 100 shown in FIG1, excluding the constituent elements used for storing information. That is, circuit 160 can also perform the above-described actions as the operation of such constituent elements.

[0449] Memory 162 is a general-purpose or dedicated memory used by memory circuit 160 to encode information of dynamic images. Memory 162 may be an electronic circuit, connected to circuit 160, or contained within circuit 160.

[0450] Furthermore, memory 162 can be an assembly of multiple electronic circuits, or it can be composed of multiple sub-memories. Furthermore, memory 162 can be a magnetic disk or optical disk, or it can be a storage device or recording medium. Furthermore, memory 162 can be either non-volatile or volatile memory.

[0451] Furthermore, for example, memory 162 can also function as one of the constituent elements of the encoding device 100 shown in FIG1, used for storing information. Specifically, memory 162 can also function as block memory 118 and frame memory 122 shown in FIG1.

[0452] Furthermore, the memory 162 may store the dynamic image to be encoded, or the bit string corresponding to the encoded dynamic image. Alternatively, the memory 162 may also store the program used by the circuit 160 to encode the dynamic image.

[0453] Furthermore, the encoding device 100 may not have all of the plurality of constituent elements shown in FIG1 installed, or may not have all of the aforementioned plurality of processes performed. A portion of the plurality of constituent elements shown in FIG1 may be included in other devices, or a portion of the aforementioned plurality of processes may be performed by other devices. Moreover, in the encoding device 100, by installing a portion of the plurality of constituent elements shown in FIG1 and performing a portion of the aforementioned plurality of processes, a prediction sample set can be appropriately derived.

[0454] Figure 31 is a flowchart illustrating an example of the operation of the encoding device 100 shown in Figure 30. For example, the encoding device 100 shown in Figure 30 performs the operation shown in Figure 31 when encoding a moving image. Specifically, the circuit 160 uses the memory 162 to perform the following operations.

[0455] First, circuit 160 converts the first quantization matrix for the complex conversion coefficients of the square block, thereby generating a second quantization matrix for the complex conversion coefficients of the rectangular block from the first quantization matrix (step S201). Next, circuit 160 quantizes the complex conversion coefficients of the rectangular block using the second quantization matrix (step S202).

[0456] Therefore, since a quantization matrix corresponding to a rectangular block is generated from a quantization matrix corresponding to a square block, it is also possible to avoid encoding the quantization matrix corresponding to the rectangular block. Thus, the amount of code is reduced, and processing efficiency is improved. Therefore, according to the encoding device 100, rectangular blocks can be quantized efficiently.

[0457] For example, circuit 160 may encode only the first quantization matrix in the first quantization matrix and the second quantization matrix in the bit stream.

[0458] This reduces the amount of code. Therefore, based on encoding device 100, processing efficiency is improved.

[0459] For example, circuit 160 can also perform downconversion processing on the first quantization matrix of the complex conversion coefficients for the square block to generate the second quantization matrix of the complex conversion coefficients for the rectangular block, wherein the aforementioned square block has one side of the same length as the long side of the processed object block, i.e., the rectangular block.

[0460] In this way, the encoding device 100 can efficiently generate a quantization matrix corresponding to a rectangular block from a quantization matrix corresponding to a square block having one side of the same length as the long side of the rectangular block.

[0461] For example, circuit 160 can also perform down-conversion processing to divide the complex matrix elements of the first quantization matrix into groups of the same number as the complex matrix elements of the second quantization matrix. For each of the complex groups, the complex matrix elements contained in the group are continuously arranged in the horizontal or vertical direction of the square block. For each of the complex groups, the matrix elements located on the lowest frequency side, the matrix elements located on the highest frequency side, or the average value of the complex matrix elements contained in the group are determined as the matrix elements in the second quantization matrix corresponding to that group.

[0462] In this way, the encoding device 100 can generate a quantization matrix corresponding to a rectangular block more efficiently from the quantization matrix corresponding to the square block.

[0463] For example, circuit 160 can also perform upconversion processing on the first quantization matrix of the complex conversion coefficients for the square block to generate the second quantization matrix of the complex conversion coefficients for the rectangular block, wherein the aforementioned square block has one side of the same length as the short side of the processed block, i.e., the rectangular block.

[0464] In this way, the encoding device 100 can efficiently generate a quantization matrix corresponding to a rectangular block from a quantization matrix corresponding to a square block having one side of the same length as the short side of the rectangular block.

[0465] For example, circuit 160 can also perform upconversion processing, (i) dividing the complex matrix elements of the second quantization matrix into groups of the same number as the complex matrix elements of the first quantization matrix, repeating the complex matrix elements contained in each group to determine the matrix elements corresponding to that group in the second quantization matrix, or (ii) determining the complex matrix elements of the second quantization matrix by performing linear interpolation between adjacent matrix elements in the complex matrix elements of the second quantization matrix.

[0466] In this way, the encoding device 100 can generate a quantization matrix corresponding to a rectangular block more efficiently from the quantization matrix corresponding to the square block.

[0467] For example, circuit 160 can also generate a second quantization matrix of a plurality of conversion coefficients for a rectangle by switching the following methods according to the ratio of the short side length to the long side length of the processing object block, i.e., the rectangular block. The switching methods include: a method of generating the matrix by performing a downconversion process on the first quantization matrix of a plurality of conversion coefficients for a square block, wherein the square block has one side with the same length as the long side of the rectangular block; and a method of generating the matrix by performing an upconversion process on the first quantization matrix of a plurality of conversion coefficients for a square block, wherein the square block has one side with the same length as the short side of the rectangular block.

[0468] In this way, the encoding device 100 can switch between downconversion and upconversion according to the block size of the processing object block, and generate a more appropriate quantization matrix corresponding to a rectangular block from the quantization matrix corresponding to a square block.

[0469] Figure 32 is a block diagram showing an example of the installation of the decoding device 200. The decoding device 200 includes a circuit 260 and a memory 262. For example, a plurality of components of the decoding device 200 shown in Figure 10 are installed by means of the circuit 260 and memory 262 shown in Figure 32.

[0470] Circuit 260 is an electronic circuit that performs information processing and can be accessed by memory 262. For example, circuit 260 is a dedicated or general-purpose electronic circuit that uses memory 262 to decode moving images. Circuit 260 can be a processor like a CPU. Alternatively, circuit 260 can be an assembly of multiple electronic circuits.

[0471] Furthermore, for example, circuit 260 can also function as one of the plurality of constituent elements of the decoding device 200 shown in FIG10, excluding the constituent element used for storing information. That is, circuit 260 can also perform the above-described actions as the operation of such constituent elements.

[0472] Memory 262 is a general-purpose or dedicated memory used by memory circuit 260 to decode information of moving images. Memory 262 can be an electronic circuit, connected to circuit 260, or contained within circuit 260.

[0473] Furthermore, memory 262 can be an assembly of multiple electronic circuits, or it can be composed of multiple sub-memories. Furthermore, memory 262 can be a magnetic disk or optical disk, or it can be a storage device or recording medium. Furthermore, memory 262 can be either non-volatile or volatile memory.

[0474] Furthermore, for example, memory 262 can also function as one of the components of the decoding device 200 shown in FIG10, serving as a component for storing information. Specifically, memory 262 can also function as block memory 210 and frame memory 214 shown in FIG10.

[0475] Furthermore, memory 262 can store either the bit string corresponding to the encoded motion image or the decoded motion image. Alternatively, memory 262 can also store the program used by circuit 260 to decode the motion image.

[0476] Furthermore, the decoding device 200 may not have all of the plurality of constituent elements shown in FIG10 installed, or may not have performed all of the aforementioned plurality of processes. A portion of the plurality of constituent elements shown in FIG10 may be included in other devices, or a portion of the aforementioned plurality of processes may be performed by other devices. Moreover, by installing a portion of the plurality of constituent elements shown in FIG10 and performing a portion of the aforementioned plurality of processes in the decoding device 200, a prediction sample set can be appropriately derived.

[0477] Figure 33 is a flowchart illustrating an example of the operation of the decoding device 200 shown in Figure 32. For example, the decoding device 200 shown in Figure 32 performs the operation shown in Figure 33 when decoding a moving image. Specifically, the circuit 260 uses the memory 262 to perform the following operations.

[0478] First, circuit 260 converts the first quantization matrix of the complex conversion coefficients for the square block, thereby generating a second quantization matrix of the complex conversion coefficients for the rectangular block from the first quantization matrix (step S301). Next, circuit 260 performs dequantization on the complex conversion coefficients of the rectangular block using the second quantization matrix (step S302).

[0479] Therefore, since a quantization matrix corresponding to a rectangular block is generated from a quantization matrix corresponding to a square block, it is not necessary to decode the quantization matrix corresponding to the rectangular block. Thus, the amount of code can be reduced, and processing efficiency is improved. Therefore, according to the decoding device 200, the rectangular block can be efficiently dequantized.

[0480] For example, circuit 260 can also decode only the first quantization matrix from the first quantization matrix and the second quantization matrix from the bit stream.

[0481] This allows for a reduction in the amount of code. Therefore, based on the decoding device 200, processing efficiency is improved.

[0482] For example, circuit 260 can also perform downconversion processing on the first quantization matrix of the complex conversion coefficients for the square block to generate the second quantization matrix of the complex conversion coefficients for the rectangular block, wherein the aforementioned square block has one side of the same length as the long side of the processed object block, i.e., the rectangular block.

[0483] In this way, the decoding device 200 can efficiently generate a quantization matrix corresponding to a rectangular block from a quantization matrix corresponding to a square block having one side of the same length as the long side of the rectangular block.

[0484] For example, circuit 260 can also perform down-conversion processing to divide the complex matrix elements of the first quantization matrix into the same number of groups as the complex matrix elements of the second quantization matrix. For each of the complex groups, the matrix element located on the lowest frequency side of the complex matrix elements contained in the group, the conversion coefficient located on the highest frequency side of the complex matrix elements contained in the group, or the average value of the complex matrix elements contained in the group are determined as the matrix elements in the second quantization matrix corresponding to that group.

[0485] In this way, the decoding device 200 can generate a quantization matrix corresponding to a rectangular block more efficiently from the quantization matrix corresponding to the square block.

[0486] For example, circuit 260 can also perform upconversion processing on the first quantization matrix of the complex conversion coefficients for the square block to generate the second quantization matrix of the complex conversion coefficients for the rectangular block, wherein the aforementioned square block has one side of the same length as the short side of the processed block, i.e., the rectangular block.

[0487] In this way, the decoding device 200 can efficiently generate a quantization matrix corresponding to a rectangular block from a quantization matrix corresponding to a square block having one side of the same length as the short side of the rectangular block.

[0488] For example, circuit 260 can also perform upconversion processing, (i) dividing the complex matrix elements of the second quantization matrix into groups of the same number as the complex matrix elements of the first quantization matrix, repeating the complex matrix elements contained in each group to determine the matrix elements in the second quantization matrix corresponding to that group, or (ii) determining the complex matrix elements of the second quantization matrix by performing linear interpolation between adjacent matrix elements in the complex matrix elements of the second quantization matrix.

[0489] In this way, the decoding device 200 can generate a quantization matrix corresponding to a rectangular block more efficiently from the quantization matrix corresponding to the square block.

[0490] For example, circuit 260 can also generate a second quantization matrix of conversion coefficients for a rectangle by switching the following methods according to the ratio of the short side length to the long side length of the processing object block, i.e., the rectangular block. The switching methods include: a method of generating the matrix by performing a downconversion process on the first quantization matrix of multiple conversion coefficients for a square block, wherein the square block has one side with the same length as the long side of the rectangular block; and a method of generating the matrix by performing an upconversion process on the first quantization matrix of multiple conversion coefficients for a square block, wherein the square block has one side with the same length as the short side of the rectangular block.

[0491] In this way, the decoding device 200 can switch between downconversion and upconversion according to the block size of the processing target block, and generate a more appropriate quantization matrix corresponding to a rectangular block from the quantization matrix corresponding to a square block.

[0492] Furthermore, as mentioned above, each component can also be a circuit. These circuits can be configured as a single circuit or as separate circuits. Moreover, each component can be implemented using a general-purpose processor or a dedicated processor.

[0493] Furthermore, other constituent elements can also perform the processing performed by a specific constituent element. Moreover, the order of processing can be changed, or multiple processes can be performed in parallel. Furthermore, the encoding / decoding device can also include an encoding device 100 and a decoding device 200.

[0494] Furthermore, the ordinal numbers 1 and 2 used in the description may be changed appropriately. Also, ordinal numbers may be repeatedly assigned or deleted for constituent elements, etc.

[0495] The above description of the encoding device 100 and decoding device 200 is based on the embodiments described, but the forms of the encoding device 100 and decoding device 200 are not limited to these embodiments. As long as they do not depart from the intent of this disclosure, any modifications conceived by those skilled in the art that are applied to this embodiment or constructed by combining the constituent elements of different embodiments can be included within the scope of the forms of the encoding device 100 and decoding device 200.

[0496] This sample may also be implemented in combination with at least a portion of other samples disclosed herein. Furthermore, a portion of the processing described in the flowchart of this sample, a portion of the device configuration, a portion of the syntax, etc., may also be implemented in combination with other samples. (Implementation Form 2)

[0497] In all the above embodiments, each functional block can generally be implemented using an MPU and memory. Furthermore, the processing of each functional block is generally achieved by a program execution unit such as a processor, which reads and executes software (programs) recorded on a recording medium such as ROM. This software can be distributed by downloading or by recording on a recording medium such as semiconductor memory. Alternatively, each functional block can of course be implemented using hardware (dedicated circuitry).

[0498] Furthermore, the processing described in each embodiment can be implemented by centralized processing using a single device (system) or by distributed processing using multiple devices. Also, the processors executing the above program can be a single number or multiple. That is, centralized processing or distributed processing is both possible.

[0499] The embodiments disclosed herein are not limited to the above embodiments and various modifications can be made, which are also included within the scope of the embodiments disclosed herein.

[0500] Further, examples of applications of the dynamic image encoding method (image encoding method) or dynamic image decoding method (image decoding method) shown in the above embodiments, and systems utilizing them, will be described here. The system is characterized by having an image encoding device utilizing the image encoding method, an image decoding device utilizing the image decoding method, and an image encoding / decoding device possessing both. Other components of the system may be appropriately modified depending on the circumstances. [Usage Example]

[0501] Figure 34 is a diagram showing the overall structure of the content delivery system ex100 that implements content publishing services. The area for providing communication services is divided into the required sizes, and fixed wireless stations, namely base stations ex106, ex107, ex108, ex109, and ex110, are set up in each cell.

[0502] In the aforementioned content delivery system ex100, various devices such as computers ex111, game consoles ex112, cameras ex113, home appliances ex114, and smartphones ex115 are connected to the Internet ex101 via Internet service provider ex102 or communication network ex104, and base stations ex106-ex110. The aforementioned content delivery system ex100 can also be connected by combining any of the above-mentioned components. The devices can also be directly or indirectly connected to each other via telephone networks or short-range wireless networks without using fixed wireless base stations ex106-ex110. Furthermore, the streaming server ex103 connects to the various devices such as computers ex111, game consoles ex112, cameras ex113, home appliances ex114, and smartphones ex115 via the Internet ex101, etc. Furthermore, the streaming server ex103 connects to terminals and other devices within a hotspot inside the aircraft ex117 via satellite ex116.

[0503] Furthermore, wireless access points or hotspots can be used to replace base stations ex106~ex110. Also, the streaming server ex103 can connect directly to the communication network ex104 without going through the Internet ex101 or the Internet service provider ex102, or connect directly to the aircraft ex117 without going through the satellite ex116.

[0504] The EX113 camera is a digital camera or similar device capable of capturing still images and moving images. The EX115 smartphone is a smartphone, mobile phone, or PHS (Personal Handyphone System) that generally supports 2G, 3G, 3.9G, 4G, and the future 5G mobile communication system.

[0505] Home appliances EX118 refers to refrigerators or other appliances included in household fuel cell cogeneration systems.

[0506] In the content delivery system ex100, a terminal with photography capabilities is connected to the streaming server ex103 via a base station ex106, enabling live streaming. For live streaming, the terminal (computer ex111, game console ex112, camera ex113, home appliance ex114, smartphone ex115, and terminal in an airplane ex117, etc.) performs the encoding processing described in the aforementioned embodiments on still images or moving images captured by the user using the terminal. The encoded image data and the corresponding encoded audio data are multiplexed, and the obtained data is then sent to the streaming server ex103. In other words, each terminal functions as an image encoding device as described in this disclosure.

[0507] Furthermore, the streaming server ex103 will stream and publish content data sent by requesting clients. Clients can be terminals within computers ex111, game consoles ex112, cameras ex113, home appliances ex114, smartphones ex115, or airplanes ex117, etc., capable of decoding the aforementioned encoded data. Each machine receiving the published data will decode and play it. That is, each machine functions as an image decoding device as described in this disclosure. [Distributed processing]

[0508] Furthermore, the streaming server ex103 can also consist of multiple servers or multiple computers, distributing the processing, recording, or distribution of data. For example, the streaming server ex103 can also be implemented using a CDN (Contents Delivery Network), which connects many edge servers distributed around the world to achieve content distribution. With a CDN, physically proximate edge servers are dynamically assigned based on the client. Then, by caching and distributing content to that edge server, latency can be reduced. Moreover, because in the event of an error, or when communication status changes due to increased traffic, processing can be distributed across multiple edge servers, or the distribution entity can be switched to another edge server to bypass the affected network portion and continue distribution, thus achieving high-speed and stable distribution.

[0509] Furthermore, the encoding of captured data can be performed not only on individual terminals but also on the server side, and the workload can be shared among different terminals. For example, encoding typically involves two processing loops. In the first loop, the image complexity or bitrate of a frame or scene unit is determined. In the second loop, processing is performed to maintain image quality and improve encoding efficiency. For instance, the terminal performs the first encoding, while the server receiving the content performs the second encoding. This reduces the processing load on each terminal while improving content quality and efficiency. If near-instantaneous reception and decoding are required, the data encoded by the terminal in the first iteration can be received and played by other terminals, thus enabling more flexible real-time publishing.

[0510] As another example, cameras such as the ex113 extract features from images, compress the relevant feature data, and send it as metadata to the server. The server uses features to determine the importance of objects and adjusts the quantization precision accordingly, compressing the image based on its meaning. Feature data is particularly effective in improving the accuracy and efficiency of motion vector prediction when compressed again on the server. Alternatively, simple encoding such as VLC (Variable Length Coding) can be performed on the terminal, while more demanding encoding methods such as CABAC (Context-Adaptive Binary Arithmetic Coding) can be used on the server.

[0511] Furthermore, as another example, in stadiums, shopping malls, or factories, there are sometimes multiple terminals capturing roughly the same scene, resulting in multiple images. In such cases, the multiple terminals taking the photos, along with other terminals and servers that are not capturing photos as needed, are used to distribute the encoding and processing data by assigning different units such as GOP (Group of Pictures), image units, or tile units derived from image segmentation. This reduces latency and achieves greater real-time performance.

[0512] Furthermore, since multiple image data points roughly depict the same scene, the server can manage and / or instruct the references between image data captured by each terminal. The server can also receive encoded data from each terminal, change the reference relationships between the multiple data points, or modify or replace the images themselves before re-encoding. This allows for the generation of streams with improved quality and efficiency for each data point.

[0513] Furthermore, the server can also transcode the video data after changing its encoding method before publishing the video data. For example, the server can convert the MPEG system encoding method to the VP system, or convert H.264 to H.265.

[0514] Thus, encoding processing can be performed via a terminal or one or more servers. Therefore, although the terms "server" or "terminal" are used below to refer to the subject of processing, some or all of the processing performed by the server can also be performed by the terminal, and some or all of the processing performed by the terminal can also be performed by the server. Furthermore, the same applies to decoding processing. [3D, Multi-angle]

[0515] In recent years, there has been an increasing trend of integrating and utilizing images or videos of different scenes, or the same scene, captured by multiple cameras such as the EX113 and / or smartphones such as the EX115, which are roughly synchronized with each other. The images captured by each terminal are integrated based on the relative positional relationship between the terminals or the regions with consistent feature points contained in the images.

[0516] The server not only encodes 2D moving images, but also automatically or at user-specified times encodes still images based on scene analysis of the moving images, and sends them to the receiving terminal. Furthermore, when the relative positions between the camera terminals are available, the server can generate the 3D shape of the scene not only from 2D moving images, but also from images of the same scene captured from different angles. Moreover, the server can additionally encode 3D data generated using point clouds, and can also use this 3D data to identify or track people or objects. Based on the identification or tracking results, it can select or reconstruct images from multiple terminals to generate the image to be sent to the receiving terminal.

[0517] In this way, users can freely select images corresponding to each camera terminal to appreciate the scene, or appreciate the content of images cut from any viewpoint using 3D data reconstructed from multiple images or videos. Furthermore, similar to images, sound can also be picked up from multiple different angles, and the server, in conjunction with the images, multiplexes and sends sound and images from specific angles or spaces.

[0518] Furthermore, in recent years, content that aligns the real world with the virtual world, such as Virtual Reality (VR) and Augmented Reality (AR), has become increasingly popular. When creating VR images, the server can create separate viewpoint images for the right and left eyes. Through Multi-View Coding (MVC), the encoding of each viewpoint image can be referenced, or they can be encoded as separate streams without reference. When decoding different streams, they are synchronized to recreate the virtual 3D space according to the user's viewpoint.

[0519] In AR image creation, the server overlays virtual object information in virtual space onto camera information in real space based on 3D position or user viewpoint movement. The decoding device can also acquire or retain virtual object information and 3D data, generating 2D images in response to user viewpoint movement, and creating overlay data through smooth stitching. Alternatively, the decoding device can send the user's viewpoint movement to the server in addition to requests for virtual object information. The server, in conjunction with the viewpoint movement received from the 3D data held on the server, creates overlay data, encodes it, and publishes it to the decoding device. Furthermore, in addition to RGB, the overlay data can also have an α value representing penetration. The server sets the α value of parts outside the object created from the 3D data to 0, encoding these parts as transparent. Alternatively, the server can also, like a chroma key, set the background to a specified RGB value and generate data where parts outside the object are set to the background color.

[0520] Similarly, the decoding of published data can be performed on individual terminals acting as clients, on the server side, or distributed among them. For example, one terminal can first send a request to the server, and then other terminals can receive the content corresponding to that request, perform decoding, and send a signal indicating completion to a device with a display. By distributing the processing and selecting appropriate content without relying on the performance of the communicating terminals themselves, high-quality data can be played. Furthermore, as another example, large-format image data can be received on TVs, etc., while simultaneously decoding the image into segmented areas such as tiles and displaying them on the viewer's personal terminal. This allows for sharing the entire image while simultaneously identifying one's own area of ​​responsibility or areas requiring more detailed examination.

[0521] Furthermore, it is anticipated that in the future, under conditions unaffected by indoor or outdoor environments, multiple short-range, medium-range, or long-range wireless communication methods will be available. Utilizing publishing system standards such as MPEG-DASH, appropriate data will be switched for each connected communication while content is received seamlessly. This will allow users to move beyond their own terminals, freely choosing between decoding or display devices such as monitors located indoors or outdoors, and switching between them in real time. Furthermore, decoding can be performed while switching between the decoding and display terminals based on location information. This allows for movement towards a destination while simultaneously displaying map information on a portion of the wall or floor of a building with embedded display devices. Additionally, the bit rate of the received data can be switched based on the ease of access to the encoded data on the network. This ease of access refers to the encoded data being cached to a server accessible to the receiving terminal for short periods, or copied to an edge server within a content delivery service. Adaptive Coding

[0522] Regarding content switching, the adaptive stream compressed and encoded using the dynamic image encoding methods described in the above embodiments, as shown in Figure 35, will be used for illustration. While it is acceptable for a server to have multiple streams with the same content but different qualities as individual streams, they can also be configured as layered encodings, as shown in the figure, thereby achieving time / space-adaptive streams and utilizing their characteristics to switch content. In other words, the decoding side determines which layer to decode based on internal performance factors and external factors such as communication band status. This allows the decoding side to freely switch between low-resolution and high-resolution content during decoding. For example, if a video viewed on a smartphone (ex115) while on the go is to be viewed on an internet TV device at home, the device only needs to decode the same stream to a different layer, thus reducing the burden on the server side.

[0523] Furthermore, besides achieving the aforementioned configuration where each layer encodes an image and an enhancement layer exists above the base layer, the enhancement layer could also contain metadata such as statistical information about the image. The decoder then uses this metadata to perform super-resolution on the base layer image, thereby generating high-quality content. Super-resolution can also refer to either increasing the S / N ratio at the same resolution or increasing the resolution. The metadata includes information specifying the linear or nonlinear filtering coefficients used in super-resolution processing, or information specifying the parameter values ​​for filtering, machine learning, or least squares operations used in super-resolution processing.

[0524] Furthermore, the image can be segmented into tiles based on the meaning of objects within it, allowing the decoder to select the tiles to be decoded, thus decoding only a portion of the area. Additionally, object attributes (people, cars, balls, etc.) and their positions within the image (coordinates within the same image, etc.) can be stored as metadata. This allows the decoder to pinpoint the location of the desired object and determine the tile containing that object based on the metadata. For example, as shown in Figure 36, metadata can be stored using a data storage structure different from pixel data, such as HEVC's SEI information. This metadata represents, for example, the position, size, or color of the main object.

[0525] Alternatively, metadata can be stored in units consisting of multiple images, such as streams, sequences, or random access units. This allows the decoder to obtain information such as the time a specific person appears in the image, and by combining this with the image unit information, it can pinpoint the image containing the object and the object's position within the image. [Webpage Optimization]

[0526] Figure 37 shows an example of a webpage display on a computer (e.g., ex111). Figure 38 shows an example of a webpage display on a smartphone (e.g., ex115). As shown in Figures 37 and 38, webpages sometimes contain multiple links to image content. The viewing experience varies depending on the browsing device. When multiple links are visible on the screen, the display device (decoding device) displays still images or I-images of each content as link images, or displays images such as GIF animations using multiple still images or I-images, or only receives the base layer to decode and display the images, until the user explicitly selects a link image, or the link image is near the center of the screen, or the entire link image enters the screen.

[0527] When a user selects a linked image, the display device prioritizes decoding the base layer. Furthermore, if the HTML of the webpage contains information indicating it is adaptable content, the display device can also decode up to the enhancement layer. Additionally, to ensure real-time performance, before selection or when communication bandwidth is very limited, the display device only decodes and displays images that reference the preceding image (I images, P images, and only B images that reference the preceding image), thereby reducing the delay between the decoding and display of the initial image (the delay from the start of content decoding to the start of display). Alternatively, the display device can intentionally ignore image reference relationships, performing coarse decoding of all B and P images with reference to the preceding image, and then performing normal decoding as time passes and more images are received. [Autonomous Driving]

[0528] Furthermore, when receiving and transmitting still images or data such as 2D or 3D map information for the purpose of vehicle autonomous driving or driving support, the receiving terminal can also receive image data belonging to more than one layer, as well as weather or construction information as metadata, and decode them accordingly. Moreover, the metadata can belong to a layer, or it can simply be multiplexed with the image data.

[0529] At this time, since the vehicle, drone, or aircraft containing the receiving terminal is moving, the receiving terminal can seamlessly receive and decode while switching base stations ex106~ex110 by sending its location information when a receiving request is made. Furthermore, the receiving terminal can dynamically switch the level of metadata reception or map information update rate according to the user's selection, the user's situation, or the status of the communication frequency band.

[0530] As described above, in the content delivery system ex100, the client can receive encoded information sent by the user in real time, decode it, and play it. [Personal Content Posting]

[0531] Furthermore, the content delivery system ex100 can handle not only high-quality, long-duration content from video publishers, but also low-quality, short-duration content from individuals, which can be unicast or multicast. Moreover, this type of personal content is expected to increase in the future. To make personal content of higher quality, the server can also perform editing before encoding. This can be achieved through, for example, the following configuration.

[0532] The server performs real-time or cumulative image processing during and after shooting, identifying and processing photographic errors, scene estimation, meaning analysis, and object detection from the original image or encoded data. Then, based on the identification results, the server manually or automatically performs the following edits: correcting out-of-focus or camera shake; deleting scenes of lower importance, such as those with lower brightness or out-of-focus areas; emphasizing object edges; and adjusting color tones. The server encodes the edited data based on the editing results. Furthermore, given that longer shooting times can lead to lower viewership, the server can also automatically edit out less important scenes and scenes with less movement, as described above, to maintain content within a specific time frame, based on image processing results and shooting time. Additionally, the server can generate and encode summaries based on the meaning analysis results of the scenes.

[0533] Furthermore, there are cases where direct broadcasting of personal content may infringe on copyrights, intellectual property rights, or portrait rights, and there are also situations where the scope of sharing exceeds the intended scope, causing inconvenience to the individual. Therefore, for example, servers can intentionally change the focus of images such as faces around the periphery of the screen or images of homes to be out of focus before re-encoding. Also, servers can identify whether the encoded image contains faces different from pre-registered individuals; if so, they can add mosaic or other processing to the faces. Furthermore, as pre-processing or post-processing for encoding, based on copyright and other considerations, users can specify the people or background areas they wish to process, and the server can replace the specified areas with other images or blur the focus. If it is a person, the face image can be replaced while tracking the person in the dynamic image.

[0534] Furthermore, viewing personal content with limited data requires immediate accessibility. Therefore, although bandwidth may vary, the decoding device prioritizes receiving the base layer for decoding and playback. During this time, the decoding device can also receive the enhancement layer, and in playback loops or after two or more playbacks, high-definition video, including the enhancement layer, will be played. Such adaptively encoded streaming provides the following experience: while the initial animation is coarse when not selected or at the beginning of viewing, the image improves as the streaming becomes smarter. In addition to adaptive encoding, combining the initial coarse stream and the second stream encoded based on the first animation into a single stream can also provide the same experience. [Other Use Cases]

[0535] Furthermore, such encoding or decoding processing is generally handled by the LSIex500 chip present in each terminal. The LSIex500 can be a single chip or composed of multiple chips. Moreover, software for encoding or decoding motion images can be integrated into a recording medium (CD-ROM, floppy disk, or hard disk, etc.) readable by a computer such as the ex111, and the encoding or decoding processing can be performed using this software. In other words, when the smartphone ex115 is equipped with a camera, it can also send motion image data captured by the camera. This motion image data is encoded by the LSIex500 chip present in the smartphone ex115.

[0536] Furthermore, the LSIex500 can also be configured to download and enable application software. In this case, the terminal first determines whether it supports the content encoding method or whether it has the capability to execute a specific service. If the terminal does not support the content encoding method or does not have the capability to execute a specific service, the terminal downloads a codec or application software, then obtains the content and plays it.

[0537] Furthermore, not limited to the content delivery system ex100 via the Internet ex101, at least one of the above-described embodiments of a motion image encoding device (image encoding device) or motion image decoding device (image decoding device) can also be incorporated into a digital broadcasting system. Since broadcasting radio waves carrying images and sound are transmitted and received using multiplexed data via satellites, etc., it is easier to configure for unicasting than the content delivery system ex100. The difference lies in its suitability for multicasting, but the same applications can be performed for encoding and decoding processing. [Hardware Configuration]

[0538] Figure 39 is a diagram showing the smartphone ex115. Figure 40 is a diagram showing an example of the configuration of the smartphone ex115. The smartphone ex115 includes: an antenna ex450 for transmitting and receiving radio waves with a base station ex110; a camera unit ex465 for capturing images and still images; and a display unit ex458 for displaying decoded data such as images captured by the camera unit ex465 and images received by the antenna ex450. The smartphone ex115 further includes: an operation unit ex466, which is a touch panel, etc.; a sound output unit ex457, which is a speaker for outputting sound or audio; a sound input unit ex456, which is a microphone for inputting sound, etc.; a memory unit ex467, which can be used to store encoded or decoded data such as captured images or still images, recorded audio, received images or still images, and emails; and a slot unit ex464, which is the interface with SIM ex468, which can be used to authenticate specific users for access to various data, primarily network data. Furthermore, an external memory unit ex467 can also be used to replace the memory unit ex467.

[0539] Furthermore, the main control unit ex460, which coordinates and controls the display unit ex458 and the operation unit ex466, is connected via bus ex470 to the power circuit unit ex461, the operation input control unit ex462, the image signal processing unit ex455, the camera interface unit ex463, the display control unit ex459, the modulation / demodulation unit ex452, the multiplexing / splitting unit ex453, the audio signal processing unit ex454, the slot unit ex464, and the memory unit ex467.

[0540] When the power button is turned on by the user, the power circuit section ex461 supplies power to all parts from the battery pack, thereby activating the smartphone ex115 into an operational state.

[0541] The smartphone ex115, controlled by the main control unit ex460 which includes a CPU, ROM, and RAM, performs call and data communication processing. During a call, the voice signal processing unit ex454 converts the audio signal received by the voice input unit ex456 into a digital audio signal. The modulation / demodulation unit ex452 performs spread spectrum processing, and the transmit / receive unit ex451 performs digital-to-analog conversion and frequency conversion before transmitting via the antenna ex450. Similarly, received data is amplified, undergoes frequency conversion and analog-to-digital conversion, is despread spectrum processed by the modulation / demodulation unit ex452, converted into an analog audio signal by the voice signal processing unit ex454, and output from the voice output unit ex457. In data communication mode, text, still images, or video data are sent to the main control unit ex460 via the operation input control unit ex462 through the main unit's operation unit ex466, and are similarly processed for transmission and reception. When transmitting images, still images, or images and sound in data communication mode, the image signal processing unit ex455 compresses and encodes the image signal stored in the memory unit ex467 or the image signal input from the camera unit ex465 using the motion picture encoding method shown in the above embodiments, and sends the encoded image data to the multiplexing / splitting unit ex453. Furthermore, the sound signal processing unit ex454 encodes the sound signal and sends the encoded sound data to the multiplexing / splitting unit ex453. This sound signal is the sound signal received by the sound input unit ex456 while the camera unit ex465 is capturing images or still images. The multiplexing / splitting unit ex453 multiplexes the encoded image data and encoded sound data in a prescribed manner, performing modulation and conversion processing by the modulation / demodulation unit (modulation / demodulation circuit unit) ex452 and the transmitting / receiving unit ex451, and then transmitting it via the antenna ex450.

[0542] When receiving images attached to emails or chat rooms, or images linked to web pages, the multiplexing / demultiplexing unit ex453 separates the multiplexed data received via antenna ex450 into bitstreams of image data and bitstreams of audio data to decode the multiplexed data. The encoded image data is then supplied to the image signal processing unit ex455 via synchronization bus ex470, and the encoded audio data is supplied to the audio signal processing unit ex454. The image signal processing unit ex455 decodes the image signal using a motion picture decoding method corresponding to the motion picture encoding method described in the above embodiments. The image or still image contained in the linked motion picture file is displayed on the display unit ex458 via the display control unit ex459. Furthermore, the audio signal processing unit ex454 decodes the audio signal and outputs audio from the audio output unit ex457. Moreover, since real-time streaming is now widespread, depending on the user's situation, there may be instances where the audio playback is socially inappropriate. Therefore, as an initial setting, it is advisable to use a configuration that plays only video signals and no audio signals. Alternatively, audio can be played only when the user performs actions such as clicking on video data.

[0543] Furthermore, although the example used here is the smartphone EX115, as a terminal, in addition to transceiver terminals with both encoders and decoders, three installation methods can also be considered: a transmitting terminal with only an encoder and a receiving terminal with only a decoder. Moreover, while the case of receiving or transmitting multiplexed data containing both video and audio data in a digital broadcasting system has been described, in addition to audio data, text data related to the video can also be multiplexed, and the video data itself can be received or transmitted, rather than being multiplexed data.

[0544] Furthermore, although the description includes the CPU's main control unit (ex460) controlling encoding or decoding processing, terminals often also have GPUs. Therefore, it's also possible to utilize GPU performance by using shared memory between the CPU and GPU, or by managing the addresses of shared memory, to process large areas at once. This shortens encoding time, ensures real-time performance, and achieves low latency. It is particularly efficient when using a GPU instead of a CPU, performing motion estimation, deblocking filtering, SAO (Sample Adaptive Offset), and conversion / quantization on a per-image basis.

[0545] Industrial availability This disclosure can be applied to, for example, televisions, digital video recorders, car navigation systems, mobile phones, digital cameras, digital camcorders, video conferencing systems, or electronic mirrors.

[0546] Blocks 10-23 100: Encoding device 102: Segmentation 104: Subtraction Department 106: Conversion Section 108: Quantitative Department 110: Entropy Coding Department 112,204: Anti-quantitative department 114,206: Reverse conversion section 116,208:Additional calculation department 118,210: Block memory 120, 212: Loop Filtering Section 122,214: Frame memory 124,216: Intra-frame prediction unit 126,218: Inter-frame prediction unit 128,220: Predictive Control Department 160, 260: Circuit 162,262: Memory 200: Decoding device 202: Entropy Decoding Department ALF: Adaptive Loop Filter AMT: Adaptive Multiple Conversion AR: Augmented Reality AST: Adaptive Secondary Transformation BIO: Bidirectional Optical Flow CCLM: Cross-component linear model CABAC: Context-Adaptive Binary Arithmetic Coding CDN: Content Delivery Network CTU: Coding Tree Unit CU: Encoding Unit Cur block: Current block DCT: Discrete Cosine Transform DF: Deblocking Filter DST: Discrete Sine Transform EMT: Explicit Multicore Conversion ex100: Content Supply System ex101: Internet ex102: Internet Service Provider ex103: Streaming Server ex104: Communication Network ex106~ex110: Base Station ex111: Computer ex112: Game console ex113: Camera ex114: Home Appliances ex115: Smartphone ex116: Satellite ex117: Airplane ex450: Antenna ex451: Transmitter / Receiver Unit ex452: Modulation / Demodulation Unit ex453: Multiplexing / Separation Section ex454: Audio Signal Processing Unit EX455: Image Signal Processing Unit ex456: Audio Input Section ex457: Audio Output Section ex458: Display Section ex459: Display Control Unit EX460: Main Control Unit ex461: Power Supply Circuit Section ex462: Operation Input Control Unit ex463: Camera Interface ex464: Slot section ex465: Camera Department ex466: Operations Department ex467: Memory Section ex468:SIM ex470: Bus, Synchronization Bus ex500:LSI FRUC: Frame Rate Boost Conversion GOP: Image Group HEVC: High Efficiency Video Encoding MBT: Multi-type Tree MV, MV0, MV1, MV_L, MV_U: Movement vectors MVC: Multi-view Coding NSST: Inseparable Secondary Transformation OBMC: Overlapping Block Movement Compensation PDPC: Location-Related Intra-Prediction Combination PMMVD: Style Matching Moving Vector Export Pred, Pred_L, Pred_U: Predicted image PU: Prediction Unit QM: Quantization matrix QP: Quantization parameter QTBT: Quadtree plus Binary Tree Ref0, Ref1: Reference images SAO: Sample Applicability Shift S101~S108, S201~S206, S301, S302, S701, S801, S1001, S1002, S1101, S1102, S1601, S1701: Steps TU: Conversion Unit v0, v1, vx, vy: movement vectors VLC: Variable Length Encoding VR: Virtual Reality

Claims

1. An encoding apparatus comprising: a circuit; and a memory connected to the circuit, wherein the circuit, using the memory, performs the following processing during operation: writing information about a first quantization matrix to a bitstream; performing an upconversion process and a downconversion process on the first quantization matrix to generate a second quantization matrix, wherein the first quantization matrix has a first number of columns and a first number of rows equal to the first number of columns, forming a square matrix, and the second quantization matrix has a second number of columns and a second number of rows different from the second number of columns, forming a rectangular matrix; and quantizing a plurality of conversion coefficients of a current block using the second quantization matrix, wherein the upconversion process is performed on the first quantization matrix in a first direction such that one of the second number of columns and the second number of rows is larger than the first number of columns, and the upconversion process is not performed on the first quantization matrix in a second direction different from the first direction. Furthermore, the aforementioned downconversion process is performed on the aforementioned first quantization matrix in the aforementioned second direction, so that the other of the aforementioned second number of columns and the aforementioned second number of rows is smaller than the aforementioned first number of columns, without performing the downconversion process on the aforementioned first quantization matrix in the aforementioned first direction.

2. A decoding apparatus comprising: a circuit; and a memory connected to the circuit, wherein the circuit, using the memory, performs the following processing during operation: interpreting information about a first quantization matrix from a one-bit serial stream; performing an upconversion and a downconversion on the first quantization matrix to generate a second quantization matrix, wherein the first quantization matrix has a first number of columns and a first number of rows equal to the first number of columns, forming a square matrix, and the second quantization matrix has a second number of columns and a second number of rows different from the second number of columns, forming a rectangular matrix; and performing dequantization on a plurality of quantization coefficients of a current block using the second quantization matrix, wherein the upconversion is performed on the first quantization matrix in a first direction such that one of the second number of columns and the second number of rows is larger than the first number of columns, and the upconversion is not performed on the first quantization matrix in a second direction different from the first direction. Furthermore, the aforementioned downconversion process is performed on the aforementioned first quantization matrix in the aforementioned second direction, so that the other of the aforementioned second number of columns and the aforementioned second number of rows is smaller than the aforementioned first number of columns, without performing the downconversion process on the aforementioned first quantization matrix in the aforementioned first direction.

3. A computer-readable non-transitory medium storing a bitstream, the bitstream comprising a plurality of quantization coefficients of a current block and information upon which a decoding device performs an inverse quantization process on the plurality of quantization coefficients, wherein the inverse quantization process includes: obtaining a first quantization matrix using the aforementioned information, the first quantization matrix having a first number of columns and a first number of rows equal to the first number of columns, forming a square matrix; generating a second quantization matrix by performing an upconversion process and a downconversion process on the first quantization matrix, the second quantization matrix having a second number of columns and a second number of rows different from the second number of columns, forming a rectangular matrix; and generating a plurality of inverse quantization coefficients of the current block from the plurality of quantization coefficients using the second quantization matrix. The aforementioned first quantization matrix is ​​up-converted in the first direction so that one of the aforementioned second number of columns and the aforementioned second number of rows is larger than the aforementioned first number of columns, without up-converting the aforementioned first quantization matrix in the second direction, which is different from the aforementioned first direction. The aforementioned first quantization matrix is ​​down-converted in the aforementioned second direction so that the other of the aforementioned second number of columns and the aforementioned second number of rows is smaller than the aforementioned first number of columns, without down-converting the aforementioned first quantization matrix in the aforementioned first direction.

Images