Image encoding apparatus and method, image decoding apparatus and method, and medium

Abstract

The present application provides an image encoding device and method, an image decoding device and method, and a medium. By adaptively using a quantization control size according to the shape of a sub-block, control of encoding of a quantization parameter can be appropriately performed not only for square sub-blocks but also for rectangular sub-blocks, and as a result, coding efficiency is improved.

Description

[0001] (This application is a divisional application of an application with the application date of June 14, 2019, the application number of 201980043647.3, and the invention name of “Image encoding apparatus, image encoding method and program, image decoding apparatus, image decoding method and program”. TECHNICAL FIELD

[0002] The present application relates to an image encoding apparatus, an image encoding method and a program, and an image decoding apparatus, an image decoding method and a program. BACKGROUND

[0003] As an encoding system for compression recording of a moving image, a HEVC (High Efficiency Video Coding) encoding system (hereinafter referred to as HEVC) is known. In order to improve the encoding efficiency, the HEVC adopts a basic block having a larger size than the existing macroblock (16 x 16 pixels). The basic block having a large size is called a CTU (Coding Tree Unit), and its size reaches 64 x 64 pixels. The CTU is further divided into subblocks as a unit of prediction and transformation. Patent Literature 1 describes a technology for making it possible to change the encoding unit of a quantization parameter by calculating a subblock size (hereinafter referred to as a quantization control size) that encodes the quantization parameter.

[0004] In recent years, activities for international standardization of a further high-efficiency encoding system as a successor version of the HEVC have started. The JVET (Joint Video Experts Team) is established between ISO IEC and ITU-T, and the standardization has been progressing as a VVC (Versatile Video Coding) encoding system (hereinafter referred to as VVC). In order to improve the encoding efficiency, in addition to the existing square subblock-based intra prediction and orthogonal transformation method, a rectangular subblock-based intra prediction and orthogonal transformation method are also studied.

[0005] For the VVC, not only a square subblock like the HEVC but also a rectangular subblock partition is studied. The quantization control size used as a reference to encode the quantization parameter in the HEVC is set assuming a square subblock. On the other hand, in the case where the rectangular subblock partition, which has been studied as the VVC, is performed, there is a case where it cannot be uniquely determined whether or not the quantization parameter is encoded. Therefore, the present application is considered to solve the above-described problem, and the object of the present application is to make it possible to appropriately control the encoding of the quantization parameter by using not only the square subblock but also the rectangular subblock.

[0006] Prior art documents

[0007] Patent documents

[0008] Patent Literature 1: Japanese Patent Application Laid-Open No. 2012-161074 SUMMARY

[0009] Solutions to the problem

[0010] To solve the above problem, an image encoding apparatus of the present application has the following structure. The image encoding apparatus divides an image into a plurality of sub-blocks and encodes the image for each of the divided sub-blocks, and includes an encoding means configured to encode a quantization parameter in a case where a smaller one of a horizontal size and a vertical size of a target sub-block to be processed among the plurality of sub-blocks is larger than or equal to a quantization control size serving as a sub-block size used to encode the quantization parameter.

[0011] Further, an image decoding apparatus of the present application has the following structure. The image decoding apparatus decodes an image composed of a plurality of sub-blocks for each of the sub-blocks, and decodes a quantization parameter in a case where a smaller one of a horizontal size and a vertical size of a target sub-block to be processed among the plurality of sub-blocks is larger than or equal to a quantization control size serving as a sub-block size used to decode the quantization parameter. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is a block diagram showing a structure of an image encoding apparatus in an embodiment.

[0013] Figure 2 is a block diagram showing a structure of an image decoding apparatus in an embodiment.

[0014] Figure 3 is a flowchart showing an image encoding process performed in the image encoding apparatus according to the embodiment.

[0015] Figure 4 is a flowchart showing an image decoding process performed in the image decoding apparatus according to the embodiment.

[0016] Figure 5 is a block diagram showing an example of a hardware structure of a computer suitable for the image encoding apparatus and the decoding apparatus of the embodiment.

[0017] Figure 6A is a diagram showing an example of a bitstream structure.

[0018] Figure 6B is a diagram showing an example of a bitstream structure.

[0019] Figure 7A is a diagram showing an example of sub-block division used in the embodiment.

[0020] Figure 7B FIG. 1 is a diagram illustrating an example of sub-block partitioning used in embodiments.

[0021] Figure 7C FIG. 1 is a diagram illustrating an example of sub-block partitioning used in embodiments.

[0022] Figure 7D FIG. 1 is a diagram illustrating an example of sub-block partitioning used in embodiments.

[0023] Figure 7E FIG. 1 is a diagram illustrating an example of sub-block partitioning used in embodiments.

[0024] Figure 7F FIG. 1 is a diagram illustrating an example of sub-block partitioning used in embodiments.

[0025] Figure 8A FIG. 1 is a diagram illustrating an example of sub-block partitioning used in embodiments.

[0026] Figure 8B FIG. 1 is a diagram illustrating an example of sub-block partitioning used in embodiments.

[0027] Figure 8C FIG. 1 is a diagram illustrating an example of sub-block partitioning used in embodiments.

[0028] Figure 8D FIG. 1 is a diagram illustrating an example of sub-block partitioning used in embodiments.

[0029] Figure 8E FIG. 1 is a diagram illustrating an example of sub-block partitioning used in embodiments.

[0030] Figure 8F FIG. 1 is a diagram illustrating an example of sub-block partitioning used in embodiments.

[0031] Figure 9A FIG. 1 is a diagram illustrating an example of sub-block partitioning used in embodiments.

[0032] Figure 9B FIG. 1 is a diagram illustrating an example of sub-block partitioning used in embodiments.

[0033] Figure 9Cis a graph illustrating a comparison between a quantization control size and a shorter one of a horizontal size and a vertical size of an object sub-block in an embodiment.

[0034] Figure 9D is a graph illustrating a comparison between a quantization control size and a shorter one of a horizontal size and a vertical size of an object sub-block in an embodiment.

[0035] Figure 9E is a graph illustrating a comparison between a quantization control size and a shorter one of a horizontal size and a vertical size of an object sub-block in an embodiment.

[0036] Figure 9F is a graph illustrating a comparison between a quantization control size and a shorter one of a horizontal size and a vertical size of an object sub-block in an embodiment.

[0037] Figure 10A is a graph illustrating a relationship between an encoding of a quantization parameter and a significant coefficient in an embodiment.

[0038] Figure 10B is a graph illustrating a relationship between an encoding of a quantization parameter and a significant coefficient in an embodiment.

[0039] Figure 10C is a graph illustrating a relationship between an encoding of a quantization parameter and a significant coefficient in an embodiment.

[0040] Figure 10D is a graph illustrating a relationship between an encoding of a quantization parameter and a significant coefficient in an embodiment.

[0041] Figure 10E is a graph illustrating a relationship between an encoding of a quantization parameter and a significant coefficient in an embodiment.

[0042] Figure 10F is a graph illustrating a relationship between an encoding of a quantization parameter and a significant coefficient in an embodiment.

[0043] Figure 11A is a graph illustrating a comparison between a quantization control size and a longer one of a horizontal size and a vertical size of an object sub-block in an embodiment.

[0044] Figure 11B is a graph illustrating a comparison between a quantization control size and a longer one of a horizontal size and a vertical size of an object sub-block in an embodiment.

[0045] Figure 11C is a graph illustrating a comparison between a quantization control size and a longer one of a horizontal size and a vertical size of an object sub-block in an embodiment.

[0046] Figure 11Dis a diagram illustrating a comparison between a quantization control size and a longer size of a horizontal size and a vertical size of an object sub-block in an embodiment.

[0047] Figure 11E is a diagram illustrating a comparison between a quantization control size and a longer size of a horizontal size and a vertical size of an object sub-block in an embodiment.

[0048] Figure 11F is a diagram illustrating a comparison between a quantization control size and a longer size of a horizontal size and a vertical size of an object sub-block in an embodiment.

[0049] Figure 12A is a diagram illustrating a comparison between a pixel count of a quantization control size and a pixel count of an object sub-block in an embodiment and in an embodiment.

[0050] Figure 12B is a diagram illustrating a comparison between a pixel count of a quantization control size and a pixel count of an object sub-block in an embodiment and in an embodiment.

[0051] Figure 12C is a diagram illustrating a comparison between a pixel count of a quantization control size and a pixel count of an object sub-block in an embodiment and in an embodiment.

[0052] Figure 12D is a diagram illustrating a comparison between a pixel count of a quantization control size and a pixel count of an object sub-block in an embodiment and in an embodiment.

[0053] Figure 12E is a diagram illustrating a comparison between a pixel count of a quantization control size and a pixel count of an object sub-block in an embodiment and in an embodiment.

[0054] Figure 12F is a diagram illustrating a comparison between a pixel count of a quantization control size and a pixel count of an object sub-block in an embodiment and in an embodiment.

[0055] Figure 13A is a diagram illustrating a depth of blocks in a sub-block partition used in an embodiment.

[0056] Figure 13B is a diagram illustrating a depth of blocks in a sub-block partition used in an embodiment.

[0057] Figure 13C is a diagram illustrating a depth of blocks in a sub-block partition used in an embodiment.

[0058] Figure 13D is a diagram illustrating a depth of blocks in a sub-block partition used in an embodiment.

[0059] Figure 13E is a diagram illustrating a depth of blocks in a sub-block partition used in an embodiment.

[0060] Figure 13F is a diagram showing a depth of a block in sub-block partitioning used in an embodiment.

[0061] Figure 14A is a diagram showing a comparison between a quantization control depth and a depth of an object sub-block in an embodiment.

[0062] Figure 14B is a diagram showing a comparison between a quantization control depth and a depth of an object sub-block in an embodiment.

[0063] Figure 14C is a diagram showing a comparison between a quantization control depth and a depth of an object sub-block in an embodiment.

[0064] Figure 14D is a diagram showing a comparison between a quantization control depth and a depth of an object sub-block in an embodiment.

[0065] Figure 14E is a diagram showing a comparison between a quantization control depth and a depth of an object sub-block in an embodiment.

[0066] Figure 15A is a diagram showing reading of a quantization parameter in a case where there is no significant coefficient in a quantization parameter sharing unit in an embodiment.

[0067] Figure 15B is a diagram showing reading of a quantization parameter in a case where there is no significant coefficient in a quantization parameter sharing unit in an embodiment.

[0068] Figure 16 is a flowchart of a quantization parameter encoding process using a quantization control depth.

[0069] Figure 17 is a flowchart of a quantization parameter decoding process using a quantization control depth.

[0070] Figure 18 is a flowchart of a quantization parameter encoding process using a quantization control size and a quantization control depth.

[0071] Figure 19 is a flowchart of a quantization parameter decoding process using a quantization control size and a quantization control depth. DETAILED DESCRIPTION

[0072] Hereinafter, the present application will be explained in detail based on suitable embodiments thereof with reference to the accompanying drawings. The structure to be explained in the following embodiments is merely one example, and the present application is not limited to the example structure.

[0073] Figure 1 is a block diagram showing an image encoding apparatus of the present embodiment. In the present embodiment, the image encoding apparatus 1 is configured to include a Figure 1In the present embodiment, the control section (control unit) 100 is a processor that controls the entire image encoding apparatus, and the terminal 101 is an input terminal for inputting image data.

[0074] The block division section (block division unit) 102 divides the input image into a plurality of elementary blocks, and outputs the image in units of elementary blocks to the subsequent stage.

[0075] The generation section (generation unit) 103 generates, for example, information on a size (quantization control size) for encoding a quantization parameter, and outputs the information. The method of generating the information on the quantization control size is not limited. The quantization control size can be input by a user, can be calculated in accordance with the characteristics of the input image, or can use a quantization control size that is specified in advance as an initial value.

[0076] The prediction section (prediction unit) 104 generates sub-blocks by dividing each elementary block, performs intra prediction as intra-frame prediction, inter prediction as inter-frame prediction, and the like in units of sub-blocks, and generates prediction image data. In addition, prediction errors are calculated from the image data indicating the input pixel values and the prediction image data, and the prediction errors are output. Information required for prediction, such as information on sub-block division, prediction mode, and motion vector, is also output together with the prediction errors. Hereinafter, the information required for prediction is referred to as prediction information.

[0077] The transform and quantization section (transform and quantization unit) 105 orthogonally transforms the residual representing the prediction error in units of sub-blocks, further performs quantization, and obtains residual coefficients representing the residual. The quantization parameter is a parameter for quantizing the transform coefficients obtained by the orthogonal transformation.

[0078] The inverse quantization and inverse transform section (inverse quantization and inverse transform unit) 106 reproduces the transform coefficients by inverse quantizing the residual coefficients output from the transform and quantization section 105, and reproduces the prediction error by applying inverse orthogonal transformation to the transform coefficients.

[0079] The frame memory 108 is a memory that stores the reproduced image data.

[0080] The image reproduction section (image reproduction unit) 107 generates prediction image data by reading the frame memory 108 as necessary using the prediction information output from the prediction section 104, and generates reproduction image data from the prediction image data and the input prediction error.

[0081] The loop filter section (loop filter unit) 109 applies loop filtering such as deblocking filtering and sample adaptive offset to the reproduced image.

[0082] The encoding section (encoding unit) 110 generates code data by encoding the residual coefficients output from the transform and quantization section 105 and the prediction information output from the prediction section 104.

[0083] The integrated encoding section (integrated encoding unit) 111 generates header code data by encoding the information on the quantization control size from the generation section 103. The integrated encoding section 111 forms a bitstream by further combining the header code data with the code data output from the encoding section 110. The terminal 112 is an output terminal for outputting the bitstream generated by the integrated encoding section 111 to the outside.

[0084] The image encoding operation in the image encoding apparatus will be described below. In the present embodiment, moving image data is input in units of frames. Alternatively, still image data of 1 frame can be input.

[0085] The image data of 1 frame input from the terminal 101 is input to the block division section 102.

[0086] The block division section 102 divides the input image data into a plurality of basic blocks, and outputs the image in units of basic blocks to the prediction section 104.

[0087] The prediction section 104 performs prediction processing on the image data input from the block division section 102. Specifically, the prediction section 104 first determines a sub-block division for dividing a basic block into further smaller sub-blocks.

[0088] Figures 7A to 7F An example of the type of division of a basic block is shown. A thick frame represented by 700 indicates a basic block. For ease of explanation, it is assumed that the size of each basic block is configured to be 32 x 32 pixels, and the rectangles in the thick frame indicate sub-blocks. Figure 7B An example of a square sub-block obtained by division is shown, and a basic block of 32 x 32 pixels is divided into sub-blocks of 16 x 16 pixels. On the other hand, Figures 7C to 7F An example of the type of rectangular sub-block obtained by division is shown. In Figure 7C , a basic block is divided into 16 x 32 pixel oblong rectangular sub-blocks. In Figure 7D , a basic block is divided into 32 x 16 pixel horizontal long sub-blocks. In Figure 7E , and Figure 7FIn this embodiment, the basic block is divided into rectangular sub-blocks in a ratio of 1:2:1. Thus, in this embodiment, the encoding process is performed using not only square sub-blocks but also rectangular sub-blocks. In this embodiment, information about this division type of the basic block is encoded as division information. In addition, in order to obtain the hierarchical structure of sub-blocks as shown on the left in FIGS. 25, 26, and 27, the information about the division type is hierarchized and encoded. Figure 15A and 15B

[0089] The prediction section 104 determines a prediction mode for each sub-block to be processed. Specifically, the prediction section 104 determines a prediction mode in units of sub-blocks, such as intra prediction using already encoded pixels in the same frame as the frame including each sub-block to be processed, or inter prediction using pixels of another already encoded frame, and the like. The prediction section 104 generates predicted image data from the determined prediction mode and the already encoded pixels, further generates prediction errors from the input image data and the predicted image data, and outputs these prediction errors to the transform and quantization section 105. The prediction section 104 also outputs information about the sub-block division and the prediction mode, and the like, as prediction information to the encoding section 110 and the image reproduction section 107.

[0090] Here, the transform process and the quantization process performed by the transform and quantization section 105 will be described more specifically. The transform and quantization section 105 applies frequency transform to the image data (pixel values) of the sub-blocks on which the prediction process has been performed by the prediction section 104, and further quantizes the image data. Figures 8A to 8F The relationship between the type of division of the block and the quantization control size is shown. The transform and quantization section 105 determines the unit of sub-blocks sharing and encoding the quantization parameter by using the object sub-block size and the quantization control size output from the generation section 103. In other words, in the subsequent encoding section 110, it is determined whether the quantization parameter should be shared between a plurality of sub-blocks according to the comparison between the quantization control size and the size of each sub-block. The encoding of the quantization parameter will be described later. The method of determining the value of the quantization parameter itself used for quantization is not limited. The user can input the quantization parameter, the quantization parameter can be calculated according to the characteristics of the input image, or a quantization parameter specified in advance as an initial value can be used.

[0091] Next, the method of determining the unit of encoding the quantization parameter will be described.

[0092] The transform and quantization section 105 compares the length of the shorter side of the horizontal side and the vertical side of each object sub-block with the quantization control size, and determines the unit of encoding the quantization parameter, that is, the unit using the same quantization parameter.

[0093] Figures 8A to 8F ​A unit in which a quantization parameter is encoded in a case where the length of the shorter side among the horizontal side and the vertical side of each object sub-block is compared with a quantization control size is shown. In Figures 8A to 8F , the length of one side of a square block is defined as the quantization control size. Specifically, Figures 8A to 8F An example in which 16 is applied as the quantization control size is shown. Figures 8A to 8F The shorter length among the horizontal length and the vertical length of each object sub-block is indicated with an arrow. Figures 8A to 8F The thick frame rectangle of the object sub-block in indicates a region that shares the quantization parameter determined as a result of comparison between each object sub-block and the quantization control size. Qp indicates the quantization parameter. For Figure 8A , Figure 8B , Figure 8C and Figure 8D , the shorter length among the horizontal length and the vertical length of each object sub-block is greater than or equal to the quantization control size (16). For this reason, quantization of each object sub-block is performed by using a corresponding quantization parameter among the respective quantization parameters (QpA to QpD). The corresponding quantization parameter among the quantization parameters (QpA to QpD) is encoded for each sub-block. On the other hand, for Figure 8E and Figure 8F , the sub-blocks whose shorter side among the horizontal side and the vertical side has a length smaller than the quantization control size are included in the basic block to be processed, and thus share the quantization parameter among the plurality of sub-blocks. Specifically, for Figure 8E and Figure 8F each, quantization of three sub-blocks is performed by using the same quantization parameter. At this time, for the quantization parameter to be encoded, one quantization parameter is not encoded for each sub-block but as a common quantization parameter. As described above, the quantization parameter is used in accordance with the size of each object sub-block and the quantization control size.

[0094] Next, an example of a case where the quantization control size is different from that of Figures 9A to 9F will be described with reference to Figures 8A to 8F . Also in Figures 9A to 9F , the length of one side of a square block is defined as the quantization control size. Specifically, Figures 9A to 9F An example in which a length 32 identical to the length of one side of the basic block to be processed is applied as the quantization control size is shown. Figures 9A to 9F The meanings of the thick frame, the arrow, and Qp in are similar to those in Figures 8A to 8F , and thus the description is omitted. In Figures 9A to 9FWhen the shorter length of the horizontal length and the vertical length of each object sub-block is compared with the quantization control size, any object sub-block is equal to or smaller than the quantization control size. For this reason, in each case, the quantization of the sub-block is performed by using the same quantization parameter. For the quantization parameter to be coded, the quantization parameter is not coded for each sub-block but as a common quantization parameter.

[0095] Referring back to Figure 1 The inverse quantization and inverse transform section 106 reproduces the transform coefficients by inverse quantizing the input residual coefficients, further reproduces the prediction errors by applying an inverse orthogonal transform to the reproduced transform coefficients, and outputs these prediction errors to the image reproduction section 107. In the inverse quantization processing of each sub-block, the same quantization parameter as that used in the transform and quantization section 105 is used.

[0096] The image reproduction section 107 reproduces the prediction image as necessary by using the prediction information input from the prediction section 104 to read out the frame memory 108. The image reproduction section 107 reproduces the image data from the reproduced prediction image and the reproduced prediction errors input from the inverse quantization and inverse transform section 106, inputs this image data to the frame memory 108, and stores this image data.

[0097] The loop filter section 109 reads out the reproduced image from the frame memory 108, and applies loop filtering such as deblocking filtering to this reproduced image. The loop filtering is performed in accordance with the prediction mode of the prediction section 104, the value of the quantization parameter used in the transform and quantization section 105, and whether or not there is a non-zero value (hereinafter referred to as a significant coefficient) in each processed sub-block after quantization, or in accordance with the sub-block division information. The loop filter section 109 inputs the filtered image again to the frame memory 108, and stores this image again.

[0098] The encoding section 110 entropy-encodes the residual coefficients generated by the transform and quantization section 105 and the prediction information input from the prediction section 104 in units of blocks, and generates code data.

[0099] No method of entropy coding is specified. Golomb coding, arithmetic coding, or Huffman coding, etc. can be used. The generated code data is output to the integrated coding section 111. When encoding the quantization parameters constituting the quantization information, an identifier indicating the difference value between the quantization parameter of the sub-block to be encoded and the quantization parameter of the sub-block encoded before that sub-block is encoded. In the present embodiment, the difference value between the quantization parameter of the sub-block immediately preceding in the encoding order as the prediction value is calculated; however, the prediction value of the quantization parameter is not limited thereto. The quantization parameter of the sub-block adjacent to the left or right of the sub-block can be used as the prediction value, or a value calculated from the quantization parameters of a plurality of sub-blocks such as an average value can be used as the prediction value.

[0100] Here, the processing of encoding the quantization parameters according to the quantization control size will be further explained with reference to Figures 10A to 10F Figures 10A to 10F The left side of each of the above shows the block partition type and the quantization parameter (Qp) used in each sub-block. The sub-blocks with diagonal hatching indicate the sub-blocks associated with the quantization parameter to be encoded. The thick frame rectangle indicates the region sharing the quantization parameter determined according to the quantization control size and the size of each object sub-block. Figures 10A to 10F The middle of each of the above indicates whether each sub-block has an important coefficient. An important coefficient means a non-zero coefficient among the residual coefficients after the transformation and quantization. In other words, having an important coefficient means that there is at least one non-zero residual coefficient in the sub-block after the transformation and quantization. Figures 10A to 10F The arrow shown in the right side of each of the above indicates the order of encoding (decoding). In the present embodiment, in the region sharing the quantization parameter, the quantization parameter is associated with and encoded in the first sub-block including the important coefficient in the encoding order. For example, in Figure 10B , the first sub-block including the important coefficient in the encoding order is the upper right sub-block, and therefore the quantization parameter associated with that sub-block is encoded. In this case, since the quantization parameter has been encoded in the upper right sub-block in the quantization parameter encoding unit, the quantization parameter is not encoded in the lower left and right sub-blocks. On the other hand, in the quantization and inverse quantization processing of the lower left and right sub-blocks, QpA which is the same quantization parameter as that of the upper right sub-block is used. Further, there is no important coefficient in the upper left sub-block, and therefore the quantization processing is not performed; however, QpA which is the same quantization parameter as that of the upper right sub-block is used in the processing using the quantization parameter such as the deblocking filter, etc. In Figure 10F , the first sub-block including the important coefficient in the encoding order is the sub-block located below, and therefore the quantization parameter is associated with and encoded in that sub-block, and for the upper and middle sub-blocks, the quantization parameter is not encoded. However, in Figure 10F ​the upper and middle sub-blocks in Figure 10B In the upper left sub-block in

[0101] Thus, in the sub-blocks in the region sharing the quantization parameter determined in accordance with the quantization control size, the quantization parameter is associated with and coded in the first sub-block including the important coefficient in the coding order.

[0102] In the integration coding section 111, information on the quantization control size is coded. The method of coding is not specified. Columbus coding, arithmetic coding, or Huffman coding, or the like can be used. A bit stream is formed by multiplexing these code data and the like input from the coding section 110. Finally, the bit stream is output to the outside from the terminal 112.

[0103] Figure 6A An example of a bit stream including the coded information on the quantization control size is shown. The information on the quantization control size is included in any of the headers of a sequence and a picture, or the like. In the present embodiment, it is assumed that the information on the quantization control size is included in the header of a picture as shown in Figure 6A However, the position to which the information on the quantization control size is coded is not limited to this, and can be included in the header of a sequence as shown in Figure 6B

[0104] Figure 3 is a flowchart showing the coding process performed in the image coding apparatus according to the present embodiment.

[0105] Initially, in step S301, the block division section 102 divides the image input in units of frames into images in units of basic blocks.

[0106] In step S302, the generation section 103 determines the quantization control size as the size for coding the quantization parameter. Then, this information is set as the quantization control size information. The quantization control size information is also coded by the integration coding section 111.

[0107] In step S303, the prediction section 104 generates sub-blocks by performing division processing on the image data generated in units of basic blocks in step S301. The prediction section 104 performs prediction processing on each of the generated sub-blocks, and generates prediction information such as block division and prediction mode, and prediction image data. In addition, the prediction error is calculated from the input image data and the prediction image data.

[0108] ​In step S304, the transform and quantization section 105 generates transform coefficients by applying orthogonal transform to the prediction error calculated in step S303. The transform and quantization section 105 further generates residual coefficients by using the quantization parameters determined in accordance with the quantization control size information generated in step S302. Specifically, as described above, whether the quantization parameters are shared between the sub-blocks in the base block is determined by comparing between the quantization control size information (e.g., the length of one side of the square block) and the size of each sub-block (e.g., the length of the short side or the long side). In accordance with the determination, the quantization of each sub-block is performed by using the quantization parameters associated with the sub-blocks in each region, and the residual coefficients of each sub-block are generated.

[0109] In step S305, the inverse quantization and inverse transform section 106 reproduces the prediction error by applying inverse quantization and inverse orthogonal transform to the residual coefficients generated in step S304. The same quantization parameters as those used in step S304 are used in the inverse quantization process in this step.

[0110] In step S306, the image reproduction section 107 reproduces the prediction image in accordance with the prediction information generated in step S303. The image reproduction section 107 further reproduces the image data in accordance with the reproduced prediction image and the prediction error generated in step S305.

[0111] In step S307, the encoding section 110 encodes the prediction information generated in step S303 and the residual coefficients generated in step S304 together with the block division information, and generates code data. The encoding section 110 also encodes the quantization parameters used in step S304 in accordance with the quantization control size information generated in step S302. The encoding section 110 generates a bit stream by further including other code data. Specifically, within each region in which the quantization parameters determined in step S304 are shared, the quantization parameters are encoded in association with the sub-blocks including at least one significant coefficient in the order of the sub-blocks to be encoded.

[0112] In step S308, the control section 100 of the image encoding apparatus judges whether the encoding of all the base blocks in the frame is completed, and when the encoding is completed, the process proceeds to step S309; otherwise, the process returns to step S303 for the next base block.

[0113] In step S309, the loop filter section 109 applies loop filtering to the image data reproduced in step S306, generates a filtered image, and ends the process.

[0114] Thus, in particular, the quantization control size information is generated in step S302, and the quantization and encoding processes are performed in accordance with the quantization control size information in steps S304 and S307, so that the quantization parameter encoding process can be appropriately enabled. As a result, the quality of the encoded image is improved while the data amount of the generated bitstream as a whole is suppressed.

[0115] In the present embodiment, the region sharing the quantization parameter is determined by comparing the shorter length between the horizontal length and the vertical length of each object sub-block with the quantization control size; however, the present application is not limited to this. For example, as shown in Figures 11A to 11F the sharing unit of the quantization parameter can be determined by comparing the longer length between the horizontal length and the vertical length of each object sub-block. In Figures 11A to 11F , the length of the long side of each sub-block is the object to be compared with the length of one side of the square block as the quantization control size (16). In Figures 11A to 11F , since the long side of all the sub-blocks is longer than the quantization control size (16), the quantization parameter is encoded for each sub-block. With this configuration, for the rectangular sub-blocks, a bitstream that does not emphasize the reduction of the code amount of the quantization parameter but emphasizes the fine control of the quantization parameter can be generated.

[0116] Further, as another embodiment, the region sharing the quantization parameter can be determined by comparing the pixel count of each object sub-block with the pixel count of the quantization control size. Figures 12A to 12F A case where the pixel count of each object sub-block is compared with the pixel count of the quantization control size is shown. In Figures 12A to 12F , the quantization control size is 16 x 16 pixels, so the pixel count is 256. For the object sub-blocks, in Figures 12A to 12F all the sub-blocks, the pixel count is greater than or equal to 256 pixels. Thus, in Figures 12A to 12F the example, the quantization parameter is encoded in each of all the sub-blocks. With this configuration, the quantization parameter control based on the pixel count in each sub-block can be achieved regardless of the shape of each sub-block.

[0117] In this embodiment, the quantization control size is assumed to be one side of a square block. Optionally, the quantization control size can be one side of a rectangular block. In this case, the width and height of the quantization control size can be specified. In this case, the vertical and horizontal lengths of each object sub-block can be compared with the width and height of the quantization control size, respectively. If two or one of these lengths is greater than or equal to the quantization control size, the quantization parameter can be encoded for each sub-block. Using this structure, different quantization parameter controls can be implemented for vertically elongated rectangular sub-blocks and horizontally elongated rectangular sub-blocks. If there are two sub-blocks in the basic block to be processed whose lengths are both less than the quantization control size, a quantization parameter to be shared among the sub-blocks that meet this condition is encoded.

[0118] In this embodiment, the unit for encoding the quantization parameters is defined by the space size; however, the structure is not limited to this. A quantization control depth (hereinafter referred to as quantization control depth) indicating the number of times a basic block is segmented can be generated, and whether to encode the quantization parameters can be determined by comparing the quantization control depth with the segmentation depth of each object sub-block. In this case, instead of Figure 6A and 6B The quantization control size information shown is used to encode the quantization control depth information.

[0119] Here, we will refer to Figures 13A to 13F This illustrates the depth of segmentation for each object's sub-blocks. Figures 13A to 13F In the diagram, D0 and D1 represent depth 0 and depth 1, respectively. Figure 13A This shows that the basic block has not been divided once and the depth of the sub-block is 0 (D0). Figure 13B This shows that the basic block has been divided into four parts and each sub-block has a depth of 1 (D1). Figure 13C and Figure 13D This shows that the basic block has been divided into two parts and each sub-block has a depth of 1 (D1). Figure 13E and Figure 13F This shows that the basic block has been divided into three parts and each sub-block has a depth of 1 (D1). Thus, the depth of each sub-block of a basic block that has been divided once increases by 1 in any of the four-part, two-part, and three-part divisions.

[0120] Next, refer to Figures 14A to 14E This explains how to encode quantization parameters based on the quantization control depth and the segmentation depth of each object's sub-blocks. Figures 14A to 14E The outermost square is shown as the basic block. Figures 14A to 14EIn each, the left side shows the split sub-blocks, and the right side shows the respective regions that share quantization parameters. D0, D1, D2, D3, or D4 within each block in these figures indicates the depth of the sub-block. For example, D0 indicates depth 0, and D4 indicates depth 4. Qp denotes quantization parameter. Figure 14A , Figure 14B , Figure 14C , Figure 14D and Figure 14E show the cases where the quantization control depth is 0, 1, 2, 3, and 4, respectively. In the case of Figure 14A , i.e., when the quantization control depth is 0, a common quantization parameter is used for all sub-blocks in this figure, and this quantization parameter is encoded in association with the first sub-block including the significant coefficient in the encoding order. In this case, the number of quantization parameters to be encoded is 1. In the case of Figure 14B , i.e., when the quantization control depth is 1, quantization parameters are shared in units of the blocks shown on the right side of Figure 14B . In addition, one quantization parameter is encoded in association with the first sub-block including the significant coefficient in the encoding order in units of blocks. In this case, the number of quantization parameters to be encoded is 4. In the case of Figure 14C , i.e., when the quantization control depth is 2, quantization parameters are shared in units of the blocks shown on the right side in this figure, and one quantization parameter is encoded in association with the first sub-block including the significant coefficient in the encoding order in these units. In this case, the number of quantization parameters to be encoded is 11. In the case of Figure 14D , i.e., when the quantization control depth is 3, quantization parameters are shared in units of the blocks shown on the right side of Figure 14D , and one quantization parameter is encoded in association with the first sub-block including the significant coefficient in the encoding order in these units. In this case, the number of quantization parameters to be encoded is 23. In the case of Figure 14E , i.e., when the quantization control depth is 4, quantization parameters are used in units of the blocks shown on the right side of Figure 14D . In the case of Figure 14E , the depth of the split of the blocks is equal to the quantization control depth, and thus the quantization parameters are encoded for each block. In this case, the number of quantization parameters to be encoded is 27.

[0121] Here, the case where there is a sub-block that does not include a significant coefficient in a region that shares quantization parameters will be described with reference to Figure 15A and 15B . Figure 15A shows the sub-block split as in Figures 14A to 14E , and D0, D1, D2, D3, or D4 indicates the depth of each sub-block. Figure 15B shows the sub-block split as in Figures 14A to 14EThe case where the quantization control depth is 2 is shown as an example. Figure 15B The case where the quantization control depth is 2 is shown as an example. Figure 15B The case where there is no significant coefficient in all three sub-blocks within the region associated with the quantization parameter QpG is shown as an example. In this case, the quantization parameter QpG of the three sub-blocks is not encoded. However, the same value as the immediately preceding encoded quantization parameter (i.e., QpF) is used for processing such as deblocking filtering. In the quantization parameter encoding unit where there is no significant coefficient, the quantization parameter encoded immediately preceding in the encoding order is used in processing using the quantization parameter; however, the present application is not limited to this. For example, QpD, which is the quantization parameter of the upper neighboring quantization parameter encoding unit, can be used, or QpF, which is the left neighboring quantization parameter, can be used. Alternatively, a value calculated from the quantization parameters of a plurality of quantization parameter encoding units, such as an average value, can be used. Alternatively, an initial value of the quantization parameter for a slice can be used. A slice means a unit of division of a frame, and is composed of at least one or a plurality of elementary blocks. In this way, the depth of each object sub-block can be compared with the quantization control depth, and in the case where the depth of each object sub-block is less than or equal to the quantization control depth, the quantization parameter can be shared. Since the sub-block division information is encoded each time the division is performed, the quantization parameter encoding control highly compatible with the sub-block division information can be achieved, and as a result, the structure of the syntax is simplified.

[0122] Figure 16 is a flowchart showing the quantization parameter encoding process using the quantization control depth.

[0123] In step S1601, the transform and quantization section 105 compares the quantization control depth with the depths of the divisions of the sub-blocks.

[0124] In step S1602, the transform and quantization section 105 determines the region of the sub-blocks whose divided depths are greater than the quantization control depth as a result of the comparison in S1601 as a region sharing one quantization parameter.

[0125] In step S1603, the transform and quantization section 105 quantizes the sub-blocks in the determined region by using the same quantization parameter.

[0126] In step S1604, the encoding section 110 encodes the quantization parameter used in step S1603 in association with the sub-blocks within the determined region that have at least one significant coefficient as a residual coefficient in the encoding order. The integration encoding section 111 encodes the quantization control depth information.

[0127] Each elementary block in a frame is divided into sub-blocks Figure 16processing.

[0128] Further, both the quantization control size and the quantization control depth can be generated and used in combination. In this case, in addition to the quantization control size information as shown in Figure 6A and 6B quantization control depth information is encoded in addition to the quantization control size information as shown in

[0129] In a case where the depth of each of the object sub-blocks is less than or equal to the quantization control depth and the length of the shorter side of the horizontal side and the vertical side of each of the object sub-blocks is greater than or equal to the quantization control size, the quantization parameter can be encoded for each of the sub-blocks. Specifically, the length of the shorter side of each of the sub-blocks is compared with the quantization control size for each of the regions associated with the quantization control depth. In each of the regions, in a case where there is a sub-block whose length of the shorter side is greater than or equal to the quantization control size, the quantization parameter is encoded for each of the sub-blocks within the region. In a case where the length of the shorter side of each of the sub-blocks within the region is not greater than or equal to the quantization control size, one quantization parameter is shared between the sub-blocks within the region, and the one quantization parameter is encoded.

[0130] Optionally, in a case where the depth of each of the object sub-blocks is less than or equal to the quantization control depth and the length of the longer side of the horizontal side and the vertical side of each of the object sub-blocks is greater than or equal to the quantization control size, the quantization parameter can be encoded for each of the sub-blocks. In this case, the size of each of the sub-blocks is compared with the quantization control size for each of the regions associated with the quantization control depth. In each of the regions, in a case where there is a sub-block whose length of the longer side is greater than or equal to the quantization control size, the quantization parameter is encoded for each of the sub-blocks within the region. In a case where the length of the longer side of each of the sub-blocks within the region is not greater than or equal to the quantization control size, one quantization parameter is shared between the sub-blocks within the region, and the one quantization parameter is encoded.

[0131] Optionally, in a case where the depth of each of the object sub-blocks is less than or equal to the quantization control depth and the pixel count of each of the object sub-blocks is greater than or equal to the pixel count of the quantization control size, the quantization parameter can be encoded for each of the sub-blocks. In this case, the size of each of the sub-blocks is compared with the quantization control size for each of the regions associated with the quantization control depth. In each of the regions, in a case where there is a sub-block whose pixel count is greater than or equal to the pixel count of the quantization control size, the quantization parameter is encoded for each of the sub-blocks within the region. In a case where the pixel count of each of the sub-blocks within the region is not greater than or equal to the pixel count of the quantization control size, one quantization parameter is shared between the sub-blocks within the region, and the one quantization parameter is encoded.

[0132] Optionally, the width and height of the quantization control size can be specified and used in combination with the quantization control depth. In this case, the vertical length and the horizontal length of each sub-block are compared with the width and height of the quantization control size, respectively, for each region associated with the quantization control depth. In this case, in the case where there are two or one of the vertical length and the horizontal length of the sub-blocks that are greater than or equal to the width and height of the quantization control size, the quantization parameter is encoded for each sub-block within the object region. In the case where there are no two or one of the vertical length and the horizontal length of the sub-blocks that are greater than or equal to the width and height of the quantization control size, one quantization parameter is shared in the region, and the one quantization parameter is encoded.

[0133] In this way, in the case of using rectangular sub-block division multiple times, by using not only the quantization control depth but also the quantization control size, it is possible to achieve quantization parameter encoding control even in extremely thin rectangular sub-blocks.

[0134] Figure 18 is a flowchart showing the quantization parameter encoding process by using both the quantization control size and the quantization control depth.

[0135] In step S1801, the transform and quantization section 105 compares the quantization control depth with the depth of the division of the sub-blocks, and determines a region having a block depth associated with the quantization control depth.

[0136] In step S1802, the transform and quantization section 105 compares the size of each sub-block included in the region determined in step S1801 with the quantization control size, for each region determined in step S1801.

[0137] In step S1803, as a result of the comparison in step S1802, the transform and quantization section 105 determines whether one quantization parameter is shared between the sub-blocks within the object region. According to the determination, the quantization of each sub-block is performed by using the quantization parameter associated with the sub-block in each region, and the residual coefficients of each sub-block are generated.

[0138] In step S1804, in the case where one quantization parameter is shared between the sub-blocks in the object region, the encoding section 110 encodes the quantization parameter in association with the first sub-block having at least one significant coefficient in the encoding order. On the other hand, the quantization parameter is not encoded in association with another sub-block. In the case where one quantization parameter is not shared between the sub-blocks in the object region, the encoding section 110 encodes the quantization parameter in association with each sub-block except for the sub-blocks not including the significant coefficient. The integration encoding section 111 encodes the quantization control size information and the quantization control depth information.

[0139] Such an encoding process is performed for each basic block.

[0140] In the case where the quantization parameter encoding process is performed by using both the quantization control size and the quantization control depth, the quantization parameter is more likely to be associated with each sub-block when the long side of each sub-block is compared with the quantization control size. In other words, the manner of comparing the long side of each sub-block with the quantization control size is more suitable for an object having an elongated shape.

[0141] In the case where the type of the division of the sub-blocks is consecutive in the hierarchy, the sharing of the quantization parameter can be prohibited regardless of the comparison result between the quantization control size and the size of each sub-block. With this configuration, the quantization of the sub-blocks suitable for an object having an elongated shape can be performed.

[0142] Figure 2 is a block diagram showing the configuration of an image decoding apparatus. In the present embodiment, for example, the decoding of the encoded data generated in the image encoding apparatus shown in Figure 1

[0143] The terminal 201 is an input terminal that inputs an encoded bitstream.

[0144] The separation and decoding section (separation and decoding unit) 202 separates the information related to the decoding process and the code data related to the residual coefficients from the bitstream, and decodes the code data in the header of the bitstream. In the present embodiment, the separation and decoding section 202 decodes the quantization control size information, and outputs the quantization control size information to the subsequent stage. The separation and decoding section 202 performs the inverse operation of the integration encoding section 111 shown in Figure 1

[0145] The decoding section (decoding unit) 203 acquires the residual coefficients and the prediction information from the code data output from the separation and decoding section 202.

[0146] The inverse quantization and inverse transform section (inverse quantization and inverse transform unit) 204 performs inverse quantization on the residual coefficients input in units of blocks, further applies inverse orthogonal transform, and acquires the prediction error.

[0147] The frame memory 206 is a memory that stores the image data of the reproduced picture.

[0148] The image reproduction section (image reproduction unit) 205 generates prediction image data by reading the frame memory 206 as necessary using the input prediction information. The image reproduction section 205 generates reproduction image data from the prediction image data and the prediction error reproduced by the inverse quantization and inverse transform section 204, and outputs the reproduction image data.

[0149] The loop filter section (loop filter unit) is 207. The loop filter section 207 performs loop filtering on the reproduction image data output from the image reproduction section 205. Figure 1 ​​The loop filter section 109 shown applies loop filtering such as deblocking filtering to the reproduced image and outputs a filtered image.

[0150] The terminal 208 is an output terminal that outputs the reproduced image data to the outside.

[0151] The image decoding operation in the image decoding apparatus will be described below. In the present embodiment, the bitstream generated in the present embodiment is decoded.

[0152] In Figure 2 The control section (control unit) 200 is a processor that controls the entire image decoding apparatus, and the bitstream input from the terminal 201 is input to the separation and decoding section 202. The separation and decoding section 202 separates information related to the decoding process and code data related to the coefficients from the bitstream, and decodes the code data in the header of the bitstream. Specifically, the decoding section 202 decodes the quantization control size information. In the present embodiment, first, the quantization control size information of the picture header of the bitstream shown is decoded. The quantization control size information thus obtained is output to the decoding section 203 and the inverse quantization and inverse transform section 204. Further, the code data in units of blocks of picture data is output to the decoding section 203. Figure 6A The decoding section 203 decodes the code data and acquires the residual coefficients, the prediction information, and the quantization parameter. The residual coefficients and the quantization parameter are output to the inverse quantization and inverse transform section 204, and the acquired prediction information is output to the image reproduction section 205.

[0153] The decoding section 203 decodes the code data and acquires the residual coefficients, the prediction information, and the quantization parameter. The residual coefficients and the quantization parameter are output to the inverse quantization and inverse transform section 204, and the acquired prediction information is output to the image reproduction section 205.

[0154] Here, the process of assigning the quantization parameter to the sub-blocks according to the quantization control size will be described with reference to Figures 10A to 10F In each of the Figures 10A to 10F In each of the left side shows the block partition type and the quantization parameter (Qp) used during encoding in each sub-block. The sub-blocks with diagonal hatching indicate the sub-blocks associated with the quantization parameter. The thick frame rectangle indicates the region sharing the quantization parameter determined according to the quantization control size and each object sub-block size. The method of comparing with the quantization control size is similar to that of the image encoding apparatus. For example, as described with reference to Figures 8A to 8F and Figures 9A to 9F The shorter side of the horizontal side and the vertical side of each object sub-block is compared with the quantization control size. The middle diagram indicates whether each sub-block has an important coefficient. The important coefficient means a non-zero coefficient among the residual coefficients. In other words, having an important coefficient means that there is at least one non-zero residual coefficient in the sub-block. The arrow shown on the right side of the diagram indicates the order of decoding. In the sub-blocks within the quantization parameter encoding unit, the quantization parameter is decoded in the first sub-block including the important coefficient in the decoding order. For example, in Figure 10BIn the decoding sequence, the first sub-block including the important coefficients is the top-right sub-block, so the quantization parameters are decoded for this sub-block. In this case, the bottom-left and bottom-right sub-blocks are in regions sharing quantization parameters, and the quantization parameters in the top-right sub-block have already been decoded. In other words, there is no encoded data for the quantization parameters associated with the bottom-left and bottom-right sub-blocks in the bitstream, and the quantization parameters associated with the bottom-left and bottom-right sub-blocks are not decoded. In the quantization and inverse quantization processing in the bottom-left and bottom-right sub-blocks, QpA, which is the same quantization parameter as that in the top-right sub-block, is used. Furthermore, there are no important coefficients in the top-left sub-block, so no quantization processing is performed; however, in processes using quantization parameters, such as deblocking filtering, QpA, which is the same quantization parameter as that in the top-right sub-block, is used. Figure 10F In the decoding sequence, the first sub-block containing the important coefficients is the sub-block below it, therefore the quantization parameters associated with this sub-block are decoded. There is no [interchangeable] ... Figure 10F The encoded data of the quantization parameters associated with the upper and middle sub-blocks is used, but the quantization parameters associated with the upper and middle sub-blocks are not decoded. However, in Figure 10F In the sub-blocks above and in the middle, with Figure 10B Similar to the upper left sub-block, QpA, which is the same quantization parameter as the quantization parameter of the lower sub-block, is used in processing involving quantization parameters, such as deblocking filtering. Thus, within the sub-block of the quantization parameter encoding unit determined according to the quantization control size, the quantization parameters are decoded for the first sub-block, including the important coefficients, in the decoding order.

[0155] The inverse quantization and inverse transform unit 204 generates orthogonal transform coefficients by inverse quantizing the input residual coefficients, further applies the inverse orthogonal transform, and reproduces the prediction error. In the inverse quantization of each sub-block, inverse quantization is performed using common quantization parameters for each region sharing quantization parameters. The acquired prediction information is output to the image reproduction unit 205.

[0156] The image reproduction unit 205 reproduces the predicted image by reading the frame memory 206 as needed using the prediction information input from the decoding unit 203. The image reproduction unit 205 reproduces image data based on the predicted image and the prediction error input from the inverse quantization and inverse transform unit 204, inputs this image data into the frame memory 206, and stores the image data in the frame memory 206. The stored image data is used as a reference during prediction.

[0157] Loop filter section (loop filter unit) 207 and Figure 1Similar to step 109, the reconstructed image is read from frame memory 206, and loop filtering, such as deblocking filtering and sample adaptive compensation, is applied to the reconstructed image. The filtered image is then input back into frame memory 206.

[0158] The reproduced image stored in frame memory 206 is eventually output to the outside from terminal 208.

[0159] Figure 4 This is a flowchart illustrating the image decoding process performed in an image decoding device.

[0160] First, in step S401, the separation and decoding unit 202 separates information related to the decoding process and code data related to the coefficients from the bit stream, decodes the code data in the header, and obtains the quantization control size information.

[0161] In step S402, the decoding unit 203 decodes the code data separated in step S401 and obtains block segmentation information, residual coefficients, prediction information and quantization parameters.

[0162] In step S403, the inverse quantization and inverse transform unit 204 performs inverse quantization on a sub-block basis, further applies an inverse orthogonal transform, and obtains the prediction error. Specifically, the quantization control size information (e.g., the length of one side of a square block) is compared with the size of each sub-block (e.g., the length of the short or long side) determined based on the obtained block segmentation information. As a result, regions (sub-blocks) sharing quantization parameters are determined. Inverse quantization is performed using quantization parameters assigned to the sub-blocks respectively.

[0163] In step S404, the image reproduction unit 205 reproduces the predicted image based on the prediction information obtained in step S402. The image reproduction unit 205 further reproduces image data based on the reproduced predicted image and the prediction error generated in step S403.

[0164] In step S405, the control unit 200 of the image decoding device determines whether the decoding of all blocks in the frame is complete, and if the decoding is complete, the process proceeds to step S406; otherwise, the process proceeds to step S402 for the next block.

[0165] In step S406, the loop filter unit 207 applies loop filtering to the image data reproduced in step S404 to generate a filtered image and ends the processing.

[0166] This operation allows for the decoding of bitstreams whose data volume has been suppressed by using quantization parameters that control the size information of the quantization parameters.

[0167] In image decoding devices, for exampleFigure 6A The quantization control size information is encoded in the picture header as shown, and the bitstream is decoded; however, the information encoding position is not limited thereto. Alternatively, the quantization size information can be encoded in the sequence header of the image as shown, or can be encoded in another position. Figure 6B

[0168] In the present embodiment, the judgment is performed by comparing the shorter length between the horizontal length and the vertical length of each object sub-block with the quantization control size; however, the present application is not limited thereto. For example, as shown in Figures 11A to 11F Figures 11A to 11F In the case shown in

[0169] Further, as another embodiment, the region sharing the quantization parameter can be determined by comparing the pixel count of each object sub-block with the pixel count of the quantization control size. Figures 12A to 12F A case where the pixel count of the size of each object sub-block is compared with the pixel count of the quantization control size is shown. In Figures 12A to 12F In the case shown in Figures 12A to 12F In the example shown in Figures 12A to 12F In the example shown in

[0170] In the present embodiment, the quantization control size is assumed to be one side of a square block. Alternatively, the quantization control size can be one side of a rectangular block. In this case, the width and the height of the quantization control size can be specified. In this case, the vertical length and the horizontal length of each object sub-block can be compared with the width and the height of the quantization control size, respectively, and in the case where two or one of these lengths is greater than or equal to the quantization control size, the quantization parameter can be decoded for each sub-block. In the case where there is a sub-block whose both lengths are smaller than the quantization control size among the basic blocks to be processed, one quantization parameter to be shared between the sub-blocks satisfying the condition is decoded. With this structure, different quantization parameter control can be implemented for the vertically long rectangular sub-block and the horizontally long rectangular sub-block. ​​

[0171] In this embodiment, the reference upon which the quantization parameters are encoded is defined by the space size; however, the structure is not limited to this. The depth of the quantization control (hereinafter referred to as the quantization control depth) that indicates the number of times the basic block is segmented is decoded from the bitstream, and this quantization control depth is compared with the segmentation depth of each object sub-block. The regions of sub-blocks sharing the quantization parameters to be decoded can be determined based on this structure. Here, reference will be made to... Figures 13A to 13F To illustrate the depth of each object's sub-block. Figures 13A to 13F In the diagram, D0 and D1 indicate depth 0 and depth 1, respectively. Figure 13A This shows that the basic block has not been divided once and the depth of the sub-block is 0 (D0). Figure 13B This shows that the basic block has been divided into four parts and each sub-block has a depth of 1 (D1). Figure 13C and Figure 13D This shows that the basic block has been divided into two parts and each sub-block has a depth of 1 (D1). Figure 13E and Figure 13F This shows that the basic block has been divided into three parts, and each sub-block has a depth of 1 (D1). Thus, the depth of each sub-block after the basic block has been divided once increases by 1 in any of the four-part, two-part, and three-part divisions. Next, refer to... Figures 14A to 14E This explains how to decode quantization parameters based on the quantization control depth and the segmentation depth of each object's sub-blocks. Figures 14A to 14E The outermost square is shown as the basic block. Figures 14A to 14E In each figure, the left side shows the segmented sub-blocks, and the right side shows the regions sharing the quantization parameter. Within each block in these figures, D0, D1, D2, D3, or D4 indicates the depth of the sub-block. For example, D0 indicates depth 0, and D4 indicates depth 4. Qp represents the quantization parameter. Figure 14A , Figure 14B , Figure 14C , Figure 14D and Figure 14E The examples show the cases where the quantization control depth is 0, 1, 2, 3, and 4, respectively. Figure 14A In the case where the quantization control depth is 0, common quantization parameters are decoded across all sub-blocks of the graph. Quantization parameters are decoded in the first sub-block, including the most important coefficients, in the decoding order. In this case, only one quantization parameter needs to be decoded. Figure 14B In the case that the quantization control depth is 1, Figure 14B The right-hand diagram shows blocks that encode common quantization parameters on a per-block basis. Each of these quantization parameters is associated with and decoded in the order of decoding, including the first sub-block containing the most important coefficients. Figure 14B In this case, the number of quantization parameters to be decoded is 4. Figure 14C In the case that the quantization control depth is 2,Figure 14C The right-hand diagram shows the encoding of common quantization parameters on a block-by-block basis. Each of these quantization parameters is associated with and decoded in the order of the first sub-block containing the most important coefficients. In this case, there are 11 quantization parameters to be decoded. Figure 14D In the case that the quantization control depth is 3, i.e., when the quantization control depth is 3, Figure 14D The right-hand diagram shows blocks that encode common quantization parameters on a unit basis. Similarly, in this case, each quantization parameter is associated with and decoded by the first sub-block, including the most important coefficients, in the decoding order. Figure 14D In this case, the number of quantization parameters to be decoded is 23. Figure 14E In the case that the quantization control depth is 4, i.e., when the quantization control depth is 4, Figure 14D The right-hand diagram shows the common quantization parameters encoded on a block-by-block basis (i.e., for each block). Quantization parameters for sub-blocks excluding importance coefficients are not decoded; however, all 27 quantization parameters are decoded when it is assumed that all sub-blocks include importance coefficients. Here, reference will be made to... Figure 15A and 15B This is to illustrate the case where there are sub-blocks in the region of shared quantization parameters that do not include importance coefficients. Figure 15A and Figures 14A to 14E The sub-blocks are shown in the same way, and D0, D1, D2, D3 or D4 indicate the depth of the sub-blocks. Figure 15B and Figures 14A to 14E The same situation illustrates an example of a shared region for quantization parameters determined by the quantization control depth and the depth of each sub-block. Figure 15A and 15B The case where the quantization control depth is 2 is shown as an example. Figure 15A and 15BIn the case where there is no significant coefficient in all three sub-blocks within the quantization parameter coding unit QoG, the quantization parameter is not decoded in the three sub-blocks. However, for a process using the quantization parameter such as deblocking filtering, the same value as the quantization parameter decoded immediately before the quantization parameter coding unit in the decoding order (i.e., QpF) is applied to the three sub-blocks in the QoG. In the quantization parameter coding unit where there is no significant coefficient, the quantization parameter coded immediately before in the process using the quantization parameter in the decoding order can be used; however, the present application is not limited thereto. For example, QpD, the quantization parameter of the upper neighboring quantization parameter coding unit, can be used, or QpF, the quantization parameter of the left neighboring quantization parameter, can be used. Alternatively, a value calculated from the quantization parameters of a plurality of quantization parameter coding units such as an average value can be used. Alternatively, an initial value of the quantization parameter for a slice can be used. The slice means a unit of division of a frame, and is composed of at least one or a plurality of elementary blocks. In this way, the depth of each object sub-block can be compared with the quantization control depth, and in the case where the depth of each object sub-block is less than or equal to the quantization control depth, the quantization parameter to be shared can be decoded. Since the sub-block division information is decoded at each division, the quantization parameter coding control highly compatible with the sub-block division information can be realized, and as a result, the bit stream whose structure of the syntax is simplified can be decoded.

[0172] Figure 17 is a flowchart showing a quantization parameter decoding process using a quantization control depth.

[0173] In step S1701, the separation and decoding section 202 decodes the quantization control size information and the information on the quantization control depth from the bit stream. The decoding section 203 decodes the division information, and obtains the information on the shape and depth of each sub-block. The inverse quantization and inverse transform section 204 compares the quantization control depth information decoded by the separation and decoding section 202 with the depth of division of each sub-block obtained from the division information decoded by the decoding section 203.

[0174] In step S1702, the inverse quantization and inverse transform section 204 determines the region of the sub-block whose depth of division is greater than the quantization control depth as a result of the comparison in S1701 as a region sharing one quantization parameter.

[0175] In step S1703, the inverse quantization and inverse transform section 204 decodes the quantization parameter associated with the sub-block having at least one significant coefficient as a residual coefficient in the sub-block within the determined region in the decoding order.

[0176] In step S1704, the inverse quantization and inverse transform unit 204 performs inverse quantization on the sub-blocks in the region determined in step S1702 by using the quantization parameter decoded in step S1703. The processing of steps S1705 to S1707 is performed on each basic block in the frame. Figure 17

[0177] Further, both the quantization control size and the quantization control depth can be generated and used in combination. In this case, in addition to the information on the quantization control size as shown in Figure 6A and 6B the information on the quantization control depth is also decoded.

[0178] For example, in a case where the depth of each of the object sub-blocks is smaller than or equal to the quantization control depth and the length of the shorter side among the horizontal side and the vertical side of each of the object sub-blocks is larger than or equal to the quantization control size, the quantization parameter is decoded for each of the sub-blocks. Specifically, the length of the shorter side of each of the sub-blocks is compared with the quantization control size for each of the regions associated with the quantization control depth. In each of the regions, in a case where there is a sub-block whose length of the shorter side is larger than or equal to the quantization control size, the quantization parameter is decoded for each of the sub-blocks in the region. In a case where the length of the shorter side of each of the sub-blocks in the region is not larger than or equal to the quantization control size, one quantization parameter shared between the sub-blocks in the region is decoded.

[0179] Alternatively, in a case where the depth of each of the object sub-blocks is smaller than or equal to the quantization control depth and the length of the longer side among the horizontal side and the vertical side of each of the object sub-blocks is larger than or equal to the quantization control size, the quantization parameter can be decoded for each of the sub-blocks. Specifically, the size of each of the sub-blocks is compared with the quantization control size for each of the regions associated with the quantization control depth. In each of the regions, in a case where there is a sub-block whose length of the longer side is larger than or equal to the quantization control size, the quantization parameter is decoded for each of the sub-blocks in the region. In a case where the length of the longer side of each of the sub-blocks in the region is not larger than or equal to the quantization control size, one quantization parameter shared between the sub-blocks in the region is decoded.

[0180] Alternatively, in a case where the depth of each of the object sub-blocks is smaller than or equal to the quantization control depth and the pixel count of each of the object sub-blocks is larger than or equal to the pixel count of the quantization control size, the quantization parameter can be decoded. In this case, the size of each of the sub-blocks is compared with the quantization control size for each of the regions associated with the quantization control depth. In each of the regions, in a case where there is a sub-block whose pixel count is larger than or equal to the pixel count of the quantization control size, the quantization parameter is decoded for each of the sub-blocks in the region. In a case where the size of each of the sub-blocks in the region is not larger than or equal to the quantization control size, one quantization parameter shared between the sub-blocks in the region is decoded.​

[0181] Optionally, the width and height of the quantization control size can be specified and used in combination with the quantization control depth. In this case, the vertical length and the horizontal length of each sub-block are compared with the width and height of the quantization control size for each region associated with the quantization control depth. In this case, the quantization parameter is decoded for each sub-block in the case where the length of one or both sides is greater than or equal to the quantization control size.

[0182] Figure 19 is a flowchart showing a quantization parameter decoding process by using both the quantization control size and the quantization control depth.

[0183] In step S1901, the separation and decoding section 202 decodes information on the quantization control size and information on the quantization control depth from the bitstream. The decoding section 203 decodes the partition information and acquires information on the shape and depth of each sub-block. The inverse quantization and inverse transform section 204 compares the quantization control depth with the depth of the partition of each sub-block and determines the region into which the sub-block is partitioned according to the depth corresponding to the quantization control depth.

[0184] In step S1902, for each region determined in step S1901, the inverse quantization and inverse transform section 204 compares the size of each sub-block included in the region with the quantization control size. The size of each sub-block can be obtained from the shape of each sub-block determined from the decoded partition information.

[0185] In step S1903, as a result of the comparison in step S1902, the inverse quantization and inverse transform section 204 determines whether one quantization parameter is shared between the sub-blocks in the region to be processed. According to the determination, the quantization of each sub-block in each region is performed by using the quantization parameter associated with the sub-block, and the residual coefficients of each sub-block are generated.

[0186] In step S1904, in the case where one quantization parameter is shared between the sub-blocks in the target region, the decoding section 203 decodes the quantization parameter associated with the first sub-block having at least one significant coefficient in the decoding order. There is no encoded data of the quantization parameter associated with another sub-block. In the case where one quantization parameter is not shared between the sub-blocks in the target region, the decoding section 203 decodes the quantization parameter associated with each sub-block except for the sub-blocks not including the significant coefficient.

[0187] Such a decoding process is performed for each elementary block.

[0188] In a case where the quantization parameter decoding process is performed by using both the quantization control size and the quantization control depth, the quantization parameter is more likely to be associated with each sub-block when the long side of each sub-block is compared with the quantization control size. In other words, the manner of comparing the long side of each sub-block with the quantization control size is more suitable for an object having an elongated rectangular shape.

[0189] With this configuration, in a case where the rectangular sub-block division is used multiple times, by using not only the quantization control depth but also the quantization control size, the quantization parameter encoding control can be achieved even in a very elongated rectangular sub-block.

[0190] In a case where the type of division of the sub-block is consecutive in the hierarchy, the sharing of the quantization parameter can be prohibited regardless of the result of the comparison between the quantization control size and the size of each sub-block. With this configuration, the quantization of the sub-blocks suitable for an object having an elongated shape can be performed. In the above-described embodiment, the processing section shown in FIGS. 1 and 2 is described as a component constituted by hardware. Alternatively, the processing performed by the processing section shown in these figures can be configured by a computer program. Figure 1 and Figure 2 The processing section shown in FIGS. 1 and 2 is described as a component constituted by hardware. Alternatively, the processing performed by the processing section shown in these figures can be configured by a computer program.

[0191] Figure 5 is a block diagram showing an example of a hardware configuration of a computer applicable to the image display apparatus according to the above-described embodiment.

[0192] The CPU 501 controls the entire computer by using a computer program and data stored in the RAM 502 or the ROM 503, and performs the above-described processing performed by the image processing apparatus according to the embodiment. In other words, the CPU 501 functions as the processing section shown in FIGS. 1 and 2. Figure 1 and Figure 2 The processing section shown in FIGS. 1 and 2 is described as a component constituted by hardware. Alternatively, the processing performed by the processing section shown in these figures can be configured by a computer program.

[0193] The RAM 502 has an area for temporarily storing a computer program and data loaded from the external storage device 506, or data acquired from the outside via the I / F (interface) 507, and the like. The RAM 502 also has a work area used when the CPU 501 performs various processing. In other words, the RAM 502 can be allocated, for example, a frame memory or provide other various areas as needed.

[0194] The ROM 503 stores therein setting data of the computer, a boot program, and the like. The operation section (operation unit) 504 is constituted by a keyboard or a mouse, or the like. The operation section 504 allows various instructions to be input to the CPU 501 when a user of the computer operates the operation section 504. The display section (display unit) 505 displays a processing result by the CPU 501. The display section 505 is constituted by, for example, a liquid crystal display.

[0195] External storage device 506 is a large-capacity information storage device, typically a hard disk drive. External storage device 506 stores the OS (operating system) and the components that enable CPU 501 to perform its functions. Figure 1 and Figure 2 The computer program shows the functions of each part. Furthermore, the image data to be processed can be stored in the external storage device 506.

[0196] The computer programs and data stored in external storage device 506 are loaded into RAM 502 as needed under the control of CPU 501 and used as objects to be processed by CPU 501. Networks such as LANs and the Internet, as well as other devices such as projectors and display devices, can be connected to I / F 507. The computer acquires or sends various information via I / F 507. 508 represents the bus connecting the above-mentioned components.

[0197] The operations comprised of the above components and described in the above flowchart are primarily controlled by CPU 501.

[0198] The object of the present invention is also achieved in the following case: a storage medium containing code of a computer program that performs the above-described functions is supplied to a system, and the system reads and executes the code of the computer program. In this case, the code of the computer program read from the storage medium itself performs the functions of the above-described embodiments, and the storage medium containing the code of the computer program constitutes the present invention. Furthermore, an operating system (OS) running on a computer can perform some or all of the actual processing according to the instructions of the computer program code, and the above-described functions can be implemented through these processes.

[0199] Furthermore, the following mode can be implemented: Computer program code read from a storage medium is written into a function expansion card inserted into the computer or into a memory included in a function expansion unit connected to the computer. The CPU or other components included in the function expansion card or function expansion unit perform some or all of the actual processing according to the instructions of the computer program code to achieve the aforementioned functions.

[0200] When this invention is applied to a storage medium, the code of the computer program corresponding to the above flowchart is stored in the storage medium.

[0201] This invention is used in encoding and decoding devices for encoding and decoding still and moving images. In particular, this invention can be applied to encoding and decoding systems that use quantization processing.

[0202] Using this invention, quantization coefficients can be appropriately encoded not only for square sub-blocks but also for rectangular sub-blocks.

[0203] The present application can be realized by supplying a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and causing one or more processors in a computer of the system or the apparatus to read out and execute the program. Alternatively, an embodiment of the present application can be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0204] Embodiments of the present application are not limited to the above-described embodiments. Various changes or modifications are applicable without departing from the spirit and scope of the present application. Therefore, the appended claims are added to show the scope of the present application.

[0205] This application claims the benefit of Japanese Patent Application No. 2018-122421, filed June 27, 2018, which is hereby incorporated by reference herein in its entirety.

Claims

1. An image encoding apparatus capable of encoding an image in units of blocks, the image encoding apparatus comprising: a first partitioning unit configured to partition an image into a plurality of coding tree units; a second partitioning unit configured to partition a coding tree unit into a plurality of blocks; a comparison unit configured to compare a partitioning value with a threshold value for determining a block group that shares a quantization parameter, the partitioning value being a value that increases as a number of times of partitioning the coding tree unit increases, the block group being capable of including a plurality of blocks for which the partitioning value is greater than the threshold value; an encoding unit configured to encode data indicating a value of a quantization parameter in accordance with a comparison result obtained by the comparison unit; and a filtering unit configured to perform deblocking filtering, wherein, in a case where at least any one of the blocks included in the block group includes a non-zero coefficient, the encoding unit is configured to encode first data in association with the block including the non-zero coefficient, the first data indicating a first value of a quantization parameter that is shared in the block group, and the filtering unit is configured to perform deblocking filtering on one or more blocks included in the block group by using the first value, and wherein, in a case where the block group consists of three blocks and the three blocks included in the block group do not include a non-zero coefficient, the encoding unit is configured not to encode the first data for the block group, and the filtering unit is configured to perform deblocking filtering on the three blocks included in the block group by using a second value of a quantization parameter, the second value being derived by calculating an average value of quantization parameters for blocks other than the block group. 2.An image decoding apparatus capable of decoding a bitstream encoded by partitioning an image into a plurality of coding tree units and partitioning a coding tree unit into a plurality of blocks, the image decoding apparatus comprising: a comparison unit configured to compare a partitioning value with a threshold value for determining a block group that shares a quantization parameter, the partitioning value being a value that increases as a number of times of partitioning the coding tree unit increases, the block group being capable of including a plurality of blocks for which the partitioning value is greater than the threshold value; a decoding unit configured to decode data indicating a value of a quantization parameter in accordance with a comparison result obtained by the comparison unit; and a filtering unit configured to perform deblocking filtering, wherein, in a case where at least any one of the blocks included in the block group includes a non-zero coefficient, the decoding unit is configured to decode first data when processing the block including the non-zero coefficient, the first data indicating a first value of a quantization parameter that is shared in the block group, and the filtering unit is configured to perform deblocking filtering on one or more blocks included in the block group by using the first value, and wherein, in a case where the block group consists of three blocks and the three blocks included in the block group do not include a non-zero coefficient, the decoding unit is configured not to decode the first data for the block group, and the filtering unit is configured to perform deblocking filtering on the three blocks included in the block group by using a second value of a quantization parameter, the second value being derived by calculating an average value of quantization parameters for blocks other than the block group. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ wherein, in a case where the block group consists of three blocks and the three blocks included in the block group do not include a non-zero coefficient, the decoding unit is configured not to decode the first data for the block group, and the filtering unit is configured to perform deblocking filtering on the three blocks included in the block group by using a second value of the quantization parameter, the second value being derived by calculating an average value of quantization parameters for blocks other than the block group.

3. An image encoding method for encoding an image in units of blocks, the image encoding method comprising: dividing an image into a plurality of coding tree units; dividing a coding tree unit into a plurality of blocks; comparing a division value with a threshold value for determining a block group that shares a quantization parameter, the division value being a value that increases as the number of times of division of the coding tree unit increases, the block group being able to include a plurality of blocks for which the division value is greater than the threshold value; and encoding data indicating a value of a quantization parameter according to a result of the comparison, wherein, in a case where at least any one of the blocks included in the block group includes a non-zero coefficient, first data is encoded in association with the block including the non-zero coefficient, the first data indicating a first value of a quantization parameter that is shared in the block group, and the image encoding method further includes performing deblocking filtering on one or more blocks included in the block group by using the first value, and wherein, in a case where the block group consists of three blocks and the three blocks included in the block group do not include a non-zero coefficient, the first data for the block group is not encoded, and the image encoding method further includes performing deblocking filtering on the three blocks included in the block group by using a second value of the quantization parameter, the second value being derived by calculating an average value of quantization parameters for blocks other than the block group.

4. An image decoding method for decoding a bitstream that is encoded by dividing an image into a plurality of coding tree units and dividing a coding tree unit into a plurality of blocks, the image decoding method comprising: comparing a division value with a threshold value for determining a block group that shares a quantization parameter, the division value being a value that increases as the number of times of division of the coding tree unit increases, the block group being able to include a plurality of blocks for which the division value is greater than the threshold value; and decoding data indicating a value of a quantization parameter according to a result of the comparison, wherein, in a case where at least any one of the blocks included in the block group includes a non-zero coefficient, first data is decoded when processing the block including the non-zero coefficient, the first data indicating a first value of a quantization parameter that is shared in the block group, and the image decoding method further includes performing deblocking filtering on one or more blocks included in the block group by using the first value, and wherein, in a case where the block group consists of three blocks and the three blocks included in the block group do not include a non-zero coefficient, the first data for the block group is not decoded, and the image decoding method further includes performing deblocking filtering on the three blocks included in the block group by using a second value of the quantization parameter, the second value being derived by calculating an average value of quantization parameters for blocks other than the block group. wherein, in a case where the block group consists of three blocks and the three blocks included in the block group do not include a non-zero coefficient, the first data for the block group is not decoded, and the image decoding method further includes performing deblocking filtering on the three blocks included in the block group by using a second value of a quantization parameter, the second value being derived by calculating an average value of quantization parameters for blocks other than the block group.

5. A non-transitory computer-readable medium storing a computer-executable program for causing a computer to perform an image encoding method for encoding an image in units of blocks, the image encoding method comprising: partitioning an image into a plurality of coding tree units; partitioning a coding tree unit into a plurality of blocks; comparing a partitioning value with a threshold value for determining a block group that shares a quantization parameter, the partitioning value being a value that increases as the number of times of partitioning the coding tree unit increases, the block group being able to include a plurality of blocks for which the partitioning value is greater than the threshold value; and encoding data indicating a value of a quantization parameter according to a result of the comparison, wherein, in a case where at least any one of the blocks included in the block group includes a non-zero coefficient, first data is encoded in association with the block including the non-zero coefficient, the first data indicating a first value of a quantization parameter that is shared in the block group, and the image encoding method further includes performing deblocking filtering on one or more blocks included in the block group by using the first value, and wherein, in a case where the block group consists of three blocks and the three blocks included in the block group do not include a non-zero coefficient, the first data for the block group is not encoded, and the image encoding method further includes performing deblocking filtering on the three blocks included in the block group by using a second value of a quantization parameter, the second value being derived by calculating an average value of quantization parameters for blocks other than the block group.

6. A non-transitory computer-readable medium storing a computer-executable program for causing a computer to perform an image decoding method for decoding a bitstream that is encoded by partitioning an image into a plurality of coding tree units and partitioning a coding tree unit into a plurality of blocks, the image decoding method comprising: comparing a partitioning value with a threshold value for determining a block group that shares a quantization parameter, the partitioning value being a value that increases as the number of times of partitioning the coding tree unit increases, the block group being able to include a plurality of blocks for which the partitioning value is greater than the threshold value; and decoding data indicating a value of a quantization parameter according to a result of the comparison, wherein, in a case where at least any one of the blocks included in the block group includes a non-zero coefficient, first data is decoded when processing the block including the non-zero coefficient, the first data indicating a first value of a quantization parameter that is shared in the block group, and the image decoding method further includes performing deblocking filtering on one or more blocks included in the block group by using the first value, and In a case where the block group is constituted by three blocks and the three blocks included in the block group do not include a non-zero coefficient, the first data for the block group is not decoded, and the image decoding method further includes performing deblocking filtering on the three blocks included in the block group by using a second value of a quantization parameter, the second value being derived by calculating an average value of quantization parameters for blocks other than the block group.

Images