Image encoding apparatus and method, image decoding apparatus and method, and medium

Abstract

The present application provides an image encoding device and method, an image decoding device and method, and a medium. By adaptively using a quantization control size according to the shape of a sub-block, control of encoding of a quantization parameter can be appropriately performed not only for square sub-blocks but also for rectangular sub-blocks, and as a result, coding efficiency is improved.

Description

[0001] (This application is a divisional application of the application filed on June 14, 2019, with application number 201980043647.3 and titled "Image Encoding Device, Image Encoding Method and Program, Image Decoding Device, Image Decoding Method and Program".) Technical Field

[0002] This invention relates to image encoding devices, image encoding methods and programs, as well as image decoding devices, image decoding methods and programs. Background Technology

[0003] The HEVC (High Efficiency Video Coding) coding system is known as a coding system for compressed recording of moving images. To improve coding efficiency, HEVC employs basic blocks with a larger size than existing macroblocks (16×16 pixels). These larger basic blocks are called CTUs (Coding Tree Units), and their size reaches 64×64 pixels. CTUs are further divided into sub-blocks as units for prediction and transformation. Patent Document 1 describes a technique for enabling the coding unit to change the quantization parameters by calculating the size of the sub-blocks encoding the quantization parameters (hereinafter referred to as the quantization control size).

[0004] In recent years, international standardization efforts have begun for a more efficient coding system following HEVC. The Joint Video Experts Team (JVET) has been established between ISO / IEC and ITU-T, and standardization is progressing as the VVC (Versatile Video Coding) coding system. To improve coding efficiency, in addition to existing intra-frame prediction and orthogonal transform methods based on square sub-blocks, intra-frame prediction and orthogonal transform methods based on rectangular sub-blocks have also been studied.

[0005] For VVC, not only square sub-blocks like HEVC have been studied, but also rectangular sub-blocks. Square sub-blocks are assumed to be used to set the quantization control size, which serves as the reference for encoding quantization parameters in HEVC. On the other hand, when performing rectangular sub-block segmentation, which has been studied in VVC, there are cases where it is not possible to uniquely determine whether to encode quantization parameters. Therefore, the present invention was conceived to solve the above-mentioned problems, and the object of the present invention is to enable appropriate control of quantization parameter encoding by using not only square sub-blocks but also rectangular sub-blocks.

[0006] Existing technical documents

[0007] Patent documents

[0008] Patent Document 1: Japanese Patent Application Publication No. 2012-161074 Summary of the Invention

[0009] Solution for solving the problem

[0010] To address the aforementioned problems, the image encoding apparatus of the present invention has the following structure. The image encoding apparatus divides an image into multiple sub-blocks and encodes the image for each sub-block, and includes an encoding component configured to encode quantization parameters when the smaller of the horizontal and vertical sizes of the object sub-block to be processed is greater than or equal to a quantization control size used to encode quantization parameters.

[0011] Furthermore, the image decoding apparatus of the present invention has the following structure. The image decoding apparatus decodes an image composed of multiple sub-blocks for each of these sub-blocks, and decodes the quantization parameters when the smaller of the horizontal and vertical sizes of the object sub-block to be processed is greater than or equal to the quantization control size used to decode the quantization parameters. Attached Figure Description

[0012] Figure 1 This is a block diagram illustrating the structure of the image encoding device in the embodiment.

[0013] Figure 2 This is a block diagram illustrating the structure of the image decoding device in the embodiment.

[0014] Figure 3 This is a flowchart illustrating image encoding processing performed in an image encoding apparatus according to an embodiment.

[0015] Figure 4 This is a flowchart illustrating the image decoding process performed in an image decoding device according to an embodiment.

[0016] Figure 5 This is a block diagram illustrating an example of the hardware structure of a computer for an image encoding device and a decoding device suitable for an embodiment.

[0017] Figure 6A This is a diagram showing an example of a bitstream structure.

[0018] Figure 6B This is a diagram showing an example of a bitstream structure.

[0019] Figure 7A This is a diagram illustrating an example of sub-block segmentation used in an embodiment.

[0020] Figure 7B This is a diagram illustrating an example of sub-block segmentation used in an embodiment.

[0021] Figure 7C This is a diagram illustrating an example of sub-block segmentation used in an embodiment.

[0022] Figure 7D This is a diagram illustrating an example of sub-block segmentation used in an embodiment.

[0023] Figure 7E This is a diagram illustrating an example of sub-block segmentation used in an embodiment.

[0024] Figure 7F This is a diagram illustrating an example of sub-block segmentation used in an embodiment.

[0025] Figure 8A This is a diagram showing a comparison between the quantized control size in the embodiment and the shorter of the horizontal and vertical sizes of the object sub-block.

[0026] Figure 8B This is a diagram showing a comparison between the quantized control size in the embodiment and the shorter of the horizontal and vertical sizes of the object sub-block.

[0027] Figure 8C This is a diagram showing a comparison between the quantized control size in the embodiment and the shorter of the horizontal and vertical sizes of the object sub-block.

[0028] Figure 8D This is a diagram showing a comparison between the quantized control size in the embodiment and the shorter of the horizontal and vertical sizes of the object sub-block.

[0029] Figure 8E This is a diagram showing a comparison between the quantized control size in the embodiment and the shorter of the horizontal and vertical sizes of the object sub-block.

[0030] Figure 8F This is a diagram showing a comparison between the quantized control size in the embodiment and the shorter of the horizontal and vertical sizes of the object sub-block.

[0031] Figure 9A This is a diagram showing a comparison between the quantized control size in the embodiment and the shorter of the horizontal and vertical sizes of the object sub-block.

[0032] Figure 9B This is a diagram showing a comparison between the quantized control size in the embodiment and the shorter of the horizontal and vertical sizes of the object sub-block.

[0033] Figure 9CThis is a diagram showing a comparison between the quantized control size in the embodiment and the shorter of the horizontal and vertical sizes of the object sub-block.

[0034] Figure 9D This is a diagram showing a comparison between the quantized control size in the embodiment and the shorter of the horizontal and vertical sizes of the object sub-block.

[0035] Figure 9E This is a diagram showing a comparison between the quantized control size in the embodiment and the shorter of the horizontal and vertical sizes of the object sub-block.

[0036] Figure 9F This is a diagram showing a comparison between the quantized control size in the embodiment and the shorter of the horizontal and vertical sizes of the object sub-block.

[0037] Figure 10A This is a graph illustrating the relationship between the encoding of the quantization parameters and the importance coefficients in the embodiment.

[0038] Figure 10B This is a graph illustrating the relationship between the encoding of the quantization parameters and the importance coefficients in the embodiment.

[0039] Figure 10C This is a graph illustrating the relationship between the encoding of the quantization parameters and the importance coefficients in the embodiment.

[0040] Figure 10D This is a graph illustrating the relationship between the encoding of the quantization parameters and the importance coefficients in the embodiment.

[0041] Figure 10E This is a graph illustrating the relationship between the encoding of the quantization parameters and the importance coefficients in the embodiment.

[0042] Figure 10F This is a graph illustrating the relationship between the encoding of the quantization parameters and the importance coefficients in the embodiment.

[0043] Figure 11A This is a diagram showing a comparison between the quantized control size in the embodiment and the longer of the horizontal and vertical sizes of the object sub-block.

[0044] Figure 11B This is a diagram showing a comparison between the quantized control size in the embodiment and the longer of the horizontal and vertical sizes of the object sub-block.

[0045] Figure 11C This is a diagram showing a comparison between the quantized control size in the embodiment and the longer of the horizontal and vertical sizes of the object sub-block.

[0046] Figure 11DThis is a diagram showing a comparison between the quantized control size in the embodiment and the longer of the horizontal and vertical sizes of the object sub-block.

[0047] Figure 11E This is a diagram showing a comparison between the quantized control size in the embodiment and the longer of the horizontal and vertical sizes of the object sub-block.

[0048] Figure 11F This is a diagram showing a comparison between the quantized control size in the embodiment and the longer of the horizontal and vertical sizes of the object sub-block.

[0049] Figure 12A This is a graph showing a comparison between the pixel count of the quantization control size and the pixel count of the object sub-block in the embodiments and examples.

[0050] Figure 12B This is a graph showing a comparison between the pixel count of the quantization control size and the pixel count of the object sub-block in the embodiments and examples.

[0051] Figure 12C This is a graph showing a comparison between the pixel count of the quantization control size and the pixel count of the object sub-block in the embodiments and examples.

[0052] Figure 12D This is a graph showing a comparison between the pixel count of the quantization control size and the pixel count of the object sub-block in the embodiments and examples.

[0053] Figure 12E This is a graph showing a comparison between the pixel count of the quantization control size and the pixel count of the object sub-block in the embodiments and examples.

[0054] Figure 12F This is a graph showing a comparison between the pixel count of the quantization control size and the pixel count of the object sub-block in the embodiments and examples.

[0055] Figure 13A This is a diagram illustrating the depth of blocks in the sub-block segmentation used in the embodiment.

[0056] Figure 13B This is a diagram illustrating the depth of blocks in the sub-block segmentation used in the embodiment.

[0057] Figure 13C This is a diagram illustrating the depth of blocks in the sub-block segmentation used in the embodiment.

[0058] Figure 13D This is a diagram illustrating the depth of blocks in the sub-block segmentation used in the embodiment.

[0059] Figure 13E This is a diagram illustrating the depth of blocks in the sub-block segmentation used in the embodiment.

[0060] Figure 13F This is a diagram illustrating the depth of blocks in the sub-block segmentation used in the embodiment.

[0061] Figure 14A This is a graph showing a comparison between the quantization control depth and the depth of the object sub-block in the embodiment.

[0062] Figure 14B This is a graph showing a comparison between the quantization control depth and the depth of the object sub-block in the embodiment.

[0063] Figure 14C This is a graph showing a comparison between the quantization control depth and the depth of the object sub-block in the embodiment.

[0064] Figure 14D This is a graph showing a comparison between the quantization control depth and the depth of the object sub-block in the embodiment.

[0065] Figure 14E This is a graph showing a comparison between the quantization control depth and the depth of the object sub-block in the embodiment.

[0066] Figure 15A This is a diagram illustrating the reading of quantization parameters in an embodiment where no significant coefficient exists in the quantization parameter sharing unit.

[0067] Figure 15B This is a diagram illustrating the reading of quantization parameters in an embodiment where no significant coefficient exists in the quantization parameter sharing unit.

[0068] Figure 16 This is a flowchart of the quantization parameter encoding process using quantization control depth.

[0069] Figure 17 This is a flowchart of the decoding process using quantization parameters that control the depth of quantization.

[0070] Figure 18 It is a flowchart of quantization parameter encoding processing using quantization control size and quantization control depth.

[0071] Figure 19 It is a flowchart of the decoding process using quantization parameters of quantization control size and quantization control depth. Detailed Implementation

[0072] The invention will now be described in detail with reference to the accompanying drawings and suitable embodiments thereof. The structures described in the following embodiments are merely examples, and the invention is not limited to the illustrated structures.

[0073] Figure 1 This is a block diagram illustrating the image encoding device of this embodiment. Figure 1In this device, the control unit (control unit) 100 is a processor that controls the entire image encoding device, and the terminal 101 is an input terminal for inputting image data.

[0074] The block segmentation unit 102 segments the input image into multiple basic blocks and outputs the image in units of basic blocks to subsequent stages.

[0075] The generation unit 103 generates, for example, information related to the size used to encode the quantization parameters (quantization control size), and outputs this information. The method for generating the information related to the quantization control size is not limited. The user can input the quantization control size, calculate the quantization control size based on the characteristics of the input image, or use a quantization control size pre-specified as an initial value.

[0076] The prediction unit 104 generates sub-blocks by dividing each basic block, and performs intra-frame prediction (as intra-frame prediction) and inter-frame prediction (as inter-frame prediction) on a sub-block basis, generating predicted image data. Furthermore, it calculates prediction errors based on image data indicating the input pixel values ​​and the predicted image data, and outputs these prediction errors. It also outputs prediction-required information (e.g., information related to sub-block segmentation, prediction mode, and motion vectors) along with the prediction errors. Hereinafter, the prediction-required information is referred to as prediction information.

[0077] The transformation and quantization unit 105 performs orthogonal transformation on the residuals representing the prediction error in sub-block units, further quantizes them, and obtains residual coefficients representing the residuals. Quantization parameters are parameters used to quantize the transformation coefficients obtained through the orthogonal transformation.

[0078] The inverse quantization and inverse transform unit (inverse quantization and inverse transform unit) 106 reproduces the transform coefficients by inverse quantization of the residual coefficients output from the transform and quantization unit 105, and reproduces the prediction error by applying an inverse orthogonal transform to the transform coefficients.

[0079] Frame memory 108 is a memory that stores the reproduced image data.

[0080] The image reproduction unit 107 generates predicted image data by reading the frame memory 108 as needed using the prediction information output from the prediction unit 104, and generates reproduced image data based on the predicted image data and the input prediction error.

[0081] The loop filter section (loop filter unit) 109 applies loop filtering, such as deblocking filtering and sample adaptive compensation, to the reproduced image.

[0082] The encoding unit 110 generates code data by encoding the residual coefficients output from the transform and quantization unit 105 and the prediction information output from the prediction unit 104.

[0083] The integrated encoding unit 111 generates header code data by encoding information related to the quantization control size from the generation unit 103. The integrated encoding unit 111 further combines the header code data with code data output from the encoding unit 110 to form a bit stream. Terminal 112 is an output terminal for outputting the bit stream generated by the integrated encoding unit 111 to an external device.

[0084] The image encoding operation in the image encoding device will be described below. In this embodiment, moving image data is input in units of frames. Optionally, one frame of still image data can be input.

[0085] One frame of image data input from terminal 101 is input to block segmentation unit 102.

[0086] The block segmentation unit 102 divides the input image data into multiple basic blocks and outputs the image in units of basic blocks to the prediction unit 104.

[0087] The prediction unit 104 performs prediction processing on the image data input from the block segmentation unit 102. Specifically, the prediction unit 104 first determines the sub-block segmentation used to divide the basic block into further smaller sub-blocks.

[0088] Figures 7A to 7F This example illustrates the type of segmentation of a basic block. A thick box, indicated by 700, represents a basic block. For clarity, it is assumed that each basic block is configured to be 32×32 pixels, and that rectangles within the thick box represent sub-blocks. Figure 7B An example of square sub-blocks obtained through segmentation is shown, where a basic block of 32×32 pixels is divided into sub-blocks of 16×16 pixels. On the other hand, Figures 7C to 7F This shows an example of the type of rectangular sub-blocks obtained through segmentation. Figure 7C In this context, the basic block is divided into 16×32 pixel vertical rectangular sub-blocks. Figure 7D In this context, the basic block is divided into horizontally elongated sub-blocks of 32×16 pixels. Figure 7E and Figure 7FIn this embodiment, the basic block is divided into rectangular sub-blocks in a 1:2:1 ratio. Thus, in this embodiment, encoding is performed using both square and rectangular sub-blocks. In this embodiment, information related to this type of basic block division is encoded as segmentation information. Furthermore, in order to obtain as described later... Figure 15A and 15B The hierarchical structure of the sub-blocks shown on the left hierarchically encodes information related to the segmentation type.

[0089] The prediction unit 104 determines a prediction mode for each sub-block to be processed. Specifically, the prediction unit 104 determines a prediction mode on a sub-block basis, such as intra-frame prediction using coded pixels from the same frame that includes each sub-block to be processed, or inter-frame prediction using pixels from another coded frame. The prediction unit 104 generates prediction image data based on the determined prediction mode and coded pixels, further generates prediction errors based on the input image data and prediction image data, and outputs these prediction errors to the transform and quantization unit 105. The prediction unit 104 also outputs information related to sub-block segmentation and prediction mode as prediction information to the encoding unit 110 and the image reproduction unit 107.

[0090] Here, the transformation and quantization processes performed by the transformation and quantization unit 105 will be explained in more detail. The transformation and quantization unit 105 applies frequency transformation to the image data (pixel values) of the sub-blocks that have undergone prediction processing by the prediction unit 104, and further quantizes the image data. Figures 8A to 8F The relationship between the type of block segmentation and the quantization control size is shown. The transform and quantization unit 105 determines the unit of sub-blocks that share and encode quantization parameters by using the object sub-block size and the quantization control size output from the generation unit 103. In other words, in the subsequent encoding unit 110, it is determined whether quantization parameters should be shared among multiple sub-blocks based on a comparison between the quantization control size and the size of each sub-block. The encoding of quantization parameters will be explained later. The method for determining the value of the quantization parameters used for quantization is not limited. The user can input quantization parameters, calculate quantization parameters based on the characteristics of the input image, or use quantization parameters that have been pre-specified as initial values.

[0091] Next, we will explain the method for determining the unit for encoding the quantization parameters.

[0092] The transformation and quantization unit 105 compares the length of the shorter side among the horizontal and vertical sides of each object sub-block with the quantization control size and determines the unit for encoding the quantization parameters, that is, using the same unit for the quantization parameters.

[0093] Figures 8A to 8FThis shows the unit used to encode the quantization parameters when comparing the length of the shorter side of the horizontal and vertical edges of each object sub-block to the quantization control size. Figures 8A to 8F In this context, the length of one side of the square block is defined as the quantization control size. Specifically, Figures 8A to 8F The example shown is an application of 16 as a quantization control size. Figures 8A to 8F Use arrows to indicate the shorter of the horizontal and vertical lengths of each object's sub-blocks. Figures 8A to 8F The thick rectangles within the object sub-blocks represent regions that share the quantization parameters determined as a result of comparisons between the individual object sub-blocks and the quantization control size. Qp represents the quantization parameter. For Figure 8A , Figure 8B , Figure 8C and Figure 8D The shorter of the horizontal and vertical lengths of each object sub-block is greater than or equal to the quantization control size (16). For this reason, each object sub-block is quantized using the corresponding quantization parameters in each quantization parameter (QpA to QpD). The corresponding quantization parameters in the quantization parameters (QpA to QpD) are encoded for each sub-block. On the other hand, for Figure 8E and Figure 8F Sub-blocks whose shorter side (horizontal or vertical) is smaller than the quantization control size are included in the base block to be processed, thus sharing quantization parameters among multiple sub-blocks. Specifically, for Figure 8E and Figure 8F Each of the three sub-blocks is quantized using the same quantization parameter. In this case, for the quantization parameter to be encoded, a single quantization parameter is encoded as a common quantization parameter, not for each sub-block. As mentioned above, the quantization parameter is used based on the size of each object sub-block and the quantization control size.

[0094] Next, refer to Figures 9A to 9F To explain and Figures 8A to 8F Examples of cases where the quantization control size is different. Also in Figures 9A to 9F In this context, the length of one side of the square block is defined as the quantization control size. Specifically, Figures 9A to 9F The example shown is an application of a length of 32, which is the same as the length of one side of the basic block to be processed, as an example of quantization control size. Figures 9A to 9F The meanings of the thick frame, arrows, and Qp in the text are... Figures 8A to 8F The bold borders, arrows, and Qp in the text have similar meanings, so explanations are omitted. Figures 9A to 9FIn this context, when comparing the shorter of the horizontal and vertical lengths of each object sub-block with the quantization control size, any object sub-block is equal to or smaller than the quantization control size. For this reason, sub-blocks are quantized using the same quantization parameter in all cases. The quantization parameter to be encoded is not specific to each sub-block but is encoded as a common quantization parameter.

[0095] Return to reference Figure 1 The inverse quantization and inverse transform unit 106 reproduces the transform coefficients by inverse quantizing the input residual coefficients, further reproduces the prediction error by applying an inverse orthogonal transform to the reproduced transform coefficients, and outputs these prediction errors to the image reproduction unit 107. In the inverse quantization processing of each sub-block, the same quantization parameters as those used in the transform and quantization unit 105 are used.

[0096] The image reproduction unit 107 reproduces the predicted image by reading the frame memory 108 as needed using the prediction information input from the prediction unit 104. The image reproduction unit 107 reproduces image data based on the reproduced predicted image and the reproduced prediction error input from the inverse quantization and inverse transform unit 106, inputs the image data into the frame memory 108, and stores the image data.

[0097] The loop filter unit 109 reads the reproduced image from the frame memory 108 and applies loop filtering, such as deblocking filtering, to the reproduced image. Loop filtering is performed based on the prediction mode of the prediction unit 104, the values ​​of the quantization parameters used in the transform and quantization unit 105, and whether there are non-zero values ​​(hereinafter referred to as significant coefficients) in each processed sub-block after quantization, or based on sub-block segmentation information. The loop filter unit 109 then inputs the filtered image back into the frame memory 108 and stores the image again.

[0098] The encoding unit 110 entropy-encodes the residual coefficients generated by the transformation and quantization unit 105 and the prediction information input from the prediction unit 104 in blocks, and generates code data.

[0099] No entropy encoding method is specified. Golomb coding, arithmetic coding, or Huffman coding can be used. The generated code data is output to the integration coding unit 111. When encoding the quantization parameters constituting the quantization information, an identifier indicating the difference between the quantization parameters of the sub-block to be encoded and the quantization parameters of the sub-block encoded before that sub-block is encoded. In this embodiment, the difference between the quantization parameters encoded immediately before that sub-block in the encoding order and the quantization parameters of that sub-block is calculated as the prediction value; however, the prediction value of the quantization parameters is not limited to this. The quantization parameters of the sub-block adjacent to the left or right side of that sub-block can be used as the prediction value, or a value calculated based on the quantization parameters of multiple sub-blocks, such as the average value, can be used as the prediction value.

[0100] Here, we will refer to Figures 10A to 10F To further explain the process of encoding quantization parameters based on the quantization control size. Figures 10A to 10F The left side of each diagram illustrates the block segmentation type and the quantization parameters (Qp) used in each sub-block. Sub-blocks with diagonal shading represent sub-blocks associated with the quantization parameters to be encoded. Bold rectangles represent regions that share quantization parameters determined by the quantization control size and the size of each object sub-block. Figures 10A to 10F The intermediate plots within each sub-block indicate whether each sub-block has a significance coefficient. A significance coefficient is a non-zero coefficient among the residual coefficients after transformation and quantization. In other words, having a significance coefficient means that there is at least one non-zero residual coefficient in the transformed and quantized sub-block. Figures 10A to 10F The arrows shown in the right-hand diagrams indicate the encoding (decoding) order. In this embodiment, in the region sharing quantization parameters, the quantization parameters are associated with and encoded by the first sub-block, including the important coefficients, in the encoding order. For example, in Figure 10B In this process, the first sub-block including the importance coefficients in the encoding order is the upper right sub-block, so the quantization parameters associated with this sub-block are encoded. In this case, since the quantization parameters have already been encoded in the upper right sub-block within the quantization parameter encoding unit, the quantization parameters are not encoded in the lower left and lower right sub-blocks. On the other hand, in the quantization and inverse quantization processing of the lower left and lower right sub-blocks, QpA, which is the same quantization parameter as that of the upper right sub-block, is used. Furthermore, since there are no importance coefficients in the upper left sub-block, no quantization processing is performed; however, in processing that uses quantization parameters, such as deblocking filtering, QpA, which is the same quantization parameter as that of the upper right sub-block, is used. Figure 10F In this encoding, the first sub-block containing the importance coefficients, in the order of encoding, is the sub-block located below. Therefore, the quantization parameters are associated with and encoded by this sub-block, while the quantization parameters are not encoded for the sub-blocks above and in the middle. However, in... Figure 10FThe top and middle sub-blocks and Figure 10B In the upper left sub-block, QpA is used as the same quantization parameter as the quantization parameter of the sub-block below in processing such as deblocking filtering.

[0101] In this way, within sub-blocks sharing a region of quantization parameters determined according to the quantization control size, the quantization parameters are associated with and encoded by the first sub-block, which includes the importance coefficients, in the encoding order.

[0102] In the integrated encoding unit 111, information related to the quantization control size is encoded. The encoding method is not specified. Columbus coding, arithmetic coding, or Huffman coding, etc., can be used. A bit stream is formed by multiplexing these codes and the code data input from the encoding unit 110. Finally, the bit stream is output to the outside from terminal 112.

[0103] Figure 6A An example of a bitstream including encoded information related to the quantization control size is shown. This information related to the quantization control size is included in any header within the headers of sequences and images, etc. In this embodiment, it is assumed that... Figure 6A As shown, information related to quantization control size is included in the image header. However, the location where information related to quantization control size is encoded is not limited to this, and as... Figure 6B As shown, it can be included in the header of the sequence.

[0104] Figure 3 This is a flowchart illustrating the encoding process performed in the image encoding device according to this embodiment.

[0105] Initially, in step S301, the block segmentation unit 102 segments the input image in units of frames into images in units of basic blocks.

[0106] In step S302, the generation unit 103 determines a quantization control size as the size used to encode the quantization parameters. This information is then set as quantization control size information. The quantization control size information is also encoded by the integration encoding unit 111.

[0107] In step S303, the prediction unit 104 generates sub-blocks by segmenting the image data generated in step S301 in units of basic blocks. The prediction unit 104 performs prediction processing on each generated sub-block and generates prediction information such as block segmentation and prediction mode, as well as prediction image data. Furthermore, it calculates the prediction error based on the input image data and the prediction image data.

[0108] In step S304, the transformation and quantization unit 105 generates transformation coefficients by applying an orthogonal transformation to the prediction error calculated in step S303. The transformation and quantization unit 105 further generates residual coefficients using quantization parameters determined based on the quantization control size information generated in step S302. Specifically, as described above, it is determined whether quantization parameters are shared between sub-blocks in the base block by comparing the quantization control size information (e.g., the length of one side of a square block) with the size of each sub-block (e.g., the length of the short or long side). Based on this determination, each sub-block is quantized using quantization parameters associated with the sub-blocks in each region, and residual coefficients for each sub-block are generated.

[0109] In step S305, the inverse quantization and inverse transform unit 106 reproduces the prediction error by applying inverse quantization and inverse orthogonal transform to the residual coefficients generated in step S304. The same quantization parameters used in the inverse quantization process of this step are employed as those used in step S304.

[0110] In step S306, the image reproduction unit 107 reproduces the predicted image based on the prediction information generated in step S303. The image reproduction unit 107 further reproduces image data based on the reproduced predicted image and the prediction error generated in step S305.

[0111] In step S307, the encoding unit 110 encodes the prediction information generated in step S303 and the residual coefficients generated in step S304 together with the block segmentation information, and generates code data. The encoding unit 110 also encodes the quantization parameters used in step S304 based on the quantization control size information generated in step S302. The encoding unit 110 generates a bitstream by further including other code data. Specifically, within each region sharing the quantization parameters determined in step S304, the quantization parameters are encoded in association with sub-blocks including at least one important coefficient in the order of the sub-blocks to be encoded.

[0112] In step S308, the control unit 100 of the image encoding device determines whether the encoding of all basic blocks in the frame is complete, and if the encoding is complete, the process proceeds to step S309; ​​otherwise, the process returns to step S303 for the next basic block.

[0113] In step S309, the loop filter unit 109 applies loop filtering to the image data reproduced in step S306 to generate a filtered image and then ends the processing.

[0114] Therefore, specifically, quantization control size information is generated in step S302, and quantization and encoding processing are performed based on the quantization control size information in steps S304 and S307, thus enabling quantization parameter encoding processing appropriately. As a result, the overall data volume of the generated bitstream is suppressed while improving the quality of the encoded image.

[0115] In this embodiment, the region sharing quantization parameters is determined by comparing the shorter of the horizontal and vertical lengths of each object sub-block with the quantization control size; however, the invention is not limited thereto. For example, as Figures 11A to 11F As shown, the shared unit of quantization parameters can be determined by comparing it with the longer of the horizontal and vertical lengths of each object sub-block. Figures 11A to 11F In this context, the length of the longer side of each sub-block is compared to the length (16) of one side of the square block that controls its size. Figures 11A to 11F In this approach, since the long side of all sub-blocks is longer than the quantization control size (16), the quantization parameters are encoded for each sub-block. Using this structure, for rectangular sub-blocks, a bitstream can be generated that prioritizes fine control of the quantization parameters rather than reducing the amount of code required.

[0116] In addition, as another embodiment, the region sharing quantization parameters can be determined by comparing the pixel count of each object sub-block with the pixel count of the quantization control size. Figures 12A to 12F This illustrates the comparison between the pixel count of each object sub-block and the pixel count of the quantization control size. Figures 12A to 12F In this context, the quantization control size is 16×16 pixels, therefore the pixel count is 256. For object sub-blocks, in Figures 12A to 12F In all sub-blocks, the pixel count is greater than or equal to 256 pixels. Therefore, in Figures 12A to 12F In the example, the quantization parameters are encoded in each sub-block within all sub-blocks. Using this structure, quantization parameter control based on the pixel count in each sub-block can be achieved regardless of the shape of the sub-blocks.

[0117] In this embodiment, the quantization control size is assumed to be one side of a square block. Optionally, the quantization control size can be one side of a rectangular block. In this case, the width and height of the quantization control size can be specified. In this case, the vertical and horizontal lengths of each object sub-block can be compared with the width and height of the quantization control size, respectively. If two or one of these lengths is greater than or equal to the quantization control size, the quantization parameter can be encoded for each sub-block. Using this structure, different quantization parameter controls can be implemented for vertically elongated rectangular sub-blocks and horizontally elongated rectangular sub-blocks. If there are two sub-blocks in the basic block to be processed whose lengths are both less than the quantization control size, a quantization parameter to be shared among the sub-blocks that meet this condition is encoded.

[0118] In this embodiment, the unit for encoding the quantization parameters is defined by the space size; however, the structure is not limited to this. A quantization control depth (hereinafter referred to as quantization control depth) indicating the number of times a basic block is segmented can be generated, and whether to encode the quantization parameters can be determined by comparing the quantization control depth with the segmentation depth of each object sub-block. In this case, instead of Figure 6A and 6B The quantization control size information shown is used to encode the quantization control depth information.

[0119] Here, we will refer to Figures 13A to 13F This illustrates the depth of segmentation for each object's sub-blocks. Figures 13A to 13F In the diagram, D0 and D1 represent depth 0 and depth 1, respectively. Figure 13A This shows that the basic block has not been divided once and the depth of the sub-block is 0 (D0). Figure 13B This shows that the basic block has been divided into four parts and each sub-block has a depth of 1 (D1). Figure 13C and Figure 13D This shows that the basic block has been divided into two parts and each sub-block has a depth of 1 (D1). Figure 13E and Figure 13F This shows that the basic block has been divided into three parts and each sub-block has a depth of 1 (D1). Thus, the depth of each sub-block of a basic block that has been divided once increases by 1 in any of the four-part, two-part, and three-part divisions.

[0120] Next, refer to Figures 14A to 14E This explains how to encode quantization parameters based on the quantization control depth and the segmentation depth of each object's sub-blocks. Figures 14A to 14E The outermost square is shown as the basic block. Figures 14A to 14EIn each figure, the left side shows the segmented sub-blocks, and the right side shows the regions sharing the quantization parameter. Within each block in these figures, D0, D1, D2, D3, or D4 indicates the depth of the sub-block. For example, D0 indicates depth 0, and D4 indicates depth 4. Qp represents the quantization parameter. Figure 14A , Figure 14B , Figure 14C , Figure 14D and Figure 14E The examples show the cases where the quantization control depth is 0, 1, 2, 3, and 4, respectively. Figure 14A In the case where the quantization control depth is 0, a common quantization parameter is used for all sub-blocks in the graph, and this quantization parameter is encoded in association with the first sub-block, including the importance coefficients, in the encoding order. In this case, the number of quantization parameters to be encoded is one. Figure 14B In the case that the quantization control depth is 1, Figure 14B The block shown on the right represents a unit-shared quantization parameter. Alternatively, a quantization parameter is encoded on a block-by-block basis, associated with the first sub-block containing the importance coefficients in the encoding order. In this case, the number of quantization parameters to be encoded is four. Figure 14C In the case where the quantization control depth is 2, quantization parameters are shared in units of blocks as shown on the right side of the diagram, and a quantization parameter is encoded in these units in association with the first sub-block, including the importance coefficients, in the encoding order. In this case, the number of quantization parameters to be encoded is 11. Figure 14D In the case that the quantization control depth is 3, i.e., when the quantization control depth is 3, Figure 14D The blocks shown on the right represent unit-shared quantization parameters, and each quantization parameter is encoded in association with the first sub-block, including the importance coefficients, in the encoding order. In this case, the number of quantization parameters to be encoded is 23. Figure 14E In the case that the quantization control depth is 4, i.e., when the quantization control depth is 4, Figure 14D The block shown on the right uses quantization parameters in units. Figure 14E In this case, the block segmentation depth equals the quantization control depth, therefore the quantization parameters are encoded for each block. In this case, the number of quantization parameters to be encoded is 27.

[0121] Here, we will refer to Figure 15A and 15B This is to illustrate the case where there are sub-blocks in the region of shared quantization parameters that do not include importance coefficients. Figure 15A and Figures 14A to 14E The sub-blocks are shown in the same way, and D0, D1, D2, D3 or D4 indicate the depth of each sub-block. Figure 15B and Figures 14A to 14EThe same situation illustrates an example of a shared region for quantization parameters determined by the quantization control depth and the depth of each sub-block. Figure 15B The case where the quantization control depth is 2 is shown as an example. Figure 15B An example is shown where no significance coefficient exists in any of the three sub-blocks within the region associated with the quantization parameter QpG. In this case, the quantization parameter QpG for the three sub-blocks is not encoded. However, for processing such as deblocking filtering, the same value as the immediately preceding quantization parameter (i.e., QpF) is used. In quantization parameter encoding units where no significance coefficient exists, the quantization parameters encoded immediately preceding the preceding quantization parameter in the encoding order are used in processing using the quantization parameters; however, the invention is not limited to this. For example, QpD, which is the quantization parameter of the upper adjacent quantization parameter encoding unit, or QpF, which is the left adjacent quantization parameter, can be used. Optionally, a value calculated from the quantization parameters of multiple quantization parameter encoding units, such as an average value, can be used. Optionally, an initial value for the quantization parameter for a slice can be used. A slice means a unit for dividing a frame and consists of at least one or more basic blocks. In this way, the depth of each object sub-block can be compared with the quantization control depth, and the quantization parameters can be shared if the depth of each object sub-block is less than or equal to the quantization control depth. Since the sub-block segmentation information is encoded at each segmentation, quantization parameter encoding control that is highly compatible with the sub-block segmentation information can be achieved, which simplifies the syntax structure.

[0122] Figure 16 This is a flowchart illustrating the quantization parameter encoding process using quantization control depth.

[0123] In step S1601, the transformation and quantization unit 105 compares the quantization control depth with the segmentation depth of each sub-block.

[0124] In step S1602, the transformation and quantization unit 105 determines the regions of sub-blocks whose segmentation depth is greater than the quantization control depth, which are the result of the comparison in S1601, as regions that share a quantization parameter.

[0125] In step S1603, the transformation and quantization unit 105 quantizes the sub-blocks in the determined region using the same quantization parameters.

[0126] In step S1604, the encoding unit 110 encodes the quantization parameters used in step S1603 in association with sub-blocks within the determined region that have at least one important coefficient as a residual coefficient in the encoding order. The integration encoding unit 111 encodes the quantization control depth information.

[0127] Perform on each basic block in the frame Figure 16The processing.

[0128] Furthermore, both quantization control size and quantization control depth can be generated and combined. In this case, besides Figure 6A and 6B In addition to the quantization control size information shown, the quantization control depth information is also encoded.

[0129] For example, if the depth of each object sub-block is less than or equal to the quantization control depth, and the shorter of the horizontal and vertical edges of each object sub-block is greater than or equal to the quantization control size, the quantization parameter can be encoded for each sub-block. Specifically, for each region associated with the quantization control depth, the length of the shorter edge of each sub-block is compared with the quantization control size. In each region, if there is a sub-block with a shorter edge length greater than or equal to the quantization control size, the quantization parameter is encoded for each sub-block within that region. If the length of the shorter edge of each sub-block within that region is not greater than or equal to the quantization control size, a single quantization parameter is shared among the sub-blocks within that region, and this single quantization parameter is encoded.

[0130] Optionally, if the depth of each object sub-block is less than or equal to the quantization control depth, and the length of the longer side of the horizontal and vertical edges of each object sub-block is greater than or equal to the quantization control size, the quantization parameters can be encoded for each sub-block. In this case, the size of each sub-block is compared with the quantization control size for each region associated with the quantization control depth. In each region, if there is a sub-block with a longer side greater than or equal to the quantization control size, the quantization parameters are encoded for each sub-block within that region. If the length of the longer side of each sub-block within that region is not greater than or equal to the quantization control size, a single quantization parameter is shared among the sub-blocks within that region, and this single quantization parameter is encoded.

[0131] Optionally, if the depth of each object sub-block is less than or equal to the quantization control depth, and the pixel count of each object sub-block is greater than or equal to the pixel count of the quantization control size, the quantization parameters can be encoded for each sub-block. In this case, the size of each sub-block is compared with the quantization control size for each region associated with the quantization control depth. In each region, if there is a sub-block with a pixel count greater than or equal to the pixel count of the quantization control size, the quantization parameters are encoded for each sub-block within that region. If the pixel count of each sub-block within that region is not greater than or equal to the quantization control size, a single quantization parameter is shared among the sub-blocks within that region, and this single quantization parameter is encoded.

[0132] Optionally, the width and height of the quantization control size can be specified and used in conjunction with the quantization control depth. In this case, for each region associated with the quantization control depth, the vertical and horizontal lengths of each sub-block are compared with the width and height of the quantization control size, respectively. In this case, if there are two or more sub-blocks whose vertical and horizontal lengths are greater than or equal to the width and height of the quantization control size, the quantization parameter is encoded for each sub-block within the object region. If there are no two or more sub-blocks whose vertical and horizontal lengths are greater than or equal to the width and height of the quantization control size, a single quantization parameter is shared within that region and encoded accordingly.

[0133] In this way, even in extremely long and narrow rectangular sub-blocks, quantization parameter encoding control can be achieved by using quantization to control both depth and size, even when using rectangular sub-blocks multiple times.

[0134] Figure 18 This is a flowchart illustrating the encoding process using quantization parameters, namely quantization control size and quantization control depth.

[0135] In step S1801, the transformation and quantization unit 105 compares the quantization control depth with the segmentation depth of the sub-block and determines the region having a block depth associated with the quantization control depth.

[0136] In step S1802, for each region determined in step S1801, the transformation and quantization unit 105 compares the size of each sub-block included in that region with the quantization control size.

[0137] In step S1803, as a result of the comparison in step S1802, the transformation and quantization unit 105 determines whether a quantization parameter is shared among sub-blocks within the target region. Based on this determination, each sub-block is quantized using the quantization parameter associated with the sub-block in each region, and residual coefficients for each sub-block are generated.

[0138] In step S1804, if a quantization parameter is shared among sub-blocks in the object region, the encoding unit 110 encodes the quantization parameter in association with the first sub-block having at least one importance coefficient in the encoding order. Conversely, if the quantization parameter is not associated with another sub-block, the encoding unit 110 encodes the quantization parameter in association with each sub-block except those that do not contain importance coefficients. The integration encoding unit 111 encodes the quantization control size information and the quantization control depth information.

[0139] This encoding process is applied to each basic block.

[0140] When encoding quantization parameters using both quantization control size and quantization control depth, the quantization parameters are more likely to be associated with each sub-block when comparing its long side with the quantization control size. In other words, comparing the long side of each sub-block with the quantization control size is more suitable for objects with elongated shapes.

[0141] When the sub-block segmentation type is hierarchically continuous, sharing quantization parameters can be prevented regardless of the comparison results between the quantization control size and the size of each sub-block. This structure allows for the quantization of sub-blocks suitable for objects with elongated shapes.

[0142] Figure 2 This is a block diagram illustrating the structure of an image decoding device. In this embodiment, for example, it will be described in... Figure 1 Decoding of encoded data generated in the image encoding device shown.

[0143] Terminal 201 is the input terminal for the encoded bit stream.

[0144] The separation and decoding unit 202 separates information related to decoding processing and code data related to residual coefficients from the bitstream, and decodes the code data in the header of the bitstream. In this embodiment, the separation and decoding unit 202 decodes the quantization control size information and outputs the quantization control size information to subsequent stages. The separation and decoding unit 202 performs... Figure 1 The inverse operation of the integrated coding unit 111 shown.

[0145] The decoding unit 203 obtains residual coefficients and prediction information from the code data output from the separation and decoding unit 202.

[0146] The inverse quantization and inverse transform unit 204 performs inverse quantization on the residual coefficients input in blocks, further applies inverse orthogonal transform, and obtains the prediction error.

[0147] Frame memory 206 is a memory that stores the image data of the reproduced picture.

[0148] The image reproduction unit 205 generates predicted image data by reading the frame memory 206 as needed using the input prediction information. The image reproduction unit 205 generates reproduced image data based on the predicted image data and the prediction error reproduced by the inverse quantization and inverse transform unit 204, and outputs the reproduced image data.

[0149] The loop filter section (loop filter unit) is 207. The loop filter section 207 and... Figure 1The loop filter section 109 shown applies loop filtering, such as deblocking filtering, to the reproduced image and outputs a filtered image.

[0150] Terminal 208 is an output terminal that outputs the reproduced image data to an external source.

[0151] The image decoding operation in the image decoding device will be described below. In this embodiment, the bitstream generated in this embodiment is decoded.

[0152] exist Figure 2 In this device, the control unit (control unit) 200 is a processor that controls the entire image decoding apparatus, and the bit stream input from terminal 201 is input to the separation and decoding unit 202. The separation and decoding unit 202 separates information related to the decoding process and code data related to coefficients from the bit stream, and decodes the code data in the header of the bit stream. Specifically, the decoding unit 202 decodes the quantization control size information. In this embodiment, firstly, the data from... Figure 6A The quantization control size information of the image header of the bitstream shown is decoded. The quantization control size information thus obtained is output to the decoding unit 203 and the inverse quantization and inverse transform unit 204. In addition, code data in blocks of image data is output to the decoding unit 203.

[0153] The decoding unit 203 decodes the code data and obtains residual coefficients, prediction information, and quantization parameters. The residual coefficients and quantization parameters are output to the inverse quantization and inverse transform unit 204, and the obtained prediction information is output to the image reproduction unit 205.

[0154] Here, we will refer to Figures 10A to 10F This explains the process of allocating quantization parameters to sub-blocks based on the quantization control size. As mentioned above, in Figures 10A to 10F Each sub-block shows the block segmentation type and the quantization parameter (Qp) used during encoding in each sub-block on the left. Sub-blocks with diagonal shading represent sub-blocks associated with quantization parameters. Bold rectangles represent regions that share quantization parameters determined based on the quantization control size and the size of each object sub-block. The method of comparison with the quantization control size is similar to that used in image encoding devices. For example, see reference... Figures 8A to 8F and Figures 9A to 9F The shorter of the horizontal and vertical sides of each object sub-block is compared with the quantization control size. The middle diagram shows whether each sub-block has an importance coefficient. An importance coefficient means a non-zero coefficient in the residual coefficients. In other words, having an importance coefficient means that there is at least one non-zero residual coefficient in the sub-block. The arrows shown on the right side of the diagram indicate the decoding order. Within the sub-blocks of the quantization parameter encoding unit, the quantization parameters are decoded in the first sub-block, including the importance coefficients, in the decoding order. For example, in Figure 10BIn the decoding sequence, the first sub-block including the important coefficients is the top-right sub-block, so the quantization parameters are decoded for this sub-block. In this case, the bottom-left and bottom-right sub-blocks are in regions sharing quantization parameters, and the quantization parameters in the top-right sub-block have already been decoded. In other words, there is no encoded data for the quantization parameters associated with the bottom-left and bottom-right sub-blocks in the bitstream, and the quantization parameters associated with the bottom-left and bottom-right sub-blocks are not decoded. In the quantization and inverse quantization processing in the bottom-left and bottom-right sub-blocks, QpA, which is the same quantization parameter as that in the top-right sub-block, is used. Furthermore, there are no important coefficients in the top-left sub-block, so no quantization processing is performed; however, in processes using quantization parameters, such as deblocking filtering, QpA, which is the same quantization parameter as that in the top-right sub-block, is used. Figure 10F In the decoding sequence, the first sub-block containing the important coefficients is the sub-block below it, therefore the quantization parameters associated with this sub-block are decoded. There is no [interchangeable] ... Figure 10F The encoded data of the quantization parameters associated with the upper and middle sub-blocks is used, but the quantization parameters associated with the upper and middle sub-blocks are not decoded. However, in Figure 10F In the sub-blocks above and in the middle, with Figure 10B Similar to the upper left sub-block, QpA, which is the same quantization parameter as the quantization parameter of the lower sub-block, is used in processing involving quantization parameters, such as deblocking filtering. Thus, within the sub-block of the quantization parameter encoding unit determined according to the quantization control size, the quantization parameters are decoded for the first sub-block, including the important coefficients, in the decoding order.

[0155] The inverse quantization and inverse transform unit 204 generates orthogonal transform coefficients by inverse quantizing the input residual coefficients, further applies the inverse orthogonal transform, and reproduces the prediction error. In the inverse quantization of each sub-block, inverse quantization is performed using common quantization parameters for each region sharing quantization parameters. The acquired prediction information is output to the image reproduction unit 205.

[0156] The image reproduction unit 205 reproduces the predicted image by reading the frame memory 206 as needed using the prediction information input from the decoding unit 203. The image reproduction unit 205 reproduces image data based on the predicted image and the prediction error input from the inverse quantization and inverse transform unit 204, inputs this image data into the frame memory 206, and stores the image data in the frame memory 206. The stored image data is used as a reference during prediction.

[0157] Loop filter section (loop filter unit) 207 and Figure 1Similar to step 109, the reconstructed image is read from frame memory 206, and loop filtering, such as deblocking filtering and sample adaptive compensation, is applied to the reconstructed image. The filtered image is then input back into frame memory 206.

[0158] The reproduced image stored in frame memory 206 is eventually output to the outside from terminal 208.

[0159] Figure 4 This is a flowchart illustrating the image decoding process performed in an image decoding device.

[0160] First, in step S401, the separation and decoding unit 202 separates information related to the decoding process and code data related to the coefficients from the bit stream, decodes the code data in the header, and obtains the quantization control size information.

[0161] In step S402, the decoding unit 203 decodes the code data separated in step S401 and obtains block segmentation information, residual coefficients, prediction information and quantization parameters.

[0162] In step S403, the inverse quantization and inverse transform unit 204 performs inverse quantization on a sub-block basis, further applies an inverse orthogonal transform, and obtains the prediction error. Specifically, the quantization control size information (e.g., the length of one side of a square block) is compared with the size of each sub-block (e.g., the length of the short or long side) determined based on the obtained block segmentation information. As a result, regions (sub-blocks) sharing quantization parameters are determined. Inverse quantization is performed using quantization parameters assigned to the sub-blocks respectively.

[0163] In step S404, the image reproduction unit 205 reproduces the predicted image based on the prediction information obtained in step S402. The image reproduction unit 205 further reproduces image data based on the reproduced predicted image and the prediction error generated in step S403.

[0164] In step S405, the control unit 200 of the image decoding device determines whether the decoding of all blocks in the frame is complete, and if the decoding is complete, the process proceeds to step S406; otherwise, the process proceeds to step S402 for the next block.

[0165] In step S406, the loop filter unit 207 applies loop filtering to the image data reproduced in step S404 to generate a filtered image and ends the processing.

[0166] This operation allows for the decoding of bitstreams whose data volume has been suppressed by using quantization parameters that control the size information of the quantization parameters.

[0167] In image decoding devices, for example Figure 6A As shown, the quantization control size information is included in the bitstream of the image header for decoding; however, the information encoding location is not limited to this. Optionally, the quantization size information can be as follows: Figure 6B The encoding shown is in the sequence header of the image, or it can be encoded in another location.

[0168] In this embodiment, the determination is made by comparing the shorter of the horizontal and vertical lengths of each object sub-block with the quantization control size; however, the invention is not limited thereto. For example, as... Figures 11A to 11F As shown, the lengths of the long sides of the horizontal and vertical dimensions of each object sub-block can be compared with the quantization control size, and the regions sharing quantization parameters can be determined. Figures 11A to 11F Since the longer side of all sub-blocks is longer than the quantization control size (16), the quantization parameters are encoded for each sub-block. In this case, for rectangular sub-blocks, instead of reducing the amount of code for the quantization parameters, quantization processing compatible with objects of varying lengths can be performed.

[0169] In addition, as another embodiment, the region sharing quantization parameters can be determined by comparing the pixel count of each object sub-block with the pixel count of the quantization control size. Figures 12A to 12F This illustrates the comparison between the pixel count of the size of each object sub-block and the pixel count of the quantization control size. Figures 12A to 12F In this configuration, the quantization control size is 16×16 pixels, therefore the pixel count is 256. Figures 12A to 12F In the example, the pixel count in all sub-blocks is greater than or equal to 256 pixels. Therefore, in Figures 12A to 12F In the example, the quantization parameters are not shared across all sub-blocks, but are used individually, and each quantization parameter is decoded. Using this structure, quantization parameter control based on the pixel count within each sub-block can be achieved regardless of the shape of the sub-blocks.

[0170] In this embodiment, the quantization control size is assumed to be one side of a square block. Optionally, the quantization control size can be one side of a rectangular block. In this case, the width and height of the quantization control size can be specified. In this case, the vertical and horizontal lengths of each object sub-block can be compared with the width and height of the quantization control size, respectively. If two or one of these lengths is greater than or equal to the quantization control size, the quantization parameter can be decoded for each sub-block. If there are sub-blocks in the basic block to be processed whose lengths are both or one of them is less than the quantization control size, a quantization parameter to be shared among the sub-blocks that meet this condition is decoded. Using this structure, different quantization parameter controls can be implemented for vertically elongated rectangular sub-blocks and horizontally elongated rectangular sub-blocks.

[0171] In this embodiment, the reference upon which the quantization parameters are encoded is defined by the space size; however, the structure is not limited to this. The depth of the quantization control (hereinafter referred to as the quantization control depth) that indicates the number of times the basic block is segmented is decoded from the bitstream, and this quantization control depth is compared with the segmentation depth of each object sub-block. The regions of sub-blocks sharing the quantization parameters to be decoded can be determined based on this structure. Here, reference will be made to... Figures 13A to 13F To illustrate the depth of each object's sub-block. Figures 13A to 13F In the diagram, D0 and D1 indicate depth 0 and depth 1, respectively. Figure 13A This shows that the basic block has not been divided once and the depth of the sub-block is 0 (D0). Figure 13B This shows that the basic block has been divided into four parts and each sub-block has a depth of 1 (D1). Figure 13C and Figure 13D This shows that the basic block has been divided into two parts and each sub-block has a depth of 1 (D1). Figure 13E and Figure 13F This shows that the basic block has been divided into three parts, and each sub-block has a depth of 1 (D1). Thus, the depth of each sub-block after the basic block has been divided once increases by 1 in any of the four-part, two-part, and three-part divisions. Next, refer to... Figures 14A to 14E This explains how to decode quantization parameters based on the quantization control depth and the segmentation depth of each object's sub-blocks. Figures 14A to 14E The outermost square is shown as the basic block. Figures 14A to 14E In each figure, the left side shows the segmented sub-blocks, and the right side shows the regions sharing the quantization parameter. Within each block in these figures, D0, D1, D2, D3, or D4 indicates the depth of the sub-block. For example, D0 indicates depth 0, and D4 indicates depth 4. Qp represents the quantization parameter. Figure 14A , Figure 14B , Figure 14C , Figure 14D and Figure 14E The examples show the cases where the quantization control depth is 0, 1, 2, 3, and 4, respectively. Figure 14A In the case where the quantization control depth is 0, common quantization parameters are decoded across all sub-blocks of the graph. Quantization parameters are decoded in the first sub-block, including the most important coefficients, in the decoding order. In this case, only one quantization parameter needs to be decoded. Figure 14B In the case that the quantization control depth is 1, Figure 14B The right-hand diagram shows blocks that encode common quantization parameters on a per-block basis. Each of these quantization parameters is associated with and decoded in the order of decoding, including the first sub-block containing the most important coefficients. Figure 14B In this case, the number of quantization parameters to be decoded is 4. Figure 14C In the case that the quantization control depth is 2, Figure 14C The right-hand diagram shows the encoding of common quantization parameters on a block-by-block basis. Each of these quantization parameters is associated with and decoded in the order of the first sub-block containing the most important coefficients. In this case, there are 11 quantization parameters to be decoded. Figure 14D In the case that the quantization control depth is 3, i.e., when the quantization control depth is 3, Figure 14D The right-hand diagram shows blocks that encode common quantization parameters on a unit basis. Similarly, in this case, each quantization parameter is associated with and decoded by the first sub-block, including the most important coefficients, in the decoding order. Figure 14D In this case, the number of quantization parameters to be decoded is 23. Figure 14E In the case that the quantization control depth is 4, i.e., when the quantization control depth is 4, Figure 14D The right-hand diagram shows the common quantization parameters encoded on a block-by-block basis (i.e., for each block). Quantization parameters for sub-blocks excluding importance coefficients are not decoded; however, all 27 quantization parameters are decoded when it is assumed that all sub-blocks include importance coefficients. Here, reference will be made to... Figure 15A and 15B This is to illustrate the case where there are sub-blocks in the region of shared quantization parameters that do not include importance coefficients. Figure 15A and Figures 14A to 14E The sub-blocks are shown in the same way, and D0, D1, D2, D3 or D4 indicate the depth of the sub-blocks. Figure 15B and Figures 14A to 14E The same situation illustrates an example of a shared region for quantization parameters determined by the quantization control depth and the depth of each sub-block. Figure 15A and 15B The case where the quantization control depth is 2 is shown as an example. Figure 15A and 15BIf no significance coefficient exists in any of the three sub-blocks within a quantization parameter encoding unit (QoG), the quantization parameters are not decoded in these three sub-blocks. However, for processing using quantization parameters, such as deblocking filtering, the same value (i.e., QpF) as the quantization parameter decoded immediately preceding the quantization parameter encoding unit in decoding order is applied to the three sub-blocks in the QoG. In quantization parameter encoding units where no significance coefficient exists, the quantization parameter encoded immediately preceding the quantization parameter in decoding order during processing using quantization parameters can be used; however, the invention is not limited to this. For example, QpD, which is the quantization parameter of the upper adjacent quantization parameter encoding unit, or QpF, which is the left adjacent quantization parameter, can be used. Optionally, a value calculated from the quantization parameters of multiple quantization parameter encoding units, such as an average value, can be used. Optionally, an initial value of the quantization parameters for a slice can be used. A slice means a unit for dividing a frame and consists of at least one or more basic blocks. In this way, the depth of each object sub-block can be compared with the quantization control depth, and if the depth of each object sub-block is less than or equal to the quantization control depth, the quantization parameters to be shared can be decoded. Since the sub-block segmentation information is decoded at each segmentation, quantization parameter encoding control that is highly compatible with the sub-block segmentation information can be achieved, resulting in the decoding of bitstreams with simplified syntax.

[0172] Figure 17 This is a flowchart illustrating the decoding process using quantization parameters with quantization control depth.

[0173] In step S1701, the separation and decoding unit 202 decodes quantization control size information and information related to quantization control depth from the bit stream. The decoding unit 203 decodes the segmentation information and obtains information related to the shape and depth of each sub-block. The inverse quantization and inverse transform unit 204 compares the quantization control depth information decoded by the separation and decoding unit 202 with the segmentation depth of each sub-block obtained from the segmentation information decoded by the decoding unit 203.

[0174] In step S1702, the inverse quantization and inverse transform unit 204 determines the regions of sub-blocks whose segmentation depth is greater than the quantization control depth, which are the result of the comparison in S1701, as regions that share a quantization parameter.

[0175] In step S1703, the inverse quantization and inverse transform unit 204 decodes the quantization parameters associated with sub-blocks in the determined region that have at least one important coefficient as a residual coefficient in the decoding order.

[0176] In step S1704, the inverse quantization and inverse transform unit 204 performs inverse quantization on the sub-blocks in the region determined in step S1702 using the quantization parameters decoded in step S1703. This process is repeated for each basic block in the frame. Figure 17 The processing.

[0177] Furthermore, both quantization control size and quantization control depth can be generated and combined. In this case, besides Figure 6A and 6B In addition to the information related to the quantization control size, information related to the quantization control depth is also decoded.

[0178] For example, if the depth of each object sub-block is less than or equal to the quantization control depth, and the shorter of the horizontal and vertical edges of each object sub-block is greater than or equal to the quantization control size, the quantization parameters are decoded for each sub-block. Specifically, for each region associated with the quantization control depth, the length of the shorter edge of each sub-block is compared with the quantization control size. In each region, if there is a sub-block with a shorter edge greater than or equal to the quantization control size, the quantization parameters are decoded for each sub-block in that region. If the length of the shorter edge of each sub-block in that region is not greater than or equal to the quantization control size, a quantization parameter shared among the sub-blocks in that region is decoded.

[0179] Optionally, if the depth of each object sub-block is less than or equal to the quantization control depth, and the length of the longer side of the horizontal and vertical edges of each object sub-block is greater than or equal to the quantization control size, the quantization parameters can be decoded for each sub-block. Specifically, the size of each sub-block is compared with the quantization control size for each region associated with the quantization control depth. In each region, if there is a sub-block with a longer side greater than or equal to the quantization control size, the quantization parameters are decoded for each sub-block in that region. If the length of the longer side of each sub-block in that region is not greater than or equal to the quantization control size, a quantization parameter shared among the sub-blocks in that region is decoded.

[0180] Optionally, quantization parameters can be decoded when the depth of each object sub-block is less than or equal to the quantization control depth, and the pixel count of each object sub-block is greater than or equal to the pixel count of the quantization control size. In this case, the size of each sub-block is compared with the quantization control size for each region associated with the quantization control depth. In each region, if there is a sub-block with a pixel count greater than or equal to the pixel count of the quantization control size, the quantization parameters are decoded for each sub-block in that region. If the size of each sub-block in each region is not greater than or equal to the quantization control size, a quantization parameter shared among the sub-blocks in that region is decoded.

[0181] Optionally, the width and height of the quantization control size can be specified and used in conjunction with the quantization control depth. In this case, the vertical and horizontal lengths of each sub-block are compared with the width and height of the quantization control size, respectively, for each region associated with the quantization control depth. In this case, if there are sub-blocks with one or two sides whose length is greater than or equal to the quantization control size, the quantization parameters are decoded for each sub-block.

[0182] Figure 19 This is a flowchart illustrating the decoding process using quantization parameters, namely quantization control size and quantization control depth.

[0183] In step S1901, the separation and decoding unit 202 decodes information related to the quantization control size and information related to the quantization control depth from the bit stream. The decoding unit 203 decodes the segmentation information and obtains information related to the shape and depth of each sub-block. The inverse quantization and inverse transform unit 204 compares the quantization control depth with the segmentation depth of each sub-block and determines the region into which the sub-block is segmented based on the depth corresponding to the quantization control depth.

[0184] In step S1902, for each region determined in step S1901, the inverse quantization and inverse transform unit 204 compares the size of each sub-block included in that region with the quantization control size. The size of each sub-block can be obtained from the shape of each sub-block determined based on the decoded segmentation information.

[0185] In step S1903, as a result of the comparison in step S1902, the inverse quantization and inverse transform unit 204 determines whether a quantization parameter is shared among the sub-blocks in the region to be processed. Based on this determination, each sub-block in each region is quantized using the quantization parameter associated with the sub-block, and residual coefficients for each sub-block are generated.

[0186] In step S1904, if a quantization parameter is shared among sub-blocks in the object region, the decoding unit 203 decodes the quantization parameter associated with the first sub-block having at least one importance coefficient in the decoding order. There is no encoded data for the quantization parameter associated with another sub-block. If a quantization parameter is not shared among sub-blocks in the object region, the decoding unit 203 decodes the quantization parameter associated with each sub-block except those that do not include importance coefficients.

[0187] This decoding process is performed on each basic block.

[0188] When decoding quantization parameters using both quantization control size and quantization control depth, the quantization parameters are more likely to be associated with each sub-block when comparing its long side with the quantization control size. In other words, comparing the long side of each sub-block with the quantization control size is more suitable for objects with long, narrow rectangular shapes.

[0189] Using this structure, even in extremely long and narrow rectangular sub-blocks, quantization parameter encoding control can be achieved by using quantization to control both depth and size, even when using rectangular sub-blocks for multiple divisions.

[0190] When the sub-block segmentation type is hierarchically continuous, sharing quantization parameters can be prohibited regardless of the comparison result between the quantization control size and the size of each sub-block. Using this structure, quantization of sub-blocks suitable for objects with elongated shapes can be performed. In the above embodiments, Figure 1 and Figure 2 The processing unit shown is described as a hardware component. Optionally, the processing performed by the processing unit shown in these figures can be configured by a computer program.

[0191] Figure 5 This is a block diagram illustrating an example of the hardware structure of a computer that can be applied to an image display device according to the above embodiments.

[0192] CPU 501 controls the entire computer by using computer programs and data stored in RAM 502 or ROM 503, and performs the aforementioned processing performed by the image processing apparatus according to the embodiment. In other words, CPU 501 acts as... Figure 1 and Figure 2 The processing unit shown.

[0193] RAM 502 has areas for temporarily storing computer programs and data loaded from external storage device 506, or data obtained from the outside via I / F (interface) 507. RAM 502 also has working areas used by CPU 501 during various processes. In other words, RAM 502 can, for example, allocate frame memory or provide various other areas as needed.

[0194] The ROM 503 stores computer settings data and boot programs. The operation unit 504 is composed of a keyboard or mouse, etc. The operation unit 504 allows the computer user to input various instructions to the CPU 501 when operating the operation unit 504. The display unit 505 displays the processing results of the CPU 501. The display unit 505 is, for example, a liquid crystal display (LCD).

[0195] External storage device 506 is a large-capacity information storage device, typically a hard disk drive. External storage device 506 stores the OS (operating system) and the components that enable CPU 501 to perform its functions. Figure 1 and Figure 2 The computer program shows the functions of each part. Furthermore, the image data to be processed can be stored in the external storage device 506.

[0196] The computer programs and data stored in external storage device 506 are loaded into RAM 502 as needed under the control of CPU 501 and used as objects to be processed by CPU 501. Networks such as LANs and the Internet, as well as other devices such as projectors and display devices, can be connected to I / F 507. The computer acquires or sends various information via I / F 507. 508 represents the bus connecting the above-mentioned components.

[0197] The operations comprised of the above components and described in the above flowchart are primarily controlled by CPU 501.

[0198] The object of the present invention is also achieved in the following case: a storage medium containing code of a computer program that performs the above-described functions is supplied to a system, and the system reads and executes the code of the computer program. In this case, the code of the computer program read from the storage medium itself performs the functions of the above-described embodiments, and the storage medium containing the code of the computer program constitutes the present invention. Furthermore, an operating system (OS) running on a computer can perform some or all of the actual processing according to the instructions of the computer program code, and the above-described functions can be implemented through these processes.

[0199] Furthermore, the following mode can be implemented: Computer program code read from a storage medium is written into a function expansion card inserted into the computer or into a memory included in a function expansion unit connected to the computer. The CPU or other components included in the function expansion card or function expansion unit perform some or all of the actual processing according to the instructions of the computer program code to achieve the aforementioned functions.

[0200] When this invention is applied to a storage medium, the code of the computer program corresponding to the above flowchart is stored in the storage medium.

[0201] This invention is used in encoding and decoding devices for encoding and decoding still and moving images. In particular, this invention can be applied to encoding and decoding systems that use quantization processing.

[0202] Using this invention, quantization coefficients can be appropriately encoded not only for square sub-blocks but also for rectangular sub-blocks.

[0203] The present invention can be implemented by supplying a program for implementing one or more functions of the above embodiments to a system or device via a network or storage medium, and causing one or more processors in the computer of the system or device to read and execute the program. Optionally, embodiments of the present invention can be implemented by circuitry (e.g., an ASIC) that implements one or more functions.

[0204] The embodiments of the present invention are not limited to those described above. Various changes or modifications are applicable without departing from the spirit and scope of the invention. Therefore, the appended claims have been added to illustrate the scope of the invention.

[0205] This application claims the benefit of Japanese Patent Application 2018-122421, filed on June 27, 2018, the entire contents of which are incorporated herein by reference.

Claims

1. An image encoding device capable of encoding an image in blocks, the image encoding device comprising: The first segmentation unit is configured to segment the image into multiple coding tree units; The second segmentation unit is configured to divide the coding tree unit into multiple blocks; A comparison unit is configured to compare a segmentation value with a threshold for determining a block group with shared quantization parameters, the segmentation value being a value that increases with the number of times the coding tree unit is segmented, the block group being capable of including multiple blocks whose segmentation value is greater than the threshold; An encoding unit is configured to encode data indicating the value of a quantization parameter based on a comparison result obtained by the comparison unit; as well as The filtering unit is configured to perform deblocking filtering. Where at least one block in the block group includes non-zero coefficients, the encoding unit is configured to encode first data in association with the block including the non-zero coefficients, the first data indicating a first value of a quantization parameter shared in the block group. Where the block group consists of three blocks and the three blocks included in the block group do not contain non-zero coefficients, the encoding unit is configured not to encode the first data for the block group, and the filtering unit is configured to deblock the three blocks included in the block group by using a second value of a quantization parameter, the second value being derived by calculating the average of the quantization parameters for blocks other than the block group. The encoding unit is configured to encode information used to specify the threshold into a bitstream.

2. An image decoding device capable of decoding a bitstream encoded by segmenting an image into multiple coding tree units and further segmenting the coding tree units into multiple blocks, the image decoding device comprising: A comparison unit is configured to compare a segmentation value with a threshold for determining a block group with shared quantization parameters, the segmentation value being a value that increases with the number of times the coding tree unit is segmented, the block group being capable of including multiple blocks whose segmentation value is greater than the threshold; A decoding unit is configured to decode data indicating the value of a quantization parameter based on a comparison result obtained by the comparison unit; as well as The filtering unit is configured to perform deblocking filtering. Where at least one block in the block group includes non-zero coefficients, the decoding unit is configured to decode first data when processing a block including the non-zero coefficients, the first data indicating a first value of a quantization parameter shared in the block group. Where the block group consists of three blocks and the three blocks included in the block group do not contain non-zero coefficients, the decoding unit is configured not to decode the first data for the block group, and the filtering unit is configured to perform deblocking filtering on the three blocks included in the block group using a second value of a quantization parameter, the second value being derived by calculating the average of the quantization parameters for blocks other than the block group. The decoding unit is configured to decode information from the bitstream used to specify the threshold.

3. An image encoding method for encoding an image in blocks, the image encoding method comprising: The image is segmented into multiple coding tree units; The coding tree unit is divided into multiple blocks; The segmentation value is compared with a threshold used to determine a block group with shared quantization parameters. The segmentation value is a value that increases with the number of times the coding tree unit is segmented. The block group can include multiple blocks whose segmentation value is greater than the threshold. as well as The data indicating the value of the quantization parameter is encoded based on the comparison results. Where at least one block in the block group includes non-zero coefficients, first data is encoded in association with the block including the non-zero coefficients, the first data indicating a first value of a quantization parameter shared in the block group. Where the block group consists of three blocks and the three blocks included in the block group do not contain non-zero coefficients, the first data for the block group is not encoded, and the image encoding method further includes deblocking filtering of the three blocks included in the block group by using a second value of a quantization parameter, the second value being derived by calculating the average of the quantization parameters for blocks other than the block group, and The image encoding method further includes encoding information used to specify the threshold into a bitstream.

4. An image decoding method for decoding a bitstream encoded by segmenting an image into multiple coding tree units and further segmenting the coding tree units into multiple blocks, the image decoding method comprising: The segmentation value is compared with a threshold used to determine a block group with shared quantization parameters. The segmentation value is a value that increases with the number of times the coding tree unit is segmented. The block group can include multiple blocks whose segmentation value is greater than the threshold. as well as The data indicating the value of the quantization parameter is decoded based on the comparison results. Where at least one block in the block group includes non-zero coefficients, the first data is decoded when processing the block including the non-zero coefficients, the first data indicating a first value of a quantization parameter shared in the block group. Where the block group consists of three blocks and the three blocks included in the block group do not contain non-zero coefficients, the first data for the block group is not decoded, and the image decoding method further includes deblocking filtering the three blocks included in the block group by using a second value of a quantization parameter, the second value being derived by calculating the average of the quantization parameters for blocks other than the block group, and The image decoding method further includes decoding information from the bitstream used to specify the threshold.

5. A non-transitory computer-readable medium storing a computer-executable program, the computer-executable program being used to cause a computer to perform an image encoding method, the image encoding method being used to encode an image in blocks, the image encoding method comprising: The image is segmented into multiple coding tree units; The coding tree unit is divided into multiple blocks; The segmentation value is compared with a threshold used to determine a block group with shared quantization parameters. The segmentation value is a value that increases with the number of times the coding tree unit is segmented. The block group can include multiple blocks whose segmentation value is greater than the threshold. as well as The data indicating the value of the quantization parameter is encoded based on the comparison results. Where at least one block in the block group includes non-zero coefficients, first data is encoded in association with the block including the non-zero coefficients, the first data indicating a first value of a quantization parameter shared in the block group. Where the block group consists of three blocks and the three blocks included in the block group do not contain non-zero coefficients, the first data for the block group is not encoded, and the image encoding method further includes deblocking filtering of the three blocks included in the block group by using a second value of a quantization parameter, the second value being derived by calculating the average of the quantization parameters for blocks other than the block group, and The image encoding method further includes encoding information used to specify the threshold into a bitstream.

6. A non-transitory computer-readable medium storing a computer-executable program, the computer-executable program being used to cause a computer to perform an image decoding method, the image decoding method being used to decode a bitstream encoded by segmenting an image into a plurality of coding tree units and further segmenting the coding tree units into a plurality of blocks, the image decoding method comprising: The segmentation value is compared with a threshold used to determine a block group with shared quantization parameters. The segmentation value is a value that increases with the number of times the coding tree unit is segmented. The block group can include multiple blocks whose segmentation value is greater than the threshold. as well as The data indicating the value of the quantization parameter is decoded based on the comparison results. Where at least one block in the block group includes non-zero coefficients, the first data is decoded when processing the block including the non-zero coefficients, the first data indicating a first value of a quantization parameter shared in the block group. Where the block group consists of three blocks and the three blocks included in the block group do not contain non-zero coefficients, the first data for the block group is not decoded, and the image decoding method further includes deblocking filtering the three blocks included in the block group by using a second value of a quantization parameter, the second value being derived by calculating the average of the quantization parameters for blocks other than the block group, and The image decoding method further includes decoding information from the bitstream used to specify the threshold.

Images