Encoding device

Abstract

This encoding device for encoding an image in block units obtained by dividing the image acquires an original image to be encoded, recursively divides the image into square blocks or rectangular blocks, and derives a cost value and an in-screen prediction mode predicted for each block. On the basis of the total value of cost values of a combined block obtained by combining first blocks, and a cost value of a second block having the same size as the combined block, the encoding device repeatedly carries out a procedure in which the combined block is selected as the block shape with regard to the size of the combined block if the total value is smaller than the cost value of the second block, and the second block is selected as the block shape with regard to the size of the combined block when the cost value of the second block is smaller than the total value. This procedure is repeated from the smallest block size to the largest block size in the recursively divided blocks on the basis of the size of the first block, whereby the encoding device determines a final block division shape.

Description

encoding device

[0001] The disclosed technology relates to an encoding device.

[0002] Conventionally, in the transmission and distribution of digital video, video encoding based on international video encoding standards such as MPEG (Moving Picture Experts Group)-2, H.264 / AVC (Advanced Video Coding), and H.265 / HEVC (High Efficiency Video Coding) has been used to reduce the amount of video data during transmission and distribution, thereby achieving high-quality video transmission and distribution in a variety of network environments.

[0003] In recent years, a new international video coding standard, Versatile Video Coding (VVC), has been standardized to achieve even higher compression in preparation for the emergence of a variety of video content, such as ultra-high definition video such as 4K video and 8K video, as well as distribution services using 360-degree video and VR (Virtual Reality).

[0004] In video coding standards such as H.265 / HEVC and VVC, each video frame to be coded is first divided into coding tree units (CTUs), which correspond to macro blocks (MBs) in MPEG-2 and H.264 / AVC.

[0005] Furthermore, each CTU is further divided into units called CUs (Coding Units). In H.265 / HEVC, each CTU is divided based on a recursive quad tree (hereinafter referred to as "QT division"), which is a tree structure in which each node is branched into four branches. Therefore, possible CU sizes include 8x8, 16x16, 32x32, and 64x64, and a CTU is formed by combining CUs of each size. Then, for each CU, PUs (Prediction Units), which are units of prediction processing, and TUs (Transform Units), which are units of transform processing, are further set, and various processes in encoding are performed.

[0006] On the other hand, in VVC, each CTU is divided not only into quadtree blocks, but also into binary tree block partitioning (hereinafter referred to as "BT partitioning"), which divides a block into two equal halves horizontally or vertically, and ternary tree block partitioning (hereinafter referred to as "TT partitioning"), which divides a block into thirds horizontally or vertically at a side ratio of 1:2:1. CUs in VVC are divided into variable-sized CUs, including not only squares but also rectangles, by repeatedly performing BT partitioning or TU partitioning horizontally or vertically for each CU generated after recursively repeating QT partitioning for the CTU. Therefore, the size of a CU in VVC can be selected from a variety of sizes, such as 32x4 or 8x16, and a CTU is formed by combining these CU sizes. Therefore, VVC does not support H.264 standard. Compared to H.265 / HEVC, VVC can take a more complex block division shape that is more suited to the picture (texture) and characteristics of the frame to be coded, allowing for highly efficient coding. Note that, in VVC, prediction processing, conversion processing, and other processes are performed on a CU basis (Non-Patent Document 1), so the units PU and TU used in H.265 / HEVC do not exist.

[0007] Jianle Chen, Yan Ye, Seung Hwan Kim, "Algorithm description for Versatile Video Coding and Test Model 11 (VTM 11)", Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 20th Meeting, by teleconference, 7-16 October 2020, VTM algorithm description (JVET-N2002-v2) https: / / jvet-experts.org / doc_end_user / current_document.php?id=10541

[0008] However, since the block patterns in VVC are much larger than those in HEVC, an enormous amount of processing is required for the encoder that performs the encoding to evaluate and compare all of the division patterns and determine the size of the CU.

[0009] This is because, when determining the size of a CU, QT partitioning, BT partitioning (vertical and horizontal), and TT partitioning (vertical and horizontal) are each tried, prediction processing, transformation and quantization processing are performed, and evaluation is performed based on the RD (Rate-Distortion) cost to determine the most efficient CU size and partition shape. Also, a method can be considered to reduce the amount of computation by terminating the process as no further partitioning attempts are necessary during the process of recursively dividing a CU, starting from a large size. However, the stage at which the process is terminated largely depends on the image pattern and characteristics of the video, and the amount of processing is not constant. If this encoding process is to be implemented using an architecture that assumes processing within a given fixed time, such as pipeline processing, it is necessary to assume the longest-time case to avoid pipeline stalls, and therefore the benefits of terminating the process cannot be fully obtained.

[0010] The disclosed technology has been developed in consideration of these circumstances, and aims to provide an encoding device that performs encoding while reducing the amount of processing required to determine block division during encoding, while maintaining a prediction mode that corresponds to the characteristics of the original image, for an encoding processing architecture that assumes a constant processing time.

[0011] A first aspect of the present disclosure is a coding device that codes an image in units of blocks into which the image is divided, and includes: an acquisition unit that acquires an original image to be coded; a narrowing down unit that recursively divides the image into square blocks or rectangular blocks; a derivation unit that derives a predicted intra-screen prediction mode and cost value for each of the blocks; and a determination unit that determines a final block division shape by repeating a procedure of adding up the cost values ​​of a combination block obtained by combining first blocks and the cost value of a second block of the same size as the combination block, selecting the combination block as a block shape for the size of the combination block if the added up value is smaller, and selecting the second block as a block shape for the size of the combination block if the cost value of the second block is smaller, with the procedure being repeated from the smallest block size to the largest block size in the recursively divided blocks as the size of the first block.

[0012] The disclosed technology has been developed in consideration of these circumstances, and for an encoding processing architecture that assumes a constant processing time, it is possible to perform encoding while reducing the amount of processing required for determining block division during encoding, while maintaining a prediction mode that corresponds to the characteristics of the original image.

[0013] 1 is a block diagram showing an example of the configuration of an encoding device. FIG. 2 is a diagram for explaining an example of block division. FIG. 3 is a diagram for explaining an intra-screen prediction mode. FIG. 4 is a block diagram showing an example of the configuration of a division unit of a first embodiment. FIG. 5 is a diagram for explaining an example of the operation of a narrowing-down unit. FIG. 6 is a diagram for explaining an example of the timing of a size determination process. FIG. 7 is a diagram for explaining an example of a size determination process. FIG. 8 is a diagram for explaining an example of a case where divided regions do not match. FIG. 9 is a block diagram showing an example of a case where divided regions match. FIG. 10 is a block diagram showing an example of the hardware configuration of an encoding device. FIG. 11 is a flowchart showing an example of the flow of encoding processing executed in the division unit of an encoding device. FIG. 12 is a block diagram showing an example of the configuration of a division unit of a second embodiment. FIG. 13 is a diagram for explaining an example of a size determination process of the second embodiment.

[0014] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same or equivalent components and parts in each drawing are given the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0015] First Embodiment First, a process of encoding video by an encoding device will be described with reference to Figures 1 to 3. Figure 1 is a block diagram showing an example of the functional configuration of an encoding device 1 used to describe the process of encoding video.

[0016] As shown in FIG. 1, the encoding device 1 includes a division unit 10, a residual signal generation unit 11, a frequency transformation unit 12, a quantization unit 13, an encoding unit 14, an inverse quantization unit 15, an inverse frequency transformation unit 16, a decoded image generation unit 17, a loop filter processing unit 18, a reference image buffer unit 19, and a predicted image generation unit 20.

[0017] The division unit 10 divides the CTUs of the original image to be encoded into CUs, determines the size of the CUs, and determines the prediction mode of each CU. FIG. 2 shows an example of block division in VVC. As shown in FIG. 2, the original image is divided into CTUs, each block having a range of 128 pixels in the horizontal and vertical directions. The CTUs are further divided into CUs, which are blocks divided recursively, and the CUs are encoded as a single unit. Hereinafter, the size of a CTU and a CU is expressed as "(number of pixels in the horizontal direction) x (number of pixels in the vertical direction)." For example, a CTU having a range of 128 pixels in the horizontal and vertical directions is expressed as 128 x 128. In addition, in the case of VVC, a CTU is divided into QT blocks at least once, so the maximum size of a CU is 64 x 64.

[0018] Furthermore, as shown in Figure 2, not only is a CTU recursively divided into four CUs, each of which is 64x64, 32x32, 16x16, and 8x8, but it may also be divided into two (BT) or three (TT) blocks in the horizontal or vertical direction. Specifically, there are six possible division shapes, as shown in Figure 2. By having such flexible block division patterns, VVC achieves more efficient encoding processing than the conventional HEVC.

[0019] The division unit 10 shown in FIG. 1 acquires an original image to be coded and a decoded image from a reference image buffer unit 19 (described later), and divides each CTU related to the original image into the above-mentioned CUs. The division unit 10 outputs the original image divided into each CU size to a residual signal generation unit 11. The division unit 10 also outputs the CU size and the prediction mode of the CU to a predicted image generation unit 20. Note that a method for determining the size of the CUs divided by the division unit 10 will be described later. Also, as an example, in this embodiment, an embodiment in which an intra-screen prediction mode is applied as the prediction mode will be described. The intra-screen prediction mode is a value for determining pixels to be referenced for coding from blocks in an image whose coding has been completed before the block to be coded, and the direction in which the referenced pixels are located is predetermined by the value (number) of the intra-screen prediction mode.

[0020] It should be noted that when determining the size of a CU and the intra-screen prediction mode during encoding, it is not necessary to use blocks within an encoded image, and it may be performed using pixels of the input original image. In this case, the prediction mode is determined using pixels different from the reference pixels used when decoding on the decoder side, and there is a possibility that the prediction accuracy may be inferior, but since there is no need to actually encode and decode to generate reference pixels, it is possible to simplify the implementation on the encoder side.

[0021] 3 shows intra-frame prediction modes in VVC. In VVC, there are a total of 67 prediction modes, including a mode in which prediction is performed by referring to neighboring pixels of the CU in question from 65 directions between 45 degrees diagonally downward to the left (mode number 2) and 45 degrees diagonally upward to the right (mode number 66), a planar mode (mode number 0) in which prediction is performed using four points of neighboring pixels, and a DC mode (mode number 1) in which prediction is performed using the average of neighboring pixels. Furthermore, wide-angle prediction has been introduced for rectangular prediction blocks, which shifts the allocation of prediction directions according to the ratio of the short side to the long side (mode numbers 67 and above in FIG. 3 or negative mode numbers are targeted).

[0022] The residual signal generation unit 11 obtains the original image divided into each CU from the division unit 10, and obtains a predicted image corresponding to each CU from the predicted image generation unit 20. The residual signal generation unit 11 outputs a difference between a CU in the obtained original image and a CU in the predicted image corresponding to the CU of the original image as a predicted residual signal to the frequency conversion unit 12.

[0023] The frequency transform unit 12 receives the prediction residual signal from the residual signal generation unit 11 and performs frequency transform using a discrete cosine transform or a discrete sine transform on the prediction image in both the horizontal and vertical directions. The frequency transform unit 12 outputs a set of derived transform coefficients as a result of the frequency transform to the quantization unit 13.

[0024] The quantization unit 13 acquires the set of transform coefficients from the frequency transform unit 12 and quantizes the transform coefficients using a predetermined quantization parameter. The quantization unit 13 outputs the quantized transform coefficients to the encoding unit 14 and the inverse quantization unit 15.

[0025] The encoding unit 14 obtains the quantized transform coefficients from the quantization unit 13, encodes the quantized transform coefficients, and outputs the resulting bit stream to an external device (not shown).

[0026] The inverse quantization unit 15 obtains the quantized transform coefficients from the quantization unit 13, and performs inverse quantization processing on the quantized transform coefficients to generate transform coefficients. The inverse quantization unit 15 outputs the transform coefficients to the inverse frequency transform unit 16 as a result of the inverse quantization processing.

[0027] The inverse frequency transform unit 16 obtains the transform coefficients from the inverse quantization unit 15, performs inverse frequency transform on the transform coefficients, and generates a prediction residual signal. The inverse frequency transform unit 16 outputs the prediction residual signal to the decoded image generation unit 17.

[0028] The decoded image generation unit 17 obtains a prediction residual signal from the inverse frequency transform unit 16, and obtains a prediction image of the original image from a prediction image generation unit 20 (described later). The decoded image generation unit 17 generates a decoded image of the original image for each CU by adding the prediction residual signal to the prediction image for each CU. The decoded image generation unit 17 outputs the decoded image of the original image to the loop filter processing unit 18 and the reference image buffer unit 19.

[0029] The loop filter processing unit 18 obtains the decoded image from the decoded image generation unit 17 and performs filter processing on the decoded image to reduce encoding distortion. The loop filter processing unit 18 outputs the decoded image after the filter processing to the reference image buffer unit 19.

[0030] The reference image buffer unit 19 acquires the decoded image before filtering from the decoded image generation unit 17 and acquires the decoded image after filtering from the loop filter processing unit 18. The reference image buffer unit 19 stores the acquired decoded image before filtering and the decoded image after filtering as reference images. In other words, the reference image buffer unit 19 stores the decoded image obtained by decoding a past original image as the reference image.

[0031] The predicted image generation unit 20 obtains the prediction mode of the CU related to the original image from the division unit 10, and obtains a past decoded image stored as a reference image from the reference image buffer unit 19. The predicted image generation unit 20 generates a predicted image of the original image based on the prediction mode of the original image and the past decoded image data. The predicted image generation unit 20 outputs the predicted image to the residual signal generation unit 11 and the decoded image generation unit 17.

[0032] Next, the functional configuration and data flow of the division unit 10 in the encoding device 1 will be described with reference to Fig. 4. Fig. 4 shows a block diagram illustrating an example of the functional configuration of the division unit 10 of this embodiment.

[0033] As described above, the division unit 10 divides the CTU of the original image to be coded into CUs, and determines the CU division shape within the CTU, the size of each CU, and the intra-frame prediction mode for each CU.

[0034] As shown in FIG. 4, the dividing unit 10 includes, as functional components, an acquiring unit 41, a narrowing down unit 42, a deriving unit 43, and a determining unit 44.

[0035] For example, the acquisition unit 41 acquires a CTU to be coded in the current frame from an original image to be coded or a reference image that has been coded in the past and stored in the reference image buffer unit. The acquisition unit 41 outputs the acquired CTU to the narrowing unit 42.

[0036] When dividing a CTU into CUs of various sizes, the narrowing down unit 42 narrows down, for each size, whether to divide a square-sized CU obtained in a certain QT division into rectangular (BT, TT) divisions vertically or horizontally, and whether to perform BT division or TT division.

[0037] FIG. 5 shows an example of a narrowing-down method. In intra-frame prediction, pixels of a CU are generated from adjacent pixel values ​​of surrounding CUs by copying pixel values ​​in the prediction direction indicated by the prediction mode. Therefore, if the CU has a texture pattern similar to that of the surrounding CUs, a better predicted image can be generated by predicting the pixels of the surrounding CUs. Here, the texture pattern refers to the pixel value arrangement pattern within a certain area. A so-called vertical striped pattern is expressed as a vertical texture pattern, and a horizontal striped pattern is expressed as a horizontal texture pattern. Similarly, when the pixel value difference between adjacent pixels in the horizontal direction is obtained, that is, when the value obtained by differentiating the pixel values ​​in the horizontal direction exceeds a predetermined threshold, it is expressed as having a vertical texture pattern. Also, when the pixel value difference between adjacent pixels in the vertical direction is obtained, that is, when the value obtained by differentiating the pixel values ​​in the vertical direction exceeds a predetermined threshold, it is expressed as having a horizontal texture pattern. A more general expression is that the texture is in direction B, which is shifted by 90 degrees from direction A (the direction in which the difference between adjacent pixels is greatest) of the pixel edge extracted by an edge extraction filter, etc. Furthermore, the difference value between adjacent pixels or the value after differential filtering obtained when deriving the direction of the texture pattern is expressed as texture intensity.

[0038] Therefore, in the narrowing-down unit 42 of this embodiment, if the CU itself has a vertical texture pattern, it will be divided vertically, if it has a horizontal texture pattern, it will be divided horizontally, or if it cannot be determined whether it is vertical or horizontal, it will not be divided vertically or horizontally.

[0039] 5, the narrowing-down unit 42 determines which direction the texture of the CU to be coded is and whether its strength is equal to or greater than a threshold. Specifically, for vertical or horizontal division, the narrowing-down unit 42 compares the sums of absolute values ​​of the vertical or horizontal texture strengths of each pixel in the CU that are equal to or greater than a threshold.

[0040] Σ|Texture intensity in the horizontal direction|>Σ|Texture intensity in the vertical direction|+Th (Equation 1) Σ|Texture intensity in the horizontal direction|+Tv<Σ|Texture intensity in the vertical direction| (Equation 2)

[0041] If the above (Equation 1) is satisfied, the texture strength in the horizontal direction is determined to be greater, and the CU is divided vertically. On the other hand, if the above (Equation 2) is satisfied, the texture strength in the vertical direction is determined to be greater, and the CU is divided horizontally.

[0042] Here, "|Horizontal Texture Intensity|" indicates that the horizontal texture intensity is equal to or greater than a threshold, "|Vertical Texture Intensity|" indicates that the vertical texture intensity is equal to or greater than a threshold, "Σ" indicates the sum, and "Th" and "Tv" are given offsets.

[0043] If both (Equation 1) and (Equation 2) are not satisfied, it is determined that dividing vertically or horizontally is inappropriate, and division is not performed. If division is not performed, evaluation is performed using only QT for that size, and the number of narrowed-down candidates for the next smaller size is increased (+1). For example, if the size of the target CU is 2N x 2N and division is not performed, no rectangular division attempt is performed for the 2N x 2N size, and the number of rectangular division attempts for the N x N size is increased by 1.

[0044] On the other hand, if it is determined that the division is vertical or horizontal, the process proceeds to step S52 or step S53 to narrow down the division to either BT division or TT division.

[0045] In step S52, the narrowing-down unit 42 calculates the average pixel value for each of the four regions (1) to (4) obtained by dividing the CU vertically into four strips, and makes a judgment using the following (Equation 3).

[0046] |(2) average pixel value - (3) average pixel value| < Max(|(1) average pixel value - (2) average pixel value|, |(3) average pixel value - (4) average pixel value|) (Equation 3)

[0047] The intention of the above (Equation 3) is that if region (2) and region (3) have similar pixel values ​​(i.e., if the value on the left side is close to 0), TT division (i.e., TTV division) is performed so that region (2) and region (3) are in the same CU. Also, if the difference between region (1) and region (2) is smaller, BT division (i.e., BTv division) is performed so that region (1) and region (2) become the same region, and if the difference between region (3) and region (4) is smaller, BT division (i.e., BTv division) is performed so that region (3) and region (4) become the same region.

[0048] Although the average pixel values ​​are used in the above (Equation 3), it is also possible to calculate the SAD (Sum of Absolute Difference) of the pixel values ​​of each region to be compared, and select a BT / TT division shape such that the regions with the smaller value are the same region.

[0049] On the other hand, step S53 is the horizontal application of the processing of step S52 described above. The narrowing-down unit 42 calculates the average pixel value for each of the four regions (1) to (4) obtained by dividing the CU horizontally into four strips, and performs a determination using the above (Equation 3). If regions (2) and (3) have similar pixel values ​​(i.e., if the value on the left side is close to 0), TT division (i.e., TTh division) is performed so that regions (2) and (3) are the same CU. Furthermore, if the difference between regions (1) and (2) is smaller, BT division (i.e., BTh division) is performed so that regions (1) and (2) are the same region. If the difference between regions (3) and (4) is smaller, BT division (i.e., BTh division) is performed so that regions (3) and (4) are the same region.

[0050] By the above process, the narrowing-down unit 42 completes the process of narrowing down the options to either vertical or horizontal, BT (BTv, BTh) division or TT (TTv, TTh) division, or whether to perform division at all, and then performs subsequent processes using CU information based on the shape determined here. The narrowing-down unit 42 outputs the CU information and texture pattern information to the derivation unit 43.

[0051] Next, the derivation unit 43 performs intra-frame prediction for each size CU obtained by the quadtree and each rectangular CU narrowed down by the narrowing down unit 42, derives an encoding cost based on the residual and the amount of code when performing the prediction, and determines a prediction mode that provides the optimal cost. The encoding cost derivation and optimal cost described here are derived by a method that calculates the minimum cost value (cost J) obtained by the following (Equation 4), such as a method using RD (Rate-Distortion) optimization, in order to select encoding parameters (prediction modes, etc.) that minimize encoding distortion for a target code amount. Note that RD optimization is a process that maximizes image quality for a target code amount by selecting optimal encoding parameters so as to minimize a cost value, which is an index obtained by linearly combining the code amount R and the encoding distortion amount D. Therefore, in this embodiment, the smaller the cost value, the better the image quality.

[0052] Cost J=D+λ×R (D: distortion, R: code amount, λ: Lagrange multiplier) (Equation 4)

[0053] The determining unit 44 outputs the cost value of the target CU (target cost value) and the sum of the cost values ​​of the divided CUs (summed cost value) to the determining unit 44 .

[0054] The determining unit 44 compares the cost values ​​calculated for each CU size from small CU size to large CU size to determine the CU size.

[0055] FIG. 6 shows an example of the timing of the size determination process. In FIG. 6, time progresses from left to right. The solid-line blocks represent CU sizes obtained by quadtree partitioning, which, in ascending order, are a 4x4 pixel CU, an 8x8 pixel CU, a 16x16 pixel CU, a 32x32 pixel CU, and a 64x64 pixel CU. The dashed-line blocks represent CU sizes obtained by binary tree (BT) or ternary tree (TT) partitioning. For example, 16x16TTh indicates a CU obtained by further dividing a 16x16 pixel CU horizontally using a ternary tree. While there are four possible sizes, namely BTh, BTv, TTh, and TTv, the narrowing-down unit 42 narrows down the CU sizes for which cost calculation is actually performed to several sizes.

[0056] Furthermore, the processing time for a 16x16 pixel CU is four times that for an 8x8 pixel CU. This is because 16x16 pixels has an area equal to four 8x8 pixels. The same applies to the relationships between other CU sizes, and processing times corresponding to the area are required.

[0057] The thick vertical lines in Figure 6 indicate the timing at which mode / size determination processing is performed. For example, at times t1 to t3, the size of an 8x8 CU is determined. Next, at time t4, size determination is performed for at least three patterns of CUs: a 16x16 CU, a rectangular CU whose cost has been calculated by narrowing down the 16x16 CU using the division narrowing down unit, and a 16x16 pixel CU (consisting of four 8x8 CUs) obtained by combining four of the 8x8 CUs after mode / size determination selected at times t1 to t3.

[0058] As an example, details of mode size determination for a 16×16 CU at time t4 are shown in Fig. 7. The rectangular CU at this time is assumed to be a 16×16 BTh CU arranged in a binary tree in the horizontal direction.

[0059] First, in step 1, the determination unit 44 performs a mode size determination by comparing cost values ​​for the upper CU ((b) in FIG. 7 ) of the region after rectangular division and the two upper regions (two 8×8 CUs, (a) in FIG. 7 ) on the quadtree division side that represent the same range as the upper CU. Here, the determination cost value for the two upper regions on the quadtree division side is calculated by adding a predetermined offset to the sum of the cost values ​​calculated for the two CUs. Then, as a result of the cost value comparison, the lower cost value is adopted as the mode size determination result for the upper region. FIG. 7 shows a case where the cost value of (a) is smaller than the cost value of (b) ((a)<(b)).

[0060] Next, in step 2, the determination unit 44 performs similar size determination on the lower CU ((d) in FIG. 7) of the region after rectangular division and the two regions below the quadtree division side (two 8×8 CUs, (c) in FIG. 7). The lower cost value is then adopted as the mode size determination result for the lower region. FIG. 7 shows a case where the cost value of (d) is smaller than the cost value of (c) ((d)<(c)).

[0061] Next, in the determination of step 3, the determination unit 44 performs a size determination between the combined results of the comparisons of step 1 and step 2 (the combined side, (a) + (d) in FIG. 7 ) and a 16×16 CU ((e) in FIG. 7 ). The size determination is performed in the same manner as in step 1 and step 2, except that on the combined side, a predetermined offset is added to the sum of the cost values ​​of the CUs obtained in each of the previous determinations, and this is used as the cost value for determination. After the determination of step 3, the 16×16 CU partition shape is determined as the determination result of the 16×16 pixel CU size. FIG. 7 shows a case where the cost value of (a) + (d) is smaller than the cost value of (e) ((a) + (d) (e)).

[0062] Although the above describes examples of quadtree (QT) partitioning and binary tree (BT) partitioning, comparison is also performed for each region in the case of quadtree (QT) partitioning and ternary tree (TT) partitioning. However, because the partition ratios are different between QT partitioning and TT partitioning, the QT partitioning does not necessarily obtain a range equal to each CU region obtained by TT partitioning. In this case, the determination is skipped, and a single size determination is performed between the 16x16 CU and the 16x16TTh or 16x16TTv CU. FIG. 8A shows an example in which the determination is skipped because the partitioned regions do not match. FIG. 8B shows an example in which the determination is performed (not skipped) because the partitioned regions match.

[0063] The above describes the size determination of a 16x16 CU by the determination unit 44. Similarly, the determination unit 44 determines the size of a 32x32 CU using the determination result of the 16x16 CU. Similarly, the determination unit 44 determines the size of a 64x64 CU using the size determination result of the 32x32 CU. For example, the determination unit 44 determines the size of a 32x32 CU at time t16 using the size determination result of the 16x16 CU at time t4, the size determination result of the 16x16 CU at time t8, the size determination result of the 16x16 CU at time t12, and the size determination result of the 16x16 CU at time t16.

[0064] As described above, the block division shape within a 64x64 CU, i.e., the size of each CU, is determined by performing tournament processing based on comparison of cost values ​​from small-sized CUs to large-sized CUs. The determination unit 44 outputs, as division information, original images obtained by dividing the determined CU size to the residual signal generation unit 11 (see FIG. 1 ). The determination unit 44 also outputs, as division information, the determined CU size and the prediction mode of the CU to the predicted image generation unit 20.

[0065] According to the encoding device 1 of this embodiment, even when determining the size of a rectangular CU using hardware with a fixed processing time, it is possible to perform the determination process while reducing the amount of calculation required.

[0066] The encoding device 1 of this embodiment can be realized by the hardware configuration shown in Fig. 9. The hardware configuration of the encoding device 1 will be described with reference to Fig. 9. Fig. 9 is a block diagram showing an example of the hardware configuration of the encoding device 1. As shown in Fig. 9, the encoding device 1 includes a CPU (Central Processing Unit) 51, a ROM (Read Only Memory) 52, a RAM (Random Access Memory) 53, a storage 54, an input unit 56, a display unit 57, and a communication I / F (Interface) 58. Each component is connected to each other via a bus 59 so as to be able to communicate with each other.

[0067] The CPU 51 is a central processing unit that executes various programs, such as an encoding program 55, stored in the storage 54, and controls each unit. That is, the CPU 51 reads the program from the ROM 52 or the storage 54, and executes the program using the RAM 53 as a work area. The CPU 51 controls the above-mentioned components and performs various arithmetic processing in accordance with the program stored in the ROM 52 or the storage 54. That is, by executing the encoding program 55, the CPU 51 functions as the above-mentioned division unit 10, residual signal generation unit 11, frequency transformation unit 12, quantization unit 13, encoding unit 14, inverse quantization unit 15, inverse frequency transformation unit 16, decoded image generation unit 17, and loop filter processing unit 18.

[0068] The ROM 52 stores various programs and various data. The RAM 53 temporarily stores programs or data as a working area. The storage 54 is configured with a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including an operating system and an encoding program 55, as well as various data. The encoding program 55 may be a single program or a group of programs configured by multiple programs or modules. The storage 54 also functions as the reference image buffer unit 19 described above.

[0069] The input unit 56 includes a pointing device such as a mouse and a keyboard, and is used to input various information to the device itself. The display unit 57 is, for example, a liquid crystal display, and displays various information. Note that a touch panel system in which the input unit 56 and the display unit 57 are integrated may also be used.

[0070] The communication I / F 58 is an interface for communication between the device itself and external devices. For this communication, a wired communication standard such as Ethernet (registered trademark) or FDDI (Fiber Distributed Data Interface) or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark) is used.

[0071] The encoding device 1 of this embodiment is implemented as a general-purpose computer device such as a server computer or a personal computer (PC).

[0072] Furthermore, the operation of the division unit 10 of the encoding device 1 will be described with reference to FIG.

[0073] 10 shows a flowchart of an example of the processing executed in the division unit 10 in the encoding processing executed in the encoding device 1 of this embodiment. The division unit 10 executes the encoding processing shown in FIG. 10 by executing an encoding program 55 stored in the storage 54.

[0074] In step S100 of FIG. 10, the acquisition unit 41 acquires the CTU to be coded as described above.

[0075] In the next step S102, the narrowing-down unit 42 divides the CTU into CUs as described above.

[0076] In the next step S104, the derivation unit 43 derives the cost values ​​of the divided CUs and the sum of the cost values, as described above.

[0077] In the next step S106, the determination unit 44 performs size determination processing on the divided CUs as described above to determine the CU size. When the processing of step S106 is completed, the encoding processing for one CTU by the division unit 10 shown in Fig. 10 is completed. Note that the encoding processing shown in Fig. 10 is performed on all CTUs obtained by dividing the video (original image) to be encoded.

[0078] [Second embodiment] In the first embodiment, the narrowing-down unit 42 narrowed down the division direction (vertical or horizontal) and the division method (binary tree division, ternary tree division) to the derivation unit 43, and then the size was determined by comparing with each CU in the determination unit 44.

[0079] In contrast to this, in this embodiment, a form will be described in which the determining unit 44 determines the size based on prediction mode information in order to further increase the number of rectangular block comparison candidates.

[0080] Note that the encoding device 1 (see FIGS. 1 and 9), the explanatory diagram of block division (see FIG. 2), and the diagram of intra-screen prediction mode (see FIG. 3) in this embodiment are the same as those in the first embodiment, and therefore descriptions thereof will be omitted. Also, descriptions of the division narrowing method based on texture pattern information (see FIG. 5) and the timing of size determination processing (see FIG. 6) will be omitted.

[0081] The configuration of the dividing unit 10 of this embodiment will be described with reference to Fig. 11. Fig. 11 shows a block diagram illustrating an example of the functional configuration of the dividing unit 10 of this embodiment.

[0082] The functions of the acquisition unit 41 and the narrowing down unit 42 are the same as those in the first embodiment (see FIG. 4), and therefore a description thereof will be omitted.

[0083] On the other hand, the derivation unit 43 and the determination unit 44 of this embodiment are different from those of the first embodiment (see FIG. 4 ), and will be described below. Unlike the first embodiment, the derivation unit 43 of this embodiment outputs, to the determination unit 44, not only the derived cost values ​​but also prediction mode information used in deriving the cost values.

[0084] 11, the determination unit 44 of this embodiment includes an integration processing unit 45. Details of the determination unit 44 of this embodiment will be described with reference to FIG.

[0085] Depending on the results of the narrowing down unit 42, rectangular blocks at a certain CU size may not be evaluated, and size determination processing may be performed only on square CUs without division (similar to the case where only the determination in step 3 of the tournament processing in Figure 7 is performed).

[0086] On the other hand, as a result of deriving cost values ​​for small-sized CUs, there may be cases where CUs pointing in similar directions are adjacent to each other. In such cases, the determination unit 44 integrates these CUs to generate a rectangular CU and incorporates it into the size determination process.

[0087] 12 shows an example of determining a size of 32×32. Note that the procedure is similar when determining a size of 16×16 or the like, except that the size of the reference CU is different.

[0088] 12, the determination unit 44 performs size determination on a CU ((a)) obtained by stacking the determination results of small-size CUs and a 32×32 CU (without division, (b)). If the cost value of (b) is lower than the cost value of (a) ((b)<(a)), (b) is adopted as the 32×32 size result.

[0089] On the other hand, if the cost value of (a) is lower than the cost value of (b) ((a)<(b)), the process proceeds to the determination process of step 2.

[0090] In the determination of step 2, the integration processing unit 45 determines whether adjacent small-size CUs can be integrated into a rectangular CU so as to use a single prediction mode, based on the prediction direction (prediction mode number) of each small-size CU constituting (a). The integration method assumes that the division is BTh, BTv, TTh, and TTv ((1) to (10) in FIG. 12). Then, it calculates whether the intra-frame prediction mode numbers of the CUs in (a) included in the assumed division area are within ±α (α is a predetermined constant) of their average value, or whether the difference value of each number is within ±β (α and β are predetermined constants). If the result is True, the CUs are integrated. Regarding the prediction mode number and cost value of the integrated CU, the prediction mode numbers of each CU before integration are used to derive cost values ​​before the timing of determining the next size, and the prediction mode number with the smallest cost value is used as the prediction mode number of the integrated CU. In addition, if there is a shortage of computing resources or this process cannot be completed in time before the judgment timing, the most frequent or average value of the prediction mode numbers of each CU before the integration will be used as the prediction mode number of the CU after the integration, and the cost value will be the cost value of each CU before the integration plus a given offset.

[0091] In this embodiment, the above-mentioned size determination process is performed using tournament processing based on cost value comparison, from small-sized CUs to large-sized CUs, to determine the block division shape within a 64x64 CU, i.e., the size of each CU.

[0092] As described above, the encoding device 1 of each of the above embodiments is an encoding device that encodes an image in units of blocks obtained by dividing the image, and includes a division unit 10 including an acquisition unit 41, a narrowing down unit 42, a derivation unit 43, and a determination unit 44. The acquisition unit 41 acquires an original image to be encoded. The narrowing down unit 42 recursively divides the image into square or rectangular blocks. The derivation unit 43 derives an intra-frame prediction mode and a cost value predicted for each block. The determination unit 44 compares a combined block obtained by combining first blocks and the sum of the cost values ​​of the first blocks with the cost value of a second block of the same size as the combined block, and selects the first block or the second block with the smaller cost value as the block shape for the size of the combined block. This procedure is repeated recursively from the smallest block size to the largest block size among the divided blocks, thereby determining the final block division shape.

[0093] The division unit 10 having the above configuration can perform encoding by obtaining a block division shape including square blocks and rectangular blocks while maintaining a prediction mode according to the characteristics of the original image, for an encoding processing architecture that is assumed to have a constant processing time, such as hardware with a pipeline processing configuration, and reducing the amount of processing required for determining block division during encoding. Therefore, the encoding device 1 of each of the above embodiments can perform encoding by maintaining a prediction mode according to the characteristics of the original image, for an encoding processing architecture that is assumed to have a constant processing time, and reducing the amount of processing required for determining block division during encoding.

[0094] Furthermore, the encoding process executed by the CPU 51 after reading the encoding program 55 in the above embodiment may be executed by various processors other than the CPU 51. Examples of such processors include a programmable logic device (PLD) (such as a field-programmable gate array (FPGA)) whose circuit configuration can be changed after manufacture, and a dedicated electrical circuit, such as an application-specific integrated circuit (ASIC), which is a processor having a circuit configuration specifically designed to execute specific processing. The encoding process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). The hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.

[0095] In the above embodiment, the encoding program 55 is pre-stored (also referred to as "installed") in the ROM 52 or the storage 54, but the present invention is not limited to this. The encoding program 55 may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The encoding program 55 may also be downloaded from an external device via a network.

[0096] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0097] The following additional notes are provided regarding the above-described embodiments.

[0098] (Supplementary Item 1) An encoding device that encodes an image in units of blocks obtained by dividing the image, comprising: an acquisition unit that acquires an original image to be encoded; a narrowing down unit that recursively divides the image into square blocks or rectangular blocks; a derivation unit that derives an intra-screen prediction mode and a cost value predicted for each of the blocks; and a determination unit that determines a final block division shape by repeating a procedure from a minimum block size to a maximum block size among the recursively divided blocks, using the size of the first block as the size of the first block, based on a sum of cost values ​​of a combination block obtained by combining first blocks and a cost value of a second block of the same size as the combination block, if the sum is smaller, selecting the combination block as a block shape for the size of the combination block, and if the cost value of the second block is smaller, selecting the second block as a block shape for the size of the combination block.

[0099] (Supplementary Item 2) The encoding device according to Supplementary Item 1, wherein the narrowing-down unit narrows down the division method or division direction for dividing the square block into the rectangular blocks based on texture or pattern information of pixel values ​​within the square block, and if the square block is not to be divided into rectangular blocks, increases the number of narrowed-down candidates for rectangular blocks among square blocks one size smaller than the square block.

[0100] (Supplementary Item 3) The encoding device according to Supplementary Item 2, wherein the determination unit, when determining the block size of a first size, compares a combination of square blocks of a second size that is one size smaller than the first size with blocks in the case of rectangular block division, and compares the blocks obtained after the determination with the square blocks of the first size.

[0101] (Supplementary Item 4) The encoding device according to Supplementary Item 1, wherein when the block division shape for the first size is obtained from a plurality of candidates, if there is no rectangular block among the candidates, the determining unit integrates the small size blocks and treats them as a rectangular block if the intra-screen prediction modes of the included blocks smaller than the first size are within a predetermined range.

[0102] (Supplementary Item 5) An encoding method for encoding an image in units of blocks obtained by dividing the image, comprising the steps of: obtaining an original image to be encoded; recursively dividing the image into square or rectangular blocks; deriving a predicted intra-screen prediction mode and cost value for each of the blocks; and, based on a sum of cost values ​​of a combination block obtained by combining first blocks and a cost value of a second block of the same size as the combination block, selecting the combination block as a block shape for the size of the combination block if the sum is smaller; and selecting the second block as a block shape for the size of the combination block if the cost value of the second block is smaller. This procedure is repeated using a minimum block size to a maximum block size among the recursively divided blocks as the size of the first block, thereby determining a final block division shape.

[0103] (Supplementary Item 6) A range control program for causing a computer to function as each part of the encoding device according to any one of Supplementary Items 1 to 4.

[0104] (Supplementary Item 7) An encoding device that encodes an image in units of blocks obtained by dividing the image, comprising: a memory; and at least one processor connected to the memory, wherein the processor is configured to: obtain an original image to be encoded; recursively divide the image into square blocks or rectangular blocks; derive a predicted intra-screen prediction mode and cost value for each block; compare a combined block obtained by combining first blocks and a sum of the cost values ​​of the first blocks with the cost value of a second block of the same size as the combined block; and determine a final block division shape by repeating the procedure of selecting the first block or the second block with the smaller cost value as the block shape for the size of the combined block, from the smallest block size to the largest block size among the divided blocks recursively.

[0105] (Supplementary Item 8) A non-transitory storage medium storing a program executable by a computer to execute an encoding process for encoding an image in units of blocks obtained by dividing the image, wherein the encoding process includes the steps of: obtaining an original image to be encoded; recursively dividing the image into square or rectangular blocks; deriving a predicted intra-screen prediction mode and a cost value for each of the blocks; comparing a combined block obtained by combining first blocks and a sum of the cost values ​​of the first blocks with the cost value of a second block of the same size as the combined block; and selecting the first block or the second block with the smaller cost value as the block shape for the size of the combined block, by repeating this procedure from the smallest block size to the largest block size among the divided blocks recursively.

[0106] REFERENCE SIGNS LIST 1 Encoding device 10 Division unit 41 Acquisition unit 42 Narrowing unit 43 Derivation unit 44 Determination unit 51 CPU 54 Storage 55 Encoding program

Claims

1. An encoding device that encodes an image in units of blocks obtained by dividing the image, comprising: an acquisition unit that acquires an original image to be encoded; a narrowing down unit that recursively divides the image into square blocks or rectangular blocks; a derivation unit that derives a predicted intra-screen prediction mode and cost value for each block; and a determination unit that determines a final block division shape by repeating the procedure of selecting the combined block as a block shape for the size of the combined block based on the sum of the cost values ​​of a combined block obtained by combining first blocks and the cost value of a second block of the same size as the combined block if the sum is smaller, and selecting the second block as a block shape for the size of the combined block if the cost value of the second block is smaller, from the smallest block size to the largest block size in the recursively divided blocks.

2. The encoding device according to claim 1, wherein the narrowing-down unit narrows down the division method or division direction for dividing the square block into the rectangular blocks based on texture or pattern information of pixel values ​​within the square block, and if the square block is not to be divided into rectangular blocks, increases the number of narrowed-down candidates for rectangular blocks among square blocks one size smaller than the square block.

3. The encoding device according to claim 2, wherein the determination unit compares a combination of the first block, which is a square block, with a block obtained after a comparison determination of the first block in the case of rectangular block division, and the second block, which is a square block, and determines the block size of the second block.

4. The encoding device of claim 1, wherein when the determination unit obtains the block division shape of the second block from multiple candidates, if there is no rectangular block among the candidates, and if the intra-screen prediction mode of a block smaller than the second block contained therein is within a predetermined range, the determination unit integrates the small block and treats it as a rectangular block.

Images