Image decoding apparatus and method, and image encoding apparatus and method

Abstract

The present application relates to an image decoding apparatus and method, and an image encoding apparatus and method. A block division section includes a 4 division section that performs 4 division by dividing an object block in a recursive division into one half in a horizontal direction and a vertical direction, and generates four blocks, and a 2-3 division section that performs 2 division or 3 division in a horizontal direction or a vertical direction on the object block in the recursive division, and generates two or three blocks, the 2-3 division section prohibiting division of the object block in the horizontal direction when the divided object block exceeds a right side of a picture boundary, and prohibiting division of the object block in the vertical direction when the divided object block exceeds a lower side of the picture boundary.

Description

[0001] This application is a divisional application based on the invention with application number 201980044819.9, application date November 29, 2019, applicant JVC Kenwood Corporation, entitled "Image Decoding Apparatus, Image Decoding Method and Image Decoding Program". Technical Field

[0002] This invention relates to a technique for encoding and decoding images in units of segmented blocks. Background Technology

[0003] In image encoding and decoding, the image is divided into blocks, which are sets of a predetermined number of pixels, and processed on a block-by-block basis. By dividing the image into appropriate blocks, the efficiency of intra-frame prediction, inter-frame prediction, orthogonal transform, entropy coding, etc., is improved, resulting in improved coding efficiency.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent No. 2015-526008.

[0007] Non-patent literature

[0008] Non-patent literature 1: JVET, Versatile Video Coding (Draft 2), July 2018. Summary of the Invention

[0009] The problem that the invention aims to solve

[0010] Encoding efficiency decreases if blocks are not segmented with appropriate size and shape. In particular, at the edges of the image, blocks containing pixels extending beyond the image boundaries become inappropriately sized and shaped, further reducing encoding efficiency.

[0011] The present invention was made in view of the following situation, and its object is to provide a technique for improving coding efficiency by performing block segmentation suitable for image encoding and decoding.

[0012] means for solving problems

[0013] One aspect of the present invention for solving the above-mentioned problems is an image decoding apparatus. This apparatus decodes an image in units of segmented blocks, including: a decoding unit (201) that decodes block segmentation information of an object block; and a block segmentation unit (202) that recursively segments the image into rectangles of a predetermined size based on the block segmentation information to generate the object block. The block segmentation unit includes: a 4-segmentation unit that divides the recursively segmented object block in half horizontally and vertically to generate 4 blocks; and a 2-3 segmentation unit that performs 2-segmentation or 3-segmentation on the recursively segmented object block in either the horizontal or vertical direction to generate 2 or 3 blocks. If the 2-3 segmentation unit segments the object block horizontally, it prohibits further horizontal segmentation when the segmented object block extends beyond the right side of the image boundary; if it segments the object block vertically, it prohibits further vertical segmentation when the segmented object block extends beyond the bottom side of the image boundary.

[0014] Another aspect of the present invention is an image decoding apparatus. This apparatus decodes an image in units of segmented blocks, comprising: a decoding unit (201) that decodes block segmentation information of an object block; and a block segmentation unit that recursively segments the image into rectangles of a predetermined size based on the block segmentation information to generate the object block. The block segmentation unit (202) includes: a 4-segmentation unit that divides the recursively segmented object block in half horizontally and vertically to generate 4 blocks; and a 2-3 segmentation unit that performs 2-segmentation or 3-segmentation on the recursively segmented object block in either the horizontal or vertical direction to generate 2 or 3 blocks. If the 2-3 segmentation unit segments the object block in any direction, block segmentation in that direction is prohibited when the segmented object block exceeds the image boundary and the block segmentation depth reaches a predetermined limit depth, or when the number or proportion of pixels exceeding the image boundary in the block to be segmented is greater than a predetermined value.

[0015] Another aspect of the present invention is an image decoding apparatus. This apparatus decodes an image in units of segmented blocks, including: a decoding unit (201) that decodes block segmentation information of an object block; and a block segmentation unit (202) that recursively segments the image into rectangles of a predetermined size based on the block segmentation information to generate the object block. The block segmentation unit includes: a 4-segmentation unit that divides the recursively segmented object block in half horizontally and vertically to generate 4 blocks; and a 2-3 segmentation unit that performs 2- or 3-segmentation on the recursively segmented object block in either the horizontal or vertical direction to generate 2 or 3 blocks. If the 2-3 segmentation unit segments the object block, it determines the segmentation direction of the object block by maximizing the number of pixels exceeding the image boundary within the segmented object block.

[0016] Another aspect of the present invention is an image decoding method. This method decodes an image in units of segmented blocks, including: a decoding step of decoding block segmentation information of an object block; and a block segmentation step of recursively segmenting the image into rectangles of a predetermined size based on the block segmentation information to generate the object block. The block segmentation step includes: a 4-segmentation step of dividing the object block in the recursive segmentation into half horizontally and half vertically to generate 4 blocks; and a 2-3 segmentation step of performing 2-segmentation or 3-segmentation on the object block in the recursive segmentation in the horizontal or vertical direction to generate 2 or 3 blocks. In the 2-3 segmentation step, if the object block is segmented horizontally, further horizontal segmentation is prohibited if the segmented object block extends beyond the right side of the image boundary; if the object block is segmented vertically, further vertical segmentation is prohibited if the segmented object block extends beyond the bottom side of the image boundary.

[0017] Furthermore, any combination of the above-mentioned constituent elements, and any transformation of the present invention among methods, apparatus, systems, recording media, computer programs, etc., are also valid embodiments of the present invention.

[0018] According to the present invention, block segmentation suitable for image encoding and decoding can be performed, thereby improving encoding efficiency. Attached Figure Description

[0019] Figure 1 This is a block diagram of an image encoding apparatus according to the first embodiment.

[0020] Figure 2 This is a block diagram of the image decoding apparatus according to the first embodiment.

[0021] Figure 3 This is a flowchart illustrating the segmentation of tree blocks and the segmentation within tree blocks.

[0022] Figure 4 This is a diagram illustrating the scenario where an input image is segmented into tree blocks.

[0023] Figure 5 This is a diagram illustrating the z-scan.

[0024] Figure 6 This is a diagram illustrating the segmentation of tree blocks.

[0025] Figure 7 This is a flowchart illustrating the process of dividing a tree block into four parts and the processing of each part after the division.

[0026] Figure 8 This is a flowchart illustrating the processing of each segment after dividing a tree block into 2 or 3 segments.

[0027] Figure 9 It is a diagram that represents the relationship between tree blocks and image boundaries.

[0028] Figure 10 It is a graph that represents the relationship between image boundaries and pixels.

[0029] Figure 11 This is a flowchart illustrating the block segmentation in the first embodiment.

[0030] Figure 12 This is a diagram illustrating block segmentation in the first embodiment.

[0031] Figure 13 This is a diagram illustrating the syntax related to block segmentation in the first implementation.

[0032] Figure 14 This is a flowchart illustrating the block segmentation in the second embodiment.

[0033] Figure 15 This is a diagram illustrating block segmentation in the second embodiment.

[0034] Figure 16 This is a flowchart illustrating the block segmentation in the third embodiment.

[0035] Figure 17 This is a diagram illustrating block segmentation in the third embodiment.

[0036] Figure 18 This is a flowchart illustrating the block segmentation in the third embodiment.

[0037] Figure 19 This is a diagram illustrating block segmentation in the fourth embodiment.

[0038] Figure 20 This is a flowchart illustrating the block segmentation in the fourth embodiment.

[0039] Figure 21 This is a diagram illustrating block segmentation in the fifth embodiment.

[0040] Figure 22 This is a diagram illustrating an example of the hardware structure of the encoding / decoding apparatus of the first embodiment. Detailed Implementation

[0041] The embodiments of the present invention provide a technique for encoding and decoding images in units of segmented blocks.

[0042] (First Implementation)

[0043] The image encoding apparatus 100 and image decoding apparatus 200 according to the first embodiment of the present invention will be described. In the first embodiment, when dividing a block into 2 or 3 segments, block division is restricted.

[0044] Figure 1 This is a block diagram of the image encoding apparatus 100 according to the first embodiment. Figure 1 Only the data stream related to the image signal is shown; data streams related to additional information other than the image signal, such as motion vectors and prediction modes, are not shown. The image signal of at least one frame is input to the image encoding device 100.

[0045] The block segmentation unit 101 segments the image into encoding target blocks, which serve as processing units for encoding, and provides the image signals within the encoding target blocks to the residual signal generation unit 103. Furthermore, in order to evaluate the consistency of the predicted image, the block segmentation unit 101 provides the image signals of the encoding target blocks to the prediction image generation unit 102.

[0046] The block segmentation unit 101 recursively segments the image into rectangles of a predetermined size, generating coded object blocks. The block segmentation unit 101 includes: a 4-segmentation unit that divides the recursively segmented object blocks into 4 segments to generate 4 blocks; and a 2-3 segmentation unit that divides the recursively segmented object blocks into 2 or 3 segments to generate 2 or 3 blocks. Detailed operation of the block segmentation unit 101 will be described later.

[0047] The prediction image generation unit 102 is provided with image signals of the encoded target block from the block segmentation unit 101 and decoded image signals from the decoded image memory 108. The prediction image generation unit 102 uses the provided signals to perform intra-frame prediction (intra-frame prediction) and inter-frame prediction (inter-frame prediction) based on prediction modes, and generates a prediction image signal. In intra-frame prediction, the image signal of an encoded block that is close to the encoded target block in the same picture (encoded picture) is provided to the prediction image generation unit 102 from the decoded image memory 108. Then, the prediction image generation unit 102 uses this image signal and the image signal of the encoded target block provided from the block segmentation unit 101 to generate a prediction image signal. In inter-frame prediction, the image signal of an encoded picture (reference picture) that is preceding or following the encoded picture in the time sequence is provided to the prediction image generation unit 102 from the decoded image memory 108. Then, the prediction image generation unit 102 uses this image signal and the encoded target block provided from the block segmentation unit 101 to evaluate consistency through block matching, etc., and calculates a motion vector representing the motion quantity. The prediction image generation unit 102 performs motion compensation from the reference image based on the motion vector to generate a prediction image signal. The prediction image generation unit 102 then provides the generated prediction image signal to the residual signal generation unit 103.

[0048] The residual signal generation unit 103 performs a subtraction operation on the image signal to be encoded and the prediction signal generated by the prediction image generation unit 102 to generate a residual signal, which is then provided to the orthogonal transformation / quantization unit 104.

[0049] The quadrature transform / quantization unit 104 performs quadrature transform / quantization on the residual signal provided from the residual signal generation unit 103. The quadrature transform / quantization unit 104 provides the quadrature transformed / quantized residual signal to the encoding unit 105 and the inverse quantization / inverse quadrature transform unit 106.

[0050] The encoding unit 105 generates an encoded bitstream corresponding to the residual signal after orthogonal transformation / quantization provided by the orthogonal transformation / quantization unit 104. Additionally, the encoding unit 105 generates corresponding encoded bitstreams for additional information such as motion vectors, prediction modes, and block segmentation information provided from each component. Then, the encoding unit 105 outputs the encoded bitstream from the image encoding apparatus 100.

[0051] The inverse quantization / inverse quadrature transformation unit 106 performs inverse quantization / inverse quadrature transformation on the residual signal after quadrature transformation / quantization provided from the quadrature transformation / quantization unit 104 to obtain the residual signal. The inverse quantization / inverse quadrature transformation unit 106 provides the residual signal to the decoded image signal overlay unit 107.

[0052] The decoded image signal overlay unit 107 overlays the predicted image signal generated by the predicted image generation unit 102 and the residual signal obtained by the inverse quantization / inverse quadrature transform unit 106 to generate a decoded image, which is then stored in the decoded image memory 108. Additionally, the decoded image signal overlay unit 107 may also perform filtering processing on the decoded image to reduce block distortion and other distortions caused by encoding, and store this information in the decoded image memory 108.

[0053] Figure 2 This is a block diagram of the image decoding apparatus 200 according to Embodiment 1. Figure 2 Only the data stream related to the image signal is shown; data streams related to additional information other than the image signal, such as motion vectors and prediction modes, are not shown. The image decoding device 200 is input with the encoded bit stream.

[0054] The decoding unit 201 decodes the provided coded bitstream and provides the residual signal after orthogonal transformation / quantization to the block segmentation unit 202. In addition, the decoding unit 201 provides additional information such as motion vectors, prediction modes, and block segmentation information to each component for processing corresponding to the additional information.

[0055] The block segmentation unit 202 determines the shape of the decoding target block based on the decoded block segmentation information, and provides the residual signal of the determined decoding target block after orthogonal transformation / quantization to the inverse quantization / inverse orthogonal transformation unit 203.

[0056] Based on the decoded block segmentation information, the block segmentation unit 202 recursively segments the image into rectangles of a specified size, generating decoded object blocks. The block segmentation unit 202 includes: a 4-segmentation unit that segments the object blocks in the recursive segmentation into 4 blocks; and a 2-3 segmentation unit that segments the object blocks in the recursive segmentation into 2 or 3 blocks. The detailed operation of the block segmentation unit 202 will be described later.

[0057] The inverse quantization / inverse quadrature transformation unit 203 performs inverse quantization / inverse quadrature transformation on the provided quadrature transformation / quantization residual signal to obtain a residual signal. This residual signal is then provided to the decoded image signal overlay unit 205.

[0058] The prediction image generation unit 204 generates a prediction image signal based on the decoded image signal provided from the decoded image memory 206, and provides it to the decoded image signal overlay unit 205.

[0059] The decoded image signal overlay unit 205 overlays the predicted image signal generated by the predicted image generation unit 204 and the residual signal obtained by the inverse quantization / inverse quadrature transform unit 203 to generate a decoded image signal. Furthermore, the decoded image signal overlay unit 205 stores the decoded image signal in the decoded image memory 206. Additionally, the decoded image signal overlay unit 205 may perform filtering processing on the decoded image to reduce block distortion caused by encoding, and store this processing in the decoded image memory 206. Then, the decoded image signal overlay unit 205 outputs the decoded image from the image decoding device 200.

[0060] Next, use Figure 3 Explain the operation of the block segmentation unit 101 in the image encoding device 100. Figure 3 This indicates that the block segmentation unit 101 segments the image into tree blocks and performs block segmentation within them.

[0061] First, the input image is segmented into tree blocks of a predetermined size (S1000). Here, the tree block is 128×128 pixels. However, the tree block is not limited to this size; any size or aspect ratio can be used as long as it is rectangular. Alternatively, the size of the tree block can be predetermined between the encoding and decoding devices. Furthermore, it can be configured such that the encoding device determines the size of the tree block and records it in the encoded bitstream, and the decoding device uses the size of the tree block recorded in the encoded bitstream. The case of segmenting the input image into tree blocks is as follows: Figure 4 As shown, tree blocks are encoded according to the raster scan order, i.e., from left to right and from top to bottom.

[0062] The interior of the tree block is further divided into rectangular blocks. The interior of the tree block is arranged according to... Figure 5 The z-scan order shown is used for encoding. The z-scan order represents the sequence of top left, top right, bottom left, and bottom right. The internal segmentation of a tree block can be 4-segment, 2-segment, or 3-segment.

[0063] like Figure 6 As shown in (a), the blocks are divided into four parts by setting the horizontal and vertical directions to half each.

[0064] A block can be divided into two or three parts by splitting it horizontally or vertically. For example, when dividing a block into two parts horizontally... Figure 6 Divide it in half as in (b), generating two blocks. Alternatively, when dividing the block into three parts horizontally, as shown... Figure 6 As in (c), the blocks are divided into a 1:2:1 ratio, generating three blocks. On the other hand, if the blocks are divided into two parts vertically, as shown in [example 1]... Figure 6 Divide it in half as in (d) to generate two blocks. Alternatively, if the block is divided into three parts vertically, as shown... Figure 6 Divide it into 1:2:1 as in (e), generating 3 blocks.

[0065] Refer again Figure 3 The operation of the block segmentation unit 101 will be explained. First, it is determined whether to perform a 4-segmentation (S1001) that divides the interior of the tree block into half horizontally and vertically.

[0066] The decision to partition a block into four parts involves considering multiple conditions to determine the optimal solution. An existing method is Rate-Distortion Optimization (RD). In RD, the coding cost is calculated based on the amount of coding and the coding distortion. Then, under multiple coding conditions, the coding cost is calculated separately, and the case with the minimum coding cost is selected. That is, the decision to partition the block into four parts is made by calculating the coding cost of partitioning the block into four parts and the coding cost of not partitioning the block into four parts, and selecting the case with the minimum coding cost. Methods other than RD can also be used to determine the optimal solution based on multiple conditions.

[0067] If it is determined that the internal structure of the tree block needs to be divided into four parts (S1001: Yes), then the internal structure of the tree block is divided into four parts (S1002). The further division of the block after the four-part division will be described later. Figure 7 ).

[0068] If it is determined that the tree block will not be divided into 4 parts (S1001: No), determine whether to divide the tree block into 2 parts or 3 parts (S1003).

[0069] If it is determined that the tree block is to be divided into 2 or 3 parts (S1003: Yes), determine whether to set the direction of the division to the vertical direction (S1004).

[0070] If the direction of the split is determined to be vertical (S1004: Yes), determine whether to split the tree block into 2 parts (S1005).

[0071] If it is determined that the block's interior is to be divided into two parts (S1005: Yes), the block's interior is divided into two parts vertically (S1006). Conversely, if it is determined that the block's interior is to be divided into three parts (S1005: No), the block's interior is divided into three parts vertically (S1007). The re-division process of blocks that have been divided into two or three parts vertically will be described later. Figure 8 ).

[0072] If the direction of the split is determined to be horizontal (S1004: No), determine whether to split the tree block into 2 parts (S1008).

[0073] If it is determined that the block's interior is to be divided into two parts (S1008: Yes), the block's interior is divided into two parts horizontally (S1009). Conversely, if it is determined that the block's interior is to be divided into three parts (S1008: No), the block's interior is divided into three parts horizontally (S1010). The re-division process of blocks that have been divided into two or three parts horizontally will be described later. Figure 8 ).

[0074] If it is determined that the internal structure of the tree block will not be split into 2 or 3 parts (S1003: No), the block splitting process ends without splitting the internal structure of the tree block (S1011).

[0075] Next, use Figure 7 The flowchart illustrates the processing of each block obtained after dividing a tree block into four halves in both the horizontal and vertical directions.

[0076] First, determine whether to perform a 4-part division of the block's interior into two halves, one horizontally and one vertically (S1101).

[0077] If it is determined that the block is to be divided into 4 parts again (S1101: Yes), the block is divided into 4 parts again (S1102).

[0078] If it is determined that the block should not be further divided into 4 parts (S1101: No), determine whether to divide the block into 2 parts or 3 parts (S1103).

[0079] If it is determined that the block is to be divided into 2 or 3 parts (S1103: Yes), determine whether to set the direction of the division to the vertical direction (S1104).

[0080] If the direction of the division is determined to be vertical (S1104: Yes), determine whether to divide the block into 2 parts (S1105).

[0081] If it is determined that the block's interior is to be divided into 2 parts (S1105: Yes), the block's interior is divided into 2 parts in the vertical direction (S1106). On the other hand, if it is determined that the block's interior is to be divided into 3 parts (S1105: No), the block's interior is divided into 3 parts in the vertical direction (S1107).

[0082] If the direction of the division is determined to be horizontal (S1104: No), determine whether to divide the block into 2 parts (S1108).

[0083] If it is determined that the block's interior is to be divided into 2 parts (S1108: Yes), the block's interior is divided into 2 parts horizontally (S1109). On the other hand, if it is determined that the block's interior is to be divided into 3 parts (S1108: No), the block's interior is divided into 3 parts horizontally (S1110).

[0084] If it is determined that the block should not be divided into 2 or 3 parts (S1103: No), the block will not be further divided and the block splitting process will end (S1111).

[0085] For each block after the 4-part division, recursively execute... Figure 7 The process is shown in the flowchart. The internal structure of the 4-segmented blocks is encoded in a z-scan order.

[0086] Next, use Figure 8 The flowchart illustrates the processing of each block after a tree block is divided into 2 or 3 segments in the vertical direction.

[0087] When a tree block is divided into 2 or 3 segments in the vertical direction, for each segmented block, it is determined whether to further divide the block into 2 or 3 segments (S1201).

[0088] If it is determined that the block is to be divided into 2 or 3 parts (S1201: Yes), determine whether to set the direction of the division to the vertical direction (S1202).

[0089] If the direction of the division is determined to be vertical (S1202: Yes), determine whether to divide the block into 2 parts (S1203).

[0090] If it is determined that the block's interior is to be divided into 2 parts (S1203: Yes), the block's interior is divided into 2 parts in the vertical direction (S1204). On the other hand, if it is determined that the block's interior is to be divided into 3 parts (S1203: No), the block's interior is divided into 3 parts in the vertical direction (S1205).

[0091] If the direction of the division is determined to be horizontal (S1202: No), determine whether to divide the block into 2 parts (S1206).

[0092] If it is determined that the block's interior is to be divided into 2 parts (S1206: Yes), the block's interior is divided into 2 parts horizontally (S1207). On the other hand, if it is determined that the block's interior is to be divided into 3 parts (S1206: No), the block's interior is divided into 3 parts horizontally (S1208).

[0093] If it is determined that the block should not be further divided into 2 or 3 parts (S1201: No), the block splitting process ends without further splitting the block (S1209).

[0094] For each block resulting from a 2- or 3-part division in the vertical direction, recursively execute... Figure 8 The process is shown in the flowchart. The internal parts of the blocks after 2 or 3 divisions are encoded in order from left to right.

[0095] Similarly, for blocks resulting from horizontal 2- or 3-part divisions, the process is executed recursively. Figure 8 The process is shown in the flowchart. The internal parts of the blocks after 2 or 3 divisions are encoded in a top-to-bottom order.

[0096] Additionally, the process of further segmenting the resulting blocks after a tree block has been explained, but the parent block may not be a tree block. For example, if a tree block (128×128 pixels) is divided into 4 parts, and the resulting 4-part block (64×64 pixels) is further divided, the same processing is applied to the segmentation of the further subdivided blocks.

[0097] Recursive block partitioning can also determine and limit the number of partitions. Alternatively, the number of partitions can be predetermined between the encoding and decoding devices. Furthermore, it can be configured such that the encoding device determines the number of partitions and records it in the encoded bitstream, and the decoding device uses the number of partitions recorded in the encoded bitstream.

[0098] Next, the block segmentation at the end of the image will be explained. Figure 9 This indicates the relationship between the image and the image boundaries when segmenting the image into tree blocks. For example... Figure 9 As shown, since the image size is not limited to an integer multiple of the tree block size, tree blocks at the edge of the image sometimes include portions inside and outside the image, separated by the image boundary. Tree blocks 1001, 1002, and 1003 are shown as tree blocks at the edge of the image. In this case, as... Figure 10 As shown, the portion of the image outside the frame boundary is treated the same as the outermost pixel within the frame. Pixels 1011, 1012, and 1013 are shown as the outermost pixels within the frame.

[0099] Pixel 1011 is the pixel located within tree block 1001 and at the top right of the image. Pixels to the right of pixel 1011 that extend beyond the image boundary are treated the same as pixel 1011.

[0100] Pixel 1012 is located within tree block 1002 and is situated at the bottom left of the image. Pixels below pixel 1012 that extend beyond the image boundary are treated the same as pixel 1012.

[0101] Pixel 1013 is located within tree block 1003 and is situated at the bottom right of the image. Pixels to the right, below, and to the bottom right of pixel 1013 that extend beyond the image boundary are treated the same as pixel 1013.

[0102] Then, when splitting blocks into 2 or 3 segments, the block segmentation is restricted. This allows the blocks at the edges of the frame to be segmented into appropriate shapes, thereby improving encoding efficiency.

[0103] The restrictions on block segmentation apply when a block is divided into 2 or 3 segments at the edge of the image. That is, Figure 3 The 2- or 3-segmentation process (S1004 to S1010) is replaced by the process described below. Additionally, Figure 7 The 2- or 3-segmentation process (S1104 to S1110) is replaced by the process described below. Furthermore, Figure 8 The 2- or 3-segmentation process (S1202 to S1208) is replaced by the process described below.

[0104] use Figure 11 The restrictions on block segmentation are explained. First, for all combinations of block segmentation, a decision is made as to whether to restrict block segmentation (S1301). All combinations of block segmentation refer to the four cases: splitting the block into two parts horizontally, splitting the block into two parts vertically, splitting the block into three parts horizontally, and splitting the block into three parts vertically. Furthermore, whether to restrict block segmentation depends on whether pixels beyond the image boundary are segmented. For example, when splitting the block into two parts horizontally, if pixels beyond the image boundary are segmented, then splitting the block into two parts horizontally is restricted.

[0105] Here, restricting block splitting to the horizontal direction means prohibiting block splitting in the horizontal direction. Similarly, restricting block splitting to the vertical direction means prohibiting block splitting in the vertical direction. Likewise, restricting block splitting by 2 means prohibiting block splitting by 2. Furthermore, restricting block splitting by 3 means prohibiting block splitting by 3.

[0106] Next, it is determined whether to restrict block partitioning in all combinations of block partitioning (S1302). If block partitioning is restricted in all combinations (S1302: Yes), no block partitioning is performed (S1314). On the other hand, if block partitioning is not restricted in all combinations (S1302: No), it is determined whether to restrict horizontal block partitioning (S1303).

[0107] If horizontal block partitioning is restricted (S1303: Yes), proceed to the next process (S1306). On the other hand, if horizontal block partitioning is not restricted (S1303: No), determine whether vertical block partitioning is restricted (S1304).

[0108] If the block segmentation is restricted to the vertical direction (S1304: Yes), proceed to the next process (S1310). On the other hand, if the block segmentation is not restricted to the vertical direction (S1304: No), determine whether to segment the block in the vertical direction (S1305).

[0109] If the direction of block division is determined to be vertical (S1305: Yes), determine whether to restrict the block to 3-part division (S1306). On the other hand, if the direction of block division is determined to be horizontal (S1305: No), proceed to the next process (S1310).

[0110] If the block is restricted to 3-part division (S1306: Yes), the block is divided into 2 parts in the vertical direction (S1308). On the other hand, if the block is not restricted to 3-part division (S1306: No), it is determined whether to divide the block into 2 parts (S1307).

[0111] If it is determined that the block should be divided into 2 parts (S1307: Yes), the block is divided into 2 parts in the vertical direction (S1308). On the other hand, if it is determined that the block should be divided into 3 parts (S1307: No), the block is divided into 3 parts in the vertical direction (S1309).

[0112] If the vertical division is restricted (S1304: Yes) and the direction of block division is determined to be horizontal (S1305: No), determine whether to restrict the 3-part division of the block (S1310).

[0113] If the block is restricted to 3-part division (S1310: Yes), the block is divided into 2 parts horizontally (S1312). On the other hand, if the block is not restricted to 3-part division (S1310: No), it is determined whether to divide the block into 2 parts (S1311).

[0114] If it is determined that the block should be divided into 2 parts (S1311: Yes), the block is divided into 2 parts horizontally (S1312). On the other hand, if it is determined that the block should be divided into 3 parts (S1311: No), the block is divided into 3 parts horizontally (S1313).

[0115] That is, when segmenting pixels beyond the image boundary using block segmentation, block segmentation in that direction is restricted.

[0116] Here is a specific example. Currently, at the bottom of the screen, the tree block is not divided into 4 parts (S1001: No), and the inside of the tree block is divided into 2 or 3 parts (S1003: Yes). Figure 12 (a) represents the case where the tree block contains both in-picture and out-of-picture portions separated by the image boundary. In this case, for the case where the tree block is segmented in all combinations, it is decided whether to restrict block segmentation (S1301).

[0117] exist Figure 12 (a) to Figure 12 (d) shows the cases where tree blocks were split in all combinations. Figure 12 As shown in (a), if the block is divided into 2 parts in the vertical direction, the block segmentation is limited because pixels beyond the image boundaries are segmented. Similarly, as Figure 12 As shown in (b), if the block is divided into 3 segments in the vertical direction, the segmentation of the block is limited because pixels beyond the image boundaries are segmented. On the other hand, as Figure 12 As shown in (c), if the block is divided into 2 parts horizontally, the block segmentation is not restricted because pixels beyond the image boundaries are not segmented. Similarly, as Figure 12 As shown in (d), if the block is divided into 3 parts in the horizontal direction, the block segmentation is not restricted because pixels beyond the image boundary are not segmented.

[0118] Therefore, block splitting is not restricted in any combination of block splitting (S1302: No). Furthermore, horizontal block splitting is not restricted (S1303: No), but vertical block splitting is restricted (S1304: Yes). Also, 3-part block splitting is not restricted (S1310: No). Therefore, it is determined whether to split the block into 2 parts (S1311). Currently, if it is determined that the block should be split into 2 parts (S1311: Yes), then the block is split into 2 parts horizontally (S1312).

[0119] By restricting the block segmentation in this way, the blocks become appropriately shaped. This is because, in a block that includes pixels outside the frame, the pixel values ​​of the off-frame portion are fixed. Therefore, the variation in pixel values ​​of the on-frame portion of that block is relatively small compared to blocks that do not include off-frame pixels. Consequently, there is less need to encode subtle pixel variations. Therefore, by making off-frame pixels into blocks that are as concentrated as possible, the amount of coding can be reduced, and coding efficiency can be improved.

[0120] The same limitation applies to the right side of the image as to this block division. Figure 12 (e) to Figure 12 (h) illustrates the case of splitting tree blocks in all combinations. At this point, as shown in (h) Figure 12 As shown in (e), if the block is divided into 2 parts horizontally, pixels extending beyond the image boundaries are segmented. Additionally, as... Figure 12 As shown in (f), if a block is divided into 3 parts horizontally, pixels that extend beyond the image boundaries are segmented. Therefore, these block segments are restricted. By restricting the block segmentation in this way, coding efficiency can be improved.

[0121] Next, the operation of the block segmentation unit 202 of the image decoding apparatus 200 will be described. The block segmentation unit 202 segments blocks according to the same processing procedure as the block segmentation unit 101 in the image encoding apparatus 100 described above. In the block segmentation unit 101, a block segmentation mode is selected, and the selected block segmentation information is output. On the other hand, the block segmentation unit 202 segments blocks using the block segmentation information decoded from the encoded bitstream. The limitations of block segmentation are the same as those in the image encoding apparatus 100 described above.

[0122] Figure 13 This represents the syntax (syntactic rules for the encoded bitstream) related to block segmentation in the first implementation. Figure 13 In this syntax, QT() represents the syntax involved in 4-segmentation of a block, and MTT() represents the syntax involved in 2-segmentation or 3-segmentation of a block. The image encoding device 100 encodes according to this syntax, and the image decoding device 200 decodes according to this syntax.

[0123] First, `QTflag` indicates whether to perform a 4-splitting operation on the block. If a 4-splitting operation is performed, `QTflag` is set to 1; otherwise, it's set to 0. If a 4-splitting operation (`QTflag` = 1) is performed, and each of the 4-splitting blocks can be further 4-splitting (`QTvalid` = 1), then the 4-splitting process is recursively performed. If a 4-splitting operation is not performed (`QTflag` = 0), whether to perform a 2-splitting or 3-splitting operation is indicated by `MTTflag`. If a 2-splitting or 3-splitting operation is performed (`MTTflag` = 1), whether to perform a vertical split is indicated by `vertical_flag`, and whether to perform a 2-splitting operation is indicated by `BTflag`. A vertical split is set to `vertical_flag` = 1, and a horizontal split is set to `vertical_flag` = 0. Additionally, a 2-splitting operation is set to `BTflag` = 1, and a 3-splitting operation is set to `BTflag` = 0. If each block after a 2- or 3-partition can be further divided into 2 or 3 parts (MTTvalid = 1), then the 2- or 3-partition process is performed recursively.

[0124] Here, we explain the variable QTvalid, which indicates whether each of the 4-segmented blocks can be further 4-segmented. QTvalid is defined for each 4-segmented block. If the block to be 4-segmented does not contain any pixels within the frame, QTvalid = 0. Otherwise, QTvalid = 1.

[0125] Additionally, the variable MTTvalid, which indicates whether each block after a 2- or 3-segment split can be further split into 2- or 3-segments, is explained. MTTvalid is defined for each block after a 2- or 3-segment split. If the block to be split into 2- or 3-segments does not contain any pixels within the frame, MTTvalid = 0. Otherwise, MTTvalid = 1.

[0126] In this implementation, since the direction of block segmentation is restricted, the vertical_flag is not needed. Therefore, it can be omitted. Figure 13 The form of vertical_flag in the code.

[0127] By restricting the block segmentation in this way, blocks at the edges of the image can be divided into appropriate shapes, which can improve encoding efficiency. In addition, it enables block segmentation suitable for image encoding and decoding.

[0128] (Second Implementation)

[0129] The image encoding apparatus and image decoding apparatus according to the second embodiment of the present invention will be described. In the second embodiment, block segmentation is restricted when the depth of block segmentation reaches a limited depth. Since the structure other than this is the same as that of the first embodiment, the description is omitted.

[0130] Here, the depth of block segmentation will be explained. In the first embodiment, it was described that after a block is segmented into 2 or 3, the process of recursively segmenting each of the resulting 2 or 3 blocks into 2 or 3 blocks was described. In this process, the first 2 or 3 segmentation process is defined as depth 0. Furthermore, the second 2 or 3 segmentation process for each block obtained by the first 2 or 3 segmentation process is defined as depth 1, and the third 2 or 3 segmentation process for each block obtained by the second 2 or 3 segmentation process is defined as depth 2, and so on. Additionally, a predetermined limit on the depth of block segmentation is defined as the limit depth.

[0131] The restrictions on block segmentation apply when a block is divided into 2 or 3 segments at the edge of the image. That is, Figure 3 The 2- or 3-segmentation process (S1004 to S1010) is replaced by the process described below. Additionally, Figure 7 The 2- or 3-segmentation process (S1104 to S1110) is replaced by the process described below. Furthermore, Figure 8 The 2- or 3-segmentation process (S1202 to S1208) is replaced by the process described below.

[0132] use Figure 14 The limitations on block partitioning are explained. First, when partitioning a block into 2 or 3 parts, it is determined whether the depth of the block partitioning has reached the limit depth (S1401).

[0133] If the depth limit is not reached (S1401: No), it is decided not to limit block partitioning (S1402). On the other hand, if the depth limit is reached (S1401: Yes), it is decided whether to limit block partitioning for all combinations of block partitioning (S1301). Since S1301 is the same as in the first embodiment, the description is omitted.

[0134] Next, it is determined whether to restrict block partitioning in all combinations of block partitioning (S1302). Since the following processing is the same as in the first embodiment, the description is omitted.

[0135] That is, when pixel segments beyond the image boundary are segmented by block segmentation, and the depth of block segmentation reaches the limit, block segmentation is restricted.

[0136] Here is a specific example. Currently, at the bottom of the screen, the tree block is not divided into 4 parts (S1001: No), but the interior of the tree block is divided into 2 or 3 parts (S1003: Yes). In addition, the depth limit is set to 1. Figure 12 (a) shows a case where a tree block contains both in-frame and out-of-frame portions across the image boundary. When the tree block is 2-segmented or 3-segmented, pixels beyond the image boundary are segmented through vertical segmentation. However, the depth is 0, not reaching the depth limit of 1 (S1401: No). Therefore, it is decided not to restrict block segmentation (S1402). Therefore, all block segmentation is not restricted (S1302: No), horizontal block segmentation is not restricted (S1303: No), and vertical block segmentation is not restricted (S1304: No). Then, it is determined whether to segment the block vertically (S1305). Currently, if it is determined that the block should be segmented vertically (S1305: Yes), since 3-segmentation of the block is not restricted (S1306: No), it is determined whether to 2-segment the block (S1307). Currently, if it is determined that the block should be 2-segmented (S1307: Yes), the block is 2-segmented vertically (S1308). The situation is as follows Figure 12 As shown in (a).

[0137] Next, among the blocks that are divided into 2s in the vertical direction, the left block is divided into 2s or 3s. At this point, the depth is 1, reaching the depth limit of 1 (S1401: Yes). Therefore, for all combinations of block division, it is decided whether to limit the block division (S1301).

[0138] Figure 15 (a) to Figure 15 (d) represents the case of segmenting blocks in all combinations. For example... Figure 15 As shown in (a), if the block is divided into 2 parts in the vertical direction, the block segmentation is limited because pixels beyond the image boundaries are segmented. Similarly, as Figure 15 As shown in (b), if the block is divided into 3 segments in the vertical direction, the segmentation of the block is limited because pixels beyond the image boundaries are segmented. On the other hand, as Figure 15 As shown in (c), if the block is divided into 2 parts horizontally, then the block segmentation is not restricted because pixels beyond the image boundaries are not segmented. Similarly, as Figure 15 As shown in (d), if the block is divided into 3 parts in the horizontal direction, the block segmentation is not restricted because pixels beyond the image boundary are not segmented.

[0139] Therefore, block partitioning is not restricted in any combination of block partitioning (S1302: No). Furthermore, horizontal block partitioning is not restricted (S1303: No), but vertical block partitioning is restricted (S1304: Yes). Also, 3-part partitioning of the block is not restricted (S1310: No). Therefore, it is determined whether to perform 2-part partitioning of the block (S1311). Currently, if it is determined that the block should be 2-part partitioned (S1311: Yes), then the block is 2-part partitioned horizontally (S1312).

[0140] This block segmentation constraint ensures that the blocks are of appropriate size and shape. This is because, in blocks containing off-screen pixels, the pixel values ​​of the off-screen portion are fixed. Therefore, the variation in pixel values ​​of the on-screen portion of this block is relatively small compared to blocks that do not contain off-screen pixels. Consequently, there is less need to encode subtle pixel variations. Therefore, by making off-screen pixels into blocks that are as concentrated as possible, the amount of coding can be reduced, and coding efficiency can be improved.

[0141] The block segmentation restriction also applies to the right edge of the image. When performing 2- or 3-segmentation on a tree block, since the depth is 0, the depth restriction of 1 is not reached, so the block segmentation is not restricted and 2- or 3-segmentation is performed. When further segmenting the segmented block, since the depth is 1, the depth restriction of 1 is reached, so it is determined whether to restrict the block segmentation. Thus, in the cases of 2-segmentation and 3-segmentation of a block in the horizontal direction, pixels exceeding the image boundary are segmented. Therefore, these block segmentations are restricted. By restricting the block segmentation in this way, coding efficiency can be improved.

[0142] In this embodiment, the depth of block segmentation is defined relative to 2-segmentation or 3-segmentation. It can also be defined for 4-segmentation. Furthermore, in this embodiment, block segmentation is limited based on the depth of block segmentation. This can be limited by the number or proportion of pixels in a block that extend beyond the image boundary. That is, if these values ​​are larger than predetermined values, the block segmentation is limited. Moreover, these values ​​can be different according to the depth of each block segmentation. This results in blocks with a small number or proportion of pixels outside the image being segmented, while blocks with a large number or proportion of pixels outside the image are not segmented. Alternatively, the values ​​related to the block segmentation limitation, such as the limiting depth, the number of pixels in a block that extend beyond the image boundary, and the proportion, can be recorded in the encoded bitstream by the encoding device, and the decoding device uses the values ​​recorded in the encoded bitstream.

[0143] In the first embodiment, block segmentation is restricted regardless of the depth of block segmentation. However, in this embodiment, block segmentation is restricted based on the depth of block segmentation. This results in blocks with a small proportion of pixels outside the frame being segmented, while blocks with a large proportion of pixels outside the frame are not segmented. Therefore, blocks at the edges of the frame can be segmented into appropriate sizes and shapes, improving encoding efficiency. Furthermore, block segmentation suitable for image encoding and decoding is possible.

[0144] (Third Implementation)

[0145] The image encoding apparatus and image decoding apparatus according to the third embodiment of the present invention will be described. In the third embodiment, block segmentation is controlled based on the number of pixels located beyond the image boundary. Since the rest of the structure is the same as in the first embodiment, the description is omitted.

[0146] Block segmentation control is applicable when dividing a block into 2 or 3 segments at the edge of the image. That is, Figure 3 The 2- or 3-segmentation process (S1004 to S1010) is replaced by the process described below. Additionally, Figure 7 The 2- or 3-segmentation process (S1104 to S1110) is replaced by the process described below. Furthermore, Figure 8 The 2- or 3-segmentation process (S1202 to S1208) is replaced by the process described below.

[0147] use Figure 16 The control of block segmentation is explained. First, for all combinations of block segmentation, the number of pixels exceeding the image boundary is counted (S1601). All combinations of block segmentation refer to the four cases: dividing the block into 2 parts horizontally, dividing the block into 2 parts vertically, dividing the block into 3 parts horizontally, and dividing the block into 3 parts vertically.

[0148] Then, for each of the cases of dividing the block into 2 and 3, the division direction with the largest number of pixels beyond the image boundary is determined (S1602).

[0149] Next, determine whether to split the block into 2 parts (S1603).

[0150] If it is determined that a 2-segmentation is to be performed (S1603: Yes), a 2-segmentation is performed in the segmentation direction determined in S1602 (S1604). On the other hand, if it is determined that a 3-segmentation is to be performed (S1603: No), a 3-segmentation is performed in the segmentation direction determined in S1602 (S1605).

[0151] That is, control the direction of block segmentation so that pixels beyond the image boundary are contained within the block to the maximum extent.

[0152] Here is a specific example. Currently, the image is 1920×1080 pixels, and the tree block is 128×128 pixels. Furthermore, the tree block is not divided into 4 parts (S1101: No), but the interior of the block is divided into 2 or 3 parts (S1103: Yes). Thus, as... Figure 17 As shown in (a), there are 72 pixels at the bottom of the image that extend beyond the image boundary and in the vertical direction. At this time, the number of pixels in each block that extend beyond the image boundary is counted when the tree blocks are divided into all combinations (S1601).

[0153] exist Figure 17 (a) to Figure 17 (d) shows the cases of splitting tree blocks in all combinations. Figure 17 As shown in (a), when the block is divided into two parts in the vertical direction, the left (500) and right (501) parts of the divided block are each 4608 pixels. On the other hand, as Figure 17 As shown in (b), if the block is divided into two parts in the horizontal direction, the upper part (510) of the divided block has 1024 pixels and the lower part (511) has 8192 pixels. That is, the part with the largest number of pixels beyond the image boundary is the lower part with 8192 pixels in the case of horizontal division. Therefore, the division direction with the largest number of pixels is the horizontal direction (S1602).

[0154] Similarly, as Figure 17 As shown in (c), if the block is divided into 3 parts vertically, the left (520) and right (522) parts of the divided block are 2304 pixels each, and the center (521) is 4608 pixels. On the other hand, as Figure 17 As shown in (d), if the block is divided into 3 parts horizontally, the top of the divided block has 0 pixels, the center (531) has 5120 pixels, and the bottom (532) has 4096 pixels. That is, the part with the largest number of pixels beyond the image boundary is the center with 5120 pixels in the case of horizontal division. Therefore, the division direction with the largest number of pixels is the horizontal direction (S1602).

[0155] Finally, it is determined whether to divide the block into two parts (S1603). If it is determined to divide the block into two parts (S1603: Yes), the block is divided into two parts in the horizontal direction, which is the determined dividing direction (S1604). On the other hand, if it is determined to divide the block into three parts (S1603: No), the block is divided into three parts in the horizontal direction, which is the determined dividing direction (S1605).

[0156] In this embodiment, when dividing a block into 2 or 3 segments, the direction of the block division is controlled to ensure that pixels extending beyond the image boundary are included within the block as much as possible. This is because, in blocks containing pixels outside the image, the pixel values ​​of the out-of-image portion are fixed, so there is little need to encode subtle pixel variations. Therefore, by concentrating such pixels into blocks that are as concentrated as possible, the amount of coding can be reduced, and coding efficiency can be improved.

[0157] Furthermore, not only the direction of block segmentation can be determined, but also the number of block segments. As mentioned above, the largest number of pixels exceeding the image boundary is 8192 pixels in the case of 2-segmentation and 5120 pixels in the case of 3-segmentation. That is, since a large number of pixels exceeding the image boundary are contained within the block in the case of 2-segmentation, the block is segmented into 2.

[0158] This is because, compared to 2-segmentation, 3-segmentation divides the block near its ends. 3-segmentation tends to have fewer pixels extending beyond the image boundaries compared to 2-segmentation. Therefore, at the edges of the image, 2-segmentation can be performed instead of 3-segmentation. This reduces the processing involved in 3-segmentation of the block, allowing for faster block segmentation processing.

[0159] Block segmentation control is applicable when dividing a block into 2 or 3 segments at the edge of the image. That is, Figure 3 The 2- or 3-segmentation process (S1004 to S1010) is replaced by the process described below. Additionally, Figure 7 The 2- or 3-segmentation process (S1104 to S1110) is replaced by the process described below. Furthermore, Figure 8 The 2- or 3-segmentation process (S1202 to S1208) is replaced by the process described below.

[0160] use Figure 18 The block segmentation control is explained. First, for all combinations of block segmentation, the number of pixels exceeding the image boundary is counted (S1651). All combinations of block segmentation refer to the two cases of dividing the block into two parts horizontally and vertically.

[0161] Next, regarding the case of dividing the block into two parts, the division direction with the largest number of pixels beyond the image boundary is determined (S1652). Then, the block is divided into two parts along the determined division direction (S1654).

[0162] Regarding the block segmentation syntax in this implementation, it is assumed to be omitted. Figure 13The `vertical_flag` in this implementation is not needed because the direction of block splitting is controlled in this embodiment. Furthermore, in this embodiment, the number of block splits can be determined, or the block can be split into three parts without using `BTflag`. In this case, `BTflag` is not needed, and therefore it can be a form where `BTflag` is omitted. The syntax related to block splitting in this embodiment can also be found in... Figure 13 same.

[0163] In the case of 1920×1080 pixels, there are no pixels extending beyond the image boundary at the right end of the frame. However, when the image width is not an integer multiple of the block size, block segmentation is controlled in the same way as described above because there are pixels extending beyond the image boundary.

[0164] In this embodiment, the direction of block segmentation is controlled based on the number of pixels extending beyond the image boundary. This can be achieved by controlling the proportion of pixels extending beyond the image boundary.

[0165] By controlling block segmentation in this way, blocks at the edges of the image can be divided into appropriate sizes and shapes, thus improving encoding efficiency. Furthermore, block segmentation suitable for image encoding and decoding can be performed.

[0166] (Fourth Implementation)

[0167] The image encoding apparatus and image decoding apparatus according to the fourth embodiment of the present invention will be described. In the fourth embodiment, the case where the image boundary is different from the image boundary in the above embodiments will be described. Since the structure other than this is the same as that in the first embodiment, the description is omitted.

[0168] Here, consider the image boundaries and Figure 12 Different patterns. However, even if the image boundaries are different, the result of determining whether block segmentation is restricted will also differ. Figure 12 Except for cases with the same pattern. For example, in Figure 12 In (a), it is assumed that the image boundary is slightly above. Even in this case, block segmentation is limited because pixels beyond the image boundary are segmented. That is, in Figure 12 In (a), even if the image boundaries change in the vertical direction, the result of the restricted block segmentation remains unchanged. This is in... Figure 12 The same applies to (b). Similarly, in Figure 12 (e) and Figure 12 In (f), even if the image boundary changes in the left and right directions, the judgment result of the restricted block segmentation does not change.

[0169] As a result, the image boundaries and Figure 12The different modes are Figure 19 from (a) to Figure 19 (d) of Figure 19 The picture boundary of (a) of Figure 12 is above that of (c) of Figure 19 The picture boundary of (b) of Figure 12 is above that of (d) of Figure 19 The picture boundaries of (c) of Figure 19 and (d) of Figure 12 are to the left of those of (g) of Figure 12 and (h) of

[0170] Figure 19 The divisions of (a) of Figure 19 and (b) of are both horizontal divisions. However, in the case of 2 - division, the pixels at positions exceeding the picture boundary are not divided, and in the case of 3 - division, the pixels at positions exceeding the picture boundary are divided. That is, only in the case of 3 - division, the block division is restricted.

[0171] Figure 19 The divisions of (c) of Figure 19 and (d) of are both vertical divisions. However, in the case of 2 - division, the pixels at positions exceeding the picture boundary are not divided, and in the case of 3 - division, the pixels at positions exceeding the picture boundary are divided. That is, only in the case of 3 - division, the block division is restricted.

[0172] That is, in 2 - division and 3 - division, the judgment for restricting block division is separated. This is because, compared with 2 - division, 3 - division divides the block at a position closer to the end of the block. There is a tendency for the pixels at positions exceeding the picture boundary to be divided in 3 - division compared with 2 - division. Therefore, at the picture end, it is possible to determine whether not to perform 3 - division on the block and to restrict the direction of 2 - division. Thus, the processing related to 3 - division of the block can be reduced, and the processing of block division can be speeded up.

[0173] The restriction on block division is applied when performing 2 - division or 3 - division on the block at the picture end. That is, Figure 3 the 2 - division or 3 - division processing (S1004 to S1010) in is replaced with the processing described below. Additionally, Figure 7 the 2 - division or 3 - division processing (S1104 to S1110) in is replaced with the processing described below. And, Figure 8 the 2 - division or 3 - division processing (S1202 to S1208) in is replaced with the processing described below.

[0174] Use Figure 20 to illustrate the restriction on block division. Figure 20 is relative to Figure 11This results in a form that omits some processing steps. Therefore, sometimes the annotation and... Figure 11 The same step numbers are used, but the descriptions are omitted.

[0175] First, for all combinations of block segmentation, a decision is made on whether to restrict block segmentation (S1901). All combinations of block segmentation refer to two cases: dividing the block into two parts horizontally and dividing the block into two parts vertically. Furthermore, whether to restrict block segmentation depends on whether pixels beyond the image boundary are segmented. For example, when dividing the block into two parts horizontally, if pixels beyond the image boundary are segmented, then the block is restricted to being divided into two parts horizontally. Since the processing after S1302 involves processes other than those involving three-part segmentation... Figure 11 The processing is the same as in the previous section, so the explanation is omitted.

[0176] Here, use Figure 19 (a) provides a specific example. In Figure 19 In case (a), for all combinations of block segmentation, it is decided whether to restrict block segmentation (S1901). Currently, when the block is divided into two parts in the horizontal direction, since pixels beyond the image boundary are not segmented, the horizontal segmentation is not restricted. However, when the block is divided into two parts in the vertical direction, since pixels beyond the image boundary are segmented, the vertical segmentation is restricted. Therefore, for all combinations of block segmentation, block segmentation is not restricted (S1302: No). Furthermore, horizontal block segmentation is not restricted (S1303: No), but vertical block segmentation is restricted (S1304: Yes). That is, the block is divided into two parts in the horizontal direction (S1312).

[0177] Here, we consider the image boundaries and... Figure 19 Compared to modes closer to the top or left, pixels extending beyond the image boundaries are segmented in any mode. Therefore, 2- and 3-segmentation of blocks can always be restricted at the edges of the image.

[0178] By controlling block segmentation in this way, blocks at the edges of the image can be divided into appropriate sizes and shapes, thereby improving encoding efficiency. Furthermore, it enables block segmentation suitable for image encoding and decoding.

[0179] (Fifth Implementation)

[0180] The image encoding apparatus and image decoding apparatus according to the fifth embodiment of the present invention will be described. In the fifth embodiment, the limitation on block segmentation at the lower right end of the screen will be described. Since the structure other than this is the same as that of the first embodiment, the description is omitted.

[0181] like Figure 9As shown, the tree blocks at the edges of the image contain both in-image and out-of-image portions across the image boundary. In particular, compared to the out-of-image portions contained in tree blocks 1001 at the right end and 1002 at the bottom end, the out-of-image portions contained in tree block 1003 at the bottom right end tend to be more numerous. Therefore, due to the limitations of block segmentation, there is greater potential to improve coding efficiency. When tree blocks are segmented, block segmentation is restricted within the segmented blocks contained within each tree block.

[0182] use Figure 20 This illustrates the limitations of the block segmentation at the bottom right of the image. The block at the bottom right of the image is equivalent to... Figure 9 Tree block 1003.

[0183] First, a decision is made regarding whether to restrict block segmentation for all possible combinations of block segmentation (S1901). All possible combinations of block segmentation refer to two cases: dividing the block into two parts horizontally and dividing the block into two parts vertically. Furthermore, whether to restrict block segmentation depends on whether pixels beyond the image boundary are segmented. For example, when dividing the block into two parts horizontally, if pixels beyond the image boundary are segmented, then dividing the block into two parts horizontally is restricted.

[0184] Currently, since it is a block at the lower right of the image, pixels beyond the image boundary are segmented for all combinations of block segmentation. When block segmentation is restricted in all combinations (S1302: Yes), the block is not segmented (S1314).

[0185] Here, use Figure 21 Provide specific examples. Figure 21 (a) and Figure 21 (b) shows all combinations of block divisions in the tree block at the bottom right of the screen. Currently, Figure 21 (a) and Figure 21 In any of (b), pixels that extend beyond the image boundary are segmented. When block segmentation is restricted in all combinations (S1302: Yes), no block segmentation is performed (S1314).

[0186] Figure 21 (c) is in relation to Figure 21 Example (a) of dividing a tree block into 4 segments at the same image boundary. The resulting 4 segments are designated as blocks 601, 602, 603, and 604 according to the z-scan order. (About...) Figure 21 (c) explains the steps involved in block partitioning.

[0187] Figure 21 (c) Block segmentation is performed through the following steps. First, in Figure 3In the processing, the image is segmented into tree blocks (S1000), and it is determined that the internal structure of each tree block will be divided into 4 segments (S1001: Yes). The tree blocks are then divided into 4 segments (S1002). For each of the 4-segmented blocks 601, 602, 603, and 604, the following steps are performed: Figure 7 The processing.

[0188] exist Figure 7 In the process, it is determined that the internal part of the segmented block will not be further divided into 4 parts (S1101: No), and it is determined that the internal part of the block will be divided into 2 or 3 parts (S1103: Yes). Figure 7 The 2- or 3-segmentation process (S1104 to S1110) is replaced by Figure 20 The processing.

[0189] exist Figure 20 In the process, for all combinations of block segmentation, it is decided whether to restrict block segmentation (S1901). Currently, since it is a block at the lower right of the image, pixels exceeding the image boundary are segmented for all combinations of block segmentation. If block segmentation is restricted in all combinations (S1302: Yes), the block is not segmented (S1314). The process concludes with these steps. Figure 21 (c) block partitioning.

[0190] Figure 21 (d) is in relation to Figure 21 (c) An example of 4-segmentation of a tree block at different image boundaries. This example illustrates a complex block segmentation combining the above embodiments. In each of the 4-segmented blocks, blocks 602 and 603 are further segmented. After the tree block is 4-segmented (S1002), the steps of block segmentation are explained for each of the 4-segmented blocks 601, 602, 603, and 604.

[0191] Block 601 proceeds Figure 7 The processing. In Figure 7 In the process, it is determined that the internal structure of the divided block will not be further divided into 4 parts (S1101: No), or it is determined that the internal structure of the block will not be further divided into 2 or 3 parts (S1103: No). Block 601 ends the block division process without further dividing its internal structure (S1111).

[0192] Block 602 proceeds Figure 7 The processing. In Figure 7 In the process, it is determined that the internal part of the segmented block will not be further divided into 4 parts (S1101: No), and it is determined that the internal part of the block will be divided into 2 or 3 parts (S1103: Yes). Figure 7 The 2- or 3-segmentation process (S1104 to S1110) is replaced by Figure 20 The processing.

[0193] exist Figure 20 In step S1901, for all combinations of block segmentation, a decision is made as to whether to restrict block segmentation. Currently, when the block is divided into two parts horizontally, the horizontal segmentation is restricted because pixels beyond the image boundary are segmented. However, when the block is divided into two parts vertically, the vertical segmentation is not restricted because pixels beyond the image boundary are not segmented. Therefore, for all combinations of block segmentation, block segmentation is not restricted (S1302: No). Furthermore, horizontal block segmentation is restricted (S1303: Yes). That is, the block is divided into two parts vertically (S1308). As a result, block 602 is vertically segmented by segmentation 612.

[0194] Block 603 proceeds Figure 7 The processing. In Figure 7 In the process, it is determined that the internal part of the segmented block will not be further divided into 4 parts (S1101: No), and it is determined that the internal part of the block will be divided into 2 or 3 parts (S1103: Yes). Figure 7 The 2- or 3-segmentation process (S1104 to S1110) is replaced by Figure 18 The processing.

[0195] exist Figure 18 In this process, for all combinations of block segmentation, the number of pixels exceeding the image boundary is counted (S1651). All combinations of block segmentation refer to the two cases: dividing the block into two parts horizontally and dividing the block into two parts vertically. Currently, in block 603, as... Figure 17 (a) and Figure 17 As shown in (b), the case where the number of pixels exceeding the image boundary is the largest is the case of horizontal segmentation. Therefore, regarding the case of 2-segmentation of the block, the segmentation direction with the largest number of pixels exceeding the image boundary is determined to be the horizontal direction (S1652). Then, the block is 2-segmented in the determined segmentation direction (S1654). As a result, block 603 is horizontally segmented by segmentation 623.

[0196] Block 604 proceeds Figure 7 The processing. In Figure 7 In the process, it is determined that the internal part of the segmented block will not be further divided into 4 parts (S1101: No), and it is determined that the internal part of the block will be divided into 2 or 3 parts (S1103: Yes). Figure 7 The 2- or 3-segmentation process (S1104 to S1110) is replaced by Figure 20 The processing.

[0197] exist Figure 20In step S1901, a decision is made regarding whether to restrict block segmentation for all combinations of block segmentation. Currently, when the block is divided into two parts horizontally by segmentation 624, the horizontal segmentation is restricted because pixels exceeding the image boundary are segmented. Similarly, when the block is divided into two parts vertically by segmentation 614, the vertical segmentation is restricted because pixels exceeding the image boundary are segmented. In other words, for all combinations of block segmentation, pixels exceeding the image boundary are segmented. When block segmentation is restricted in all combinations (S1302: Yes), the block is not segmented (S1314).

[0198] By controlling block segmentation in this way, blocks at the edges of the image can be divided into appropriate sizes and shapes, thus improving encoding efficiency. Furthermore, block segmentation suitable for image encoding and decoding can be performed.

[0199] In all the embodiments described above, the object of control block segmentation is set to a position exceeding the image boundary. This allows for the determination of any boundary, with block segmentation controlled for positions exceeding that boundary. Furthermore, as an arbitrary boundary, pixels with higher importance compared to surrounding pixels can be identified, and positions exceeding that boundary can be designated as the object of control block segmentation. Moreover, the position exceeding any boundary is not limited to the bottom or right edge of the image; it can also be the top or left edge, or even not at the edge. In this case, even if it is not at the edge of the image, the block can be segmented into an appropriate size and shape, improving coding efficiency.

[0200] All of the above-described embodiments can also be combined in multiple ways.

[0201] In all the embodiments described above, the encoded bitstream output by the image encoding device has a specific data format so that it can be decoded according to the encoding method used in the embodiment. The encoded bitstream can be provided by recording on a computer-readable recording medium such as an HDD, SSD, flash memory, or optical disc, or it can be provided from a server via a wired or wireless network. Therefore, the image decoding device corresponding to this image encoding device can decode the encoded bitstream of this specific data format without a providing unit.

[0202] In the case of exchanging encoded bitstreams between an image encoding device and an image decoding device using wired or wireless networks, the encoded bitstreams can be converted into a data format suitable for the transmission method of the communication path for transmission. In this case, the device includes: a transmitting device that converts the encoded bitstream output by the image encoding device into encoded data suitable for the transmission method of the communication line and transmits it to the network; and a receiving device that receives the encoded data from the network, restores the encoded data to the encoded bitstream, and provides it to the image decoding device.

[0203] The transmitting device includes: a memory for buffering the encoded bitstream output by the image encoding device; a packet processing unit for packetizing the encoded bitstream; and a transmitting unit for transmitting the packetized encoded data via a network. The receiving device includes: a receiving unit for receiving the packetized encoded data via a network; a memory for buffering the received encoded data; and a packet processing unit for packetizing the encoded data to generate an encoded bitstream and providing it to the image decoding device.

[0204] When wired or wireless networks are used to exchange encoded bitstreams between image encoding and image decoding devices, a relay device is provided in addition to the transmitting and receiving devices. The relay device receives encoded data sent by the transmitting device and provides the encoded data to the receiving device. The relay device includes: a receiving unit for receiving packetized encoded data sent by the transmitting device; a memory for buffering the received encoded data; and a transmitting unit for sending the packetized encoded data to the network. Furthermore, the relay device may also include: a receiving packet processing unit for processing the packetized encoded data to generate an encoded bitstream; a recording medium for storing the encoded bitstream; and a transmitting packet processing unit for packetizing the encoded bitstream.

[0205] Alternatively, a display unit that displays the image decoded by the image decoding device can be added to the configuration to serve as the display device. In this case, the display unit reads the decoded image signal generated by the decoded image signal overlay unit 205 and stored in the decoded image memory 206, and displays it on the screen.

[0206] Alternatively, an imaging unit can be added to the structure to input the captured image into an image encoding device, thus serving as an imaging device. In this case, the imaging unit inputs the captured image signal into the block segmentation unit 101.

[0207] Figure 22 An example of the hardware structure of the encoding / decoding apparatus of this application is shown. The encoding / decoding apparatus includes the configuration of the image encoding apparatus and the image decoding apparatus according to embodiments of the present invention. The encoding / decoding apparatus 9000 has a CPU 9001, an encoder / decoder IC 9002, an I / O interface 9003, a memory 9004, an optical disc drive 9005, a network interface 9006, and a video interface 9009, all of which are connected via a bus 9010.

[0208] The image encoding unit 9007 and the image decoding unit 9008 are typically installed as a codec IC 9002. In the image encoding apparatus according to embodiments of the present invention, image encoding processing is performed by the image encoding unit 9007, and in the image decoding apparatus according to embodiments of the present invention, image decoding processing is performed by the image encoding unit 9007. The I / O interface 9003 is implemented, for example, via a USB interface, and is connected to an external keyboard 9104, mouse 9105, etc. The CPU 9001 controls the encoding / decoding apparatus 9000 based on user operations input through the I / O interface 9003 to execute actions desired by the user. User operations via the keyboard 9104, mouse 9105, etc., include selecting a function for encoding or decoding, setting the encoding quality, setting the input / output destination of the encoded stream, and setting the input / output destination of the image.

[0209] When a user desires to reproduce an image recorded on the disc recording medium 9100, the optical disc drive 9005 reads the encoded bitstream from the inserted disc recording medium 9100 and sends the read encoded stream to the image decoding unit 9008 of the codec IC 9002 via the bus 9010. The image decoding unit 9008 performs image decoding processing in the image decoding apparatus according to embodiments of the present invention on the input encoded bitstream and sends the decoded image to an external monitor 9103 via the video interface 9009. Additionally, the codec apparatus 9000 has a network interface 9006, which can connect to an external distribution server 9106 or portable terminal 9107 via network 9101. When a user desires to reproduce an image recorded on the distribution server 9106 or portable terminal 9107 instead of an image recorded on the disc recording medium 9100, the network interface 9006 obtains the encoded stream from the network 9101 instead of reading the encoded bitstream from the input disc recording medium 9100. Furthermore, if a user wishes to reproduce an image recorded in memory 9004, image decoding processing according to an embodiment of the present invention is performed on the encoded stream recorded in memory 9004.

[0210] When a user wishes to encode and record an image captured by an external camera 9102 in memory 9004, the video interface 9009 inputs the image from the camera 9102 and sends it to the image encoding unit 9007 of the codec IC 9002 via bus 9010. The image encoding unit 9007 performs image encoding processing according to the image encoding apparatus of this embodiment on the image input via the video interface 9009, generating an encoded bitstream. Then, the encoded bitstream is sent to memory 9004 via bus 9010. If the user wishes to record the encoded stream on a disc recording medium 9100 instead of memory 9004, the optical disc drive 9005 writes the encoded stream to the inserted disc recording medium 9100.

[0211] It is also possible to implement a hardware structure that has an image encoding device but no image decoding device, or a hardware structure that has an image decoding device but no image encoding device. Such a hardware structure can be implemented, for example, by replacing the codec IC 9002 with an image encoding unit 9007 or an image decoding unit 9008, respectively.

[0212] The processing related to the above encoding and decoding can, of course, be implemented using hardware transmission, storage, and receiving devices, or through firmware stored in ROM (Read-Only Memory), flash memory, or software such as computers. This firmware or software program can be provided by recording it on a readable recording medium such as a computer, or by providing it from a server via wired or wireless networks, or by providing it as data broadcast via terrestrial wave or satellite digital broadcasting.

[0213] The present invention has been described above based on embodiments. The embodiments are illustrative; various modifications can be made to the combination of these constituent elements and processing procedures, and such modifications are also within the scope of the present invention, as will be understood by those skilled in the art.

[0214] Industrial availability

[0215] This invention enables the use of techniques for encoding and decoding images.

[0216] Symbol Explanation

[0217] 100 Image encoding device, 101 Block segmentation unit, 102 Predictive image generation unit, 103 Residual signal generation unit, 104 Orthogonal transform / quantization unit, 105 Encoding unit, 106 Inverse quantization / inverse orthogonal transform unit, 107 Decoded image signal overlap unit, 108 Decoded image memory, 200 Image decoding device, 201 Decoding unit, 202 Block segmentation unit, 203 Inverse quantization / inverse orthogonal transform unit, 204 Predictive image generation unit, 205 Decoded image signal overlap unit, 206 Decoded image memory.

Claims

1. An image decoding apparatus for decoding an image in units of segmented blocks, characterized in that it comprises: The decoding unit decodes the block segmentation information, which includes a flag indicating that the object block is to be split into 2 or 3 segments; as well as The block segmentation unit recursively segments the image into rectangles of a specified size based on the block segmentation information, thereby generating the object block. The block segmentation portion includes: The 4-segmentation section divides the object block in the recursive segmentation into half horizontally and half vertically to generate 4 blocks to be processed in Z-scan order. as well as The 2-3 segmentation section performs 2- or 3-segmentation on the object block in the recursive segmentation, either horizontally or vertically, to generate 2 or 3 blocks to be processed in Z-scan order. The 4-segmentation unit determines whether to perform 4-segmentation on the object block. When it is determined that the object block should not be divided into four parts, the 2-3 segmentation unit determines whether to perform a two-part or three-part segmentation on the object block. If the 2-3 segmentation part divides the object block into two parts using a horizontal dividing line, then when the divided object block extends beyond the right side of the image boundary; or if the 2-3 segmentation part divides the object block into two parts using a vertical dividing line, then when the divided object block extends beyond the bottom side of the image boundary, then further 3-3 segmentation of the object block is prohibited. If the 2-3 segmentation part divides the object block into two parts using a horizontal dividing line, then when the divided object block extends beyond the right side of the image boundary, dividing the object block into two parts using a horizontal dividing line is prohibited; dividing the object block into two parts using a vertical dividing line is permitted. If the 2-3 segmentation part divides the object block into two parts using a vertical dividing line, then when the divided object block extends beyond the lower edge of the image boundary, dividing the object block into two parts using a vertical dividing line is prohibited; dividing the object block into two parts using a horizontal dividing line is permitted. If the tree block to be processed in raster scan order (128×128 pixels) after image segmentation exceeds the lower right edge of the image boundary, and therefore the tree block is prohibited from being segmented by horizontal and vertical dividing lines, then the 2-3 segmentation unit will not segment the tree block. The decoding unit does not decode the flag when the object block is prohibited from being divided into three parts.

2. An image decoding method, which decodes an image in units of segmented blocks, characterized in that it includes: The decoding step involves decoding the block segmentation information, which includes flags indicating whether the object block is split into 2 or 3 segments. as well as The block segmentation step recursively segments the image into rectangles of a specified size based on the block segmentation information, thereby generating the object blocks. The block segmentation step includes one of a 4-segmentation step or a 2-3-segmentation step. In the four-segmentation step, the object block in the recursive segmentation is divided into half horizontally and half vertically, resulting in four blocks to be processed in Z-scan order. In the 2-3 segmentation step, the object block in the recursive segmentation is divided into 2 or 3 segments in the horizontal or vertical direction, generating 2 or 3 blocks to be processed in Z-scan order. In the 4-segmentation step, it is determined whether the object block should be 4-segmented. If it is determined that the object block will not be divided into four parts, then in the 2-3 division step, it is determined whether to divide the object block into two or three parts. In the 2-3 segmentation step, if the object block is segmented horizontally, then segmenting it beyond the right side of the image boundary is prohibited; or if the object block is segmented vertically, then segmenting it beyond the bottom side of the image boundary is prohibited. In the 2-3 segmentation step, if the object block is divided into two parts by a horizontal dividing line, then if the segmented object block extends beyond the right side of the image boundary, dividing the object block into two parts by a horizontal dividing line is prohibited; dividing the object block into two parts by a vertical dividing line is permitted. In the 2-3 segmentation step, if the object block is divided into two parts by a vertical dividing line, then if the segmented object block extends beyond the lower edge of the image boundary, further division by a vertical dividing line is prohibited; however, division by a horizontal dividing line is permitted. In the 2-3 segmentation steps, if the 128×128 pixel tree block to be processed in raster scan order after segmenting the image exceeds the lower right side of the image boundary, thus prohibiting the 2-segmentation of the tree block using horizontal and vertical segmentation lines, then the tree block is not segmented. In the decoding step, the flag is not decoded if the object block is prohibited from being divided into three parts.

3. An image encoding apparatus for encoding an image in units of segmented blocks, characterized in that it comprises: The block segmentation unit recursively segments the image into rectangles of a specified size to generate object blocks; as well as, The encoding unit encodes block segmentation information, which includes flags indicating whether the object block is divided into 2 or 3 segments. The block segmentation portion includes: The 4-segmentation section divides the object block in the recursive segmentation into half horizontally and half vertically to generate 4 blocks to be processed in Z-scan order. as well as The 2-3 segmentation section performs 2- or 3-segmentation on the object block in the recursive segmentation, either horizontally or vertically, to generate 2 or 3 blocks to be processed in Z-scan order. The 4-segmentation unit determines whether to perform 4-segmentation on the object block. When it is determined that the object block should not be divided into four parts, the 2-3 segmentation unit determines whether to perform a two-part or three-part segmentation on the object block. If the 2-3 segmentation part divides the object block into two parts using a horizontal dividing line, then when the divided object block extends beyond the right side of the image boundary; or if the 2-3 segmentation part divides the object block into two parts using a vertical dividing line, then when the divided object block extends beyond the bottom side of the image boundary, then further 3-3 segmentation of the object block is prohibited. If the 2-3 segmentation part divides the object block into two parts using a horizontal dividing line, then when the divided object block extends beyond the right side of the image boundary, dividing the object block into two parts using a horizontal dividing line is prohibited; dividing the object block into two parts using a vertical dividing line is permitted. If the 2-3 segmentation part divides the object block into two parts using a vertical dividing line, then when the divided object block extends beyond the lower edge of the image boundary, dividing the object block into two parts using a vertical dividing line is prohibited; dividing the object block into two parts using a horizontal dividing line is permitted. If the tree block to be processed in raster scan order (128×128 pixels) after image segmentation exceeds the lower right edge of the image boundary, and therefore the tree block is prohibited from being segmented by horizontal and vertical dividing lines, then the 2-3 segmentation unit will not segment the tree block. The encoding unit does not encode the flag when the object block is prohibited from being divided into three parts.

4. An image encoding method, wherein the image is encoded in units of segmented blocks, the image encoding method being characterized by comprising: The block segmentation step recursively segments the image into rectangles of a specified size to generate object blocks; as well as The encoding step involves encoding the block segmentation information, which includes flags indicating whether the object block is divided into 2 or 3 segments. The block segmentation step includes one of a 4-segmentation step or a 2-3-segmentation step. In the four-segmentation step, the object block in the recursive segmentation is divided into half horizontally and half vertically, resulting in four blocks to be processed in Z-scan order. In the 2-3 segmentation step, the object block in the recursive segmentation is divided into 2 or 3 segments in the horizontal or vertical direction, generating 2 or 3 blocks to be processed in Z-scan order. In the 4-segmentation step, it is determined whether the object block should be 4-segmented. If it is determined that the object block will not be divided into four parts, then in the 2-3 division step, it is determined whether to divide the object block into two or three parts. In the 2-3 segmentation step, if the object block is segmented horizontally, then segmenting it beyond the right side of the image boundary is prohibited; or if the object block is segmented vertically, then segmenting it beyond the bottom side of the image boundary is prohibited. In the 2-3 segmentation step, if the object block is divided into two parts by a horizontal dividing line, then if the segmented object block extends beyond the right side of the image boundary, dividing the object block into two parts by a horizontal dividing line is prohibited; dividing the object block into two parts by a vertical dividing line is permitted. In the 2-3 segmentation step, if the object block is divided into two parts by a vertical dividing line, then if the segmented object block extends beyond the lower edge of the image boundary, further division by a vertical dividing line is prohibited; however, division by a horizontal dividing line is permitted. In the 2-3 segmentation steps, if the 128×128 pixel tree block to be processed in raster scan order after segmenting the image exceeds the lower right side of the image boundary, thus prohibiting the 2-segmentation of the tree block using horizontal and vertical segmentation lines, then the tree block is not segmented. In the encoding step, the flag is not encoded if the object block is prohibited from being divided into three parts.

5. A method for storing a bit stream, characterized in that, The image encoding method of claim 4 is used to generate a bitstream; and the bitstream is stored.

6. A method for transmitting a bit stream, characterized in that, The image encoding method of claim 4 is used to generate a bit stream; and the bit stream is transmitted.

Images