Image processing apparatus and method

By dividing parallel encoding tree units and transformation blocks in the image processing device, combining entropy coding delay and prediction mode limitation, the problem of parallelism reduction caused by the increase in CTU size is solved, and the efficiency of image processing is improved.

CN119052502APending Publication Date: 2024-11-29SONY GROUP CORP
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202410974097.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2018-12-21
Filing Date
2019-12-06
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

As the image resolution increases, the CTU size processed in encoding increases, resulting in a decrease in parallel between encoding and decoding, thereby increasing processing time.

Method used

By dividing the image into multiple parallel encoding tree unit CTU rows in the image processing device, each CTU is divided into multiple blocks, and a transformation block is used as a processing unit during the encoding and decoding process, using entropy encoding delay and within the frame / Limitation of inter-frame prediction, optimized parallelization of encoding and decoding.

Benefits of technology

The parallelism of encoding and decoding is effectively suppressed, processing time is reduced, and image processing efficiency is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119052502A_ABST
    Figure CN119052502A_ABST
Patent Text Reader

Abstract

The present disclosure relates to an image processing apparatus, comprising: at least one processor configured to: divide an image into a plurality of coding tree units (CTUs) having a plurality of parallel rows of coding tree units (CTUs); dividing each CTU into a plurality of blocks, where each of the blocks is a processing unit that is smaller in size than the CTU; determining whether to apply entropy coding for each CTU row, where the entropy coding delays one CTU between adjacent CTU rows; and based on determining to apply entropy coding of each CTU row: referring to a corresponding adjacent block of at least one adjacent upper CTU of an adjacent upper CTU row, performing intra prediction on a left upper block of the current CTU of the current CTU row, and referring to the corresponding adjacent block of the adjacent upper CTU of the upper CTU row and not referring to a left lower block of a non-adjacent right upper CTU of the adjacent upper CTU row, and performing intra-frame prediction on the right upper block of the current CTU.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of the PCT application with international application number PCT / JP2019 / 047782 filed on December 6, 2019 and invention name “Image Processing Device and Method”. The date on which the PCT application entered the Chinese national phase is June 11, 2021, and the national application number is 201980082665.2. Technical Field

[0002] The present disclosure relates to an image processing device and an image processing method, and particularly to an image processing device and an image processing method capable of suppressing a decrease in the parallelism of encoding and decoding. Background Art

[0003] WPP (Wavefront Parallel Processing) is routinely introduced into HEVC (High Efficiency Video Coding) as a tool that parallelizes CABAC (Context-based Adaptive Binary Arithmetic Code) encoding in each CTU (Coding Tree Unit) row while minimizing the reduction in coding efficiency (see, for example, non-patent document 1).

[0004] In addition, a method has been proposed in which a 64×64 block is used as a processing unit for a luminance component and a 32×32 block is used as a processing unit for a chrominance component to perform pipelining on individual processing steps such as encoding (decoding), transform and quantization (inverse quantization and inverse transform), deblocking, and SAO (Sample Adaptive Offset) (for example, see Non-Patent Document 2 and Non-Patent Document 3).

[0005] [Prior art literature]

[0006] [Non-patent literature]

[0007] [Non-patent document 1]

[0008] "High efficiency video coding", H.265, from the telecommunications standardization department of the ITU (International Telecommunication Union) in December 2016;

[0009] [Non-patent document 2]

[0010] Tzu-Der Chuang, Ching-Yeh Chen, Yu-Wen Huang, Shaw-Min Lei, “CE1-related: Separate tree partitioning at 64×64-luma / 32×32-chroma unit level”, JVET-K0230-v3, Joint Video Experts Group (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 11th meeting: SI, Ljubljana, July 10-18, 2018;

[0011] [Non-patent document 3]

[0012] Chia-Ming Tsai, Chih-Wei Hsu, Tzu-Der Chuang, Ching-Yeh Chen, Yu-Wen Huang, Shaw-Min Lei, “CE1.2.1: Constraint for binary and ternary partitions”, JVET-L0081-v2, Joint Video Experts Group (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG11, 12th meeting: Macau, China, October 3-12, 2018. Summary of the invention

[0013] [Technical issues]

[0014] However, as image resolution increases, there is a recent need to process larger CTU sizes in encoding. For example, in the case of HEVC described in Non-Patent Document 1, the CTU size is a maximum of 64×64. However, in the case of the methods described in Non-Patent Document 2 and Non-Patent Document 3, the CTU size is a maximum of 128×128.

[0015] In WPP, in the same picture (specifically, in the case of the same resolution), as described above, the increased CTU size reduces the number of CTU rows in a frame. Therefore, the parallelism of encoding and decoding (in other words, the number of CTUs processed in parallel) is reduced, which may cause an increase in the delay in the parallel pipeline (in other words, the processing time is extended).

[0016] In addition, according to the methods described in non-patent document 2 and non-patent document 3, individual processing steps such as encoding (decoding), transform and quantization (inverse quantization and inverse transform), deblocking and SAO can be pipelining by using blocks of reduced size as processing units, however, it is difficult to perform parallel pipelining on the encoding and decoding of each block.

[0017] In view of such circumstances, an object of the present disclosure is to make it possible to suppress a decrease in the degree of parallelism of encoding and decoding.

[0018] [Solution to the problem]

[0019] An image processing device according to one aspect of the present technology includes a coding unit that encodes coefficient data related to an image in parallel for each row of a coding tree unit, each coding tree unit including a transform block and corresponding to a highest coding block of a tree structure, and the coding unit delays encoding of each row by an amount corresponding to one coding tree unit relative to encoding of an immediately previous row of a row to be encoded.

[0020] An image processing method according to one aspect of the present technology includes encoding coefficient data related to an image in parallel according to each row of a coding tree unit, each coding tree unit including a transform block and corresponding to a highest coding block of a tree structure, and delaying the encoding of each row relative to the encoding of the immediately previous row of the row to be encoded by an amount corresponding to one coding tree unit.

[0021] An image processing device according to another aspect of the present technology includes a decoding section that decodes encoded data obtained by encoding coefficient data related to an image in parallel for each row of a coding tree unit, each coding tree unit including a transform block and corresponding to a highest-order coding block of a tree structure, and the decoding section performs decoding of each row by an amount corresponding to one coding tree unit with respect to decoding of an immediately previous row of a row to be decoded.

[0022] An image processing method according to another aspect of the present technology includes: decoding, in parallel, encoded data obtained by encoding coefficient data associated with an image in each row of a coding tree unit that includes a transform block and corresponds to an uppermost coding block in a tree structure, and performing decoding on each row so that the decoding is delayed by an amount corresponding to one coding tree unit relative to decoding of a row immediately above the row to be decoded.

[0023] In an image processing device and an image processing method according to one aspect of the present technology, coefficient data related to an image is encoded in parallel in each row of a coding tree unit that includes a transform block and corresponds to an uppermost coding block in a tree structure, and encoding is performed on each row so that the encoding is delayed by an amount corresponding to one coding tree unit relative to the encoding of the row immediately above the row to be encoded.

[0024] In an image processing device and an image processing method according to another aspect of the present technology, encoded data obtained by encoding coefficient data associated with an image is decoded in parallel according to each row of a coding tree unit, each coding tree unit including a transform block and being a highest coding block of a tree structure, and decoding of each row is delayed by an amount corresponding to one coding tree unit relative to decoding of an immediately previous row to be decoded.

[0025] An image processing device according to one aspect of the present technology includes: at least one processor, which is configured to: divide an image into a plurality of coding tree units having a plurality of parallel coding tree unit CTU rows; divide each CTU into a plurality of blocks, wherein each of the blocks is a processing unit having a size smaller than that of the CTU; determine whether to apply entropy coding of each CTU row, wherein the entropy coding is delayed by one CTU between adjacent CTU rows; and based on the determination to apply entropy coding of each CTU row: refer to a corresponding neighboring block of at least one adjacent upper CTU of an adjacent upper CTU row, perform intra-frame prediction on an upper left block of a current CTU of a current CTU row, and refer to a corresponding neighboring block of an adjacent upper CTU of the upper CTU row without referring to a lower left block of a non-adjacent upper right CTU of the adjacent upper CTU row. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] [ Figure 1 ]

[0027] Figure 1 is a diagram showing an example of WPP.

[0028] [ Figure 2 ]

[0029] Figure 2 is a diagram showing an example of the relationship between the degree of parallelism and the block size.

[0030] [ Figure 3 ]

[0031] Figure 3 is a diagram showing an example of a VPDU.

[0032] [ Figure 4 ]

[0033] Figure 4 is a diagram showing an example of pipelining using VPDU.

[0034] [ Figure 5 ]

[0035] Figure 5 is a diagram showing a method for parallel pipelining of encoding and decoding.

[0036] [ Figure 6 ]

[0037] Figure 6 is a diagram showing an example of parallelization in each CTU row by using VPDU.

[0038] [ Figure 7 ]

[0039] Figure 7 is a diagram showing an example of processing timing of each thread.

[0040] [ Figure 8 ]

[0041] Figure 8 is a diagram showing an example of reference restriction for intra prediction.

[0042] [ Fig. 9 ]

[0043] Fig. 9 An example of reference restriction for intra prediction is shown.

[0044] [ Fig.10 ]

[0045] Fig.10 An example of reference restriction for intra prediction is shown.

[0046] [ Fig.11 ]

[0047] Fig.11 An example of reference restriction for inter prediction is shown.

[0048] [ Fig.12 ]

[0049] Fig.12 is a diagram showing an example of reference restriction for inter prediction.

[0050] [ Fig.13 ]

[0051] Fig.13 is a block diagram showing a main configuration example of an image encoding device.

[0052] [ Fig.14 ]

[0053] Fig.14 is a flowchart showing an example of the flow of image encoding processing.

[0054] [ Fig.15 ]

[0055] Fig.15 is a flowchart showing an example of the flow of prediction processing.

[0056] [ Fig.16 ]

[0057] Fig.16 is a flowchart showing an example of the flow of encoding processing.

[0058] [ Fig.17 ]

[0059] Fig.17 is a flowchart showing an example of the flow of CTU processing.

[0060] [ Fig.18 ]

[0061] Fig.18 is a flowchart showing an example of the flow of CTU processing.

[0062] [ Fig.19 ]

[0063] Fig.19 is a block diagram showing a main configuration example of an image decoding device.

[0064] [ Fig. 20 ]

[0065] Fig. 20 is a flowchart showing an example of the flow of image decoding processing.

[0066] [ Fig.21 ]

[0067] Fig.21 is a flowchart showing an example of the flow of a decoding process.

[0068] [ Fig. 22 ]

[0069] Fig. 22 is a flowchart showing an example of the flow of CTU processing.

[0070] [ Fig.23 ]

[0071] Fig.23 is a flowchart showing an example of the flow of CTU processing.

[0072] [ Fig.24 ]

[0073] Fig.24 is from Fig.23 Continuing with the flowchart, and Fig.24 An example of the flow of CTU processing is shown.

[0074] [ Fig.25 ]

[0075] Fig.25 is a block diagram showing a main configuration example of a computer. DETAILED DESCRIPTION

[0076] Modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described below. The description is given in the following order.

[0077] 1. Documents supporting technical content and technical terminology, etc.

[0078] 2.WPP and VPDU

[0079] 3. Concept

[0080] 4. Method 1

[0081] 5. First Embodiment (Image Coding Device and Image Decoding Device)

[0082] 6. Supplementary features

[0083] <1. Documents supporting technical content and technical terminology, etc.>

[0084] The scope disclosed in the present technology includes the contents described in the non-patent literature listed below, the contents known at the time of filing, and the contents described in the examples.

[0085] Non-patent document 1: (listed above)

[0086] Non-patent document 2: (listed above)

[0087] Non-patent document 3: (listed above)

[0088] Non-patent document 4: ITU (International Telecommunication Union) Telecommunication Standardization Sector, "Advanced video coding for generic audiovisual services", H.264, 04 / 2017

[0089] Non-Patent Literature 5: Jianle Chen, Elena Alshina, Gary J. Sullivan, Jens-Rainer, Jill Boyce, “Algorithm Description of Joint Exploration Test Model 4”, VET-G1001_v1, Joint Video Exploration Team (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 Seventh Meeting: Turin, Italy (IT), July 13-21, 2017

[0090] Non-Patent Literature 6: Benjamin Bross, Jianle Chen, Shan Liu, “Versatile Video Coding (Draft 2)”, JVET-K1001-v7, Joint Video Experts Group (JVET) of ITU-T SG 16WP 3, and ISO / IEC JTC 1 / SC 29 / WG 11 11th Meeting: Ljubljana, Slovenia (SI), July 10-18, 2018

[0091] In other words, the contents described in the non-patent documents listed above also constitute a basis for determining the support requirements. For example, the quadtree block structure described in non-patent document 1 and the QTBT (quadtree plus binary tree) block structure described in non-patent document 5 are intended to fall within the scope of the present disclosure and also meet the support requirements in the claims, even if the examples do not include a direct description of such structures. In addition, for example, technical terms such as parsing, grammar, and semantics are similarly intended to fall within the scope of the present disclosure and meet the support requirements in the claims, even if the examples do not include a direct description of such terms.

[0092] In addition, unless otherwise specified, the "block" (not referring to the block of the processing unit) used in this document to describe a partial area or a processing unit of an image (picture) indicates any partial area in the picture, and the block size, shape, characteristics, etc. are not limited. For example, "block" is intended to include any partial area (processing unit), such as TB (transform block), TU (transform unit), PB (prediction block), PU (prediction unit), SCU (minimum coding unit), CU (coding unit), LCU (maximum coding unit), CTB (coding tree block), CTU (coding tree unit), transform block, sub-block, macroblock, tile or slice described in non-patent documents 1 to non-patent documents 6 listed above.

[0093] In addition, when specifying the size of such a block, indirect specification of the block size as well as direct specification of the block size can be used. For example, identification information for identifying the size can be used to specify the block size. In addition, for example, the block size can be specified by using the size ratio of the block to the reference block (e.g., LCU, SCU, etc.) or the difference between the block and the reference block. For example, in the case of sending information that specifies the block size as a syntax element, etc., information that indirectly specifies the size as described above can be used as the information. This makes it possible to reduce the amount of information and can allow for improved coding efficiency. In addition, the specification of the block size includes the specification of the range of the block size (e.g., the specification of the allowable range of the block size, etc.).

[0094] In addition, the encoding used in this article includes partial processing and overall processing for transforming an image into a bit stream. For example, encoding includes: general processing of quantization and arithmetic coding; comprehensive processing of prediction processing, quantization and arithmetic coding, etc.; and comprehensive processing of prediction processing, orthogonal transformation, quantization and arithmetic coding, etc. Similarly, decoding includes partial processing and overall processing for transforming a bit stream into an image. For example, decoding includes: comprehensive processing of inverse arithmetic decoding and inverse quantization; comprehensive processing of inverse arithmetic decoding, inverse quantization and prediction processing, etc.; and comprehensive processing of inverse arithmetic decoding, inverse quantization, inverse orthogonal transformation, prediction processing, etc.

[0095] <2.WPP and VPDU>

[0096] <wpp>

[0097] For example, as described in non-patent document 1, WPP (Wavefront Parallel Processing) is conventionally introduced into HEVC (High Efficiency Video Coding) as a tool that parallelizes CABAC (Context-based Adaptive Binary Arithmetic Code) encoding in each row of CTU (Coding Tree Unit) while minimizing the reduction in coding efficiency.

[0098] For example, Figure 1 A portion of a picture to be encoded is shown, and each square indicates a CTU. Figure 1 As shown, CTUs are arranged in a matrix. When WPP is applied, the steps of encoding and decoding (entropy encoding and entropy decoding) in each row of CTU (also referred to as each CTU row) are parallelized (the processing steps in each CTU row are performed in parallel). In each CTU row, as shown by the dotted arrows in the figure, CTUs are processed one by one starting from the leftmost CTU (entropy encoding or entropy decoding).

[0099] Each CTU is processed by using the occurrence probability derived in the processing of the previous CTU as context. However, the leftmost CTU in the topmost row of the image is processed by using the context with the initial value. In addition, as Figure 1 As shown by the solid square and arrow in FIG. 1 , the leftmost CTU in each of the immediately above and subsequent CTU rows is processed by using the occurrence probability (also referred to as the learned occurrence probability) derived when processing the second CTU from the left in the CTU row immediately above the CTU row to be processed as a context. In other words, Figure 1 As shown by the gray squares in , each CTU row is processed while being delayed by an amount corresponding to two CTUs relative to the CTU row immediately above the CTU row.

[0100] This allows processing of individual CTU rows in parallel (even with timing gaps, processing can be achieved so that at least the processing timings partially overlap with each other (there is a point in time when CTUs in multiple CTU rows are processed)). Therefore, encoding and decoding can be performed faster than when all CTUs in an image are processed serially.

[0101] <Reduced parallelism>

[0102] However, as image resolution increases, there has recently been a demand for larger CTU sizes to be processed in encoding. For example, in the case of HEVC described in Non-Patent Document 1, the CTU size is a maximum of 64×64. However, in the case of the methods described in Non-Patent Document 2 and Non-Patent Document 3, the CTU size is a maximum of 128×128.

[0103] In WPP, in the same image (specifically, in the case of the same resolution), as described above, the increased CTU size reduces the number of CTU rows in a frame. Therefore, the parallelism of encoding and decoding (in other words, the number of CTUs processed in parallel) is reduced, which may lead to increased latency in the parallel pipeline (in other words, extended processing time).

[0104] For example, it is assumed that the same processing time is used for all CTUs. As shown in formula (1), the maximum parallelism is defined, which indicates the maximum number of CTUs processed at the same time point. In formula (1), CTUw indicates the number of CTUs in the horizontal direction in the image to be processed. In addition, CTUh indicates the number of CTUs in the vertical direction in the image to be processed. In addition, the function ceil indicates rounding to the nearest integer.

[0105] [Mathematical formula 1]

[0106] Maximum parallelism = min(ceil(CTUw / 2),CTUh)...(1)

[0107] In addition, as shown in formula (2), an average parallelism is defined, which indicates the average number of CTUs processed at the same time point. In formula (2), p(x, y) indicates the parallelism achieved when processing CTU(x, y). In addition, CTUw and CTUh are the same as CTUw and CTUh in formula (1).

[0108] [Mathematical formula 2]

[0109] Average parallelism = Σp(x,y) / (CTUw*CTUh)...(2)

[0110] For example, Figure 2 The table in indicates comparison results between maximum parallelism and average parallelism when processing a 4K image (an image with a resolution of 3840×2160) through WPP, for example, when the CTU size is at most 64×64 and when the CTU size is at most 128×128.

[0111] like Figure 2 As shown in the table, when the maximum CTU size is 64×64, CTUw and CTUh are 60 and 34, respectively. Therefore, the above equations (1) and (2) indicate that the maximum parallelism is 30, while the average parallelism is 21.2. On the other hand, when the maximum CTU size is 128×128, CTUw and CTUh are 30 and 17, respectively. Therefore, the above equations (1) and (2) indicate that the maximum parallelism is 15, while the average parallelism is 10.6. Therefore, the parallelism (maximum parallelism and average parallelism) is reduced to half. Therefore, as described above, when the maximum CTU size is 128×128, there is a possibility that the delay of the parallel pipeline increases and the processing time is prolonged compared to the case where the maximum CTU size is 64×64.

[0112] Additionally, as the processing load varies between individual CTUs being processed in parallel, wait times may be required to coordinate the processing timing of CTU rows with each other. The granularity of parallelism increases as the CTU size increases, which makes it more difficult to distribute the variations in the CTU processing load. In other words, as the CTU size increases, wait times may increase due to the variations, and the increased wait times may further reduce the actual parallelism, and therefore, the reduced actual parallelism may further extend the processing time.

[0113] <vpdu>

[0114] In addition, as described in non-patent document 2 and non-patent document 3, the following method is proposed: a 64×64 block is used as a processing unit for a luminance component, and a 32×32 block is used as a processing unit for a chrominance component to perform pipelining on individual processing steps such as encoding (decoding), transformation and quantization (inverse quantization and inverse transformation), deblocking, and SAO (sample adaptive offset).

[0115] For example, Figure 3 As shown, for the luminance component, the 128×128 CTB is divided into four parts (the CTB is divided into two parts in the vertical direction and into two parts in the horizontal direction), and a 64×64 block (Virtual Pipeline Data Unit, also called VPDU) is used as a processing unit. In addition, for the color difference component, the 64×64 CTB is divided into four parts (the CTB is divided into two parts in the vertical direction and into two parts in the horizontal direction), and a 32×32 block (also called VPDU) is used as a processing unit.

[0116] like Figure 4 As shown in the example of , by using such VPDU as a processing unit, pipelined processing steps such as encoding (decoding), transform and quantization (inverse quantization and inverse transform), deblocking and SAO are performed. This allows suppressing an increase in the total processing time.

[0117] However, according to this method, it is difficult to perform parallel pipelining on encoding and decoding in each block.

[0118] <3. Concept>

[0119] <WPP in each transform block row>

[0120] Therefore, if Figure 5 As in method 1 described in the top row of the table, WPP compatible with VPDU is implemented, that is, by parallelization in each CTU row using VPDU. A transform block is a processing unit for transform processing such as an orthogonal transform performed on image data, and a CTU row indicates a row (line) of a CTU arranged in a matrix.

[0121] As in Figure 5 As in method 1-1 described in the second upper row of the table, encoding and decoding can be parallelized in each row of the CTU including the VPDU, and the delay (pipeline delay) between rows can be set to an amount corresponding to one CTU.

[0122] For example, coefficient data associated with an image is encoded in parallel in each row of a coding tree unit, each coding tree unit includes a single or multiple transform blocks and corresponds to the highest coding block in the tree structure, and the encoding of each row is performed with a delay corresponding to one coding tree unit relative to the encoding of the row immediately above the row. In addition, for example, the image processing device includes a coding unit that encodes coefficient data associated with an image in parallel according to each row of a coding tree unit, each coding tree unit includes a single or multiple transform blocks and corresponds to the highest coding block in the tree structure, and encoding is performed on each row so that the encoding is delayed relative to the encoding of the row immediately above the row by an amount corresponding to one coding tree unit. Compared with WPP in HEVC, this greatly reduces the delay between CTU rows, thereby allowing the reduction in the parallelism of encoding to be suppressed. This allows the increase in encoding processing time to be suppressed.

[0123] In addition, for example, in each row of the coding tree units of coefficient data associated with an image, the coding tree units may be encoded one by one in order starting from the leftmost coding tree unit.

[0124] In addition, for example, each coding tree unit may be encoded for each transform block.

[0125] In addition, for example, each transform block may be entropy encoded by using the occurrence probability derived from entropy encoding of the previous transform block.

[0126] In addition, for example, the first transform block in the leftmost coding tree unit in the row of the uppermost coding tree unit in the image may be entropy encoded by using the initial value of the occurrence probability. In addition, for example, the first transform block in the leftmost coding tree unit in each of the next upper and subsequent rows of the coding tree units in the image may be entropy encoded by using the occurrence probability derived from the entropy encoding of the last transform block in the leftmost coding tree unit in the row of the coding tree unit immediately above the row.

[0127] In addition, for example, the transformation block may be a VPDU (Virtual Pipeline Data Unit).

[0128] In addition, the encoded data obtained by encoding the coefficient data associated with the image is decoded in parallel in each row of the coding tree unit, each coding tree unit includes a single or multiple transform blocks and corresponds to the highest coding block in the tree structure, and the decoding of each row is delayed by an amount corresponding to one coding tree unit relative to the decoding of the row immediately above the row while being performed. In addition, for example, the image processing device includes a decoding unit that decodes the encoded data obtained by encoding the coefficient data associated with the image in parallel in each row of the coding tree unit, each coding tree unit includes a single or multiple transform blocks and corresponds to the highest coding block in the tree structure, and performs decoding by delaying the decoding of each row relative to the decoding of the row immediately above the row by an amount corresponding to one coding tree unit. Compared with WPP in HEVC, this greatly reduces the delay between CTU rows, thereby allowing the reduction in the parallelism of decoding to be suppressed. This allows the increase in decoding processing time to be suppressed.

[0129] In addition, for example, the encoded data in each row of the coding tree unit of coefficient data related to the image may be decoded so that the coding tree units are decoded one by one in sequence starting from the leftmost coding tree unit.

[0130] In addition, for example, the encoded data of each coding tree unit may be decoded for each transform block.

[0131] In addition, for example, the encoded data of each transform block may be entropy-decoded by using the occurrence probability derived from the entropy decoding of the encoded data of the previous transform block.

[0132] In addition, for example, the coded data of the first transform block in the leftmost coding tree unit in the topmost row of the coding tree unit in the image can be entropy decoded by using the initial value of the occurrence probability. In addition, for example, the coded data of the first transform block in each of the leftmost coding tree units in each of the next upper and subsequent rows of the coding tree unit in the image can be entropy decoded by using the occurrence probability derived from entropy decoding of the last transform block in the leftmost coding tree unit in the row of the coding tree unit immediately above the row.

[0133] In addition, for example, the transformation block may be a VPDU (Virtual Pipeline Data Unit).

[0134] In addition, as in Figure 5 In the method 1-2 described in the third paragraph from the top in the table of , intra prediction can be restricted. Figure 5 As in method 1-2-1 described in the fourth paragraph from the top in the table in , in the intra-frame prediction of the upper right VPDU in the CTU to be processed (also called the current CTU), reference to the VPDU (CTU) at the upper right of the upper right VPDU can be prohibited (invalid).

[0135] For example, in the intra prediction of the upper right transform block of the coding tree unit, the reference to the transform block at the upper right of the upper right transform block can be invalidated. This makes it possible to reduce the dependency relationship between CTU rows based on the reference to the intra prediction. Therefore, the increase in waiting time can be suppressed.

[0136] In addition, as in Figure 5 In the method 1-3 described in the fifth paragraph from the top in the table in, inter-frame prediction can be restricted. For example, as in Figure 5 As in method 1-3-1 described in the sixth paragraph from the top in the table in , for inter-frame prediction in which the upper right VPDU in the current CTU is set as the prediction block to be processed (also referred to as the current prediction block), reference to the VPDU (CTU) at the upper right of the prediction block may be prohibited (invalid). In other words, restrictions may be imposed similarly to the case of the above-mentioned method 1-2-1 (i.e., intra-frame prediction may be restricted).

[0137] For example, in the inter-frame prediction of the upper right transform block of the coding tree unit, the reference to the transform block at the upper right of the upper right transform block can be invalidated. This makes it possible to reduce the dependency relationship between CTU rows based on the reference to the inter-frame prediction, thereby suppressing the increase in waiting time.

[0138] In addition, as in Figure 5 As in method 1-3-2 described in the seventh paragraph from the top in the table in , in inter-frame prediction in which the current prediction block has a block size of 128×N, reference to the block at the upper right of the current prediction block can be prohibited (invalid).

[0139] For example, in the inter-frame prediction of a prediction block corresponding to the processing unit of the inter-frame prediction and having the same horizontal length as the horizontal length of the coding tree unit, the reference to the transform block at the upper right of the prediction block can be invalidated. This makes it possible to reduce the dependency relationship between CTU rows based on the reference to the inter-frame prediction, thereby suppressing the increase in waiting time.

[0140] <4. Method 1>

[0141] <4-1: Method 1-1>

[0142] Next, the above-mentioned "Method 1" will be described in more detail. Figure 6 1 shows a portion of a picture to be encoded. The picture is divided into CTUs 111 corresponding to the highest coding blocks in the tree structure. Figure 6 , each area surrounded by a thick square is a CTU 111. In WPP in HEVC, processing (entropy encoding and entropy decoding) is parallelized in each row of CTU 111. In each CTU row, CTUs are processed one by one from left to right along the CTU row, for example, in the order of CTU 111-1, CTU 111-2, CTU 111-3, and CTU 111-4.

[0143] In addition, each CTU 111 is divided into VPDUs 121. Figure 6 , each area surrounded by a thin square is a VPDU 121. In this example, each CTU 111 is divided into four parts (two parts in the vertical direction and two parts in the horizontal direction). For example, CTU 111-1 is divided into VPDU 121-1, VPDU 121-2, VPDU 121-3 and VPDU 121-4. In other words, one CTU row is provided with two VPDU rows. Note that the size of CTU 111 is optional. As long as VPDU 121 is smaller than CTU 111, the size of VPDU 121 is also optional. In the description of this article, it is assumed that the CTU size is 128×128, and the VPDU size is assumed to be 64×64.

[0144] In method 1-1, if Figure 6 As shown by the dotted arrows in FIG. 1 , the processing (entropy encoding and entropy decoding) is parallelized in each row of the CTUs 111 each including the VPDU 121. Furthermore, in each CTU row, the CTUs 111 are processed one by one starting from the leftmost CTU. In addition, the processing of each CTU row is performed while being delayed by an amount corresponding to one CTU with respect to the CTU row immediately above the CTU row.

[0145] exist Figure 6 In FIG. 1 , the numbers in each VPDU 121 indicate the order of processing. In other words, each CTU 111 is processed (entropy encoding and entropy decoding) for each VPDU 121.

[0146] In this regard, entropy coding and entropy decoding are lossless coding and lossless decoding using the probability of occurrence derived from the previous process as context. For example, entropy coding and entropy decoding may be arithmetic coding and arithmetic decoding such as CABAC. The description herein assumes that CABAC is applied. In other words, each VPDU is processed by using the probability of occurrence derived from the processing of the previous VPDU as context.

[0147] However, the first VPDU (upper left VPDU) in the leftmost CTU in the uppermost CTU row of the image is processed by using the context having the initial value. In addition, the first VPDU (upper left VPDU) in each of the leftmost CTUs in the next upper CTU row and the subsequent CTU rows is processed by using the occurrence probability derived during the processing of the last VPDU (lower left VPDU) in the leftmost CTU in the CTU row immediately above the CTU row to be processed (also referred to as the learned occurrence probability) as the context, as shown in FIG. Figure 6 Indicated by the solid box and arrow.

[0148] Therefore, if Figure 7 As shown, the processing thread of each CTU row is delayed by an amount corresponding to one CTU relative to the processing thread of the CTU row immediately above the CTU row. Although there is a time interval, multiple threads are executed in parallel.

[0149] In addition, restrictions on intra-frame prediction and inter-frame prediction allow the delay between CTU rows to be adjusted to one CTU. This allows the reduction in the parallelism of encoding and decoding due to the increase in CTU size to be suppressed. Therefore, the increase in delay in the parallel pipeline can be suppressed, and thus the increase in processing time can be suppressed. For example, in the case of an image with an image size of 4K (3840×2160) to be processed, the maximum parallelism is 17 and the average parallelism is 13.8.

[0150] Note that, as described above, in the case where encoding and decoding in each row of a CTU each including a plurality of VPDUs are parallelized, restrictions on prediction blocks and transform blocks are restricted similarly to the case of pipeline processing of VPDUs described in Non-Patent Document 2 and Non-Patent Document 3.

[0151] <4-2: Method 1-2>

[0152] In method 1-2, some modes are restricted in intra prediction. When a block to be processed is in intra prediction mode, the mode is predicted by referring to the intra prediction mode of a neighboring block, and a predicted image is created by referring to pixels of the neighboring block.

[0153] <4-2-1: Method 1-2-1>

[0154] In method 1-2-1, if Figure 8 As shown by the dotted arrow in FIG. 1 , in the intra prediction of the upper right VPDU in the CTU, the reference to the VPDU (CTU) at the upper right of the upper right VPDU is prohibited (useless). Specifically, the prohibition of reference is Figure 8 The dotted arrows in FIG. 4 indicate the intra prediction mode and the predicted pixels.

[0155] For example, when encoding and decoding are not parallelized (during WPP OFF), in the intra prediction of VPDU[b] corresponding to the upper right VPDU 121 in the current CTU 111, the intra prediction modes of VPDU[a], VPDU[f], VPDU[g], and VPDU[h] and the adjacent pixels (black belt portion in the figure) are referred to, such as Fig. 9 As shown in A in . In this case, the intra prediction block size is equal to or smaller than 64×64 (equal to or smaller than VPDU 121), and therefore, when the intra prediction block size is equal to VPDU 121, the reference range is maximum.

[0156] In contrast, in method 1-2-1, in the case where encoding and decoding are parallelized in each row of CTUs each including a VPDU as in method 1-1 (during WPP processing of the VPDU), Fig. 9 As shown in B, in the intra prediction of VPDU [b] corresponding to the upper right VPDU 121 in CTU 111, the intra prediction mode of reference VPDU [h] and adjacent pixels is prohibited (reference is invalidated).

[0157] In method 1-1, processing is parallelized in each CTU row, and the delay between rows corresponds to one CTU, so a CTU with VPDU[b] and a CTU with VPDU[h] can be processed simultaneously. However, the above reference relationship may require that the processing of VPDU[b] wait until the processing of VPDU[h] is completed (in other words, the waiting time may be extended).

[0158] Similar to the restriction in method 1-2-1, the restriction on reference can prevent the creation of such a reference relationship for intra prediction (VPDU[b] can be intra predicted independently of VPDU[h]), thereby suppressing the increase in waiting time.

[0159] For other VPDUs 121 in the CTU 111, even during the WPP processing of the VPDU, reference can be made similarly to the case during the WPP off period. For example, in the intra prediction of VPDU[a] corresponding to the top left VPDU 121 in the current CTU 111, as shown in FIG. Fig.10 As shown in A, the intra prediction modes of VPDU[e], VPDU[f], VPDU[g], VPDU[i] and VPDU[j] and the adjacent pixels (black bands in the figure) are referenced during WPP OFF and during WPP processing of VPDU. VPDU[g] at the upper right of VPDU[a] is also processed before VPDU[a] during WPP processing of VPDU, and can therefore be referenced.

[0160] In addition, for example, Fig.10 As shown in B, in the intra prediction of VPDU[c] corresponding to the lower left VPDU 121 in the current CTU 111, the intra prediction modes of VPDU[a], VPDU[b], VPDU[i], VPDU[j] and adjacent pixels (black bands in the figure) will be referenced during WPP off and during WPP processing of the VPDU. VPDU[b] at the upper right of VPDU[c] is also processed before VPDU[c] during WPP processing of the VPDU, and can therefore be referenced.

[0161] In addition, for example, Fig.10 As shown in FIG. 3 , in the intra prediction of VPDU[d] corresponding to the lower right VPDU 121 in the current CTU 111, the intra prediction modes of VPDU[a], VPDU[b], and VPDU[c] and the adjacent pixels (black band portion in the figure) are referenced during the WPP off period and during the WPP period of processing the VPDU. The VPDU at the upper right of VPDU[d] is processed after VPDU[d] during the WPP off period and during the WPP period of processing the VPDU, and is therefore prevented from being referenced.

[0162] <4-3: Method 1-3>

[0163] In methods 1-3, inter prediction is limited. When a block to be processed is in inter prediction mode, the mode is predicted by referring to the inter prediction mode of a neighboring block, and the motion vector of the block to be processed is predicted by referring to the motion vector of the neighboring block.

[0164] <4-3-1: Method 1-3-1>

[0165] In method 1-3-1, if Figure 8 As shown by the dotted arrow in FIG. 1 , in the inter-frame prediction of the upper right VPDU in the CTU as the current prediction block, the VPDU at the upper right of the reference upper right VPDU is prohibited (invalid). Specifically, the reference is prohibited by Figure 8 The inter prediction mode and motion vector indicated by the dotted arrow in . In other words, the reference relationship similar to the reference relationship of the intra prediction is invalidated (restrictions are imposed similar to the case of method 1-2-1).

[0166] This can prevent the creation of such a reference relationship for inter-frame prediction, thereby suppressing an increase in waiting time.

[0167] For inter prediction where each other VPDU 121 in CTU 111 is used as the current prediction block, similar to the case of intra prediction ( Fig.10 ), a reference similar to the reference during WPP off can also be made during WPP processing of VPDU. Note that in the case where the inter-frame prediction block size is equal to or less than 64×64 (equal to or less than VPDU 121), it is sufficient to limit the reference in method 1-3-1.

[0168] However, in the inter prediction, the block size of the prediction block may be set to be larger than 64×64, and the block size may be set to, for example, 128×64, 64×128, 128×128, or the like.

[0169] <4-3-2: Method 1-3-2>

[0170] In method 1-3-2, in the case where the current prediction block has a block size of 128×N, reference to a block at the upper right of the current prediction block is invalidated.

[0171] For example, in the inter-frame prediction of a prediction block of a block size of 128×64, it is prohibited to refer to the inter-frame prediction mode and motion vector of the block at the upper right of the prediction block (the reference is invalidated). Fig.11 The prediction block [ab] including the upper left VPDU [a] and the upper right VPDU [b] in the current CTU 111 shown in A is the current prediction block 131. In this case, during the WPP off period, the inter-frame prediction mode and motion vector of VPDU [e], VPDU [f], VPDU [g], VPDU [h], VPDU [i] and VPDU [j] are referred to, as shown in FIG. Fig.11 As shown in A.

[0172] In contrast, in method 1-3-2, in the inter prediction of the prediction block [ab], it is prohibited to refer to the inter prediction mode and motion vector of the VPDU [h] at the upper right of the prediction block [ab] (the reference is invalidated) during the processing of the WPP of the VPDU, as shown in FIG. Fig.11 As shown in B.

[0173] In method 1-1, processing is parallelized in each CTU row, and the delay between rows corresponds to one CTU, so VPDU[a] (i.e., prediction block [ab]) and VPDU[h] can be processed simultaneously. However, the above reference relationship may require waiting for the processing of prediction block [ab] until the processing of VPDU[h] is completed (in other words, the waiting time is extended).

[0174] Similar to the restriction in method 1-3-2, the restriction on reference can prevent the creation of such a reference relationship for inter-frame prediction (inter-frame prediction can be performed on prediction block [ab] independently of VPDU[h]), thereby suppressing the increase in waiting time.

[0175] In addition, in the inter-frame prediction of the prediction block with a block size of 128×128, the inter-frame prediction mode and motion vector of the block at the upper right of the prediction block are similarly prohibited from being referred to (the reference is invalidated). Fig.11 The prediction block [abcd] including all VPDUs (VPDU [a], VPDU [b], VPDU [c] and VPDU [d]) in the current CTU 111 shown in FIG. C is the current prediction block 131. In this case, during the WPP off period, the inter-frame prediction mode and motion vector of VPDU [e], VPDU [f], VPDU [g], VPDU [h], VPDU [i], VPDU [j] are referred to, as shown in FIG. Fig.11 As shown in C.

[0176] In contrast, in method 1-3-2, in the inter-frame prediction of the prediction block [abcd], the inter-frame prediction mode and motion vector of the VPDU [h] at the upper right of the reference prediction block [abcd] are prohibited during the processing of the WPP of the VPDU (the reference is invalidated), as shown in FIG. Fig.11 As shown in D.

[0177] In method 1-1, processing is parallelized in each CTU row, and the delay between rows corresponds to one CTU, so VPDU[a] (i.e., prediction block [abcd] and VPDU[h]) can be processed simultaneously. However, the above reference relationship may require waiting for the processing of prediction block [abcd] until the processing of VPDU[h] is completed (in other words, the waiting time may be extended).

[0178] Restrictions on reference similar to those in method 1-3-2 can prevent the creation of such a reference relationship for inter-frame prediction (inter-frame prediction can be performed on the prediction block [abcd] independently of VPDU[h]), thereby suppressing an increase in waiting time.

[0179] Note that in the case where the prediction block [cd] including the lower left VPDU [c] and the lower right VPDU [d] of the current CTU 111 is used as the current prediction block 131, similarly to the reference during WPP off, it is also possible to Fig.12 Reference is made during WPP of the processing VPDU shown (reference is made to the inter-frame prediction mode and motion vectors of VPDU[a], VPDU[b], VPDU[i] and VPDU[j]) (the upper right block is processed later and therefore is not referenced even during WPP OFF).

[0180] <5. First Embodiment>

[0181] <5-1: Image Coding Device>

[0182] The present technology described above can be applied to any apparatus, device, system, etc. For example, the present technology described above can be applied to an image encoding apparatus that encodes image data.

[0183] Fig.13 : is a block diagram showing a configuration example of an image encoding device corresponding to one aspect of an image processing device to which the present technology is applied. Fig.13 The image encoding device 500 shown is a device for encoding image data of a moving image. For example, the image encoding device 500 implements the technology described in non-patent documents 1 to 6, and encodes the image data of the moving image by using a method that conforms to the standard described in any of these documents.

[0184] Notice, Fig.13 The main elements such as the processing unit and the data flow are shown, but not all elements of the image encoding device 500 are shown. In other words, in the image encoding device 500, there may be Fig.13 The processing section shown in the box, or there may be Fig.13 The processing flow or data flow shown by arrows etc.

[0185] like Fig.13 As shown, the image encoding device 500 includes a control unit 501, a sorting buffer 511, a calculation unit 512, an orthogonal transform unit 513, a quantization unit 514, an encoding unit 515, a storage buffer 516, an inverse quantization unit 517, an inverse orthogonal transform unit 518, a calculation unit 519, an in-loop filter unit 520, a frame memory 521, a prediction unit 522 and a rate control unit 523.

[0186] <Control Unit>

[0187] Based on the block size in the external processing unit or in the processing unit specified in advance, the control section 501 divides the moving image data held by the sorting buffer 511 into blocks in the processing unit (CU, PU, ​​transform block, etc.). In addition, for example, the control section 501 determines encoding parameters (header information Hinfo, prediction mode information Pinfo, transform information Tinfo, filter information Finfo, etc.) to be provided to each block based on RDO (rate distortion optimization).

[0188] The details of the encoding parameters will be described below. The control unit 501 determines the encoding parameters as described above, and then feeds the encoding parameters to each block. Specifically, it is performed as described below.

[0189] The header information Hinfo is fed to each block. The prediction mode information Pinfo is fed to the encoding section 515 and the prediction section 522. The transform information Tinfo is fed to the encoding section 515, the orthogonal transform section 513, the quantization section 514, the inverse quantization section 517, and the inverse orthogonal transform section 518. The filter information Finfo is fed to the in-loop filter section 520.

[0190] <Sort Buffer>

[0191] The image encoding device 500 receives the respective fields of the motion image data (input image) in the order of reproduction (display) of the respective fields. The sorting buffer 511 acquires the respective input images in the order of reproduction (display) of the images, and saves (stores) the acquired input images. The sorting buffer 511 sorts the input images into the order of encoding (decoding) under the control of the control unit 501, and divides the input images into blocks in the processing unit. The sorting buffer 511 feeds the respective input images thus processed into the calculation unit 512. In addition, the sorting buffer 511 also feeds the respective input images (original images) to the prediction unit 522 and the in-loop filter unit 520.

[0192] <Calculation Department>

[0193] The calculation unit 512 receives as input an image I corresponding to a block in the processing unit and a predicted image P fed by the prediction unit 522, and subtracts the predicted image P from the image I as shown in equation (3) to derive a prediction residual resi. The calculation unit 512 feeds the prediction residual resi to the orthogonal transformation unit 513.

[0194] [Mathematical formula 3]

[0195] resi=IP...(3)

[0196] <Orthogonal Transformation Unit>

[0197] The orthogonal transform section 513 receives the prediction residual resi fed from the calculation section 512 and the transform information Tinfo fed from the control section 501 as input, and performs an orthogonal transform on the prediction residual resi based on the transform information Tinfo to derive a transform coefficient coef. The orthogonal transform section 513 feeds the obtained transform coefficient coef to the quantization section 514.

[0198] <Quantitative Department>

[0199] The quantization section 514 receives as input the transform coefficient coef fed from the orthogonal transform section 513 and the transform information Tinfo fed from the control section 501, and scales (quantizes) the transform coefficient coef based on the transform information Tinfo. Note that the rate of quantization is controlled by the rate control section 523. The quantization section 514 feeds the quantized transform coefficient obtained by quantization as described above, that is, the quantized transform coefficient level qcoef, to the encoding section 515 and the inverse quantization section 517.

[0200] <Coding Department>

[0201] The encoding section 515 receives as input the quantized transform coefficient level qcoef fed from the quantization section 514, various types of encoding parameters (header information Hinfo, prediction mode information Pinfo, transform information Tinfo, filter information Finfo, etc.) fed from the control section 501, information related to the filter (e.g., filter coefficients fed from the in-loop filter section 520), and information related to the optimal prediction mode fed from the prediction section 522. The encoding section 515 encodes the quantized transform coefficient level qcoef (e.g., performs arithmetic encoding such as CABAC) to generate a bit sequence.

[0202] In addition, the encoding unit 515 derives residual information Rinfo from the quantized transform coefficient level qcoef, and encodes the residual information Rinfo to generate a bit sequence.

[0203] Furthermore, the encoding section 515 includes, in the filter information Finfo, information related to the filter fed from the in-loop filter section 520, and includes, in the prediction mode information Pinfo, information related to the optimal prediction mode fed from the prediction section 522. Then, the encoding section 515 encodes the above-described various types of encoding parameters (header information Hinfo, prediction mode information Pinfo, transform information Tinfo, filter information Finfo, etc.) to generate a bit sequence.

[0204] In addition, the encoding section 515 multiplexes the bit sequences of the various types of information generated as described above to generate encoded data. The encoding section 515 feeds the encoded data to the storage buffer 516.

[0205] <Storage Buffer>

[0206] The storage buffer 516 temporarily stores the encoded data obtained from the encoding section 515. The storage buffer 516 outputs the encoded data stored in the storage buffer 516 to the outside of the image encoding device 500 in the form of a bit stream or the like at a predetermined timing. For example, the encoded data is sent to the decoding side via an optional recording medium, an optional transmission medium, an optional image processing device, etc. In other words, the storage buffer 516 also functions as a transmission section that transmits the encoded data (bit stream).

[0207] <Inverse Quantization Section>

[0208] The inverse quantization section 517 performs processing related to inverse quantization. For example, the inverse quantization section 517 receives as input the quantized transform coefficient level qcoef fed from the quantization section 514 and the transform information Tinfo fed from the control section 501 to scale (inverse quantize) the quantized transform coefficient level qcoef based on the transform information Tinfo. Note that inverse quantization is the inverse process of quantization performed in the quantization section 514. The inverse quantization section 517 feeds the transform coefficient coefI obtained by inverse quantization as described above to the inverse orthogonal transform section 518.

[0209] <Inverse Orthogonal Transformation Unit>

[0210] The inverse orthogonal transform section 518 performs processing related to the inverse orthogonal transform. For example, the inverse orthogonal transform section 518 receives as input the transform coefficient coefI fed from the inverse quantization section 517 and the transform information Tinfo fed from the control section 501, and performs an inverse orthogonal transform on the transform coefficient coefI based on the transform information Tinfo to derive the prediction residual resiI. Note that the inverse orthogonal transform is the inverse process of the orthogonal transform performed in the orthogonal transform section 513. The inverse orthogonal transform section 518 feeds the prediction residual resiI obtained by the inverse orthogonal transform to the calculation section 519 as described above. Note that the inverse orthogonal transform section 518 is similar to the inverse orthogonal transform section on the decoding side (described below), and therefore the description of the decoding side (given below) can be applied to the inverse orthogonal transform section 518.

[0211] <Calculation Department>

[0212] The calculation unit 519 receives as input the prediction residual resiI fed from the inverse orthogonal transform unit 518 and the prediction image P fed from the prediction unit 522. The calculation unit 519 adds the prediction residual resiI and the prediction image P corresponding to the prediction residual resiI to derive a local decoded image Rlocal. The calculation unit 519 feeds the derived local decoded image Rlocal to the in-loop filter unit 520 and the frame memory 521.

[0213] <In-loop filter section>

[0214] The in-loop filter section 520 performs processing related to the in-loop filter processing. For example, the in-loop filter section 520 receives as input the local decoded image Rlocal fed from the calculation section 519, the filter information Finfo fed from the control section 501, and the input image (original image) fed from the sorting buffer 511. Note that any information is input to the in-loop filter section 520, and information other than the above may be input to the in-loop filter section 520. For example, the in-loop filter section 520 may receive as input information such as a prediction mode, motion information, a target value for the amount of code, a quantization parameter QP, a picture type, or a block (CU, CTU, etc.) as needed.

[0215] The in-loop filter section 520 appropriately performs a filter process on the local decoded image Rlocal based on the filter information Finfo. The in-loop filter section 520 uses the input image (original image) and any other input information for the filter process as necessary.

[0216] For example, the in-loop filter unit 520 sequentially applies four in-loop filters, including a bilateral filter, a deblocking filter (DBF), an adaptive offset filter (SAO (sample adaptive offset)), and an adaptive in-loop filter (ALF). Please note that which filter to apply and in what order the filters are applied are optional, and the filters and order can be selected as needed.

[0217] It goes without saying that the filtering process performed by the in-loop filter section 520 is optional and is not limited to the above example. For example, the in-loop filter section 520 may apply a Wiener filter or the like.

[0218] The in-loop filter section 520 feeds the filtered local decoded image Rlocal to the frame memory 521. Note that, for example, in the case where information related to the filter such as filter coefficients is sent to the decoding side, the in-loop filter section 520 supplies the information related to the filter to the encoding section 515.

[0219] <Frame Memory>

[0220] The frame memory 521 performs processing related to storing image-related data. For example, the frame memory 521 receives as input the local decoded image Rlocal fed from the calculation section 519 and the filtered local decoded image Rlocal fed from the in-loop filter section 520, and saves (stores) the local decoded image Rlocal. In addition, the frame memory 521 reconstructs the decoded image R of each picture unit using the local decoded image Rlocal, and saves the decoded image R (storing the decoded image R in a buffer in the frame memory 521). In response to a request from the prediction section 522, the frame memory 521 feeds the decoded image R (or a portion of the decoded image R) to the prediction section 522.

[0221] <Forecasting Department>

[0222] The prediction section 522 performs processing related to the generation of a predicted image. For example, the prediction section 522 receives as input the prediction mode information Pinfo fed from the control section 501, the input image (original image) fed from the sorting buffer 511, and the decoded image R (or a portion of the decoded image R) read out from the frame memory 521. The prediction section 522 uses the prediction mode information Pinfo and the input image (original image) to perform prediction processing such as inter-frame prediction and intra-frame prediction, and performs prediction by referring to the decoded image R as a reference image. Based on the prediction result, the prediction section 522 performs motion compensation processing to generate a predicted image P. The prediction section 522 feeds the generated predicted image P to the calculation section 512 and the calculation section 519. In addition, the prediction section 522 feeds information related to the prediction mode selected by the above-mentioned processing (i.e., information related to the optimal prediction mode) to the encoding section 515 as needed.

[0223] <Rate Control Section>

[0224] The rate control section 523 performs processing related to rate control. For example, the rate control section 523 controls the rate of the quantization operation of the quantization section 514 based on the code amount of the encoded data stored in the storage buffer 516 to prevent overflow or underflow.

[0225] <Application of this technology>

[0226] The present technology described in <3. Concept> and <4. Method 1> above is applied to the image encoding device 500 configured as described above. For example, "Method 1-1" can be applied to the image encoding device 500. For example, the control section 501 can set the VPDU in the CTU 111, and the encoding section 515 can perform encoding in parallel in each row of the CTU 111 including the VPDU 121. In addition, the encoding section 515 can perform processing per CTU row to delay the processing per CTU row by an amount corresponding to one CTU relative to the CTU row immediately above the CTU row.

[0227] For example, the encoding unit 515 may encode coefficient data associated with the image in parallel for each row of coding tree units, each of which includes a single or multiple transform blocks and corresponds to the top coding block in the tree structure, and may perform encoding on each row to delay the encoding by an amount corresponding to one coding tree unit relative to the row immediately above the row.

[0228] In addition, for example, in each row of the coding tree unit of coefficient data related to the image, the encoding unit 515 may sequentially encode the coding tree units one by one starting from the leftmost coding tree unit.

[0229] In addition, for example, the encoding unit 515 may encode each coding tree unit for each transform block.

[0230] In addition, for example, the encoding part 515 may entropy encode each transform block by using the occurrence probability derived from the entropy encoding of the previous transform block.

[0231] In addition, for example, the encoding unit 515 may entropy encode the first transform block in the leftmost coding tree unit in the topmost row of the coding tree units in the image by using the initial value of the occurrence probability. In addition, for example, the encoding unit 515 may entropy encode the first transform block in each of the leftmost coding tree units in the next upper row and subsequent rows of the coding tree units in the image by using the occurrence probability derived from entropy encoding the last transform block in the leftmost coding tree unit in the row of the coding tree unit immediately above the row.

[0232] In addition, "method 1-2" can be applied to the image encoding device 500. Specifically, the prediction unit 522 can limit certain modes in the intra-frame prediction mode. For example, "method 1-2-1" can be applied to the image encoding device 500. Specifically, in the intra-frame prediction of the upper right VPDU in the CTU, the control unit 501 can invalidate the reference to the VPDU at the upper right of the upper right VPDU, and the prediction unit 522 can perform intra-frame prediction under the control of the control unit 501. In other words, for the upper right VPDU in the CTU, the prediction unit 522 can perform intra-frame prediction while invalidating the reference to the VPDU at the upper right of the upper right VPDU.

[0233] For example, in the intra prediction of the upper right transform block of the coding tree unit, the prediction part 522 may invalidate the reference to the transform block at the upper right of the upper right transform block.

[0234] In addition, "Method 1-3" can be applied to the image encoding device 500. Specifically, the prediction unit 522 can limit the inter-frame prediction. For example, "Method 1-3-1" can be applied to the image encoding device 500. Specifically, in the inter-frame prediction in which the upper right VPDU in the CTU is used as the current prediction block, the control unit 501 can invalidate the reference to the inter-frame prediction mode and motion vector of the VPDU at the upper right of the upper right VPDU, and the prediction unit 522 can perform the inter-frame prediction under the control of the control unit 501. In other words, in the case of using the upper right VPDU in the CTU as the current prediction block, the prediction unit 522 can perform the inter-frame prediction while invalidating the reference to the inter-frame prediction mode and motion vector of the VPDU at the upper right of the upper right VPDU.

[0235] For example, in the inter prediction of the upper right transform block of the coding tree unit, the prediction part 522 may invalidate the reference to the transform block at the upper right of the upper right transform block.

[0236] In addition, for example, "Method 1-3-2" may be applied to the image encoding device 500. Specifically, in the inter-frame prediction of the prediction block having a block size of 128×N, the control section 501 may prohibit reference to the inter-frame prediction mode and motion vector of the block at the upper right of the prediction block (invalidate reference to the inter-frame prediction mode and motion vector of the block at the upper right of the prediction block), and the prediction section 522 may perform the inter-frame prediction under the control of the control section 501. In other words, the prediction section 522 may perform the inter-frame prediction while invalidating reference to the block at the upper right of the current prediction block having a block size of 128×N.

[0237] For example, in inter prediction of a prediction block serving as a processing unit of inter prediction and having the same horizontal length as that of a coding tree unit corresponding to an uppermost coding block in a tree structure, the prediction section 522 may invalidate reference to a transform block at the upper right of the prediction block.

[0238] Such a configuration enables the image encoding device 500 to suppress a decrease in the degree of parallelism in encoding, thereby being able to suppress an increase in processing time.

[0239] <5-2: Processing Flow>

[0240] <Flow of Image Coding Process>

[0241] Next, the flow of each type of processing performed by the image encoding device 500 as described above will be described. Fig.14 An example of the process of image encoding processing will be described with reference to the flowchart in FIG.

[0242] When the image encoding process is started, in step S101 , the sorting buffer 511 sorts frames in the input moving image data under the control of the control section 501 to change the order of the frames from the display order to the encoding order.

[0243] In step S102 , the control section 501 sets a processing unit (performs block division) for the input image held by the sorting buffer 511 .

[0244] In step S103, the control unit 501 determines (sets) the encoding parameters of the input image stored in the sorting buffer 511. At this time, the control unit 501 makes settings related to the restrictions of the above-mentioned intra-frame prediction and inter-frame prediction as needed. The settings are fed to the prediction unit 522 as prediction mode information Pinfo, for example.

[0245] In step S104, under the control of the control section 501, the prediction section 522 performs prediction processing to generate a predicted image in the optimal prediction mode, etc. The details of the prediction processing will be described below.

[0246] In step S105, the calculation unit 512 calculates the difference between the input image and the predicted image in the optimal mode selected by the prediction process in step S104. In other words, the calculation unit 512 generates a prediction residual resi between the input image and the predicted image. The prediction residual resi thus determined has a smaller amount of data than the original image data. Therefore, compared with the case where the image is directly encoded, this can reduce the amount of data.

[0247] In step S106, the orthogonal transform section 513 performs an orthogonal transform process on the prediction residual resi generated by the process in step S105 to derive a transform coefficient coef.

[0248] In step S107 , the quantization unit 514 quantizes the transform coefficient coef obtained by the process in step S106 using, for example, the quantization parameter calculated by the control unit 501 , thereby deriving a quantized transform coefficient level qcoef.

[0249] In step S108 , the inverse quantization unit 517 inversely quantizes the quantized transform coefficient level qcoef generated by the process in step S107 using characteristics corresponding to the characteristics of the quantization in step S107 , thereby deriving a transform coefficient coefI.

[0250] In step S109, the inverse orthogonal transform section 518 performs inverse orthogonal transform on the transform coefficient coefI obtained by the process in step S108 using a method corresponding to the orthogonal transform process in step S106, thereby deriving a prediction residual resiI. Note that the inverse orthogonal transform process is similar to the inverse orthogonal transform section performed on the decoding side (described below), and therefore the description of the decoding side (given below) can be applied to the inverse orthogonal transform process in step S109.

[0251] In step S110 , the calculation section 519 adds the predicted image obtained by the prediction process in step S104 and the prediction residual resiI derived from the process in step S109 to generate a locally decoded decoded image.

[0252] In step S111 , the in-loop filter unit 520 performs an in-loop filter process on the decoded image locally decoded and derived from the process in step S110 .

[0253] In step S112 , the frame memory 521 stores the decoded image locally decoded and derived from the process in step S110 and the decoded image locally decoded and filtered in step S111 .

[0254] In step S113, the encoding section 515 performs an encoding process to encode the quantized transform coefficient level qcoef obtained by the process in step S107. For example, the encoding section 515 encodes the quantized transform coefficient level qcoef corresponding to the information related to the image by arithmetic coding or the like to generate encoded data. In addition, at this time, the encoding section 515 encodes various types of encoding parameters (header information Hinfo, prediction mode information Pinfo, and transform information Tinfo). In addition, the encoding section 515 derives residual information Rinfo from the quantized transform coefficient level qcoef, and encodes the residual information Rinfo. The details of the encoding process will be described.

[0255] In step S114, the storage buffer 516 stores the encoded data thus obtained, and outputs the encoded data as a bit stream, for example, to the outside of the image encoding device 500. The bit stream is transmitted to the decoding side via a transmission path or a recording medium, for example. In addition, the rate control unit 523 performs rate control as needed.

[0256] When the processing in step S114 ends, the image encoding processing ends.

[0257] In the encoding process performed in step S113 of the image encoding process, the encoding section 515 encodes coefficient data related to the image in parallel for each row of the coding tree units each including a single or multiple transform blocks and corresponding to the highest coding block in the tree structure, and encodes each row by delaying the encoding of the row immediately before the row by an amount corresponding to one coding tree unit. This enables the image encoding device 500 to suppress a decrease in the degree of parallelism of encoding, thereby suppressing an increase in processing time.

[0258] <Flow of prediction processing>

[0259] Next, we will refer to Fig.15 The flowchart in the description is in Fig.14 An example of the flow of prediction processing performed in step S104 in .

[0260] When the prediction process is started, in step S131, for intra prediction of the top right VPDU in the CTU, the control section 501 invalidates the reference to the VPDU at the top right of the top right VPDU. In step S132, the prediction section 522 performs intra prediction according to the setting in step S131. In other words, the prediction section 522 performs intra prediction of each VPDU, and in the case where the top right VPDU in the CTU is to be processed, the prediction section 522 performs intra prediction while invalidating the reference to the VPDU at the top right of the top right VPDU.

[0261] In step S133, for the inter-frame prediction of the prediction block including the VPDU at the upper right of the CTU, the control section 501 invalidates the reference to the VPDU at the upper right of the prediction block. In addition, in step S134, for the inter-frame prediction of the prediction block with a block size of 128×N, the control section 501 invalidates the reference to the block at the upper right of the prediction block. In step S135, the prediction section 522 performs inter-frame prediction according to the settings in steps S133 and S134. In other words, the prediction section 522 performs inter-frame prediction for each prediction block, and in the case of processing a prediction block including the upper right VPDU in the CTU, the prediction section 522 performs inter-frame prediction while invalidating the reference to the VPDU at the upper right of the prediction block, and in the case of processing a prediction block with a block size of 128×N, the prediction section 522 performs inter-frame prediction while invalidating the reference to the block at the upper right of the prediction block.

[0262] In step S136, the prediction section 522 selects the optimal prediction mode based on the processing result in step S132 and the processing result in step S135. In other words, the prediction section 522 performs intra prediction in the optimal intra prediction mode or the like to generate a predicted image, performs inter prediction in the optimal inter prediction mode or the like to generate a predicted image, and selects the optimal prediction mode from the above modes based on a cost function or the like.

[0263] When the processing in step S136 ends, the prediction processing ends, and the processing returns to Fig.14 .

[0264] <Encoding Process Flow>

[0265] Next, we will refer to Fig.16 The flowchart in the description is in Fig.14 An example of the flow of the encoding process performed in step S113 in .

[0266] When the encoding process is started, in step S151 , the encoding section 515 sets a variable T indicating the position of a CTU row to be processed (row number starting from the top row in the image in order) to an initial value (=1).

[0267] In step S152, the encoding section 515 performs CTU processing on CTU row 1 corresponding to the uppermost row in the image. This processing will be described in detail below. When the processing performed on CTU row 1 ends, the processing proceeds to step S153.

[0268] In step S153, the encoding section 515 determines whether there is any unprocessed CTU line. In the case where the processing up to the lowermost CTU line has not been completed and the encoding section 515 determines that there is an unprocessed CTU line, the process proceeds to step S154.

[0269] In step S154, the encoding section 515 adds 1 to the value of the variable T (T=T+1). When the value of the variable T increases, the process proceeds to step S155.

[0270] In step S155, the encoding unit 515 performs CTU processing on the CTU line T corresponding to the Tth line from the top line in the image. This processing will be described in detail below. When the processing performed on the CTU line T ends, the processing returns to step S153.

[0271] In other words, the processing in steps S153 to S155 is repeated until all CTU rows are processed. Then, in the case where the encoding unit 515 determines in step S153 that there is no unprocessed CTU row (all CTU rows have been processed), the encoding process ends and the process returns to Fig.14 .

[0272] <Flow of CTU Processing for CTU Row 1>

[0273] Next, we will refer to Fig.17 The flowchart in the description is in Fig.16 An example of the process of CTU processing on CTU row 1 performed in step S152.

[0274] When the CTU process is started, in step S171, the encoding section 515 sets the variable i indicating the position of the current CTU (the number of CTUs from the leftmost CTU in the image) to an initial value (=1). In addition, the encoding section 515 sets the value N to the number of CTUs in the horizontal direction. In other words, the value N indicates the number of CTUs in CTU row 1.

[0275] In step S172 , the encoding unit 515 initializes the processing thread context (occurrence probability).

[0276] In step S173 , the encoding section 515 determines whether the variable i is equal to or less than N (i<=N). In the case where it is determined that the variable i is equal to or less than N and it is determined that CTU row 1 includes an unprocessed CTU, the process proceeds to step S174 .

[0277] In step S174, the encoding section 515 encodes the i-th CTU from the leftmost side. The encoding section 515 encodes the i-th CTU from the leftmost side for each VPDU included in the i-th CTU. When the i-th CTU from the leftmost side is encoded, the process proceeds to step S175.

[0278] In step S175 , the encoding section 515 determines whether the value of the variable i is 1 (i==1). In a case where the value of the variable i is determined to be 1 (in other words, the leftmost CTU has been encoded in the process of step S174 ), the process proceeds to step S176 .

[0279] In step S176, the encoding section 515 saves the CABAC context (occurrence probability) of the last VPDU in the leftmost CTU in the picture, which has been generated in step S174. When the context is saved, the process proceeds to step S177.

[0280] In addition, in step S175 , in the case where it is determined that the value of the variable i is not 1 (the leftmost CTU in the image is not a processing target), the process of step S176 is skipped, and the process proceeds to step S177 .

[0281] In step S177 , the encoding section 515 determines whether the value of the variable i is equal to or greater than 1 (i>=1). In the case where it is determined that the value of the variable i is equal to or greater than 1, the process proceeds to step S178 .

[0282] In step S178 , the encoding section 515 notifies the CTU row 2 processing thread corresponding to the thread processing CTU row 2 of the completion of the CTU processing. When the completion of the CTU processing is notified, the process proceeds to step S179 .

[0283] In addition, in the event that it is determined in step S177 that the value of the variable i is less than 1 (the leftmost CTU has not yet completed encoding), the processing of step S178 is skipped and the processing proceeds to step S179.

[0284] In step S179, the encoding unit 515 increases the value of the variable i by 1 (i=i+1). When the value of the variable i increases, the process returns to step S173.

[0285] In other words, the processing in steps S173 to S179 is repeated until all CTUs in CTU row 1 are processed. Then, in step S173, if the encoding section 515 determines that there is no unprocessed CTU (all CTUs in CTU row 1 have been processed), the CTU processing ends and the processing returns to Fig.16 .

[0286] <Flow of CTU Processing for CTU Row T>

[0287] Next, we will refer to Fig.18 The flowchart in the description is in Fig.16 An example of the process of CTU processing of CTU row T performed in step S155, etc.

[0288] When CTU processing is started, in step S191, the encoding unit 515 waits until the encoding unit 515 receives a notification of CTU processing completion from a CTU row T-1 processing thread corresponding to a thread processing CTU row T-1 immediately above CTU row T. When the notification of CTU processing completion is obtained from the CTU row T-1 processing thread, the process proceeds to step S192.

[0289] In step S192 , the encoding unit 515 sets the variable i to an initial value (=1). In addition, the encoding unit 515 sets the value N to the number of CTUs in the horizontal direction.

[0290] In step S193 , the encoding unit 515 takes over the CABAC context (occurrence probability) saved when the CTU process is performed on the CTU row T−1 immediately above the CTU row T, and initializes CABAC.

[0291] In step S194 , the encoding section 515 determines whether the variable i is equal to or less than N (i<=N). In the case where it is determined that the variable i is equal to or less than N and it is determined that the CTU row T includes an unprocessed CTU, the process proceeds to step S195 .

[0292] In step S195, in order to control the processing timing, the encoding section 515 waits until the encoding section 515 receives a notification of CTU processing completion from the CTU row T-1 processing thread. When the notification of CTU processing completion is obtained from the CTU row T-1 processing thread, the process proceeds to step S196.

[0293] In step S196, the encoding section 515 encodes the i-th CTU from the leftmost side for each VPDU included in the i-th CTU. When the i-th CTU from the leftmost side is encoded, the process proceeds to step S197.

[0294] In step S197 , the encoding section 515 determines whether the value of the variable i is 1 (i==1). In a case where the value of the variable i is determined to be 1 (ie, it is determined that the leftmost CTU is processed), the process proceeds to step S198 .

[0295] In step S198, the encoding section 515 saves the CABAC context (occurrence probability) of the last VPDU in the leftmost CTU of the picture, which CABAC context has been generated in step S196. When the context is saved, the process proceeds to step S199.

[0296] In addition, in step S197 , in the case where it is determined that the value of the variable i is not 1 (the leftmost CTU in the image is not a processing target), the processing of step S198 is skipped and the processing proceeds to step S199 .

[0297] In step S199, the encoding section 515 determines whether the value of the variable i is equal to or greater than 1 (i>=1). In the case where it is determined that the value of the variable i is equal to or greater than 1, the process proceeds to step S200.

[0298] In step S200 , the encoding section 515 notifies the CTU row T+1 processing thread corresponding to the thread processing CTU row T+1 immediately below CTU row T of the completion of the CTU processing. When the completion of the CTU processing is notified, the process proceeds to step S201 .

[0299] In addition, in the event that it is determined in step S199 that the value of the variable i is less than 1 (ie, the leftmost CTU has not yet completed encoding), the process of step S200 is skipped and the process proceeds to step S201 .

[0300] In step S201, the encoding unit 515 increases the value of the variable i by 1 (i=i+1). When the value of the variable i increases, the process returns to step S194.

[0301] In other words, the processing of steps S194 to S201 is repeated until all CTUs in the CTU row T are processed. Then, in step S194, if the encoding unit 515 determines that there is no unprocessed CTU (all CTUs in the CTU row T have been processed), the CTU processing ends and the processing returns to Fig.16 .

[0302] By executing each processing step, the image encoding device 500 can suppress a decrease in the degree of parallelism of encoding.

[0303] <5-3: Image Decoding Device>

[0304] In addition, the present technology described above can be applied to, for example, an image decoding device that decodes encoded data obtained by encoding image data.

[0305] Fig.19 : is a block diagram showing an example of a configuration of an image decoding device corresponding to one aspect of an image processing device to which the present technology is applied. As in AVC or HEVC, Fig.25 The image decoding device 600 shown decodes the coded data obtained by encoding the prediction residual between the image and the predicted image of the image. For example, the image decoding device 600 implements the technology described in non-patent document 1 to non-patent document 6, and decodes the coded data obtained by encoding the image data of the moving image by using a method that conforms to the standard described in any of these documents. For example, the image decoding device 600 decodes the coded data (bit stream) generated by the above-mentioned image coding device 500.

[0306] Notice, Fig.19 The main elements such as the processing unit and the data flow are shown, but not all elements of the image decoding device 600 are shown. In other words, in the image decoding device 600, there may be Fig.19 The processing section shown in the box, or there may be Fig.19 The processing flow or data shown by arrows etc.

[0307] exist Fig.19 , the image decoding device 600 includes a storage buffer 611, a decoding section 612, an inverse quantization section 613, an inverse orthogonal transformation section 614, a calculation section 615, an in-loop filter section 616, a sorting buffer 617, a frame memory 618, and a prediction section 619. Note that the prediction section 619 includes an intra-frame prediction section and an inter-frame prediction section, which are not shown. The image decoding device 600 is a device configured to generate motion image data by decoding encoded data (bit stream).

[0308] <Storage Buffer>

[0309] The storage buffer 611 acquires and holds (stores) a bit stream input to the image decoding device 600. For example, the storage buffer 611 feeds the stored bit stream to the decoding section 612 at a predetermined timing or when a predetermined condition is satisfied.

[0310] <Decoding section>

[0311] The decoding section 612 performs processing related to decoding of an image. For example, the decoding section 612 receives a bit stream fed from the storage buffer 611 as input, and variable-length-decodes syntax values ​​of various syntax elements from a bit sequence according to the definition in a syntax table to derive parameters.

[0312] The parameters derived from the syntax elements and the syntax values ​​of the syntax elements include, for example, header information Hinfo, prediction mode information Pinfo, transform information Tinfo, residual information Rinfo, filter information Finfo, etc. In other words, the decoding unit 612 parses (analyzes and obtains) this information from the bitstream. This information will be described below.

[0313] <Header information Hinfo>

[0314] The header information Hinfo includes header information such as VPS (video parameter set) / SPS (sequence parameter set) / PPS (picture parameter set) / SH (slice header). The header information Hinfo includes information specifying, for example, the following: image size (horizontal width PicWidth and vertical width PicHeight), bit depth (luminance bitDepthY and chrominance bitDepthC), chrominance array type ChromaArrayType, maximum value MaxCUSize of CU size / minimum value MinCUSize of CU size, maximum depth MaxQTDepth / minimum depth MinQTDepth of quadtree partitioning, maximum depth MaxBTDepth / minimum depth MinBTDepth of binary tree partitioning, maximum value MaxTSSize of transform skip block (also called maximum transform skip block size), on / off flag of each encoding tool (also called enable flag), etc.

[0315] For example, the on / off flag of the coding tool included in the header information Hinfo includes an on / off flag related to the transformation and quantization processing described below. Note that the on / off flag of the coding tool can also be interpreted as a flag indicating whether there is a syntax related to the coding tool in the coded data. In addition, an on / off flag with a value of 1 (true) indicates that the coding tool is available, while an on / off flag with a value of 0 (false) indicates that the coding tool is invalid. Note that the flag value can be interpreted inversely.

[0316] The inter-component prediction enable flag (ccp_enabled_flag) is flag information indicating whether inter-component prediction (CCP, also referred to as CC prediction) is available. For example, the flag information is "1" (true) indicating that CCP is valid, while the flag information is "0" (false) indicating that CCP is invalid.

[0317] Note that CCP is also known as Inter-Component Linear Prediction (CCLM or CCLMP).

[0318] <Prediction mode information Pinfo>

[0319] The prediction mode information Pinfo includes information such as processing target PB (prediction block) size information PBSize (prediction block size), intra prediction mode information IPinfo, and motion prediction information MVinfo.

[0320] The intra prediction mode information IPinfo includes, for example, prev_intra_luma_pred_flag, mpm_idx, and rem_intra_pred_mode in 7.3.8.5 Coding Unit Syntax of JCTVC-W1005, and the luma intra prediction mode IntraPredModeY derived from the syntax.

[0321] In addition, the intra-frame prediction mode information IPinfo includes, for example, the inter-component prediction flag (ccp_flag(cclmp_flag)), the multi-class linear prediction mode flag (mclm_flag), the chroma sample location type identifier (chroma_sample_loc_type_idx) and the chroma MPM identifier (chroma_mpm_idx) and the luma intra-frame prediction mode (IntraPredModeC) derived from these syntaxes.

[0322] The inter-component prediction flag (ccp_flag(cclmp_flag)) is flag information indicating whether inter-component linear prediction is applied. For example, ccp_flag==1 indicates that inter-component linear prediction is to be applied, and ccp_flag==0 indicates that inter-component linear prediction is not to be applied.

[0323] The multi-class linear prediction mode flag (mclm_flag) is information related to the mode of linear prediction (linear prediction mode information). More specifically, the multi-class linear prediction mode flag (mclm_flag) is flag information indicating whether the multi-class linear prediction mode is applied. For example, "0" indicates a one-class mode (single-class mode) (e.g., CCLMP), and "1" indicates a two-class mode (multi-class mode) (e.g., MCLMP).

[0324] The chroma sample position type identifier (chroma_sample_loc_type_idx) is an identifier that identifies the type of pixel position of the chroma component (also referred to as the chroma sample position type). For example, when the chroma array type (ChromaArrayType) corresponding to the information related to the color format indicates the 420 format, the chroma sample position type identifier is assigned as shown in the following equation (4).

[0325] [Formula 4]

[0326] chroma_sample_loc_type_idx==0: Type 2

[0327] chroma_sample_loc_type_idx == 1: Type 3

[0328] chroma_sample_loc_type_idx == 2: Type 0

[0329] chroma_sample_loc_type_idx == 3: Type 1

[0330] …(4)

[0331] Note that the color difference sample position type identifier (chroma_sample_loc_type_idx) (stored in the information related to the pixel position of the color difference component and) is transmitted as the information related to the pixel position of the color difference component (chroma_sample_loc_info()).

[0332] The chroma MPM identifier (chroma_mpm_idx) is an identifier indicating which prediction mode candidate in the chroma intra prediction candidate list (intraPredModeCandListC) is designated as the chroma intra prediction mode.

[0333] The motion prediction information MVinfo includes, for example, information such as merge_idx, merge_flag, inter_pred_idc, ref_idx_LX, mvp_1X_flag, X={0,1}, and mvd (for example, see 7.3.8.6 Prediction Unit Syntax of JCTVC-W1005).

[0334] It goes without saying that the information included in the prediction mode information Pinfo is optional and may include information other than the above.

[0335] <Transformation information Tinfo>

[0336] The transformation information Tinfo includes, for example, information described below. It goes without saying that the information included in the transformation information Tinfo is optional, and may include information other than the following.

[0337] The horizontal width size TBWSize and the vertical width TBHSize of the transform block to be processed (or the logarithmic values ​​of TBWSize and TBHSize with base 2 as log2TBWSize and log2TBHSize). Transform skip flag (ts_flag): A flag indicating whether to skip (inverse) primary transform and (inverse) secondary transform.

[0338] Scan Identifier (scanIdx)

[0339] Quantization parameter (qp)

[0340] Quantization matrix (scaling_matrix (e.g., 7.3.4 Scaling List Data Syntax of JCTVC-W1005))

[0341] <Residual information Rinfo>

[0342] The residual information Rinfo (for example, see 7.3.8.11 Residual Coding Syntax in JCTVC-W1005) includes, for example, the following syntax.

[0343] cbf(coded_block_flag): residual data presence / absence flag

[0344] last_sig_coeff_x_pos: the last non-zero coefficient X coordinate

[0345] last_sig_coeff_y_pos: the last non-zero coefficient Y coordinate

[0346] coded_sub_block_flag: Sub-block non-zero coefficient presence / absence flag

[0347] sig_coeff_flag: Non-zero coefficient presence / absence flag

[0348] gr1_flag: A flag indicating whether the level of the non-zero coefficient is greater than 1 (also called the GR1 flag)

[0349] gr2_flag: A flag indicating whether the level of non-zero coefficients is greater than 2 (also called the GR2 flag)

[0350] sign_flag: A code indicating whether a nonzero coefficient is positive or negative (also called a sign code)

[0351] coeff_abs_level_remaining: The remaining level of non-zero coefficients (also called the non-zero coefficient remaining level)

[0352] etc.

[0353] It goes without saying that the information included in the residual information Rinfo is optional and may include information other than the above-mentioned information.

[0354] <Filter information Finfo>

[0355] The filter information Finfo includes, for example, control information related to various filtering processes described below.

[0356] Control information related to the deblocking filter (DBF)

[0357] Control information related to the adaptive offset filter (SAO)

[0358] Control information related to the adaptive loop filter (ALF)

[0359] Control information associated with any other linear or nonlinear filter

[0360] More specifically, the filter information Finfo includes, for example, a picture to which each filter is applied, information specifying an area within the picture, filter on / off control information in units of CUs, slices, filter on / off control information related to a boundary between tiles, etc. It goes without saying that the information included in the filter information Finfo is optional and may include information other than the above information.

[0361] Returning to the description of the decoding section 612 , the decoding section 612 refers to the residual information Rinfo to derive the quantized transform coefficient level qcoef in each coefficient position within each transform block. The decoding section 612 feeds the quantized transform coefficient level qcoef to the inverse quantization section 613 .

[0362] In addition, the decoding section 612 feeds each block with the parsed header information Hinfo, prediction mode information Pinfo, quantized transform coefficient level qcoef, transform information Tinfo, and filter information Finfo. Specifically, the information is fed as described below.

[0363] The header information Hinfo is fed to the inverse quantization section 613, the inverse orthogonal transform section 614, the prediction section 619, and the in-loop filter section 616. The prediction mode information Pinfo is fed to the inverse quantization section 613 and the prediction section 619. The transform information Tinfo is fed to the inverse quantization section 613 and the inverse orthogonal transform section 614. The filter information Finfo is fed to the in-loop filter section 616.

[0364] It goes without saying that the above examples are illustrative rather than restrictive. For example, each encoding parameter can be fed to any processing unit. In addition, other information can be fed to any processing unit.

[0365] <Inverse Quantization Section>

[0366] The inverse quantization unit 613 performs inverse quantization-related processing. For example, the inverse quantization unit 613 receives the transform information Tinfo and the quantized transform coefficient level qcoef fed from the decoding unit 612 as inputs, and scales (inverse quantizes) the value of the quantized transform coefficient level qcoef based on the transform information Tinfo, thereby deriving the inversely quantized transform coefficient coefI.

[0367] Note that inverse quantization is performed as an inverse process of quantization performed by the quantization section 514. In addition, inverse quantization is a process similar to inverse quantization performed by the inverse quantization section 517. In other words, the inverse quantization section 517 performs a process (inverse quantization) similar to the process performed by the inverse quantization section 613.

[0368] The inverse quantization section 613 feeds the derived transform coefficient coefI to the inverse orthogonal transform section 614 .

[0369] <Inverse Orthogonal Transformation Unit>

[0370] The inverse orthogonal transform unit 614 performs processing related to the inverse orthogonal transform. For example, the inverse orthogonal transform unit 614 receives the transform coefficient coefI fed from the inverse quantization unit 613 and the transform information Tinfo fed from the decoding unit 612 as input, and performs an inverse orthogonal transform on the transform coefficient coefI based on the transform information Tinfo, thereby deriving a prediction residual resiI.

[0371] Note that the inverse orthogonal transform is performed as an inverse process of the orthogonal transform performed by the orthogonal transform section 513. In addition, the inverse orthogonal transform is a process similar to the inverse orthogonal transform performed by the inverse orthogonal transform section 518. That is, the inverse orthogonal transform section 518 performs a process (inverse orthogonal transform) similar to the process performed by the inverse orthogonal transform section 614.

[0372] The inverse orthogonal transform unit 614 feeds the derived prediction residual resiI to the calculation unit 615.

[0373] <Calculation Department>

[0374] The calculation unit 615 performs processing related to the addition of image-related information. For example, the calculation unit 615 receives as input the prediction residual resiI fed from the inverse orthogonal transform unit 614 and the predicted image P fed from the prediction unit 619. The calculation unit 615 adds the prediction residual resiI to the predicted image P (prediction signal) corresponding to the prediction residual resiI to derive a local decoded image Rlocal, as shown in the following equation (5).

[0375] [Formula 5]

[0376] Rlocal=resiI+P...(5)

[0377] The calculation unit 615 feeds the derived local decoded image Rlocal to the in-loop filter unit 616 and the frame memory 618 .

[0378] <In-loop filter section>

[0379] The in-loop filter section 616 performs processing related to the in-loop filter processing. For example, the in-loop filter section 616 receives as input the local decoded image Rlocal fed from the calculation section 615 and the filter information Finfo fed from the decoding section 612. Note that the information input to the in-loop filter section 616 is optional, and information other than the above information may be input.

[0380] The in-loop filter unit 616 appropriately performs filtering processing on the local decoded image Rlocal based on the filter information Finfo.

[0381] For example, the in-loop filter unit 616 sequentially applies four in-loop filters, which include a bilateral filter, a deblocking filter (DBF), an adaptive offset filter (SAO (sample adaptive offset)), and an adaptive in-loop filter (ALF). Please note that which filter is applied and the order of application are optional and can be determined by appropriate selection.

[0382] The in-loop filter unit 616 performs a filtering process corresponding to the filtering process performed on the encoding side (e.g., the in-loop filter unit 520 of the image encoding device 500). It goes without saying that the filtering process performed by the in-loop filter unit 616 is optional and is not limited to the above example. For example, the in-loop filter unit 616 can apply a Wiener filter or the like.

[0383] The in-loop filter section 616 feeds the filtered local decoded image Rlocal to the sorting buffer 617 and the frame memory 618 .

[0384] <Sort Buffer>

[0385] The sorting buffer 617 receives the local decoded image Rlocal fed from the in-loop filter section 616 as an input, and saves (stores) the local decoded image Rlocal. The sorting buffer 617 reconstructs the decoded image R in units of pictures using the local decoded image Rlocal, and saves the decoded image R (stores the decoded image R in the buffer). The sorting buffer 617 sorts the obtained decoded image R to change the order of the decoded images R from the decoding order to the reproduction order. The sorting buffer 617 outputs a group of sorted decoded images R as motion image data to the outside of the image decoding device 600.

[0386] <Frame Memory>

[0387] The frame memory 618 performs processing related to the storage of image-related data. For example, the frame memory 618 receives the local decoded image Rlocal fed from the calculation section 615 as input, reconstructs the decoded image R in units of pictures, and stores the decoded image R in a buffer in the frame memory 618.

[0388] In addition, the frame memory 618 receives as input the local decoded image Rlocal that has been subjected to in-loop filtering and is fed from the in-loop filter section 616, reconstructs the decoded image R in units of pictures, and stores the decoded image R in a buffer in the frame memory 618. The frame memory 618 appropriately feeds any decoded image R (or a portion of the decoded image R) stored in the frame memory 618 to the prediction section 619 as a reference image.

[0389] Note that the frame memory 618 may store header information Hinfo, prediction mode information Pinfo, transform information Tinfo, filter information Finfo, and the like related to generation of a decoded image.

[0390] <Forecasting Department>

[0391] The prediction section 619 performs processing related to the generation of a prediction image. For example, the prediction section 619 receives the prediction mode information Pinfo fed from the decoding section 612 as an input, and performs prediction by using the prediction method specified by the prediction mode information Pinfo, thereby deriving a prediction image P. During the deriving, the prediction section 619 uses the filtered or unfiltered decoded image R (or a part of the decoded image R) stored in the frame memory 618 and specified by the prediction mode information Pinfo as a reference image. The prediction section 619 feeds the derived prediction image P to the calculation section 615.

[0392] <Application of this technology>

[0393] The present technology described above in <3. Concept> and <4. Method 1> is applied to the image decoding device 600 configured as described above. For example, "Method 1-1" is applied to the image decoding device 600. For example, the decoding section 612 may set the VPDU 121 in the CTU 111, and decode the encoded data in parallel in each row of the CTU 111 including the VPDU 121. In addition, the decoding section 612 performs the processing of each CTU row by an amount corresponding to one CTU with respect to the CTU row immediately above the CTU row.

[0394] For example, the decoding unit 612 decodes the encoded data obtained by encoding coefficient data related to the image in parallel according to each row of the coding tree unit, each coding tree unit includes a single or multiple transform blocks and corresponds to the highest coding block of the tree structure, and the decoding unit 612 delays the decoding of each row relative to the decoding of the immediately previous row by an amount corresponding to one coding tree unit.

[0395] In addition, for example, the decoding section 612 may decode the encoded data of each row of the coding tree unit of coefficient data related to the image so that the coding tree units are decoded one by one in order starting from the leftmost coding tree unit.

[0396] In addition, for example, the decoding unit 612 may decode the coded data of each coding tree unit for each transform block.

[0397] In addition, for example, the decoding section 612 may entropy-decode the encoded data of each transform block by using the occurrence probability derived from the entropy decoding of the encoded data of the previous transform block.

[0398] In addition, for example, the decoding unit 612 may entropy decode the encoded data of the first transform block in the leftmost coding tree unit in the topmost row of the coding tree unit in the image using the initial value of the occurrence probability. In addition, for example, the decoding unit 612 may entropy decode the encoded data of the first transform block in each of the leftmost coding tree units in the next upper row and subsequent rows of the coding tree unit in the image by using the occurrence probability derived from entropy decoding the last transform block in the leftmost coding tree unit in the row of the coding tree unit immediately above the row.

[0399] In addition, "method 1-2" may be applied to the image decoding device 600. Specifically, the prediction unit 619 may limit certain modes in intra prediction. For example, "method 1-2-1" may be applied to the image decoding device 600. Specifically, in the intra prediction of the upper right VPDU in the CTU, the prediction unit 619 may perform intra prediction while invalidating the reference to the VPDU at the upper right of the upper right VPDU.

[0400] For example, in the intra prediction of the upper right transform block of the coding tree unit, the prediction part 619 may invalidate the reference to the transform block at the upper right of the upper right transform block.

[0401] In addition, "method 1-3" may be applied to the image decoding device 600. Specifically, the prediction unit 619 may limit inter-frame prediction. For example, "method 1-3-1" may be applied to the image decoding device 600. Specifically, in inter-frame prediction in which the upper right VPDU in the CTU is used as the current prediction block, the prediction unit 619 may perform inter-frame prediction while invalidating the reference to the inter-frame prediction mode and motion vector of the VPDU at the upper right of the upper right VPDU.

[0402] For example, in the inter prediction of the upper right transform block in the coding tree unit, the prediction part 619 may invalidate the reference to the transform block at the upper right of the upper right transform block.

[0403] In addition, for example, "Method 1-3-2" can be applied to the image decoding device 600. Specifically, in the inter prediction of the prediction block having a block size of 128×N, the prediction unit 619 can perform the inter prediction while invalidating the reference to the inter prediction mode and motion vector of the block at the upper right of the prediction block.

[0404] For example, in inter prediction of a prediction block corresponding to a processing unit of inter prediction and having the same horizontal length as that of a coding tree unit, the prediction section 619 may invalidate reference to a transform block at the upper right of the prediction block.

[0405] Such a configuration enables the image decoding device 600 to suppress a decrease in the parallelism of decoding, thereby suppressing an increase in processing time.

[0406] <5-4: Processing Flow>

[0407] <Flow of Image Decoding Process>

[0408] Next, the flow of each type of processing performed by the image decoding device 600 as described above will be described below. Fig. 20 The flowchart in describes an example of the flow of image decoding processing.

[0409] When the image decoding process is started, in step S301 , the storage buffer 611 acquires encoded data (bit stream) fed from the outside of the image decoding device 600 , and saves (stores) the encoded data.

[0410] In step S302, the decoding unit 612 performs a decoding process to decode the coded data (bit stream) to obtain the quantized transform coefficient level qcoef. In addition, the decoding unit 612 performs decoding to parse (analyze and obtain) various types of coding parameters from the coded data (bit stream). The details of the decoding process will be described below.

[0411] In step S303 , the inverse quantization section 613 performs inverse quantization corresponding to the inverse process of quantization performed on the encoding side on the quantized transform coefficient level qcoef obtained by the process in step S302 , thereby obtaining a transform coefficient coefI.

[0412] In step S304, the inverse orthogonal transform section 614 performs an inverse orthogonal transform process corresponding to the inverse process of the orthogonal transform process performed on the encoding side on the transform coefficient coefI obtained by the process in step S303, thereby obtaining a prediction residual resiI.

[0413] In step S305 , the prediction section 619 performs prediction processing using the prediction method specified by the encoding side based on the information parsed in step S302 , and generates a predicted image P by referring to a reference image stored in the frame memory 618 , for example.

[0414] In step S306 , the calculation section 615 adds the prediction residual resiI obtained by the process in step S304 to the predicted image P obtained by the process in step S305 to derive the local decoded image Rlocal.

[0415] In step S307 , the in-loop filter section 616 performs in-loop filter processing on the local decoded image Rlocal obtained by the processing in step S306 .

[0416] In addition, in step S308 , the frame memory 618 stores at least the local decoded image Rlocal obtained by the process in step S306 or the filtered local decoded image Rlocal obtained by the process in step S307 .

[0417] In step S309 , the sorting buffer 617 derives a decoded image R using the filtered local decoded image Rlocal obtained by the process in step S307 , and sorts the group of decoded images R to change the order of the decoded images R from the decoding order to the reproduction order.

[0418] In step S310, the sorting buffer 617 outputs the group of decoded images R sorted in the reproduction order as a moving image to the outside of the image decoding device 600. When the process in step S310 ends, the image decoding process ends.

[0419] In the decoding process performed in step S302 of the image decoding process as described above, the decoding section 612 decodes the coded data obtained by encoding the coefficient data related to the image in parallel for each row of the coding tree unit, each coding tree unit including a single or multiple transform blocks and corresponding to the highest coding block of the tree structure, and the decoding section performs the decoding of each row by an amount corresponding to one coding tree unit with respect to the decoding of the row immediately before the row. This enables the image decoding device 600 to suppress a decrease in the parallelism of decoding, thereby suppressing an increase in processing time.

[0420] <Decoding Process Flow>

[0421] Next, we will refer to Fig.21 The flowchart in the description is in Fig. 20 An example of the flow of the decoding process performed in step S302.

[0422] When the decoding process is started, in step S331 , the decoding section 612 sets a variable T indicating the position of a CTU row to be processed (the number of rows arranged sequentially from the top row in the image) to an initial value (=1).

[0423] In step S332, the decoding section 612 performs CTU processing on CTU row 1 corresponding to the uppermost row in the image. The details of the processing will be described below. When the processing on CTU row 1 is completed, the processing proceeds to step S333.

[0424] In step S333, the decoding section 612 determines whether there is any unprocessed CTU line. In the case where the decoding section 612 determines that the processing up to the lowermost CTU line has not been performed and there is an unprocessed CTU line, the process proceeds to step S334.

[0425] In step S334, the decoding section 612 adds 1 to the value of the variable T (T=T+1). When the value of the variable T is increased, the process proceeds to step S335.

[0426] In step S335, the decoding unit 612 performs CTU processing on the CTU line T corresponding to the Tth line from the top line in the image. The details of the processing will be described below. When the processing on the CTU line T is completed, the processing returns to step S333.

[0427] In other words, the processing in steps S333 to S335 is repeated until all CTU rows are processed. Then, in step S333, if the decoding unit 612 determines that there are no unprocessed CTU rows (all CTU rows have been processed), the decoding process ends and the process returns to step S334. Fig. 20 .

[0428] <Flow of processing VPDU of CTU row 1>

[0429] Next, we will refer to Fig. 22 The flowchart in the description is in Fig.21 An example of the process of CTU processing on CTU row 1 performed in step S332.

[0430] When starting CTU processing, in step S351, the decoding unit 612 sets the variable i indicating the position of the current CTU to an initial value (=1) (CTU number in order starting from the leftmost CTU in the image). In addition, the decoding unit 612 sets the value N to the number of CTUs in the horizontal direction.

[0431] In step S352, the decoding unit 612 initializes the CABAC context (occurrence probability).

[0432] In step S353 , the decoding section 612 determines whether the variable i is equal to or less than N (i<=N). In a case where the decoding section 612 determines that the variable i is equal to or less than N and CTU row 1 includes an unprocessed CTU, the process proceeds to step S354 .

[0433] In step S354 , the decoding unit 612 decodes the encoded data of the i-th CTU starting from the leftmost side.

[0434] In step S355 , the decoding section 612 determines whether the value of the variable i is 1 (i==1). In the case where the value of the variable i is determined to be 1 (ie, in step S354 , the leftmost CTU in the image is processed), the process proceeds to step S356 .

[0435] In step S356, the decoding section 612 saves the CABAC context (occurrence probability) of the last VPDU in the leftmost CTU of the picture, which has been generated in step S354. When the context is saved, the process proceeds to step S357.

[0436] In addition, in step S355, when it is determined that the value of the variable i is not 1 (in step S354, CTUs other than the leftmost CTU in the image have been processed, or the processing of the leftmost CTU in the image has not been completed), the processing of step S356 is skipped and the processing proceeds to step S357.

[0437] In step S357 , the decoding section 612 determines whether the value of the variable i is equal to or greater than 1 (i>=1). In the case where it is determined that the value of the variable i is equal to or greater than 1, the process proceeds to step S358 .

[0438] In step S358 , the decoding section 612 notifies the completion of the CTU process of the CTU row 2 processing thread corresponding to the thread processing CTU row 2. When the completion of the CTU process is notified, the process proceeds to step S359 .

[0439] In addition, in step S357, in the case where it is determined that the value of the variable i is less than 1 (the processing of the leftmost CTU has not been completed), the processing of step S358 is skipped and the processing proceeds to step S359.

[0440] In step S359, the decoding unit 612 increases the value of the variable i by 1 (i=i+1). When the value of the variable i increases, the process returns to step S353.

[0441] In other words, the processing in steps S353 to S359 is repeated until all CTUs in CTU row 1 are processed. Then, in step S353, if the decoding section 612 determines that there is no unprocessed CTU (all CTUs in CTU row 1 have been processed), the CTU processing ends and the processing returns to Fig.21 .

[0442] <Flow of CTU Processing for CTU Row T>

[0443] Next, we will refer to Fig.23 and Fig.24 The flowchart in the description is in Fig.21 An example of the process of CTU processing of CTU row T performed in step S335.

[0444] When CTU processing is started in this case, in step S371, the decoding section 612 waits until the decoding section 612 receives a notification of CTU processing completion from a CTU row T-1 processing thread corresponding to a thread processing CTU row T-1 immediately above the CTU row. When the notification of CTU processing completion is received from the CTU row T-1 processing thread, the process proceeds to step S372.

[0445] In step S372, the decoding unit 612 sets the variable i to an initial value (=1). In addition, the decoding unit 612 sets the value N to the number of CTUs in the horizontal direction.

[0446] In step S373 , the decoding section 612 takes over the CABAC context (occurrence probability) saved when CTU processing is performed on the CTU row T−1 immediately above the CTU row T, and initializes CABAC.

[0447] In step S374 , the decoding section 612 determines whether the variable i is equal to or less than N (i<=N). In the case the decoding section 612 determines that the variable i is equal to or less than N and the CTU row T includes an unprocessed CTU, the process proceeds to step S375 .

[0448] In step S375, in order to control the processing timing, the decoding section 612 waits until the decoding section 612 receives a notification of CTU processing completion from the CTU row T-1 processing thread. When the notification of CTU processing completion is received from the CTU row T-1 processing thread, the process proceeds to step S376.

[0449] In step S376, the decoding unit 612 invalidates the reference to the CTU at the upper right of the current CTU. Fig. 20 Step S305 in the above description includes a case where the upper right VPDU in the CTU is used as the inter-frame prediction of the current prediction block, and the prediction unit 619 can perform inter-frame prediction while invalidating the reference to the inter-frame prediction mode and motion vector of the VPDU at the upper right of the current prediction block (i.e., the CTU at the upper right of the current CTU).

[0450] In step S377, the decoding unit 612 decodes the coded data of the i-th CTU starting from the leftmost side. When the decoding is completed, the process proceeds to Fig.24 Step S381 in .

[0451] In step S381, the decoding section 612 determines whether the i-th CTU is in the inter 128×N mode (i.e., whether the i-th CTU is included in a 128×N prediction block on which inter prediction has been performed). In the case where the decoding section 612 determines that the i-th CTU is in the inter 128×N mode, the process proceeds to step S382.

[0452] In step S382, in this case, the current CTU is included in the prediction block of block size 128×N on which inter prediction has been performed, and therefore, the decoding section 612 invalidates the reference to the block at the upper right of the prediction block. When the processing in step S382 ends, the processing proceeds to step S383.

[0453] In addition, in step S381, in the case where it is determined that the i-th CTU is not in the inter 128×N mode, the process of step S382 is skipped (omitted) and the process proceeds to step S383.

[0454] In step S383, the decoding section 612 determines whether the value of the variable i is 1 (i==1). In the case where the value of the variable i is determined to be 1, the process proceeds to step S384.

[0455] In step S384, the decoding unit 612 saves the CABAC context (probability of occurrence) of the current VPDU (the last VPDU in the leftmost CTU in the image), which is already in Fig.23 The CABAC encoding is generated in step S377. When the context is saved, the process proceeds to step S385.

[0456] In addition, in step S383, in the case where it is determined that the value of the variable i is not 1 (the leftmost CTU in the image is not a processing target), the processing of step S384 is skipped and the processing proceeds to step S385.

[0457] In step S385, the decoding section 612 determines whether the value of the variable i is equal to or greater than 1 (i>=1). In the case where it is determined that the value of the variable i is equal to or greater than 1, the process proceeds to step S386.

[0458] In step S386, the decoding unit 612 notifies the CTU row T+1 processing thread corresponding to the thread processing CTU row T+1 immediately below CTU row T of the completion of the CTU processing. When the completion of the CTU processing is notified, the process proceeds to step S387.

[0459] In addition, in step S385, in the case where it is determined that the value of the variable i is less than 1 (the leftmost CTU in the image as the processing target has not yet been processed), processing step S386 is skipped and the processing proceeds to step S387.

[0460] In step S387, the decoding unit 612 increases the value of the variable i by 1 (i=i+1). When the value of the variable i increases, the process returns to Fig.23 Step S374 in .

[0461] In other words, repetition Fig.23 The processing in steps S374 to S377 in Fig.24 The processing in steps S381 to S387 in is repeated until all CTUs in CTU row T are processed. Then, in step S374, the decoding unit 612 determines that there is no unprocessed CTU (all CTUs in CTU row T have been processed), the CTU processing ends, and the processing returns to Fig.21 .

[0462] By performing the processing described above, the image decoding device 600 can suppress a decrease in the degree of parallelism of decoding.

[0463] <6. Supplementary functions>

[0464] <Computer>

[0465] The sequence of processing steps may be performed by hardware or software. In the case where the sequence of processing steps is performed by software, the program included in the software is installed in a computer. Here, examples of computers include computers integrated in dedicated hardware, for example, general-purpose computers that can perform various functions by installing various programs.

[0466] Fig.25 : is a block diagram showing a configuration example of hardware of a computer that executes the sequence of the above-described processing steps according to a program.

[0467] exist Fig.25 In the illustrated computer 800 , a CPU (Central Processing Unit) 801 , a ROM (Read Only Memory) 802 , and a RAM (Random Access Memory) 803 are connected together via a bus 804 .

[0468] An input / output interface 810 is also connected to the bus 804. The input / output interface 810 is connected to an input section 811, an output section 812, a storage section 813, a communication section 814, and a drive 815.

[0469] The input unit 811 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 812 includes, for example, a display, a speaker, an output terminal, etc. The storage unit 813 includes, for example, a hard disk, a RAM disk, a nonvolatile memory, etc. The communication unit 814 includes, for example, a network interface. The driver 815 drives a removable medium 821, such as a magnetic medium, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0470] In the computer configured as described above, the CPU 801 loads the program stored in the storage section 813 into the RAM 803 via the input / output interface 810 and the bus 804 and executes the program, thereby executing the sequence of the above-mentioned processing steps. The RAM 803 also stores data required for the CPU 801 to execute various processing steps, for example, as needed.

[0471] For example, the program executed by the computer (CPU 801) may be recorded in a removable medium 821 serving as a package medium of an application or the like. In this case, by mounting the removable medium 821 to the drive 815, the program can be installed in the storage section 813 via the input / output interface 810.

[0472] In addition, the program may be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. In this case, the program may be received by the communication section 814 and installed in the storage section 813.

[0473] Otherwise, the program may be installed in the ROM 802 or the storage section 813 in advance.

[0474] <Units of Information and Processing>

[0475] The data units provided with the above-mentioned various types of information and the data units corresponding to the targets of the various types of processing are optional and are not limited to the above-mentioned examples. For example, the above-mentioned information and processing can be set for each of TU (transform unit), TB (transform block), PU (prediction unit), PB (prediction block), CU (coding unit), LCU (maximum coding unit), sub-block, block, tile, slice, picture, sequence or component, or data in such data units. It goes without saying that a data unit can be set for each piece of information and each processing step, and it is not necessary to use a common data unit for all information and processing. Note that this information is stored in any location and can be stored in a header, parameter set, etc. in the above-mentioned data unit. In addition, information can be stored in multiple locations.

[0476] <Control Information>

[0477] Control information related to the present technology and described in the above-mentioned embodiments may be sent from the encoding side to the decoding side. For example, control information (e.g., enabled_flag) for controlling whether to allow (or prohibit) the application of the above-mentioned present technology may be sent. In addition, for example, control information indicating a target to which the above-mentioned present technology is applied (or a target to which the above-mentioned present technology is not applied) may be sent. For example, control information may be sent that specifies a block size (upper limit or lower limit or both), a frame, a component, a layer, etc., to which the present technology is to be applied (or to which application is allowed or prohibited).

[0478] <Application target of this technology>

[0479] The present technology can be applied to any image encoding and decoding scheme. Specifically, within the scope consistent with the above-mentioned present technology, the specification of various types of processing related to image encoding and decoding, such as transformation (inverse transformation), quantization (inverse quantization), encoding (decoding), and prediction, is optional, and the description is not limited to the above examples. In addition, within the scope consistent with the above-mentioned present technology, some of these types of processing can be omitted.

[0480] In addition, the present technology can be applied to a multi-view image encoding and decoding system for encoding and decoding a multi-view image including images in a plurality of views. In this case, it is sufficient to apply the present technology to encoding and decoding in each view.

[0481] In addition, the present technology can be applied to a layered image coding (scalable coding) and decoding system for encoding and decoding a layered image with multiple layers (with a hierarchical structure) to provide a scalability function for some parameters. In this case, it is sufficient to apply the present technology to the coding and decoding of each layer.

[0482] The image processing device, image encoding device and image decoding device according to the above-mentioned embodiments can be applied to various types of electronic devices, including, for example, transmitters and receivers in satellite broadcasting, broadcasting such as cable television, distribution on the Internet, and distribution to terminals via cellular communications (for example, television receivers and cellular phones) or devices that record images in storage media such as optical disks, magnetic disks and flash memories and copy images from these storage media (for example, hard disk recorders and camera devices).

[0483] In addition, the present technology can be implemented as any configuration installed in a device or in any device included in a system, such as a processor used as a system LSI (large scale integration) (e.g., a video processor), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), a collection of units to which any other functions are added (e.g., a video set), etc. (i.e., as a configuration corresponding to a part of the device).

[0484] In addition, the present technology can be applied to a network system including multiple devices. For example, the present technology can be applied to a cloud service that provides image (moving image) related services to any terminal such as a computer, AV (audio-visual) equipment, a portable information processing terminal, or an IoT (Internet of Things) device.

[0485] Note that the systems, devices, processing units, etc. to which the present technology is applied can be used in any field such as transportation, medical treatment, crime prevention, agriculture, animal husbandry, mining, beauty care, factories, home appliances, weather services, and natural monitoring. In addition, they can be used for any purpose.

[0486] For example, the present technology can be applied to systems and devices for providing content for viewing, etc. In addition, the present technology can be applied to systems and devices for traffic purposes such as management of traffic conditions and automatic operation control. In addition, for example, the present technology can be applied to systems and devices for security purposes. In addition, for example, the present technology can be applied to systems and devices for automatic control of machines, etc. In addition, for example, the present technology can be applied to systems and devices for agriculture and animal husbandry. In addition, the present technology can be applied to systems and devices for monitoring natural conditions such as volcanoes, forests, oceans, wildlife, etc. In addition, for example, the present technology can be applied to systems and devices for sports purposes.

[0487] <Others>

[0488] Note that the "flag" used in this article refers to information for distinguishing multiple states from each other, and includes information for distinguishing three or more states and two states of true (1) and false (0) from each other. Therefore, the value that the "flag" can take can be a binary value of 1 / 0, or three or more values. Specifically, the number of bits included in the "flag" is optional, and the "flag" includes one or more bits. In addition, it is assumed that the identification information (including the flag) has the following form: the difference information between the identification information and the information used as a reference is included in the bit stream; and has the following form: the identification information is included in the bit stream. The "flag" and "identification information" used in this article include not only the information itself, but also the difference information relative to the reference information.

[0489] In addition, various types of information (metadata, etc.) related to the coded data (bitstream) can be sent or recorded in any form, as long as the information is associated with the coded data. Here, the term "associated" refers to configuring two pieces of data so that, for example, when processing one piece of data, another piece of data can be used for (linked to) the processing. In other words, multiple pieces of data associated with each other can be compiled into one piece of data or used as separate data. For example, information associated with the coded data (image) can be transmitted through a transmission path different from the transmission path of the coded data (image). In addition, for example, information associated with the coded data (image) can be recorded in a recording medium different from the recording medium in which the coded data (image) is recorded (or can be recorded in different recording areas in the same recording medium). Note that "associated" can be performed on a part of the data rather than the entire data. For example, an image and the information corresponding to the image can be associated with each other in any unit such as a plurality of frames, a frame or a part within a frame.

[0490] Note that the terms "synthesize", "multiplex", "add", "integrate", "include", "store", "put into", "insert into", "insert" and the like indicate the bringing together of multiple objects, such as compiling encoded data and metadata into one data, and indicate a method of the above-mentioned "association".

[0491] In addition, the embodiment of the present technology is not limited to the above-described embodiment, and various modifications may be made without departing from the scope of the present technology.

[0492] In addition, for example, a configuration described as one device (or processing unit) may be divided into a plurality of devices (or processing units). In contrast, the configuration described above as a plurality of devices (or processing units) may be combined into one device (or processing unit). Furthermore, it goes without saying that the configuration of each device (or each processing unit) may include other components in addition to the above components. Furthermore, a portion of the configuration of one device (or processing unit) may be included in the configuration of another device (or another processing unit), as long as the devices are substantially the same in configuration or operation of the entire system.

[0493] Note that the system used in this article refers to a group of multiple components (devices, modules (components), etc.), regardless of whether all the components are located in the same housing. Therefore, multiple devices housed in separate housings and connected together via a network and one device with multiple modules housed in one housing are both systems.

[0494] In addition, for example, the present technology may adopt a configuration of cloud computing in which one function is shared by a plurality of devices via a network and processed jointly.

[0495] In addition, for example, the above-mentioned program can be executed in any device. In this case, it is sufficient to provide the device with the required functions (functional blocks, etc.) and enable the device to obtain the required information.

[0496] In addition, for example, each of the above steps in the flowchart may be shared by a plurality of devices instead of being performed by one device. Furthermore, in the case where one step includes multiple types of processing, the multiple types of processing included in one step may be shared by a plurality of devices instead of being performed by one device. In other words, the multiple types of processing included in one step may be performed as multiple processing steps. In contrast, processing described as multiple steps may be put into one step.

[0497] Note that for a program executed by a computer, the processing steps describing the program may be performed in chronological order according to the order described herein, may be performed in parallel, or may be performed separately at a desired timing, such as at the timing of invoking the operation. In other words, the processing steps may be performed in an order different from the order described above within the scope consistent with the present technology. In addition, the processing steps describing the program may be performed in parallel or in combination with the processing of another program.

[0498] Note that, within the scope consistent with the present technology, the multiple present technologies described herein can be independently implemented as a single technology. It goes without saying that any multiple present technologies can be implemented together. For example, part or all of the present technologies described in any embodiment can be combined with part or all of the present technologies described in another embodiment and can be implemented. In addition, part or all of any of the above-mentioned current technologies can be implemented together with another technology not described above.

[0499] Note that the present technology can also adopt the following configurations. (1)

[0501] An image processing device, comprising:

[0502] An encoding section encodes coefficient data related to an image in parallel for each row of a transform block, each of which is used as a unit for transforming image data into coefficient data. (2)

[0504] The image processing device according to (1),

[0505] The encoding unit encodes each row of the transform block of coefficient data related to the image so that the transform blocks are encoded one by one in sequence starting from the leftmost transform block, and the encoding unit entropy encodes each transform block by using the occurrence probability derived from the previous entropy encoding. (3)

[0507] The image processing device according to (2),

[0508] The encoding unit entropy encodes the leftmost transform block in the row of the topmost transform block in the image by using an initial value of the occurrence probability, and the encoding unit entropy encodes the first transform block in the leftmost transform block in each of the second upper row and subsequent rows of transform blocks in the image by using the occurrence probability derived from the entropy encoding of the second left transform block in the row of the transform block immediately above the row to be encoded. (4)

[0510] The image processing device according to (2),

[0511] The encoding unit entropy encodes the leftmost transform block in the row of the uppermost transform block in the image by using an initial value of the occurrence probability, and the encoding unit entropy encodes the leftmost transform block in the row of transform blocks belonging to the same coding tree unit as the transform block immediately above the transform block to be encoded by using the occurrence probability derived from entropy encoding the second left transform block in the row of the transform block immediately above the transform block to be encoded, the row corresponding to the second upper row and subsequent rows of the transform blocks in the image, the coding tree unit corresponding to the uppermost coding block in the tree structure, and the encoding unit entropy encodes the leftmost transform block in the row of transform blocks belonging to a coding tree unit different from the coding tree unit of the transform block immediately above the transform block to be encoded by using the occurrence probability derived from entropy encoding the third transform block from the left in the row of the transform block immediately above the transform block to be encoded, the row corresponding to one of the second upper and subsequent rows of the transform block in the image. (5)

[0513] The image processing device according to (1),

[0514] The transformation block is a VPDU (Virtual Pipeline Data Unit). (6)

[0516] The image processing device according to (1), further comprising:

[0517] a prediction unit, the prediction unit performing intra-frame prediction on the image,

[0518] Here, in intra prediction of an upper left transform block in a coding tree unit corresponding to a highest coding block in a tree structure, the prediction section invalidates a reference to a lower right transform block in a previous coding tree structure. (7)

[0520] The image processing device according to (1), further comprising:

[0521] a prediction unit, the prediction unit performing inter-frame prediction on the image,

[0522] Here, in inter-frame prediction of an upper left transform block in a coding tree unit corresponding to a highest coding block in a tree structure, the prediction section invalidates reference to a motion vector of a lower right transform block in a previous coding tree. (8)

[0524] The image processing device according to (1), further comprising:

[0525] a prediction unit, the prediction unit performing inter-frame prediction on the image,

[0526] In which, in the inter-frame prediction of a prediction block corresponding to the processing unit of the inter-frame prediction and having the same horizontal length as the horizontal length of the coding tree unit corresponding to the highest coding block in the tree structure, the prediction unit invalidates the reference to the motion vector of the transform block at the upper right of the prediction block. (9)

[0528] The image processing device according to (1),

[0529] Among them, for the prediction block corresponding to the processing unit of the inter-frame prediction and having the same vertical length as the vertical length of the coding tree unit corresponding to the highest coding block in the tree structure, the encoding unit encodes the mode information indicating the mode of the inter-frame prediction for each prediction block, and encodes the residual data between the image and the predicted image for each transform block included in the prediction block. (10)

[0531] An image processing method, comprising:

[0532] Coefficient data associated with an image are encoded in parallel per row of transform blocks each corresponding to a unit for transforming image data into coefficient data. (11)

[0534] An image processing device, comprising:

[0535] A decoding section decodes, in parallel, coded data obtained by coding coefficient data associated with an image in each row of transform blocks each corresponding to a unit for transforming image data into coefficient data. (12)

[0537] The image processing device according to (11),

[0538] The decoding unit decodes the encoded data in each row of the transform block of coefficient data related to the image, so that the transform blocks are decoded one by one in sequence starting from the leftmost transform block, and the encoded data of each transform block is entropy decoded by using the occurrence probability derived from the previous entropy decoding. (13)

[0540] The image processing device according to (12),

[0541] The decoding unit entropy decodes the coded data of the leftmost transform block in the top row of the transform blocks in the image by using the initial value of the occurrence probability, and entropy decodes the coded data of the leftmost transform block in each of the next upper and subsequent rows of the transform blocks in the image by using the occurrence probability derived from the entropy decoding of the coded data of the second left transform block in the row of the transform block immediately above the row to be transformed. (14)

[0543] The image processing device according to (12),

[0544] wherein the decoding unit entropy decodes the coded data of the leftmost transform block in the uppermost row of the transform block in the image by using the initial value of the occurrence probability, the decoding unit entropy decodes the coded data of the leftmost transform block in the row of the transform block belonging to the same coding tree unit as the coding tree unit of the transform block immediately above the transform block to be decoded by using the occurrence probability derived from the entropy decoding of the coded data of the second leftmost transform block in the row of the transform block immediately above the transform block to be decoded, the row corresponding to the second upper and subsequent row of the transform block in the image, the coding tree unit corresponding to the uppermost coded block in the tree structure, and the decoding unit entropy decodes the coded data of the leftmost transform block in the row of the transform block belonging to the coding tree unit different from the coding tree unit of the transform block immediately above the transform block to be decoded by using the occurrence probability derived from the entropy decoding of the coded data of the third transform block from the left in the row of the transform block immediately above the transform block to be decoded, the row corresponding to one of the second upper and subsequent rows of the transform block in the image. (15)

[0546] The image processing device according to (11),

[0547] The transformation block is a VPDU (Virtual Pipeline Data Unit). (16)

[0549] The image processing device according to (11),

[0550] a prediction unit, the prediction unit performing intra-frame prediction on the image,

[0551] Here, in intra prediction of an upper left transform block in a coding tree unit corresponding to a highest coding block in a tree structure, the prediction unit invalidates a reference to a lower right transform block in a previous decoding of the coding tree unit. (17)

[0553] The image processing device according to (11), further comprising:

[0554] a prediction unit, the prediction unit performing inter-frame prediction on the image,

[0555] Here, in inter-frame prediction of an upper left transform block in a coding tree unit corresponding to a highest coding block in a tree structure, the prediction unit invalidates a reference to a motion vector of a lower right transform block in a previous decoding of the coding tree unit. (18)

[0557] The image processing device according to (11), further comprising:

[0558] a prediction unit, the prediction unit performing inter-frame prediction on the image,

[0559] In which, in the inter-frame prediction of a prediction block corresponding to the processing unit of the inter-frame prediction and having the same horizontal length as the horizontal length of the coding tree unit corresponding to the highest coding block in the tree structure, the prediction unit invalidates the reference to the motion vector of the transform block at the upper right of the prediction block. (19)

[0561] The image processing device according to (11),

[0562] Among them, for the prediction block corresponding to the processing unit of the inter-frame prediction and having the same vertical length as the vertical length of the coding tree unit corresponding to the highest coding block in the tree structure, the decoding unit decodes the encoded data of the mode information indicating the mode of the inter-frame prediction for each prediction block, and decodes the encoded data of the residual data between the image and the predicted image for each transform block included in the prediction block. (20)

[0564] An image processing method, comprising:

[0565] The encoded data obtained by encoding the coefficient data associated with the image is decoded in parallel in each row of the transform blocks each corresponding to a unit of transforming the image data into the coefficient data.

[0566] [reference numerals list]

[0567] 500: Image encoding device

[0568] 501: Control Department

[0569] 515: Coding Department

[0570] 522: Forecasting Department

[0571] 600: Image decoding device

[0572] 612: Decoding Department

[0573] 619: Forecasting Department< / vpdu> < / wpp>

Claims

1. An image processing device, comprising: at least one processor configured to: Dividing the image into a plurality of coding tree units having a plurality of parallel coding tree unit CTU rows; Dividing each CTU into a plurality of blocks, wherein each of the blocks is a processing unit having a size smaller than a size of the CTU; determining whether to apply entropy coding of each CTU row, wherein the entropy coding is delayed by one CTU between adjacent CTU rows; and Apply entropy coding to each CTU row based on the determination: Referring to a corresponding neighboring block of at least one neighboring upper CTU of an adjacent upper CTU row, intra-predicting an upper left block of a current CTU of a current CTU row, and The upper right block of the current CTU is intra-predicted with reference to a corresponding adjacent block of an adjacent upper CTU of an upper CTU row without reference to a lower left block of a non-adjacent upper right CTU of an adjacent upper CTU row.

Citation Information

Patent Citations

  • Emulsified oil and fat composition for bakery products and bakery product prepared therewith

    JP2019047782A