Encoding device, decoding device, and non-transitory storage medium

Abstract

The invention provides an encoding device, a decoding device, and a non-transitory storage medium. The encoding device encodes a flag by context-adaptive encoding if the number of processing times of context-adaptive encoding is limited to allow context-adaptive encoding of a plurality of coefficient information flags together in both cases of applying a different syntax and skipping an orthogonal transform in residual encoding of a current block, and then performs encoding of the flag by the context-adaptive encoding if the number of processing times of the context-adaptive encoding is limited to allow context-adaptive encoding of a plurality of coefficient information flags. And encoding the residual value of the coefficient by Golomb encoding, if not allowed, skipping the encoding of the flag, encoding the value of the coefficient by Golomb encoding, and if the orthogonal transformation is skipped, encoding the coefficient information flag other than the plurality of absolute value flags after encoding and before encoding the residual value of the coefficient. A plurality of absolute value flags are encoded by context adaptive coding, the flags including a flag indicating whether the value of the coefficient is zero or non-zero and a flag indicating whether the coefficient is odd or even, and also including a flag indicating whether the coefficient is positive or negative if the orthogonal transformation is skipped.

Description

[0001] This application is a divisional of the patent application with application number 202080030228.9, filed on April 24, 2020, and titled “Encoding apparatus, decoding apparatus, encoding method, and decoding method”. TECHNICAL FIELD

[0002] The present application relates to video encoding, for example, systems, components, and methods in encoding and decoding of moving images, and the like. BACKGROUND

[0003] Video encoding technology has progressed from H.261 and MPEG-1 to H.264 / AVC (Advanced Video Coding), MPEG-LA, H.265 / HEVC (High Efficiency Video Coding), and H.266 / VVC (Versatile Video Codec). Along with this progress, in order to handle the increasing amount of digital video data in various uses, there has been a constant need to provide improvements and optimizations in video encoding technology.

[0004] Furthermore, Non-Patent Literature 1 relates to an example of an existing standard related to the above-described video encoding technology.

[0005] Prior Art Documents

[0006] Non-Patent Literature

[0007] Non-Patent Literature 1: H.265 (ISO / IEC 23008-2 HEVC) / HEVC (High Efficiency Video Coding) SUMMARY

[0008] Problems to be Solved by the Invention

[0009] With regard to the above-described encoding method, it is desirable to propose a new method for improvement of encoding efficiency, improvement of picture quality, reduction of processing amount, reduction of circuit size, or appropriate selection of elements or actions such as filters, blocks, sizes, motion vectors, reference pictures, or reference blocks, and the like.

[0010] The present application provides a structure or method that can contribute to one or more of, for example, improvement of encoding efficiency, improvement of picture quality, reduction of processing amount, reduction of circuit size, improvement of processing speed, and appropriate selection of elements or actions. Furthermore, the present application can include a structure or method that can contribute to benefits other than the above.

[0011] Means for Solving the Problems

[0012] For example, an encoding device of an aspect of the present application includes: a circuit; and a memory connected to the circuit, in residual encoding of a current block, in a case where application of an orthogonal transform using different syntaxes and a case where the orthogonal transform is skipped, in a case where a context-adaptive encoding process number limit allows context-adaptive encoding of a plurality of coefficient information flags related to a coefficient included in the current block, the circuit encodes the plurality of coefficient information flags by the context-adaptive encoding, and encodes a residual value of the coefficient by Golomb encoding, the residual value of the coefficient being a value used to reconstruct a value of the coefficient using the plurality of coefficient information flags, in a case where the process number limit does not allow the context-adaptive encoding of the plurality of coefficient information flags, the circuit skips encoding of the plurality of coefficient information flags, and encodes a value of the coefficient by the Golomb encoding, in the case where the orthogonal transform is skipped, the circuit encodes a plurality of absolute value flags by the context-adaptive encoding after encoding coefficient information flags other than the plurality of absolute value flags among the plurality of coefficient information flags and before encoding the residual value of the coefficient, the plurality of absolute value flags being flags related to whether an absolute value of the coefficient is larger than a predetermined value, and the predetermined value being an integer larger than 1, the plurality of coefficient information flags including a flag indicating whether the value of the coefficient is zero or nonzero and a flag indicating whether the coefficient is odd or even, and in the case where the orthogonal transform is skipped, the plurality of coefficient information flags further include a flag indicating whether the coefficient is positive or negative.

[0013] The installation of the several embodiments of the present application can improve the encoding efficiency, can simplify the encoding / decoding process, can speed up the encoding / decoding processing speed, and can efficiently select appropriate constituent elements / operations used in the encoding and decoding, such as appropriate filters, block sizes, motion vectors, reference pictures, reference blocks, and the like.

[0014] Further advantages and effects of an aspect of the present application are clear from the description and the drawings. The advantages and / or effects are obtained by the features described in the several embodiments and the description and the drawings, but it is not necessary to provide all of them in order to obtain one or more of the advantages and / or effects.

[0015] Furthermore, these general or specific aspects can also be implemented by system, method, integrated circuit, computer program, recording medium, or any combination thereof.

[0016] Effects of the Invention

[0017] The structure or method of an aspect of the present application can contribute to one or more of, for example, improvement of coding efficiency, improvement of image quality, reduction of processing amount, reduction of circuit size, improvement of processing speed, and appropriate selection of elements or actions. In addition, the structure or method of an aspect of the present application can contribute to benefits other than the above. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a block diagram showing a functional structure of an encoding apparatus of an embodiment.

[0019] Figure 2 is a flowchart showing an example of overall encoding processing by an encoding apparatus.

[0020] Figure 3 is a conceptual diagram showing an example of block partitioning.

[0021] Figure 4A is a conceptual diagram showing an example of the structure of a slice.

[0022] Figure 4B is a conceptual diagram showing an example of the structure of a tile.

[0023] Figure 5A is a table showing transform basis functions corresponding to various transform types.

[0024] Figure 5B is a conceptual diagram showing an example of an SVT (Spatially Varying Transform).

[0025] Figure 6A is a conceptual diagram showing an example of the shape of a filter used in an ALF (adaptive loop filter).

[0026] Figure 6B is a conceptual diagram showing another example of the shape of a filter used in an ALF.

[0027] Figure 6C is a conceptual diagram showing another example of the shape of a filter used in an ALF.

[0028] Figure 7 is a block diagram showing an example of the detailed structure of a loop filter section functioning as a DBF (deblocking filter).

[0029] Figure 8 is a conceptual diagram showing an example of deblocking filtering having a filter characteristic symmetrical with respect to a block boundary.

[0030] Figure 9is a conceptual diagram for explaining a concept of a block boundary on which deblocking filtering processing is performed.

[0031] Figure 10 is a conceptual diagram showing an example of a Bs value.

[0032] Figure 11 is a flowchart showing an example of processing performed by a prediction processing section of an encoding apparatus.

[0033] Figure 12 is a flowchart showing another example of processing performed by a prediction processing section of an encoding apparatus.

[0034] Figure 13 is a flowchart showing another example of processing performed by a prediction processing section of an encoding apparatus.

[0035] Figure 14 is a conceptual diagram showing an example of 67 intra prediction modes in intra prediction of an embodiment.

[0036] Figure 15 is a flowchart showing an example of a basic process of inter prediction.

[0037] Figure 16 is a flowchart showing an example of motion vector derivation.

[0038] Figure 17 is a flowchart showing another example of motion vector derivation.

[0039] Figure 18 is a flowchart showing another example of motion vector derivation.

[0040] Figure 19 is a flowchart showing an example of inter prediction based on a normal inter mode.

[0041] Figure 20 is a flowchart showing an example of inter prediction based on a merge mode.

[0042] Figure 21 is a conceptual diagram for explaining an example of motion vector derivation processing based on a merge mode.

[0043] Figure 22 is a flowchart showing an example of FRUC (frame rate up conversion) processing.

[0044] Figure 23 is a conceptual diagram for explaining an example of pattern matching (bi-directional matching) between 2 blocks along a motion trajectory.

[0045] Figure 24is a conceptual diagram for explaining an example of pattern matching (template matching) between a template within a current picture and a block within a reference picture.

[0046] Figure 25A is a conceptual diagram for explaining an example of derivation of a motion vector of a sub-block unit based on motion vectors of a plurality of neighboring blocks.

[0047] Figure 25B is a conceptual diagram for explaining derivation of a motion vector of a sub-block unit in an affine mode with 3 control points.

[0048] Figure 26A is a conceptual diagram for explaining an affine merge mode.

[0049] Figure 26B is a conceptual diagram for explaining an affine merge mode with 2 control points.

[0050] Figure 26C is a conceptual diagram for explaining an affine merge mode with 3 control points.

[0051] Figure 27 is a flowchart representing an example of a process of an affine merge mode.

[0052] Figure 28A is a conceptual diagram for explaining an affine inter mode with 2 control points.

[0053] Figure 28B is a conceptual diagram for explaining an affine inter mode with 3 control points.

[0054] Figure 29 is a flowchart representing an example of a process of an affine inter mode.

[0055] Figure 30A is a conceptual diagram for explaining an affine inter mode in which a current block has 3 control points and a neighboring block has 2 control points.

[0056] Figure 30B is a conceptual diagram for explaining an affine inter mode in which a current block has 2 control points and a neighboring block has 3 control points.

[0057] Figure 31A is a flowchart representing a merge mode including DMVR (decoder motion vector refinement).

[0058] Figure 31B is a conceptual diagram for explaining an example of a DMVR process.

[0059] Figure 32 is a flowchart representing an example of generation of a prediction image.

[0060] Figure 33 is a flowchart showing another example of generation of a prediction image.

[0061] Figure 34 is a flowchart showing another example of generation of a prediction image.

[0062] Figure 35 is a flowchart for explaining an example of prediction image correction processing based on OBMC (overlapped block motion compensation) processing.

[0063] Figure 36 is a conceptual diagram for explaining an example of prediction image correction processing based on OBMC processing.

[0064] Figure 37 is a conceptual diagram for explaining generation of a prediction image of 2 triangles.

[0065] Figure 38 is a conceptual diagram for explaining a model assuming constant velocity straight line motion.

[0066] Figure 39 is a conceptual diagram for explaining an example of a prediction image generation method using brightness correction processing based on LIC (local illumination compensation) processing.

[0067] Figure 40 is a block diagram showing an example of a configuration of an encoding device.

[0068] Figure 41 is a block diagram showing a functional configuration of a decoding device of an embodiment.

[0069] Figure 42 is a flowchart showing an example of overall decoding processing by a decoding device.

[0070] Figure 43 is a flowchart showing an example of processing by a prediction processing section of a decoding device.

[0071] Figure 44 is a flowchart showing another example of processing by a prediction processing section of a decoding device.

[0072] Figure 45 is a flowchart showing an example of inter prediction based on a normal inter mode in a decoding device.

[0073] Figure 46 is a block diagram showing an example of a configuration of a decoding device.

[0074] Figure 47 is a flowchart showing the basic coefficient encoding method of the first mode.

[0075] Figure 48 is a flowchart showing the basic first encoding method of the first mode.

[0076] Figure 49 is a flowchart showing the basic second encoding method of the first mode.

[0077] Figure 50 is a flowchart showing the coefficient encoding method of the first example of the first mode.

[0078] Figure 51 is a flowchart showing the coefficient encoding method of the second example of the first mode.

[0079] Figure 52 is a flowchart showing the coefficient encoding method of the first example of the second mode.

[0080] Figure 53 is a flowchart showing the coefficient encoding method of the second example of the second mode.

[0081] Figure 54 is a syntax diagram showing the basic first encoding method of the third mode.

[0082] Figure 55 is a syntax diagram showing the basic second encoding method of the third mode.

[0083] Figure 56 is a syntax diagram showing the second encoding method of the first example of the third mode.

[0084] Figure 57 is a syntax diagram showing the second encoding method of the second example of the third mode.

[0085] Figure 58 is a flowchart showing the action of the encoding apparatus of the embodiment.

[0086] Figure 59 is a flowchart showing the action of the decoding apparatus of the embodiment.

[0087] Figure 60 is a block diagram showing the overall structure of a content supply system that implements a content distribution service.

[0088] Figure 61 is a conceptual diagram showing an example of the encoding structure at the time of scalable encoding.

[0089] Figure 62 is a conceptual diagram showing an example of the encoding structure at the time of scalable encoding.

[0090] Figure 63is a conceptual diagram showing an example of a display screen of a web page.

[0091] Figure 64 is a conceptual diagram showing an example of a display screen of a web page.

[0092] Figure 65 is a block diagram showing an example of a smart phone.

[0093] Figure 66 is a block diagram showing an example of a structure of a smart phone. DETAILED DESCRIPTION

[0094] For example, in encoding of a block of an image, an encoding device can sometimes transform the block into data that is easy to compress by applying an orthogonal transform to the block. On the other hand, in encoding of a block of an image, an encoding device can sometimes reduce processing delay by not applying an orthogonal transform to the block.

[0095] Furthermore, the characteristics of a block to which an orthogonal transform is applied and the characteristics of a block to which an orthogonal transform is not applied are different from each other. The encoding method used for a block to which an orthogonal transform is applied and the encoding method used for a block to which an orthogonal transform is not applied can be different from each other.

[0096] However, in a case where an inappropriate encoding method is used for a block to which an orthogonal transform is applied, or in a case where an inappropriate encoding method is used for a block to which an orthogonal transform is not applied, an increase in the amount of encoding or an increase in processing delay, or the like can occur. Furthermore, in a case where there is a large difference between the encoding method used for a block to which an orthogonal transform is applied and the encoding method used for a block to which an orthogonal transform is not applied, processing can be complicated, and the circuit size can increase.

[0097] Therefore, for example, an encoding device of one aspect of the present technology includes a circuit and a memory connected to the circuit, the circuit encodes a block of an image while limiting the number of times of processing of context adaptive encoding in operation, in encoding of the block, in both a case where an orthogonal transform is applied to the block and a case where an orthogonal transform is not applied to the block, a sub-block flag encoding process is performed without being included in the number of times of processing, the sub-block flag encoding process encodes a sub-block flag indicating whether or not a sub-block included in the block includes a non-zero coefficient by context adaptive encoding.

[0098] Thus, regardless of whether or not an orthogonal transform is applied and regardless of the number of times of processing of context adaptive encoding being limited, a sub-block flag is likely to be encoded by context adaptive encoding. Therefore, the amount of encoding is likely to be reduced. Furthermore, the difference between the encoding method used for a block to which an orthogonal transform is applied and the encoding method used for a block to which an orthogonal transform is not applied is likely to be small, and the circuit size is likely to be small.

[0099] Further, for example, the circuit further performs, without including in the number of times of processing, a position parameter encoding process that encodes, by context adaptive encoding, a parameter indicating a position of a first non-zero coefficient in the block in a case where the orthogonal transform is applied to the block.

[0100] Therefore, in a case where the orthogonal transform is applied, the parameter indicating the position of the first non-zero coefficient is likely to be encoded by the context adaptive encoding regardless of whether the number of times of processing of the context adaptive encoding is limited. Therefore, the amount of encoding is likely to be reduced.

[0101] Further, for example, the circuit further determines a range of limitation of the number of times of processing by the position of the first non-zero coefficient in a case where the orthogonal transform is applied to the block.

[0102] Therefore, in a case where the orthogonal transform is applied, it is possible to appropriately determine the number of times of limitation of processing. Therefore, it is possible to appropriately adjust a balance between reduction of the amount of encoding and reduction of processing delay.

[0103] Further, for example, a decoding device of an aspect of the present technology includes a circuit and a memory connected to the circuit, the circuit decodes a block of an image while limiting a number of times of processing of context adaptive decoding in operation, and in decoding of the block, performs, without including in the number of times of processing, a sub-block flag decoding process that decodes, by context adaptive decoding, a sub-block flag indicating whether a sub-block included in the block includes a non-zero coefficient in both a case where an inverse orthogonal transform is applied to the block and a case where the inverse orthogonal transform is not applied to the block.

[0104] Thus, it is possible to decode the sub-block flag by the context adaptive decoding regardless of whether the inverse orthogonal transform is applied and whether the number of times of processing of the context adaptive decoding is limited. Therefore, the amount of encoding is likely to be reduced. Further, a difference between a decoding method used in a block to which the inverse orthogonal transform is applied and a decoding method used in a block to which the inverse orthogonal transform is not applied is likely to be small, and a circuit scale is likely to be small.

[0105] Further, for example, the circuit further performs, without including in the number of times of processing, a position parameter decoding process that encodes, by context adaptive encoding, a parameter indicating a position of a first non-zero coefficient in the block in a case where the orthogonal transform is applied to the block.

[0106] Thus, in the case where the inverse orthogonal transform is applied, the parameter indicating the position of the first nonzero coefficient is likely to be decoded by the context adaptive decoding regardless of whether or not the number of times of processing of the context adaptive decoding is limited. Therefore, the amount of encoding is likely to be reduced.

[0107] Further, for example, the circuit further determines the range of limitation of the number of times of processing by the position of the first nonzero coefficient in the case where the inverse orthogonal transform is applied to the block.

[0108] Thus, in the case where the inverse orthogonal transform is applied, the number of times of processing is likely to be appropriately determined. Therefore, the balance between the reduction of the amount of encoding and the reduction of the processing delay is likely to be appropriately adjusted.

[0109] Further, for example, an encoding method of one aspect of the present application encodes a block of an image while limiting the number of times of processing of context adaptive encoding, and in the encoding of the block, in both a case where an orthogonal transform is applied to the block and a case where the orthogonal transform is not applied to the block, subblock flag encoding processing is performed without being included in the number of times of processing, the subblock flag encoding processing encoding, by context adaptive encoding, a subblock flag indicating whether or not a subblock included in the block includes a nonzero coefficient.

[0110] Thus, in both the case where the orthogonal transform is applied and the case where the number of times of processing of the context adaptive encoding is limited, the subblock flag is likely to be encoded by the context adaptive encoding. Therefore, the amount of encoding is likely to be reduced. Further, the difference between the encoding manner used in the block to which the orthogonal transform is applied and the encoding manner used in the block to which the orthogonal transform is not applied is likely to be small, and the circuit scale is likely to be small.

[0111] Further, for example, a decoding method of one aspect of the present application decodes a block of an image while limiting the number of times of processing of context adaptive decoding, and in the decoding of the block, in both a case where an inverse orthogonal transform is applied to the block and a case where the inverse orthogonal transform is not applied to the block, subblock flag decoding processing is performed without being included in the number of times of processing, the subblock flag decoding processing decoding, by context adaptive decoding, a subblock flag indicating whether or not a subblock included in the block includes a nonzero coefficient.

[0112] Thus, in both the case where the inverse orthogonal transform is applied and the case where the number of times of processing of the context adaptive decoding is limited, the subblock flag is likely to be decoded by the context adaptive decoding. Therefore, the amount of encoding is likely to be reduced. Further, the difference between the decoding manner used in the block to which the inverse orthogonal transform is applied and the decoding manner used in the block to which the inverse orthogonal transform is not applied is likely to be small, and the circuit scale is likely to be small.

[0113] Further, for example, an encoding device of an aspect of the present application includes a circuit and a memory connected to the circuit, and in operation, in a case where orthogonal transform is applied to a block of an image to be encoded and in a case where the orthogonal transform is not applied to the block, in a case where the number of times of processing by context adaptive encoding is within a limit range of the number of times of processing, encodes a coefficient information flag indicating a property of a coefficient included in the block by context adaptive encoding, in a case where the number of times of processing is not within the limit range of the number of times of processing, skips encoding of the coefficient information flag, in a case where the coefficient information flag is encoded, encodes residual value information for reconstructing a value of the coefficient using the coefficient information flag by Golomb encoding, and in a case where encoding of the coefficient information flag is skipped, encodes the value of the coefficient by Golomb encoding.

[0114] Thus, it is possible to skip encoding of the coefficient information flag in accordance with the limit of the number of times of processing by context adaptive encoding, regardless of whether or not the orthogonal transform is applied. Therefore, it is possible to suppress an increase in processing delay and to suppress an increase in the amount of encoding. Further, it is possible to make a difference between the encoding method used in the block to which the orthogonal transform is applied and the encoding method used in the block to which the orthogonal transform is not applied small, and to make the circuit scale small.

[0115] Further, for example, the coefficient information flag is a flag indicating whether or not the value of the coefficient is greater than 1.

[0116] Thus, it is possible to skip encoding of the coefficient information flag indicating whether or not the value of the coefficient is greater than 1 in accordance with the limit of the number of times of processing by context adaptive encoding, regardless of whether or not the orthogonal transform is applied. Therefore, it is possible to suppress an increase in processing delay and to suppress an increase in the amount of encoding.

[0117] Further, for example, a decoding device of an aspect of the present application includes a circuit and a memory connected to the circuit, and in operation, in a case where inverse orthogonal transform is applied to a block of an image to be decoded and in a case where the inverse orthogonal transform is not applied to the block, in a case where the number of times of processing by context adaptive decoding is within a limit range of the number of times of processing, decodes a coefficient information flag indicating a property of a coefficient included in the block by context adaptive decoding, in a case where the number of times of processing is not within the limit range of the number of times of processing, skips decoding of the coefficient information flag, in a case where the coefficient information flag is decoded, decodes residual value information for reconstructing a value of the coefficient using the coefficient information flag by Golomb decoding, and in a case where decoding of the coefficient information flag is skipped, decodes the value of the coefficient by Golomb decoding.

[0118] Thus, it is possible to skip decoding of the coefficient information flag in accordance with the limit on the number of times of processing of context-adaptive decoding, regardless of whether or not the inverse orthogonal transform is applied. Therefore, it is possible to suppress an increase in processing delay and an increase in the amount of encoding. Furthermore, it is possible to make a difference between the decoding method used in the block in which the inverse orthogonal transform is applied and the decoding method used in the block in which the inverse orthogonal transform is not applied smaller, and it is possible to make the circuit size smaller.

[0119] Furthermore, for example, the coefficient information flag is a flag indicating whether or not the value of the coefficient is greater than 1.

[0120] Thus, it is possible to skip decoding of the coefficient information flag indicating whether or not the value of the coefficient is greater than 1 in accordance with the limit on the number of times of processing of context-adaptive decoding, regardless of whether or not the inverse orthogonal transform is applied. Therefore, it is possible to suppress an increase in processing delay and an increase in the amount of encoding.

[0121] Furthermore, for example, the coefficient information flag is a flag indicating whether or not the value of the coefficient is greater than 1.

[0122] Thus, it is possible to skip decoding of the coefficient information flag in accordance with the limit on the number of times of processing of context-adaptive decoding, regardless of whether or not the inverse orthogonal transform is applied. Therefore, it is possible to suppress an increase in processing delay and an increase in the amount of encoding. Furthermore, it is possible to make a difference between the decoding method used in the block in which the inverse orthogonal transform is applied and the decoding method used in the block in which the inverse orthogonal transform is not applied smaller, and it is possible to make the circuit size smaller.

[0123] Further, for example, a decoding method of an aspect of the present application, in a case where inverse orthogonal transform is applied to a block of a decoding target image and in a case where inverse orthogonal transform is not applied to the block, decodes, in a case where the number of times of processing of context adaptive decoding is within a limit range of the number of times of processing, a coefficient information flag indicating a property of a coefficient included in the block by context adaptive decoding, skips decoding of the coefficient information flag in a case where the number of times of processing is not within the limit range of the number of times of processing, decodes, in a case where the coefficient information flag is decoded, residual value information for reconstructing a value of the coefficient using the coefficient information flag by Golomb-Rice decoding, and decodes, in a case where decoding of the coefficient information flag is skipped, the value of the coefficient by Golomb-Rice decoding.

[0124] Thus, it is possible to skip decoding of the coefficient information flag in accordance with the limit of the number of times of processing of context adaptive decoding, regardless of whether or not inverse orthogonal transform is applied. Therefore, it is possible to suppress an increase in processing delay and to suppress an increase in the amount of encoding. Further, it is possible to make a difference between a decoding method used in a block to which inverse orthogonal transform is applied and a decoding method used in a block to which inverse orthogonal transform is not applied smaller, and to make a circuit size smaller.

[0125] Further, for example, an encoding device of an aspect of the present application includes a circuit and a memory connected to the circuit, and in operation, limits the number of times of processing of context adaptive encoding to encode a block of an image, and in the encoding of the block, in a case where orthogonal transform is not applied to the block, determines whether a plurality of coefficient information flags respectively indicating a plurality of properties of a coefficient included in the block satisfy a processing condition, in a case where it is determined that the processing condition is satisfied, encodes the plurality of coefficient information flags by context adaptive encoding, the processing condition being a condition in which the number of times of processing is within a limit range of the number of times of processing in a case where the number of times of processing is added to the number of the plurality of coefficient information flags.

[0126] Thus, in a case where orthogonal transform is not applied, it is possible to collectively determine whether or not it is possible to use context adaptive encoding with respect to a plurality of coefficient information flags. Therefore, it is possible to simplify processing and to reduce processing delay. Further, in a case where similar processing is performed with respect to a block to which orthogonal transform is applied, it is possible to make a difference between an encoding method used in a block to which orthogonal transform is applied and an encoding method used in a block to which orthogonal transform is not applied smaller, and to make a circuit size smaller.

[0127] Further, for example, the plurality of coefficient information flags include a coefficient information flag indicating whether or not a value of the coefficient is greater than 3 and a coefficient information flag indicating whether or not a value of the coefficient is greater than 5.

[0128] Thus, it is possible to make a general determination of a plurality of coefficient information flags including a coefficient information flag indicating whether the value of the coefficient is greater than 3 and a coefficient information flag indicating whether the value of the coefficient is greater than 5. Therefore, it is possible to simplify the processing, and it is possible to reduce the processing delay.

[0129] In addition, for example, the plurality of coefficient information flags further include a coefficient information flag indicating whether the value of the coefficient is greater than 7 and a coefficient information flag indicating whether the value of the coefficient is greater than 9.

[0130] Thus, it is possible to make a general determination of a plurality of coefficient information flags including a coefficient information flag indicating whether the value of the coefficient is greater than 3, a coefficient information flag indicating whether the value of the coefficient is greater than 5, a coefficient information flag indicating whether the value of the coefficient is greater than 7, and a coefficient information flag indicating whether the value of the coefficient is greater than 9. Therefore, it is possible to simplify the processing, and it is possible to reduce the processing delay.

[0131] Further, for example, a decoding device of an aspect of the present application includes a circuit and a memory connected to the circuit, and the circuit, in operation, limits the number of times of context-adaptive decoding to decode a block of an image, and in the decoding of the block, in a case where inverse orthogonal transform is not applied to the block, determines whether a plurality of coefficient information flags respectively indicating a plurality of properties of coefficients included in the block satisfy a processing condition, and in a case where it is determined that the processing condition is satisfied, decodes the plurality of coefficient information flags by context-adaptive decoding, the processing condition being a condition in which the number of times of processing, in a case where the number of times of processing is added to the number of the plurality of coefficient information flags, is within a range of limitation of the number of times of processing.

[0132] Thus, it is possible to make a general determination of a plurality of coefficient information flags respectively indicating a plurality of properties of coefficients included in the block. Therefore, it is possible to simplify the processing, and it is possible to reduce the processing delay. Further, in a case where similar processing is performed on a block to which inverse orthogonal transform is applied, it is possible to make the difference between the decoding method used in the block to which inverse orthogonal transform is applied and the decoding method used in the block to which inverse orthogonal transform is not applied smaller, and it is possible to make the circuit scale smaller.

[0133] In addition, for example, the plurality of coefficient information flags include a coefficient information flag indicating whether the value of the coefficient is greater than 3 and a coefficient information flag indicating whether the value of the coefficient is greater than 5.

[0134] Thus, it is possible to make a general determination of a plurality of coefficient information flags including a coefficient information flag indicating whether the value of the coefficient is greater than 3 and a coefficient information flag indicating whether the value of the coefficient is greater than 5. Therefore, it is possible to simplify the processing, and it is possible to reduce the processing delay.

[0135] In addition, for example, the plurality of coefficient information flags include a coefficient information flag indicating whether the value of the coefficient is greater than 7 and a coefficient information flag indicating whether the value of the coefficient is greater than 9.

[0136] Thus, it is possible to make a general determination of the plurality of coefficient information flags including the four coefficient information flags indicating whether the value of the coefficient is greater than 3, whether the value of the coefficient is greater than 5, whether the value of the coefficient is greater than 7, and whether the value of the coefficient is greater than 9. Therefore, it is possible to simplify the processing and reduce the processing delay.

[0137] Further, for example, an encoding method of an aspect of the present application encodes a block of an image while limiting the number of times of processing of context adaptive encoding, and in the encoding of the block, in a case where the block is not subjected to orthogonal transform, determines whether a plurality of coefficient information flags respectively indicating a plurality of properties of a coefficient included in the block satisfy a processing condition, encodes the plurality of coefficient information flags by context adaptive encoding in a case where it is determined that the processing condition is satisfied, the processing condition being a condition in which the number of times of processing, to which the number of the plurality of coefficient information flags is added, is within a range of limitation of the number of times of processing.

[0138] Thus, in a case where the orthogonal transform is not applied, it is possible to make a general determination of the plurality of coefficient information flags as to whether it is possible to use context adaptive encoding. Therefore, it is possible to simplify the processing and reduce the processing delay. Further, in a case where similar processing is performed on a block to which the orthogonal transform is applied, it is possible to make the difference between the encoding method used in the block to which the orthogonal transform is applied and the encoding method used in the block to which the orthogonal transform is not applied small, and it is possible to make the circuit size small.

[0139] Further, for example, a decoding method of an aspect of the present application decodes a block of an image while limiting the number of times of processing of context adaptive decoding, and in the decoding of the block, in a case where the block is not subjected to inverse orthogonal transform, determines whether a plurality of coefficient information flags respectively indicating a plurality of properties of a coefficient included in the block satisfy a processing condition, decodes the plurality of coefficient information flags by context adaptive decoding in a case where it is determined that the processing condition is satisfied, the processing condition being a condition in which the number of times of processing, to which the number of the plurality of coefficient information flags is added, is within a range of limitation of the number of times of processing.

[0140] Thus, without applying the inverse orthogonal transform, it is possible to comprehensively determine whether or not it is possible to use context adaptive decoding for a plurality of coefficient information flags. Therefore, the processing can be simplified, and the processing delay can be reduced. Further, in the case where similar processing is performed on a block to which the inverse orthogonal transform is applied, the difference between the decoding method used in the block to which the inverse orthogonal transform is applied and the decoding method used in the block to which the inverse orthogonal transform is not applied can be small, and the circuit size can be small.

[0141] Further, for example, an encoding apparatus of an aspect of the present application includes a division section, an intra prediction section, an inter prediction section, a prediction control section, a transform section, a quantization section, an entropy encoding section, and a loop filter section.

[0142] The division section divides an encoding target picture constituting a moving image into a plurality of blocks. The intra prediction section performs intra prediction that generates a prediction picture of an encoding target block in the encoding target picture using a reference picture in a reference picture different from the encoding target picture. The inter prediction section performs inter prediction that generates the prediction picture of the encoding target block using a reference picture in the reference picture different from the encoding target picture.

[0143] The prediction control section controls the intra prediction performed by the intra prediction section and the inter prediction performed by the inter prediction section. The transform section transforms a prediction residual signal between the prediction picture generated by the intra prediction section or the inter prediction section and a picture of the encoding target block, and generates a transform coefficient signal of the encoding target block. The quantization section quantizes the transform coefficient signal. The entropy encoding section encodes the quantized transform coefficient signal. The loop filter section applies a filter to the encoding target block.

[0144] Further, for example, the entropy encoding section encodes a block of a picture while limiting the number of times of processing of context adaptive encoding in the operation, and in encoding of the block, in both a case where an orthogonal transform is applied to the block and a case where the orthogonal transform is not applied to the block, sub-block flag encoding processing that encodes, by context adaptive encoding, a sub-block flag indicating whether or not a sub-block included in the block includes a non-zero coefficient is performed without being included in the number of times.

[0145] Further, for example, the entropy encoding section encodes, in operation, a coefficient information flag indicating properties of coefficients included in a block of an image to be encoded by context adaptive encoding in a case where a number of times of processing of context adaptive encoding is within a limit range of the number of times of processing in both a case where orthogonal transform is applied to the block and a case where orthogonal transform is not applied to the block, skips encoding of the coefficient information flag in a case where the number of times of processing is not within the limit range of the number of times of processing, encodes, in a case where the coefficient information flag is encoded, residual value information for reconstructing values of the coefficients using the coefficient information flag by Golomb encoding, and encodes, in a case where encoding of the coefficient information flag is skipped, the values of the coefficients by Golomb encoding.

[0146] Further, for example, the entropy encoding section encodes, in operation, a block of an image while limiting a number of times of processing of context adaptive encoding, determines, in encoding of the block, whether a plurality of coefficient information flags respectively indicating a plurality of properties of coefficients included in the block satisfy a processing condition in a case where orthogonal transform is not applied to the block, encodes, in a case where it is determined that the processing condition is satisfied, the plurality of coefficient information flags by context adaptive encoding, the processing condition being a condition where the number of times of processing is within a limit range of the number of times of processing in a case where the number of times of processing is added to a number of the plurality of coefficient information flags.

[0147] Further, for example, a decoding apparatus of one aspect of the present application decodes a moving image using a prediction image, the decoding apparatus including an entropy decoding section, an inverse quantization section, an inverse transform section, an intra prediction section, an inter prediction section, a prediction control section, an addition section (reconstruction section), and a loop filter section.

[0148] The entropy decoding section decodes a quantized transform coefficient signal of a decoding target block in a decoding target picture constituting the moving image. The inverse quantization section inverse quantizes the quantized transform coefficient signal. The inverse transform section inverse transforms the transform coefficient signal to obtain a prediction residual signal of the decoding target block.

[0149] The intra prediction section performs intra prediction using a reference image in the decoding target picture to generate the prediction image of the decoding target block. The inter prediction section performs inter prediction using a reference image in a reference picture different from the decoding target picture to generate the prediction image of the decoding target block. The prediction control section controls the intra prediction performed by the intra prediction section and the inter prediction performed by the inter prediction section.

[0150] The addition section reconstructs an image of the decoding target block by adding the prediction image generated by the intra prediction section or the inter prediction section to the prediction residual signal. The loop filter section applies a filter to the decoding target block.

[0151] Further, for example, the entropy decoding section decodes a block of an image while limiting the number of times of processing of context adaptive decoding, in decoding of the block, in a case where inverse orthogonal transform is applied to the block and in a case where inverse orthogonal transform is not applied to the block, performs sub-block flag decoding processing which decodes, by context adaptive decoding, a sub-block flag indicating whether or not a sub-block included in the block includes a non-zero coefficient, without including the sub-block flag decoding processing in the number of times of processing.

[0152] Further, for example, the entropy decoding section decodes a block of an image while limiting the number of times of processing of context adaptive decoding, in decoding of the block, in a case where inverse orthogonal transform is applied to the block and in a case where inverse orthogonal transform is not applied to the block, performs sub-block flag decoding processing which decodes, by context adaptive decoding, a sub-block flag indicating whether or not a sub-block included in the block includes a non-zero coefficient, without including the sub-block flag decoding processing in the number of times of processing.

[0153] Further, for example, the entropy decoding section decodes a block of an image while limiting the number of times of processing of context adaptive decoding, in decoding of the block, in a case where inverse orthogonal transform is applied to the block and in a case where inverse orthogonal transform is not applied to the block, performs sub-block flag decoding processing which decodes, by context adaptive decoding, a sub-block flag indicating whether or not a sub-block included in the block includes a non-zero coefficient, without including the sub-block flag decoding processing in the number of times of processing.

[0154] Moreover, these inclusive or specific modes can be realized by a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a CD-ROM readable by a computer, and can be realized by any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0155] Hereinafter, the embodiments will be specifically described with reference to the drawings. In addition, the embodiments described below each show an inclusive or specific example. The numerical values, shapes, materials, component configurations, arrangement positions and connection modes of components, steps, relationships and orders of steps, and the like shown in the embodiments below are one example, and are not intended to limit the scope of the claims.

[0156] Hereinafter, the embodiments of the encoding apparatus and the decoding apparatus will be described. The embodiments are examples of the encoding apparatus and the decoding apparatus capable of applying the processes and / or structures described in each aspect of the present application. The processes and / or structures can be implemented in the encoding apparatus and the decoding apparatus different from the embodiments. For example, with respect to the processes and / or structures applied to the embodiments, for example, any one of the following can be performed.

[0157] (1) Any one of the plurality of components of the encoding apparatus or the decoding apparatus of the embodiments described in each aspect of the present application can be replaced with another component described in any one of the aspects of the present application, or a combination thereof;

[0158] (2) In the encoding apparatus or the decoding apparatus of the embodiments, arbitrary changes such as addition, replacement, deletion, and the like of the functions or processes performed by part of the plurality of components of the encoding apparatus or the decoding apparatus can be made. For example, any one of the functions or processes is replaced with another function or process described in any one of the aspects of the present application, or a combination thereof;

[0159] (3) In the method implemented by the encoding apparatus or the decoding apparatus of the embodiments, arbitrary changes such as addition, replacement, deletion, and the like can be made with respect to part of the plurality of processes included in the method. For example, any one of the processes in the method is replaced with another process described in any one of the aspects of the present application, or a combination thereof;

[0160] (4) Part of the plurality of components of the encoding apparatus or the decoding apparatus of the embodiments can be combined with a component described in any one of the aspects of the present application, a component having part of the functions described in any one of the aspects of the present application, or a component implementing part of the processes implemented by the components described in any one of the aspects of the present application;

[0161] (5) A configuration element that has a part of the function of the encoding apparatus or the decoding apparatus of the embodiment, or a configuration element that performs a part of the processing of the encoding apparatus or the decoding apparatus of the embodiment, is combined with or replaced by a configuration element described in any of the aspects of the present application, a configuration element that has a part of the function described in any of the aspects of the present application, or a configuration element that performs a part of the processing described in any of the aspects of the present application.

[0162] (6) In the method performed by the encoding apparatus or the decoding apparatus of the embodiment, a part of the plurality of processes included in the method is replaced by a process described in any of the aspects of the present application or the same process, or a combination thereof.

[0163] (7) A part of the plurality of processes included in the method performed by the encoding apparatus or the decoding apparatus of the embodiment can also be combined with the process described in any of the aspects of the present application.

[0164] (8) The aspects of the present application are not limited to the encoding apparatus or the decoding apparatus of the embodiment. For example, the processes and / or structures can also be implemented in an apparatus used for a purpose different from the motion picture encoding or the motion picture decoding disclosed in the embodiment.

[0165] [Encoding Apparatus]

[0166] First, the encoding apparatus of the embodiment will be described. Figure 1 is a block diagram showing the functional structure of the encoding apparatus 100 of the embodiment. The encoding apparatus 100 is a motion picture encoding apparatus that encodes a motion picture in a block unit.

[0167] As shown in Figure 1 , the encoding apparatus 100 is an apparatus that encodes an image in a block unit, and includes a division section 102, a subtraction section 104, a transform section 106, a quantization section 108, an entropy encoding section 110, an inverse quantization section 112, an inverse transform section 114, an addition section 116, a block memory 118, a loop filter section 120, a frame memory 122, an intra prediction section 124, an inter prediction section 126, and a prediction control section 128.

[0168] The encoding apparatus 100 is realized by, for example, a general-purpose processor and a memory. In this case, when a software program stored in the memory is executed by the processor, the processor functions as the division section 102, the subtraction section 104, the transform section 106, the quantization section 108, the entropy encoding section 110, the inverse quantization section 112, the inverse transform section 114, the addition section 116, the loop filter section 120, the intra prediction section 124, the inter prediction section 126, and the prediction control section 128. Further, the encoding apparatus 100 can also be realized as one or more electronic circuits dedicated to the division section 102, the subtraction section 104, the transform section 106, the quantization section 108, the entropy encoding section 110, the inverse quantization section 112, the inverse transform section 114, the addition section 116, the loop filter section 120, the intra prediction section 124, the inter prediction section 126, and the prediction control section 128.

[0169] Hereinafter, after the flow of the overall processing of the encoding apparatus 100 is explained, each constituent element included in the encoding apparatus 100 is explained.

[0170] [Overall flow of encoding processing]

[0171] Figure 2 is a flowchart showing an example of the overall encoding processing performed by the encoding apparatus 100.

[0172] First, the division section 102 of the encoding apparatus 100 divides each picture included in an input image that is a moving image into a plurality of blocks of a fixed size (for example, 128 x 128 pixels) (step Sa_1). Then, the division section 102 selects a division pattern (also referred to as a block shape) for the blocks of the fixed size (step Sa_2). That is, the division section 102 further divides the blocks of the fixed size into a plurality of blocks constituting the selected division pattern. Then, the encoding apparatus 100 performs the processing of steps Sa_3 to Sa_9 for each of the plurality of blocks, on the block (i.e., the encoding target block).

[0173] That is, a prediction processing section constituted by all or a part of the intra prediction section 124, the inter prediction section 126, and the prediction control section 128 generates a prediction signal (also referred to as a prediction block) of the encoding target block (also referred to as a current block) (step Sa_3).

[0174] Next, the subtraction section 104 generates a difference between the encoding target block and the prediction block as a prediction residual (also referred to as a difference block) (step Sa_4).

[0175] Next, the transform section 106 and the quantization section 108 generate a plurality of quantization coefficients by performing transform and quantization on the difference block (step Sa_5). Further, a block constituted by the plurality of quantization coefficients is also referred to as a coefficient block.

[0176] Next, the entropy coding section 110 generates an encoded signal by encoding (specifically, entropy encoding) the coefficient block and a prediction parameter related to the generation of the prediction signal (step Sa_6). The encoded signal is also referred to as an encoded bitstream, a compressed bitstream, or a stream.

[0177] Next, the inverse quantization section 112 and the inverse transform section 114 reproduce a plurality of prediction residuals (i.e., difference blocks) by inverse quantizing and inverse transforming the coefficient block (step Sa_7).

[0178] Next, the addition section 116 reconstructs the current block into a reconstructed image (also referred to as a reconstructed block or a decoded image block) by adding the prediction block to the reproduced difference block (step Sa_8). Thereby, the reconstructed image is generated.

[0179] When the reconstructed image is generated, the loop filter 120 filters the reconstructed image as necessary (step Sa_9).

[0180] Then, the encoding apparatus 100 determines whether or not the encoding of the entire picture has been completed (step Sa_10), and in the case where it is determined that the encoding has not been completed (NO in step Sa_10), the processing from step Sa_2 is repeated.

[0181] In addition, in the example described above, the encoding apparatus 100 selects one partitioning pattern for the fixed-size block and encodes each block according to the partitioning pattern, but each block can be encoded according to each of a plurality of partitioning patterns. In this case, the encoding apparatus 100 can evaluate the cost for each of the plurality of partitioning patterns, and for example, can select the encoded signal obtained by encoding according to the partitioning pattern of the smallest cost as the output encoded signal.

[0182] As illustrated, the processing of these steps Sa_1 to Sa_10 is sequentially performed by the encoding apparatus 100. Alternatively, a part of the plurality of processes in these processes can be performed in parallel, or the order of these processes can be changed.

[0183] [Partitioning Section]

[0184] The division section 102 divides each picture included in the input moving image into a plurality of blocks, and outputs each block to the subtraction section 104. For example, the division section 102 first divides a picture into blocks of a fixed size (e.g., 128 x 128). Other fixed block sizes can also be used. The fixed-size blocks are referred to as coding tree units (CTUs). Also, the division section 102 divides each of the fixed-size blocks into blocks of variable sizes (e.g., 64 x 64 or less) based on, for example, recursive quadtree and / or binary tree block division. That is, the division section 102 selects a division pattern. The variable-size blocks are referred to as coding units (CUs), prediction units (PUs), or transform units (TUs). In addition, in various processing examples, CUs, PUs, and TUs do not need to be distinguished, and a portion or all of the blocks within a picture can be used as processing units of CUs, PUs, and TUs.

[0185] Figure 3 is a conceptual diagram showing an example of block division in an embodiment. In Figure 3 , a solid line indicates a block boundary based on quadtree block division, and a dashed line indicates a block boundary based on binary tree block division.

[0186] Here, the block 10 is a square block of 128 x 128 pixels (128 x 128 block). The 128 x 128 block 10 is first divided into four square 64 x 64 blocks (quadtree block division).

[0187] The upper left 64 x 64 block is further vertically divided into two rectangular 32 x 64 blocks, and the left 32 x 64 block is further vertically divided into two rectangular 16 x 64 blocks (binary tree block division). As a result, the upper left 64 x 64 block is divided into two 16 x 64 blocks 11 and 12 and a 32 x 64 block 13.

[0188] The upper right 64 x 64 block is horizontally divided into two rectangular 64 x 32 blocks 14 and 15 (binary tree block division).

[0189] The lower left 64 x 64 block is divided into four square 32 x 32 blocks (quadtree block division). The upper left block and the lower right block among the four 32 x 32 blocks are further divided. The upper left 32 x 32 block is vertically divided into two rectangular 16 x 32 blocks, and the right 16 x 32 block is further horizontally divided into two 16 x 16 blocks (binary tree block division). The lower right 32 x 32 block is horizontally divided into two 32 x 16 blocks (binary tree block division). As a result, the lower left 64 x 64 block is divided into a 16 x 32 block 16, two 16 x 16 blocks 17 and 18, two 32 x 32 blocks 19 and 20, and two 32 x 16 blocks 21 and 22.

[0190] The 64x64 block 23 at the lower right is not split.

[0191] As above, in Figure 3 , the block 10 is split into 13 blocks 11 to 23 of variable sizes based on the recursive quad-tree and binary-tree block splitting. Such splitting is referred to as QTBT (quad-tree plus binary tree) splitting.

[0192] In addition, in Figure 3 , 1 block is split into 4 or 2 blocks (quad-tree or binary-tree block splitting), but the splitting is not limited to these. For example, it can be that 1 block is split into 3 blocks (ternary-tree splitting). The splitting including such ternary-tree splitting is referred to as MBT (multi type tree) splitting.

[0193] [Structure slice / tile of picture]

[0194] In order to decode a picture in parallel, a picture is sometimes configured in a slice unit or in a tile unit. A picture configured in a slice unit or a tile unit can be configured by the splitting section 102.

[0195] A slice is a basic unit of encoding that configures a picture. A picture is configured by 1 or more slices, for example. In addition, a slice is configured by 1 or more consecutive CTUs (Coding Tree Units).

[0196] Figure 4A is a conceptual diagram that shows an example of the structure of a slice. For example, a picture includes 11x8 CTUs, and is split into 4 slices (slices 1 to 4). Slice 1 is configured by 16 CTUs, slice 2 is configured by 21 CTUs, slice 3 is configured by 29 CTUs, and slice 4 is configured by 22 CTUs. Here, each CTU within the picture belongs to any one of the slices. The shape of the slice becomes a shape that splits the picture in the horizontal direction. The boundary of the slice need not be the picture end, and can be an arbitrary position in the boundary of the CTU within the picture. The processing order (encoding order or decoding order) of the CTUs in the slice is, for example, a raster scan order. Furthermore, a slice contains header information and encoded data. In the header information, the CTU address of the beginning of the slice, the slice type, and the like, which are characteristics of the slice, can be described.

[0197] A tile is a unit of a rectangular region that configures a picture. Each tile can be assigned a number called TileId in a raster scan order.

[0198] Figure 4Bis a conceptual diagram showing an example of a structure of a tile. For example, a picture includes 11 x 8 CTUs, and is divided into 4 tiles (tiles 1 to 4) of rectangular regions. In a case where tiles are used, the processing order of CTUs is changed compared to a case where tiles are not used. In a case where tiles are not used, a plurality of CTUs within a picture are processed in a raster scan order. In a case where tiles are used, in each of a plurality of tiles, at least one CTU is processed in a raster scan order. For example, as shown in Figure 4B

[0199] In addition, one tile sometimes contains one or more slices, and one slice sometimes contains one or more tiles.

[0200] [Subtracting section]

[0201] The subtracting section 104 subtracts a prediction signal (a prediction sample input from the prediction control section 128 shown below) from an original signal (an original sample) in units of blocks input from and divided by the dividing section 102. That is, the subtracting section 104 calculates a prediction error (also called a residual) of a coding target block (hereinafter referred to as a current block). Also, the subtracting section 104 outputs the calculated prediction error (residual) to the transforming section 106.

[0202] The original signal is an input signal of the encoding apparatus 100, and is a signal (for example, a luma signal and two chroma signals) showing an image constituting each picture of a moving image. Hereinafter, there is also a case where the signal showing the image is referred to as a sample.

[0203] [Transforming section]

[0204] The transforming section 106 transforms the prediction error in the spatial domain into a transform coefficient in the frequency domain, and outputs the transform coefficient to the quantizing section 108. Specifically, the transforming section 106, for example, performs a predetermined discrete cosine transform (DCT) or a discrete sine transform (DST) on the prediction error in the spatial domain. The predetermined DCT or DST can also be determined in advance.

[0205] In addition, the transforming section 106 can also adaptively select a transform type from among a plurality of transform types, and transform the prediction error into a transform coefficient using a transform basis function corresponding to the selected transform type. Such a transform has a case where it is called EMT (explicit multiple core transform) or AMT (adaptive multiple transform).​

[0206] The plurality of transform types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII. Figure 5A is a table indicating a transform basis function corresponding to each of the transform types. In Figure 5A In, N indicates the number of input pixels. The selection of the transform type from among the plurality of transform types can depend, for example, on the kind of prediction (intra prediction and inter prediction) or on the intra prediction mode.

[0207] The information indicating whether to apply such EMT or AMT (for example, referred to as an EMT flag or an AMT flag) and the information indicating the selected transform type are generally signaled at the CU level. In addition, the signaling of these information does not need to be limited to the CU level, but can be another level (for example, a bit sequence level, a picture level, a slice level, a tile level, or a CTU level).

[0208] Further, the transform section 106 can also perform a re-transformation on the transform coefficients (transform result). Such a re-transformation has a case called AST (adaptive secondary transform) or NSST (non-separable secondary transform). For example, the transform section 106 performs a re-transformation on each sub-block (for example, a 4x4 sub-block) included in a block of transform coefficients corresponding to the intra prediction error. The information indicating whether to apply NSST and the information related to the transform matrix used in NSST are generally signaled at the CU level. In addition, the signaling of these information does not need to be limited to the CU level, but can be another level (for example, a sequence level, a picture level, a slice level, a tile level, or a CTU level).

[0209] The transform section 106 can apply both a separable transform and a non-separable transform. The separable transform refers to a manner of performing multiple transformations by separating each direction corresponding to the number of dimensions of the input. The non-separable transform refers to a manner of performing a transformation on two or more dimensions together by regarding them as one dimension when the input is multi-dimensional.

[0210] For example, as an example of the non-separable transform, there is a case where, when the input is a 4x4 block, it is regarded as one arrangement having 16 elements, and a transform process is performed on the arrangement with a 16x16 transform matrix.

[0211] Furthermore, in a further example of the non-separable transformation, the 4×4 input block can be viewed as a permutation of 16 elements, and then a transformation involving multiple Givens rotations of that permutation can be performed (HypercubeGivens Transform).

[0212] In the transformation within the transformation unit 106, the type of the base to be transformed into a frequency region can be switched according to the region within the CU. As an example, there is SVT (Spatially Varying Transform). In SVT, such as... Figure 5B As shown, the CU is bisected horizontally or vertically, and only one region is transformed to the frequency region. The type of transform basis can be set for each region, such as using DST7 and DCT8. In this example, only one of the two regions within the CU is transformed, while the other is not transformed; however, both regions can also be transformed. Furthermore, the segmentation method is not limited to bisecting; it can be more flexible, such as quartering or encoding the segmentation information separately, and signaling it in the same way as the CU segmentation. Additionally, SVT is sometimes referred to as SBT (Sub-block Transform).

[0213] [Quantitative Department]

[0214] The quantization unit 108 quantizes the transform coefficients output from the transform unit 106. Specifically, the quantization unit 108 scans the transform coefficients of the current block in a predetermined scan order, quantizing the transform coefficients based on the quantization parameters (QP) corresponding to the scanned transform coefficients. Furthermore, the quantization unit 108 outputs the quantized transform coefficients (hereinafter referred to as quantized coefficients) of the current block to the entropy encoding unit 110 and the inverse quantization unit 112. The predetermined scan order can also be predetermined.

[0215] The specified scan order is the order in which the transform coefficients are quantized / dequantized. For example, the specified scan order can also be defined by ascending frequency (from low frequency to high frequency) or descending frequency (from high frequency to low frequency).

[0216] The quantization parameter (QP) is a parameter that defines the quantization step size (quantization width). For example, if the value of the quantization parameter increases, the quantization step size also increases. That is, if the value of the quantization parameter increases, the quantization error increases.

[0217] In addition, in quantization, a quantization matrix is sometimes used. For example, a plurality of quantization matrices are sometimes used in correspondence with a frequency transform size such as 4 x 4 and 8 x 8, a prediction mode such as intra prediction and inter prediction, a pixel component such as luminance and chrominance, and the like. In addition, quantization refers to digitizing a value sampled at a prescribed interval in correspondence with a prescribed level, and in this technical field, other expressions such as rounding, rounding, and scaling can be used in reference, and rounding, rounding, and scaling can be adopted. The prescribed interval and the level can also be determined in advance.

[0218] As a method of using a quantization matrix, there is a method of using a quantization matrix directly set on an encoding device side and a method of using a default quantization matrix (default matrix). On the encoding device side, by directly setting a quantization matrix, it is possible to set a quantization matrix corresponding to the characteristics of an image. However, in this case, there is a disadvantage that the amount of encoding increases due to encoding of the quantization matrix.

[0219] On the other hand, there is also a method of quantizing in such a manner that the coefficients of high frequency components and the coefficients of low frequency components are all the same without using a quantization matrix. In addition, this method is equivalent to a method of using a quantization matrix in which all the coefficients are the same value (flat matrix).

[0220] A quantization matrix can be specified by, for example, SPS (Sequence Parameter Set) or PPS (Picture Parameter Set). SPS contains parameters used for a sequence, and PPS contains parameters used for a picture. SPS and PPS are sometimes referred to simply as a parameter set.

[0221] [Entropy encoding section]

[0222] The entropy encoding section 110 generates an encoded signal (encoded bit stream) based on the quantization coefficients input from the quantization section 108. Specifically, the entropy encoding section 110, for example, binarizes the quantization coefficients, arithmetically encodes the binarized signal, and outputs a compressed bit stream or sequence.

[0223] [Inverse quantization section]

[0224] The inverse quantization section 112 inverse quantizes the quantization coefficients input from the quantization section 108. Specifically, the inverse quantization section 112 inverse quantizes the quantization coefficients of the current block in a prescribed scan order. Also, the inverse quantization section 112 outputs the inverse quantized transform coefficients of the current block to the inverse transform section 114. The prescribed scan order can also be determined in advance.

[0225] [Inverse transform section]

[0226] The inverse transform unit 114 restores the prediction error (residual) by performing inverse transform on the transform coefficients input from the inverse quantization unit 112. Specifically, the inverse transform unit 114 restores the prediction error of the current block by performing inverse transform on the transform coefficients corresponding to the transform performed by the transform unit 106. The inverse transform unit 114 outputs the restored prediction error to the addition unit 116.

[0227] In addition, the restored prediction error generally does not match the prediction error calculated by the subtraction unit 104 because information is lost by quantization. That is, the restored prediction error generally includes quantization error.

[0228] [Addition Unit]

[0229] The addition unit 116 reconstructs the current block by adding the prediction error input from the inverse transform unit 114 to the prediction sample input from the prediction control unit 128. The addition unit 116 outputs the reconstructed block to the block memory 118 and the loop filter unit 120. The reconstructed block is also referred to as a local decoded block.

[0230] [Block Memory]

[0231] The block memory 118 is, for example, a storage unit that stores a block in an encoded object picture (referred to as a current picture) that is referred to in intra prediction. Specifically, the block memory 118 stores the reconstructed block output from the addition unit 116.

[0232] [Frame Memory]

[0233] The frame memory 122 is, for example, a storage unit that stores a reference picture used in inter prediction, and is also referred to as a frame buffer. Specifically, the frame memory 122 stores the reconstructed block filtered by the loop filter unit 120.

[0234] [Loop Filter Unit]

[0235] The loop filter unit 120 applies loop filtering to the block reconstructed by the addition unit 116, and outputs the filtered reconstructed block to the frame memory 122. Loop filtering refers to filtering used within an encoding loop (in-loop filtering), and includes, for example, deblocking filtering (DF or DBF), sample adaptive offset (SAO), adaptive loop filtering (ALF), and the like.

[0236] In ALF, a least square error filter for removing encoding distortion is adopted, and, for example, one filter selected from a plurality of filters based on the direction of a gradient and the activity in each 2x2 sub-block in the current block is adopted.

[0237] Specifically, first, a sub-block (e.g., 2x2 sub-block) is classified into a plurality of classes (e.g., 15 or 25 classes). The classification of the sub-block is performed based on the direction and activity of the gradient. For example, using the direction value D (e.g., 0~2 or 0~4) of the gradient and the activity value A (e.g., 0~4) of the gradient, a classification value C (e.g., C=5D+A) is calculated. And, based on the classification value C, the sub-block is classified into a plurality of classes.

[0238] The direction value D of the gradient is derived, for example, by comparing the gradients of a plurality of directions (e.g., horizontal, vertical, and 2 diagonal directions). Further, the activity value A of the gradient is derived, for example, by adding the gradients of a plurality of directions, and quantizing the addition result.

[0239] Based on the result of such classification, a filter for the sub-block is decided from among a plurality of filters.

[0240] As the shape of the filter used in the ALF, for example, a circularly symmetric shape is used. Figure 6A-6C is a diagram showing a plurality of examples of the shape of the filter used in the ALF. Figure 6A shows a 5x5 diamond shape filter, Figure 6B shows a 7x7 diamond shape filter, Figure 6C shows a 9x9 diamond shape filter. The information showing the shape of the filter is usually signaled at the picture level. In addition, the signaling of the information showing the shape of the filter need not be limited to the picture level, and can be another level (e.g., sequence level, slice level, tile level, CTU level, or CU level).

[0241] The on / off of the ALF can also be decided, for example, at the picture level or the CU level. For example, with respect to the luminance, whether to adopt the ALF can be decided at the CU level, and with respect to the color difference, whether to adopt the ALF can be decided at the picture level. The information showing the on / off of the ALF is usually signaled at the picture level or the CU level. In addition, the signaling of the information showing the on / off of the ALF need not be limited to the picture level or the CU level, and can be another level (e.g., sequence level, slice level, tile level, or CTU level).

[0242] The coefficient set of the selectable plurality of filters (e.g., up to 15 or 25 filters) is usually signaled at the picture level. In addition, the signaling of the coefficient set need not be limited to the picture level, and can be another level (e.g., sequence level, slice level, tile level, CTU level, CU level, or sub-block level).

[0243] [Loop filter> deblocking filter]

[0244] In the deblocking filter, the loop filter 120 reduces distortion generated at the block boundary of the reconstructed image by performing a filter process on the block boundary.

[0245] Figure 7 is a block diagram showing an example of a detailed configuration of the loop filter 120 that functions as a deblocking filter.

[0246] The loop filter 120 includes a boundary determination section 1201, a filter determination section 1203, a filter processing section 1205, a processing determination section 1208, a filter characteristic decision section 1207, and switches 1202, 1204, and 1206.

[0247] The boundary determination section 1201 determines whether or not there is a pixel (i.e., an object pixel) that is subjected to deblocking filter processing in the vicinity of a block boundary. Then, the boundary determination section 1201 outputs the determination result to the switch 1202 and the processing determination section 1208.

[0248] In a case where it is determined by the boundary determination section 1201 that the object pixel exists in the vicinity of the block boundary, the switch 1202 outputs the image before the filter processing to the switch 1204. In contrast, in a case where it is determined by the boundary determination section 1201 that the object pixel does not exist in the vicinity of the block boundary, the switch 1202 outputs the image before the filter processing to the switch 1206.

[0249] The filter determination section 1203 determines whether or not the object pixel is subjected to deblocking filter processing, on the basis of pixel values of at least one peripheral pixel located in the periphery of the object pixel. Then, the filter determination section 1203 outputs the determination result to the switch 1204 and the processing determination section 1208.

[0250] In a case where it is determined by the filter determination section 1203 that the object pixel is subjected to deblocking filter processing, the switch 1204 outputs the image before the filter processing, which is acquired via the switch 1202, to the filter processing section 1205. In contrast, in a case where it is determined by the filter determination section 1203 that the object pixel is not subjected to deblocking filter processing, the switch 1204 outputs the image before the filter processing, which is acquired via the switch 1202, to the switch 1206.

[0251] In a case where the image before the filter processing is acquired via the switches 1202 and 1204, the filter processing section 1205 executes deblocking filter processing having a filter characteristic decided by the filter characteristic decision section 1207, with respect to the object pixel. Then, the filter processing section 1205 outputs the pixel after the filter processing to the switch 1206.

[0252] The switch 1206 selectively outputs the pixel that is not subjected to deblocking filter processing and the pixel that is subjected to deblocking filter processing by the filter processing section 1205, in accordance with the control of the processing determination section 1208.

[0253] The processing determination section 1208 controls the switch 1206 based on the respective determination results of the boundary determination section 1201 and the filter determination section 1203. That is, the processing determination section 1208 outputs a pixel after the deblocking filter process from the switch 1206 in a case where it is determined by the boundary determination section 1201 that the object pixel exists in the vicinity of the block boundary and it is determined by the filter determination section 1203 that the deblocking filter process is performed on the object pixel. In addition, the processing determination section 1208 outputs a pixel which is not subjected to the deblocking / filter process from the switch 1206 in a case other than the above. By repeating the output of such a pixel, an image after the filter process is output from the switch 1206.

[0254] Figure 8 is a conceptual diagram showing an example of deblocking filter having a filter characteristic symmetrical with respect to a block boundary.

[0255] In the deblocking filter process, for example, using a pixel value and a quantization parameter, either one of two deblocking filters having different characteristics, that is, a strong filter and a weak filter, is selected. In the strong filter, as shown in Figure 8 , in a case where there are pixels p0 to p2 and pixels q0 to q2 across the block boundary, the pixel values of the pixels q0 to q2 are changed to pixel values q'0 to q'2, respectively, by performing an operation shown by the following expression, for example.

[0256] q'0 = (p1 + 2 x p0 + 2 x q0 + 2 x q1 + q2 + 4) / 8

[0257] q'1 = (p0 + q0 + q1 + q2 + 2) / 4

[0258] q'2 = (p0 + q0 + q1 + 3 x q2 + 2 x q3 + 4) / 8

[0259] Further, in the above expression, p0 to p2 and q0 to q2 are the pixel values of the pixels p0 to p2 and the pixels q0 to q2, respectively. In addition, q3 is the pixel value of a pixel q3 adjacent to the pixel q2 on the side opposite to the block boundary. Further, on the right side of each of the above expressions, the coefficients multiplied by the pixel values of each pixel used in the deblocking filter process are filter coefficients.

[0260] Furthermore, in the deblocking filter process, clipping processing can be performed in a manner that the pixel value after the operation does not exceed a threshold value set. In this clipping processing, the pixel value after the operation based on the above expression is clipped to "operation object pixel value ± 2 x threshold value" using the threshold value decided in accordance with the quantization parameter. Thereby, excessive smoothing can be prevented.

[0261] Figure 9 is a conceptual diagram for explaining a block boundary on which the deblocking filter process is performed. Figure 10 is a conceptual diagram showing an example of a Bs value.

[0262] The block boundaries for deblocking filtering are, for example, Figure 9 The boundaries of the PU (Prediction Unit) or TU (Transform Unit) of the 8×8 pixel block are shown. Deblocking filtering can be performed in units of 4 rows or 4 columns. First, for Figure 9 Blocks P and Q are shown, as follows Figure 10 That determines the Bs (Boundary Strength) value.

[0263] according to Figure 10 The Bs value determines whether deblocking filtering of different intensities is applied even to block boundaries belonging to the same image. When the Bs value is 2, deblocking filtering is applied to the chrominance signal. When the Bs value is 1 or higher and certain conditions are met, deblocking filtering is applied to the luminance signal. These conditions can be predetermined. Furthermore, the criteria for determining the Bs value are not limited to... Figure 10 The conditions shown can also be determined based on other parameters.

[0264] [Prediction Processing Unit (Intra-Frame Prediction Unit / Inter-Frame Prediction Unit / Prediction Control Unit)]

[0265] Figure 11 This is a flowchart illustrating an example of the processing performed by the prediction processing unit of the coding apparatus 100. Furthermore, the prediction processing unit is composed of all or part of the constituent elements of the intra-frame prediction unit 124, the inter-frame prediction unit 126, and the prediction control unit 128.

[0266] The prediction processing unit generates a prediction image for the current block (step Sb_1). This prediction image is also called a prediction signal or a prediction block. Additionally, the prediction signal may include, for example, an intra-frame prediction signal or an inter-frame prediction signal. Specifically, the prediction processing unit generates a prediction image for the current block using the reconstructed image obtained through generating prediction blocks, generating difference blocks, generating coefficient blocks, restoring difference blocks, and generating decoded image blocks.

[0267] The reconstructed image can be, for example, an image of a reference image, or an image containing the current block, i.e., an image of the encoded blocks within the current image. The encoded blocks within the current image can be, for example, neighboring blocks of the current block.

[0268] Figure 12 This is a flowchart illustrating another example of the processing performed by the prediction processing unit of the encoding device 100.

[0269] The prediction processing section generates a prediction image by a first method (step Sc la), generates a prediction image by a second method (step Sc lb), and generates a prediction image by a third method (step Sc lc). The first method, the second method, and the third method are mutually different methods for generating a prediction image, and can be, for example, an inter prediction method, an intra prediction method, and another prediction method, respectively. In such a prediction method, the above-described reconstructed image can also be used.

[0270] Next, the prediction processing section selects any one of the plurality of prediction images generated in steps Sc la, Sc lb, and Sc lc (step Sc 2). The selection of the prediction image, that is, the selection of the method or mode for obtaining the final prediction image can also be performed by calculating a cost with respect to each of the generated prediction images and based on the cost. Further, the selection of the prediction image can be performed based on a parameter for the encoding processing. The encoding apparatus 100 can signal information for determining the selected prediction image, method, or mode as an encoded signal (also referred to as an encoded bit stream). The information can be, for example, a flag or the like. Thereby, a decoding apparatus can generate a prediction image in the method or mode selected in the encoding apparatus 100 based on the information. Further, in the example shown in FIG. 8, the prediction processing section selects any one of the prediction images after generating the prediction images by the respective methods. However, the prediction processing section can select a method or mode based on a parameter for the above-described encoding processing before generating the prediction images, and can generate the prediction images in accordance with the method or mode. Figure 12

[0271] For example, the first method and the second method are intra prediction and inter prediction, respectively, and the prediction processing section can select a final prediction image for the current block from the prediction images generated in accordance with these prediction methods.

[0272] Figure 13 is a flowchart showing another example of processing performed by the prediction processing section of the encoding apparatus 100.

[0273] First, the prediction processing section generates a prediction image by intra prediction (step Sd la), and generates a prediction image by inter prediction (step Sd lb). Further, the prediction image generated by intra prediction is also referred to as an intra prediction image, and the prediction image generated by inter prediction is also referred to as an inter prediction image.

[0274] ​Next, the prediction processing section evaluates each of the intra prediction image and the inter prediction image (step Sd_2). In this evaluation, a cost can also be used. That is, the prediction processing section calculates the cost C of each of the intra prediction image and the inter prediction image. This cost C can be calculated by the equation of the R-D optimization model, for example, C = D + λ x R. In this equation, D is the encoding distortion of the prediction image, and is expressed by, for example, the sum of the absolute values of the differences between the pixel values of the current block and the pixel values of the prediction image. In addition, R is the amount of encoding for generating the prediction image, specifically, the amount of encoding required for encoding the motion information and the like for generating the prediction image. In addition, λ is, for example, the undetermined multiplier of Lagrange.

[0275] Then, the prediction processing section selects the prediction image for which the minimum cost C is calculated from among the intra prediction image and the inter prediction image as the final prediction image of the current block (step Sd_3). That is, the prediction mode or the like for generating the prediction image of the current block is selected.

[0276] [Intra prediction section]

[0277] The intra prediction section 124 performs intra prediction (also referred to as in-picture prediction) of the current block with reference to the blocks within the current picture saved in the block memory 118, thereby generating a prediction signal (intra prediction signal). Specifically, the intra prediction section 124 generates an intra prediction signal by performing intra prediction with reference to the samples (e.g., luminance values, color difference values) of the blocks adjacent to the current block, and outputs the intra prediction signal to the prediction control section 128.

[0278] For example, the intra prediction section 124 performs intra prediction using one of a plurality of prescribed intra prediction modes. The plurality of intra prediction modes generally includes one or more non-directional prediction modes and a plurality of directional prediction modes. The prescribed plurality of modes can also be determined in advance.

[0279] The one or more non-directional prediction modes include, for example, the Planar prediction mode and the DC prediction mode prescribed by the H.265 / HEVC specification.

[0280] The plurality of directional prediction modes include, for example, the 33-direction prediction modes prescribed by the H.265 / HEVC specification. In addition, the plurality of directional prediction modes can also include 32-direction prediction modes in addition to the 33 directions (a total of 65 directional prediction modes). Figure 14 is a conceptual diagram showing all 67 intra prediction modes (2 non-directional prediction modes and 65 directional prediction modes) that can be used in intra prediction. The solid arrows indicate the 33 directions prescribed by the H.265 / HEVC specification, and the dashed arrows indicate the additional 32 directions (the 2 non-directional prediction modes are not shown in Figure 14 ).

[0281] In various processing examples, in the intra prediction of a color difference block, a luminance block can also be referred to. That is, the color difference component of a current block can also be predicted based on the luminance component of the current block. Such intra prediction is a case where it is referred to as CCLM (cross-component linear model) prediction. Such an intra prediction mode of a color difference block that refers to a luminance block (for example, referred to as a CCLM mode) can also be added as one of the intra prediction modes of a color difference block.

[0282] The intra prediction section 124 can also correct the pixel values after the intra prediction based on the gradient of the reference pixels in the horizontal / vertical direction. Intra prediction with such correction is a case where it is referred to as PDPC (position dependent intra prediction combination). Information indicating whether or not PDPC is applied (for example, referred to as a PDPC flag) is generally signaled at the CU level. In addition, the signaling of this information does not need to be limited to the CU level, and can be another level (for example, a sequence level, a picture level, a slice level, a tile level, or a CTU level).

[0283] [Inter prediction section]

[0284] The inter prediction section 126 performs inter prediction (also referred to as inter picture prediction) of a current block with reference to a reference picture different from the current picture stored in the frame memory 122, thereby generating a prediction signal (inter prediction signal). Inter prediction is performed in units of a current block or a current sub-block (for example, a 4x4 block) within the current block. For example, the inter prediction section 126 performs motion search within the reference picture for a current block or a current sub-block, and finds a reference block or sub-block most consistent with the current block or the current sub-block. Also, the inter prediction section 126 acquires motion information (for example, a motion vector) that compensates for motion or change from the reference block or sub-block to the current block or sub-block. The inter prediction section 126 performs motion compensation (or motion prediction) based on the motion information, thereby generating an inter prediction signal of the current block or sub-block. Also, the inter prediction section 126 outputs the generated inter prediction signal to the prediction control section 128.

[0285] The motion information used in the motion compensation is signaled as an inter prediction signal in various forms. For example, a motion vector can be signaled. As another example, a difference between a motion vector and a prediction motion vector (motion vector predictor) can be signaled.

[0286] [Basic flow of inter prediction]

[0287] Figure 15 is a flowchart showing an example of a basic procedure of inter prediction.

[0288] The inter prediction section 126 first generates a prediction image (steps Se_1 to Se_3). Next, the subtraction section 104 generates a difference between the current block and the prediction image as a prediction residual (step Se_4).

[0289] Here, in the generation of the prediction image, the inter prediction section 126 generates the prediction image by performing determination of a motion vector (MV) of the current block (steps Se_1 and Se_2) and motion compensation (step Se_3). Further, in the determination of the MV, the inter prediction section 126 determines the MV by performing selection of a candidate motion vector (candidate MV) (step Se_1) and derivation of the MV (step Se_2). The selection of the candidate MV is performed, for example, by selecting at least one candidate MV from a candidate MV list. Further, in the derivation of the MV, the inter prediction section 126 can further select at least one candidate MV from the at least one candidate MV, and determine the selected at least one candidate MV as the MV of the current block. Alternatively, the inter prediction section 126 can determine the MV of the current block by searching a region of a reference picture indicated by each of the selected at least one candidate MV. Further, the act of searching the region of the reference picture can be referred to as motion estimation.

[0290] Further, in the above example, steps Se_1 to Se_3 are performed by the inter prediction section 126, but the processing of, for example, step Se_1 or step Se_2 can be performed by another constituent element included in the encoding apparatus 100.

[0291] [Flow of derivation of motion vector]

[0292] Figure 16 is a flowchart showing an example of motion vector derivation.

[0293] The inter prediction section 126 derives the MV of the current block in a mode in which motion information (e.g., MV) is encoded. In this case, for example, the motion information is encoded as a prediction parameter, and is signaled. That is, the encoded motion information is included in an encoded signal (also referred to as an encoded bitstream).

[0294] Alternatively, the inter prediction section 126 derives the MV in a mode in which the motion information is not encoded. In this case, the motion information is not included in the encoded signal.

[0295] Here, the mode of the MV derivation can also be the normal inter mode, the merge mode, the FRUC mode, the affine mode, and the like described later. Among these modes, the modes in which the motion information is coded are the normal inter mode, the merge mode, and the affine mode (specifically, the affine inter mode and the affine merge mode), and the like. Further, the motion information can include not only the MV but also the prediction motion vector selection information described later. Further, the modes in which the motion information is not coded are the FRUC mode, and the like. The inter prediction section 126 selects the mode for deriving the MV of the current block from among these multiple modes, and derives the MV of the current block using the selected mode.

[0296] Figure 17 is a flowchart showing another example of the motion vector derivation.

[0297] The inter prediction section 126 derives the MV of the current block in the mode in which the differential MV is coded. In this case, for example, the differential MV is coded as the prediction parameter, and is signaled. That is, the coded differential MV is included in the coded signal. The differential MV is the difference between the MV of the current block and the prediction MV thereof.

[0298] Alternatively, the inter prediction section 126 derives the MV in the mode in which the differential MV is not coded. In this case, the coded differential MV is not included in the coded signal.

[0299] Here, as described above, the mode of the MV derivation is the normal inter mode, the merge mode, the FRUC mode, the affine mode, and the like described later. Among these modes, the modes in which the differential MV is coded are the normal inter mode, and the affine mode (specifically, the affine inter mode), and the like. Further, the modes in which the differential MV is not coded are the FRUC mode, the merge mode, and the affine mode (specifically, the affine merge mode), and the like. The inter prediction section 126 selects the mode for deriving the MV of the current block from among these multiple modes, and derives the MV of the current block using the selected mode.

[0300] [Flow of the derivation of the motion vector]

[0301] Figure 18is a flowchart showing another example of motion vector derivation. The mode of MV derivation, i.e., the inter prediction mode, has a plurality of modes, which are roughly classified into a mode in which a differential MV is coded and a mode in which a differential motion vector is not coded. The mode in which a differential MV is not coded has a merge mode, an FRUC mode, and an affine mode (specifically, an affine merge mode). Details of these modes will be described later, but simply, the merge mode is a mode in which a MV of a current block is derived by selecting a motion vector from peripheral coded blocks, the FRUC mode is a mode in which a MV of a current block is derived by searching among coded regions, and the affine mode is a mode in which a plurality of sub-blocks constituting a current block are assumed to be subjected to affine transformation and a MV of each of the sub-blocks is derived as a MV of the current block.

[0302] Specifically, as shown in the drawing, in a case where the inter prediction mode information indicates 0 (0 in Sf_1), the inter prediction section 126 derives a motion vector based on the merge mode (Sf_2). Further, in a case where the inter prediction mode information indicates 1 (1 in Sf_1), the inter prediction section 126 derives a motion vector according to the FRUC mode (Sf_3). Further, in a case where the inter prediction mode information indicates 2 (2 in Sf_1), the inter prediction section 126 derives a motion vector according to the affine mode (specifically, the affine merge mode) (Sf_4). Further, in a case where the inter prediction mode information indicates 3 (3 in Sf_1), the inter prediction section 126 derives a motion vector according to a mode in which a differential MV is coded (for example, a normal inter mode) (Sf_5).

[0303] [MV derivation > normal inter mode]

[0304] The normal inter mode is an inter prediction mode in which a MV of a current block is derived based on a block similar to the image of the current block from a region of a reference picture represented by a candidate MV. Further, in this normal inter mode, a differential MV is coded.

[0305] Figure 19 is a flowchart showing an example of inter prediction based on the normal inter mode.

[0306] First, the inter prediction section 126 acquires a plurality of candidate MVs for a current block based on information of MVs and the like of a plurality of coded blocks located around the current block in time or space (step Sg_1). That is, the inter prediction section 126 creates a candidate MV list.

[0307] Next, the inter prediction section 126 extracts N (N is an integer of 2 or more) candidate MVs as prediction motion vector candidates (also referred to as prediction MV candidates) in a prescribed priority order from the plurality of candidate MVs acquired in step Sg_1 (step Sg_2). Note that the priority order can be determined in advance for each of the N candidate MVs.

[0308] Next, the inter prediction section 126 selects one of the N prediction motion vector candidates as a prediction motion vector (also referred to as a prediction MV) for the current block (step Sg_3). At this time, the inter prediction section 126 encodes prediction motion vector selection information for identifying the selected prediction motion vector into a stream. Note that the stream is the coded signal or the coded bitstream described above.

[0309] Next, the inter prediction section 126 refers to the coded reference picture to derive the MV of the current block (step Sg_4). At this time, the inter prediction section 126 also encodes a difference value between the derived MV and the prediction motion vector as a difference MV into the stream. Note that the coded reference picture is a picture composed of a plurality of blocks reconstructed after coding.

[0310] Finally, the inter prediction section 126 generates a prediction image of the current block by performing motion compensation on the current block using the derived MV and the coded reference picture (step Sg_5). Note that the prediction image is the inter prediction signal described above.

[0311] Further, information indicating the inter prediction mode (in the example described above, the normal inter mode) used in the generation of the prediction image included in the coded signal is encoded as, for example, a prediction parameter.

[0312] Note that the candidate MV list can also be used in common with the list used in other modes. Further, the processing related to the candidate MV list can be applied to the processing related to the list used in other modes. The processing related to the candidate MV list is, for example, extraction or selection of a candidate MV from the candidate MV list, rearrangement of the candidate MVs, or deletion of a candidate MV, and the like.

[0313] [MV derivation > merge mode]

[0314] The merge mode is an inter prediction mode in which a candidate MV is selected from a candidate MV list as an MV of a current block to derive the MV.

[0315] Figure 20 is a flowchart indicating an example of inter prediction based on the merge mode.

[0316] First, the inter prediction section 126 acquires a plurality of candidate MVs for the current block based on information of a plurality of coded block MVs and the like located around the current block in time or space (step Sh_l). That is, the inter prediction section 126 creates a candidate MV list.

[0317] Next, the inter prediction section 126 derives the MV of the current block by selecting one candidate MV from the plurality of candidate MVs acquired in step Sh_l (step Sh_2). At this time, the inter prediction section 126 encodes MV selection information for identifying the selected candidate MV into the stream.

[0318] Finally, the inter prediction section 126 generates a prediction image of the current block by performing motion compensation on the current block using the derived MV and the coded reference picture (step Sh_3).

[0319] Further, information indicating the inter prediction mode (in the above example, the merge mode) used in the generation of the prediction image included in the coded signal is encoded as, for example, a prediction parameter.

[0320] Figure 21 is a conceptual diagram for explaining an example of the motion vector derivation process of the current picture based on the merge mode.

[0321] First, a prediction MV list in which candidates of a prediction MV are registered is generated. As the candidates of the prediction MV, there are: a spatial neighboring prediction MV, which is an MV possessed by a plurality of coded blocks located in the periphery of the space of the target block; a temporal neighboring prediction MV, which is an MV possessed by a block in the vicinity of the target block projected in the coded reference picture; a combined prediction MV, which is an MV generated by combining the MV values of the spatial neighboring prediction MV and the temporal neighboring prediction MV; and a zero prediction MV, which is an MV having a value of zero, and the like.

[0322] Next, the MV for the target block is decided by selecting one prediction MV from the plurality of prediction MVs registered in the prediction MV list.

[0323] Further, in the variable length coding section, a signal indicating which prediction MV is selected, that is, merge_idx, is described in the stream and encoded.

[0324] In addition, the prediction MVs registered in the prediction MV list explained in Figure 21 may be a number different from that in the drawing, or a structure not containing a part of the kinds of the prediction MVs in the drawing, or a structure in which prediction MVs other than the kinds of the prediction MVs in the drawing are added.

[0325] The MV of the object block derived by the merge mode can also be used to determine the final MV by performing a DMVR (decoder motion vector refinement) process described later.

[0326] In addition, the candidate of the prediction MV is the above-described candidate MV, and the prediction MV list is the above-described candidate MV list. Furthermore, the candidate MV list can also be referred to as a candidate list. Furthermore, the merge_idx is the MV selection information.

[0327] [MV derivation FRUC mode]

[0328] The motion information can also not be signaled from the encoding device side but derived at the decoding device side. In addition, as described above, the merge mode prescribed by the H.265 / HEVC specification can also be used. Furthermore, the motion information can also be derived, for example, by performing a motion search at the decoding device side. In an embodiment, the motion search is performed at the decoding device side without using the pixel values of the current block.

[0329] Here, a mode in which motion estimation is performed at the decoding device side will be described. This mode in which motion estimation is performed at the decoding device side is a case where it is referred to as a PMMVD (pattern matched motion vector derivation) mode or a FRUC (frame rate up-conversion) mode.

[0330] In the form of a flowchart Figure 22The FRUC processing is an example. First, a plurality of candidates each having a predicted motion vector (MV) are generated by referring to motion vectors of coded blocks adjacent to the current block in space or time (step Si_1). Next, a best candidate MV is selected from among the plurality of candidates registered in the candidate MV list (step Si_2). For example, evaluation values of the respective candidate MVS included in the candidate MV list are calculated, and one candidate is selected based on the evaluation values. Further, a motion vector for the current block is derived based on the motion vector of the selected candidate (step Si_4). Specifically, for example, the motion vector of the selected candidate (best candidate MV) is used as it is as the motion vector for the current block. Alternatively, for example, the motion vector for the current block can be derived by performing pattern matching in a peripheral region of a position in a reference picture corresponding to the motion vector of the selected candidate. That is, the peripheral region of the best candidate MV can be searched using pattern matching in the reference picture and evaluation values, and in the case where there is an MV having a better evaluation value, the best candidate MV is updated to the above-mentioned MV, which is used as the final MV for the current block. It can also be configured not to perform the process of updating to an MV having a better evaluation value.

[0331] Finally, the inter prediction unit 126 generates a predicted image of the current block by performing motion compensation on the current block using the derived MV and the coded reference picture (step Si_5).

[0332] The same processing can also be performed in the case where processing is performed in sub-block units.

[0333] The evaluation value can be calculated by various methods. For example, a reconstructed image of a region in a reference picture corresponding to a motion vector is compared with a reconstructed image of a prescribed region (for example, the region can be a region of another reference picture or a region of a neighboring block of the current picture). The prescribed region can be determined in advance.

[0334] Then, a difference in pixel values of the two reconstructed images can be calculated as the evaluation value for the motion vector. Alternatively, other information can be used in addition to the difference value to calculate the evaluation value.

[0335] Next, an example of pattern matching is described in detail. First, one of the candidate MVS included in the candidate MV list (for example, the merge list) is selected as a starting point of search based on pattern matching. For example, as the pattern matching, first pattern matching or second pattern matching can be used. The first pattern matching and the second pattern matching are respectively referred to as bilateral matching and template matching.

[0336] [MV derivation] FRUC bi-directional matching

[0337] In the 1st pattern matching, pattern matching is performed between two blocks along the motion trajectory of the current block in different two reference pictures. Thus, in the 1st pattern matching, as the prescribed region for the calculation of the evaluation value for the candidate, a region in the other reference picture along the motion trajectory of the current block is used. The prescribed region can also be determined in advance.

[0338] Figure 23 is a conceptual diagram for explaining the 1st pattern matching (bi-directional matching) between two blocks in two reference pictures along the motion trajectory. As shown in Figure 23 , in the 1st pattern matching, two motion vectors (MV0, MV1) are derived by searching for the most matching pair among pairs of two blocks in different two reference pictures (Ref0, Ref1) along the motion trajectory of the current block (Cur block). Specifically, for the current block, the difference between the reconstructed image at the specified position in the 1st encoded reference picture (Ref0) designated by the candidate MV and the reconstructed image at the specified position in the 2nd encoded reference picture (Ref1) designated by the symmetric MV obtained by scaling the above-mentioned candidate MV by the display time interval is calculated, and the evaluation value is calculated using the obtained difference value. The candidate MV whose evaluation value is the best value can be selected from among a plurality of candidate MVs as the final MV, and good results can be obtained.

[0339] Under the assumption of the continuity of the motion trajectory, the motion vectors (MV0, MV1) indicating the two reference blocks are proportional to the temporal distance (TD0, TD1) between the current picture (Cur Pic) and the two reference pictures (Ref0, Ref1). For example, in the case where the current picture is located between the two reference pictures in time, and the temporal distance from the current picture to the two reference pictures is equal, in the 1st pattern matching, mirror-symmetric bi-directional motion vectors are derived.

[0340] [MV derivation] FRUC template matching

[0341] In the 2nd pattern matching (template matching), pattern matching is performed between a template (a block adjacent to the current block (e.g., an upper and / or left adjacent block) in the current picture) in the current picture and a block in the reference picture. Thus, in the 2nd pattern matching, as the prescribed region for the calculation of the evaluation value for the candidate, a block adjacent to the current block in the current picture is used.

[0342] Figure 24is a conceptual diagram for explaining an example of pattern matching (template matching) between a template within a current picture and a block within a reference picture. As shown in Figure 24 In the 2nd pattern matching, a motion vector of a current block is derived by searching a reference picture (Ref0) for a block that best matches a block adjacent to the current block (Cur block) within a current picture (Cur Pic). Specifically, for the current block, a difference between a reconstructed image of a coded region of both or one of a left-adjacent and an above-adjacent and a reconstructed image at a corresponding position within a coded reference picture (Ref0) specified by a candidate MV is calculated, a value of a cost function is calculated using the resulting difference value, and a candidate MV for which the value of the cost function is the best value is selected as a best candidate MV from among a plurality of candidate MVs.

[0343] Such information indicating whether to employ the FRUC mode (e.g., referred to as a FRUC flag) is signaled at a CU level. Further, in the case of employing the FRUC mode (e.g., in the case of the FRUC flag being true), information indicating a method of pattern matching that can be employed (the 1st pattern matching or the 2nd pattern matching) is signaled at a CU level. In addition, the signaling of such information does not need to be limited to a CU level, and can be another level (e.g., a sequence level, a picture level, a slice level, a tile level, a CTU level, or a sub-block level).

[0344] [MV derivation > affine mode]

[0345] Next, an affine mode of deriving a motion vector in a sub-block unit based on motion vectors of a plurality of adjacent blocks will be described. This mode is sometimes referred to as an affine motion compensation prediction mode.

[0346] Figure 25A is a conceptual diagram for explaining an example of derivation of a motion vector in a sub-block unit based on motion vectors of a plurality of adjacent blocks. In Figure 25A , a current block includes 16 4x4 sub-blocks. Here, a motion vector v0 of a top-left control point of the current block is derived based on motion vectors of adjacent blocks, and similarly, a motion vector v1 of a top-right control point of the current block is derived based on motion vectors of adjacent sub-blocks. Then, according to the following equation (1A), the two motion vectors v0 and v1 can be projected, and motion vectors (v x , v y ) of each sub-block within the current block can be derived.

[0347] [Equation 1]

[0348]

[0349] Here, x and y respectively represent the horizontal position and the vertical position of the sub-block, and w represents a prescribed weight coefficient. The prescribed weight coefficient can also be determined in advance.

[0350] Information indicating such an affine mode (for example, referred to as an affine flag) can be signaled as a signal at the CU level. Furthermore, the signaling of the information indicating the affine mode need not be limited to the CU level, and can be at another level (for example, the sequence level, the picture level, the slice level, the tile level, the CTU level, or the sub-block level).

[0351] In addition, in such an affine mode, several modes different in the derivation method of the motion vectors of the upper-left and upper-right corner control points can also be included. For example, in the affine mode, there are two modes, an affine inter (also referred to as an affine normal inter) mode and an affine merge mode.

[0352] [MV derivation > affine mode]

[0353] Figure 25B is a conceptual diagram for explaining an example of the derivation of the motion vectors of the sub-block units in the affine mode having three control points. In Figure 25B , the current block includes 16 4x4 sub-blocks. Here, the motion vector v0 of the upper-left corner control point of the current block is derived based on the motion vectors of the neighboring blocks, and similarly, the motion vector v1 of the upper-right corner control point of the current block is derived based on the motion vectors of the neighboring blocks, and the motion vector v2 of the lower-left corner control point of the current block is derived based on the motion vectors of the neighboring blocks. Then, according to the following equation (1B), the three motion vectors v0, v1, and v2 can be projected, and the motion vectors (v x , v y ) of the respective sub-blocks within the current block can be derived.

[0354] [Equation 2]

[0355]

[0356] Here, x and y respectively represent the horizontal position and the vertical position of the center of the sub-block, w represents the width of the current block, and h represents the height of the current block.

[0357] Affine modes different in the number of control points (for example, two and three) can also be signaled at the CU level. In addition, information indicating the number of control points of the affine mode used at the CU level can also be signaled at another level (for example, the sequence level, the picture level, the slice level, the tile level, the CTU level, or the sub-block level).

[0358] In addition, in such an affine mode having 3 control points, several modes different in the derivation method of the motion vector of the upper left, upper right, and lower left control points can also be included. For example, in the affine mode, there are 2 modes, an affine inter (also referred to as affine normal inter) mode and an affine merge mode.

[0359] [MV derivation] affine merge mode

[0360] Figure 26A Figure 26B and Figure 26C are conceptual diagrams for explaining the affine merge mode.

[0361] In the affine merge mode, as shown in Figure 26A , for example, the prediction motion vector of each of the control points of the current block is calculated based on the plurality of motion vectors corresponding to the blocks coded in the affine mode among the neighboring blocks A (left), B (upper), C (upper right), D (lower left), and E (upper left) of the current block. Specifically, the blocks coded in the affine mode are determined in the order of the blocks A (left), B (upper), C (upper right), D (lower left), and E (upper left), and the prediction motion vector of the control point of the current block is calculated based on the plurality of motion vectors corresponding to the determined blocks.

[0362] For example, as shown in Figure 26B , in the case where the block A neighboring the left side of the current block is coded in the affine mode having 2 control points, the motion vectors v3 and v4 projected to the positions of the upper left and upper right corners of the coded block including the block A are derived. Then, the prediction motion vector v0 of the control point of the upper left corner and the prediction motion vector v1 of the control point of the upper right corner of the current block are calculated from the derived motion vectors v3 and v4.

[0363] For example, as shown in Figure 26C , when the block A neighboring the left side of the current block is coded in the affine mode having 3 control points, the motion vectors v3, v4, and v5 projected to the positions of the upper left, upper right, and lower left corners of the coded block including the block A are derived. Then, the prediction motion vector v0 of the control point of the upper left corner, the prediction motion vector v1 of the control point of the upper right corner, and the prediction motion vector v2 of the control point of the lower left corner of the current block are calculated from the derived motion vectors v3, v4, and v5.

[0364] In addition, in the derivation of the prediction motion vector of each of the control points of the current block in step Sj_1 of the process described later Figure 29 , the prediction motion vector derivation method can also be used.

[0365] Figure 27 is a flowchart showing an example of the affine merge mode.​

[0366] In the affine merge mode, as shown in the drawing, first, the inter prediction section 126 derives the prediction MVs of the control points of the current block (step Sk_1). The control points are the upper left and upper right points of the current block, as shown in Figure 25A , or the upper left, upper right and lower left points of the current block, as shown in Figure 25B .

[0367] That is, as shown in Figure 26A , the inter prediction section 126 checks the coded blocks in the order of block A (left), block B (upper), block C (upper right), block D (lower left) and block E (upper left), and determines the initial effective block coded in the affine mode.

[0368] Then, in the case where block A is determined and block A has 2 control points, as shown in Figure 26B , the inter prediction section 126 calculates the motion vector v0 of the control point of the upper left corner and the motion vector v1 of the control point of the upper right corner of the current block from the motion vectors v3 and v4 of the upper left and upper right corners of the coded blocks containing block A. For example, by projecting the motion vectors v3 and v4 of the upper left and upper right corners of the coded blocks onto the current block, the inter prediction section 126 calculates the prediction motion vector v0 of the control point of the upper left corner and the prediction motion vector v1 of the control point of the upper right corner of the current block.

[0369] Or, in the case where block A is determined and block A has 3 control points, as shown in Figure 26C , the inter prediction section 126 calculates the motion vector v0 of the control point of the upper left corner, the motion vector v1 of the control point of the upper right corner and the motion vector v2 of the control point of the lower left corner of the current block from the motion vectors v3, v4 and v5 of the upper left, upper right and lower left corners of the coded blocks containing block A. For example, by projecting the motion vectors v3, v4 and v5 of the upper left, upper right and lower left corners of the coded blocks onto the current block, the inter prediction section 126 calculates the prediction motion vector v0 of the control point of the upper left corner, the prediction motion vector v1 of the control point of the upper right corner and the motion vector v2 of the control point of the lower left corner of the current block.

[0370] Next, the inter prediction section 126 performs motion compensation for each of the plurality of sub-blocks included in the current block. That is, the inter prediction section 126 calculates, for each of the plurality of sub-blocks, a motion vector of the sub-block as an affine MV using the two prediction motion vectors v0 and vl and the above-described equation (1A), or the three prediction motion vectors v0, vl, and v2 and the above-described equation (1B) (step Sk_2). Then, the inter prediction section 126 performs motion compensation for the sub-block using the affine MV and the encoded reference picture (step Sk_3). As a result, the current block is motion-compensated, and a predicted image of the current block is generated.

[0371] [MV derivation: affine inter mode]

[0372] Figure 28A is a conceptual diagram for explaining an affine inter mode having two control points.

[0373] In this affine inter mode, as shown in Figure 28A , a motion vector selected from motion vectors of the encoded blocks A, B, and C adjacent to the current block is used as a prediction motion vector v0 of a control point of the upper left corner of the current block. Likewise, a motion vector selected from motion vectors of the encoded blocks D and E adjacent to the current block is used as a prediction motion vector vl of a control point of the upper right corner of the current block.

[0374] Figure 28B is a conceptual diagram for explaining an affine inter mode having three control points.

[0375] In this affine inter mode, as shown in Figure 28B , a motion vector selected from motion vectors of the encoded blocks A, B, and C adjacent to the current block is used as a prediction motion vector v0 of a control point of the upper left corner of the current block. Likewise, a motion vector selected from motion vectors of the encoded blocks D and E adjacent to the current block is used as a prediction motion vector vl of a control point of the upper right corner of the current block. In addition, a motion vector selected from motion vectors of the encoded blocks F and G adjacent to the current block is used as a prediction motion vector v2 of a control point of the lower left corner of the current block.

[0376] Figure 29 is a flowchart showing an example of the affine inter mode.

[0377] As shown in the figure, in the affine inter mode, first, the inter prediction section 126 derives prediction MVs (v0, vl) or (v0, vl, v2) of two or three control points of the current block (step Sj_1). As shown in Figure 25A or Figure 25B , the control points are points of the upper left corner, the upper right corner, or the lower left corner of the current block.

[0378] That is, the inter prediction section 126 derives the prediction motion vectors (v0, v1) or (v0, v1, v2) of the control points of the current block by selecting the motion vector of a certain block among the encoded blocks in the vicinity of the control points of the current block Figure 28A Or Figure 28B of the current block. At this time, the inter prediction section 126 encodes the prediction motion vector selection information for identifying the selected two motion vectors into the stream.

[0379] For example, the inter prediction section 126 can decide which motion vector of the encoded blocks adjacent to the current block to select as the prediction motion vector of the control point by using a cost evaluation or the like, and can describe a flag indicating which prediction motion vector is selected in the bit stream.

[0380] Next, the inter prediction section 126 performs a motion search (steps Sj_3 and Sj_4) while updating the prediction motion vectors selected or derived in step Sj_1 (step Sj_2). That is, the inter prediction section 126 calculates (step Sj_3) the motion vectors of the respective sub-blocks corresponding to the prediction motion vectors to be updated as affine MVs, and uses the above-described equation (1A) or equation (1B). Then, the inter prediction section 126 performs motion compensation on the respective sub-blocks using these affine MVs and the encoded reference picture (step Sj_4). As a result, in the motion search loop, the inter prediction section 126 decides, for example, the prediction motion vector that can obtain the minimum cost as the motion vector of the control point (step Sj_5). At this time, the inter prediction section 126 also encodes the difference value between this decided MV and the prediction motion vector as a difference MV into the stream.

[0381] Finally, the inter prediction section 126 generates a prediction image of the current block by performing motion compensation on the current block using the decided MV and the encoded reference picture (step Sj_6).

[0382] [MV derivation > affine inter mode]

[0383] In a case where affine modes with different numbers of control points (for example, two and three) are signaled at the CU level, the number of control points is sometimes different between the encoded block and the current block. Figure 30A And Figure 30B are conceptual diagrams for explaining a method of deriving the prediction vectors of the control points in a case where the number of control points is different between the encoded block and the current block.

[0384] For example, as Figure 30AAs shown, when the current block has three control points (top left, top right, and bottom left), and the block A adjacent to the left of the current block is encoded in an affine pattern with two control points, motion vectors v3 and v4 are derived, projected onto the top left and top right corners of the encoded block containing block A. Then, based on the derived motion vectors v3 and v4, the predicted motion vector v0 for the top left control point and the predicted motion vector v1 for the top right control point of the current block are calculated. Furthermore, the predicted motion vector v2 for the bottom left control point is calculated based on the derived motion vectors v0 and v1.

[0385] For example, such as Figure 30B As shown, when the current block has two control points (top left and top right), and the block A adjacent to the left of the current block is encoded in an affine pattern with three control points, motion vectors v3, v4, and v5 are derived to project onto the top left, top right, and bottom left corners of the encoded block containing block A. Then, based on the derived motion vectors v3, v4, and v5, the predicted motion vector v0 for the top left control point and the predicted motion vector v1 for the top right control point of the current block are calculated.

[0386] exist Figure 29 The same method for deriving predicted motion vectors can also be used in the process of deriving the predicted motion vectors of the control points of the current block in step Sj_1.

[0387] [MV Export > DMVR]

[0388] Figure 31A This is a flowchart illustrating the merging modes and the relationship between DMVR.

[0389] The inter-frame prediction unit 126 derives the motion vector of the current block in merge mode (step S1_1). Next, the inter-frame prediction unit 126 determines whether to perform a motion vector search, i.e., a motion search (step S1_2). Here, when it is determined that no motion search should be performed (no in step S1_2), the inter-frame prediction unit 126 determines the motion vector derived in step S1_1 as the final motion vector for the current block (step S1_4). That is, in this case, the motion vector of the current block is determined in merge mode.

[0390] On the other hand, when it is determined in step Sl_1 that motion search is to be performed (yes in step Sl_2), the inter-frame prediction unit 126 derives the final motion vector for the current block by searching the surrounding region of the reference image represented by the motion vector derived in step Sl_1 (step Sl_3). That is, in this case, the motion vector of the current block is determined by the DMVR.

[0391] Figure 31B This is a conceptual diagram used to illustrate an example of DMVR processing for determining MV.

[0392] First, the best MVP set for the current block (e.g., in merge mode) is set as the candidate MV. Then, based on the candidate MV (L0), reference pixels are determined according to the encoded image in the L0 direction, i.e., the first reference image (L0). Similarly, based on the candidate MV (L1), reference pixels are determined according to the encoded image in the L1 direction, i.e., the second reference image (L1). The template is generated by averaging these reference pixels.

[0393] Next, using the template described above, the surrounding areas of the candidate MVs in the first reference image (L0) and the second reference image (L1) are searched respectively, and the MV with the lowest cost is determined as the final MV. In addition, the cost value can also be calculated using, for example, the difference between the pixel values ​​of the template and the pixel values ​​of the search area, as well as the candidate MV values.

[0394] Furthermore, typically, the structure and operation of the processing described herein are essentially the same in both the encoding device and the decoding device, which will be described later.

[0395] Even if it's not the processing example described here, any processing method can be used as long as it can search for the surrounding elements of candidate MVs and derive the final MV.

[0396] [Motion Compensation > BIO / OBMC]

[0397] In motion compensation, there are patterns for generating a predicted image and then correcting that predicted image. Examples of such patterns include BIO and OBMC, which will be discussed later.

[0398] Figure 32 This is a flowchart illustrating an example of generating a predicted image.

[0399] The inter-frame prediction unit 126 generates a prediction image (step Sm_1) and corrects the prediction image using, for example, any of the modes described above (step Sm_2).

[0400] Figure 33 This is a flowchart illustrating another example of generating a predicted image.

[0401] The inter-frame prediction unit 126 determines the motion vector of the current block (step Sn_1). Next, the inter-frame prediction unit 126 generates a prediction image (step Sn_2) and determines whether to perform correction processing (step Sn_3). Here, when it is determined that correction processing should be performed (yes in step Sn_3), the inter-frame prediction unit 126 generates a final prediction image by correcting the prediction image (step Sn_4). On the other hand, when it is determined that correction processing should not be performed (no in step Sn_3), the inter-frame prediction unit 126 outputs the prediction image as the final prediction image without correcting the prediction image (step Sn_5).

[0402] Further, in the motion compensation, there is a mode of correcting the luminance at the time of generating the prediction image. This mode is, for example, LIC described later.

[0403] Figure 34 is a flowchart showing another example of generating a prediction image.

[0404] The inter prediction section 126 derives the motion vector of the current block (step So_1). Next, the inter prediction section 126 determines whether or not to perform the luminance correction processing (step So_2). Here, when it is determined to perform the luminance correction processing (Yes in step So_2), the inter prediction section 126 generates the prediction image while performing the luminance correction (step So_3). That is, the prediction image is generated by LIC. On the other hand, when it is determined not to perform the luminance correction processing (No in step So_2), the inter prediction section 126 generates the prediction image by the usual motion compensation without performing the luminance correction (step So_4).

[0405] [Motion compensation > OBMC]

[0406] Not only the motion information of the current block obtained by the motion search but also the motion information of the neighboring block can be used to generate the inter prediction signal. Specifically, it is also possible to generate the inter prediction signal in the sub-block unit within the current block by weightedly adding a prediction signal based on the motion information obtained by the motion search (within the reference picture) and a prediction signal based on the motion information of the neighboring block (within the current picture). Such inter prediction (motion compensation) is sometimes referred to as OBMC (overlapped block motion compensation).

[0407] In the OBMC mode, information indicating the size of the sub-block for the OBMC (for example, referred to as OBMC block size) can also be signaled at the sequence level. Also, information indicating whether or not to apply the OBMC mode (for example, referred to as OBMC flag) can also be signaled at the CU level. In addition, the level of the signaling of these information does not need to be limited to the sequence level and the CU level, and can be another level (for example, the picture level, the slice level, the tile level, the CTU level, or the sub-block level).

[0408] An example of the OBMC mode is described more specifically. Figure 35 and Figure 36 is a flowchart and a conceptual diagram for explaining the outline of the prediction image correction processing based on the OBMC processing.

[0409] First, as shown in Figure 36 , the prediction image (Pred) based on the usual motion compensation is obtained using the motion vector (MV) assigned to the processing target (current) block. In theFigure 36 In this case, the arrow "MV" points to the reference picture and indicates which block of the current picture refers to in order to obtain the prediction image.

[0410] Next, the motion vector (MV_L) that has been derived for the already coded left neighboring block is applied (reused) to the block to be coded to obtain a prediction image (Pred_L). The motion vector (MV_L) is indicated by the arrow "MV_L" pointing from the current block to the reference picture. Then, a first correction of the prediction image is performed by overlapping the two prediction images Pred and Pred_L. This has the effect of blending the boundaries between neighboring blocks.

[0411] Likewise, the motion vector (MV_U) that has been derived for the already coded upper neighboring block is applied (reused) to the block to be coded to obtain a prediction image (Pred_U). The motion vector (MV_U) is indicated by the arrow "MV_U" pointing from the current block to the reference picture. Then, a second correction of the prediction image is performed by overlapping the prediction image Pred_U with the prediction image that has been subjected to the first correction (e.g., Pred and Pred_L). This has the effect of blending the boundaries between neighboring blocks. The prediction image obtained by the second correction is the final prediction image of the current block with the boundaries to the neighboring blocks being blended (smoothed).

[0412] Further, the above-described example is a two-path correction method using the left and upper neighboring blocks, but the correction method can also be a three-path or more path correction method using the right and / or lower neighboring blocks as well.

[0413] In addition, the region in which the overlapping is performed can not be the entire pixel region of the block, but only a partial region near the block boundary.

[0414] In addition, the prediction image correction processing of the OBMC that is described herein is for obtaining one prediction image Pred by overlapping one reference picture with the additional prediction images Pred_L and Pred_U. However, in the case where the prediction image is corrected based on multiple reference images, the same processing can be applied to the multiple reference pictures respectively. In this case, by performing the OBMC image correction based on multiple reference images, after the corrected prediction images are obtained from the respective reference pictures, the final prediction image is obtained by further overlapping the obtained multiple corrected prediction images.

[0415] In addition, in the OBMC, the unit of the block to be processed can be the prediction block unit, or the sub-block unit obtained by further dividing the prediction block.

[0416] As a method of determining whether to apply the OBMC process, for example, there is a method of using a signal indicating whether to apply the OBMC process, that is, obmc_flag. As a specific example, the encoding device can determine whether the target block belongs to a region of complex motion. The encoding device encodes while applying the OBMC process by setting a value 1 to obmc_flag in the case of belonging to the region of complex motion, and encodes without applying the OBMC process by setting a value 0 to obmc_flag in the case of not belonging to the region of complex motion. On the other hand, in the decoding device, the obmc_flag described in the stream (for example, a compressed sequence) is decoded, and whether to apply the OBMC process is switched according to the value to decode.

[0417] In the example described above, the inter prediction section 126 generates one rectangular prediction image for the rectangular current block. However, the inter prediction section 126 can generate a plurality of prediction images having shapes different from a rectangle for the rectangular current block, and can generate a final rectangular prediction image by combining the plurality of prediction images. The shapes different from a rectangle can be, for example, triangles.

[0418] Figure 37 is a conceptual diagram for explaining generation of prediction images of two triangles.

[0419] The inter prediction section 126 generates a triangular prediction image by performing motion compensation using the first MV of the first triangular partition within the current block. Similarly, the inter prediction section 126 generates a triangular prediction image by performing motion compensation using the second MV of the second triangular partition within the current block. Then, the inter prediction section 126 generates a rectangular prediction image having the same shape as the current block by combining the prediction images.

[0420] Further, in the example shown in Figure 37 In the example shown in Figure 37 In the example shown in

[0421] In addition, the first partition and the second partition can be repeated. That is, the first partition and the second partition can include the same pixel region. In this case, the prediction image in the first partition and the prediction image in the second partition can be used to generate the prediction image of the current block.

[0422] In addition, in this example, an example in which both the first partition and the second partition generate prediction images by inter prediction is shown, but prediction images can be generated by intra prediction for at least one of the partitions.

[0423] [motion compensation > BIO]

[0424] Next, a method of deriving a motion vector is described. First, a mode of deriving a motion vector based on a model assuming constant velocity straight line motion is described. This mode is sometimes referred to as a BIO (bi-directional optical flow) mode.

[0425] Figure 38 is a conceptual diagram for explaining the model assuming constant velocity straight line motion. In Figure 38 , (v x , v y ) represents a velocity vector, τ0, τ1 represent distances in time between a current picture (Cur Pic) and two reference pictures (Ref0, Ref1), respectively. (MVx0, MVy0) represents a motion vector corresponding to the reference picture Ref0, and (MVx1, MVy1) represents a motion vector corresponding to the reference picture Ref1.

[0426] At this time, it is also possible that (MVx0, MVy0) and (MVx1, MVy1) are represented as (vxτ0, vyτ0) and (−vxτ1, −vyτ1), respectively, under the assumption of constant velocity straight line motion of the velocity vector (v x , v y ), and the following optical flow equation (2) is used.

[0427] [Equation 3]

[0428]

[0429] Here, I(k) represents a luminance value of a reference picture k (k = 0, 1) after motion compensation. This optical flow equation represents that the sum of (i) a time differential of a luminance value, (ii) a product of a horizontal component of a velocity and a horizontal component of a spatial gradient of a reference picture, and (iii) a product of a vertical component of a velocity and a vertical component of a spatial gradient of a reference picture is equal to zero. It is also possible that a block unit motion vector obtained from a merge list or the like is corrected in a pixel unit based on a combination of this optical flow equation and Hermite interpolation.

[0430] In addition, it is also possible to derive a motion vector on the decoding apparatus side by a method different from the derivation of a motion vector based on a model assuming constant velocity straight line motion. For example, it is also possible to derive a motion vector in a sub-block unit based on motion vectors of a plurality of neighboring blocks.

[0431] [motion compensation > LIC]

[0432] Next, an example of a mode using LIC (local illumination compensation) processing to generate a prediction image (prediction) will be described.

[0433] Figure 39 is a conceptual diagram for explaining an example of a prediction image generation method using luminance correction processing based on LIC processing.

[0434] First, the MV is derived from the coded reference picture, and the reference image corresponding to the current block is obtained.

[0435] Next, information indicating how luminance values change in the reference picture and the current picture is extracted for the current block. The extraction is performed based on luminance pixel values of a coded left neighboring reference region (peripheral reference region) and a coded upper neighboring reference region (peripheral reference region) in the current picture, and luminance pixel values at equivalent positions in the reference picture specified by the derived MV. Then, using the information indicating how luminance values change, a luminance correction parameter is calculated.

[0436] By applying the above-described luminance correction parameter to the reference image in the reference picture specified by the MV, a prediction image for the current block is generated.

[0437] In addition, Figure 39 The shape of the above-described peripheral reference region in is an example, and shapes other than this can also be used.

[0438] Further, the processing of generating a prediction image from one reference picture has been described here, but the same applies in the case of generating a prediction image from a plurality of reference pictures, and a prediction image can also be generated after performing luminance correction processing on the reference images obtained from each of the reference pictures in the same manner as described above.

[0439] As a method of determining whether or not to adopt LIC processing, for example, there is a method of using lic_flag as a signal indicating whether or not to adopt LIC processing. As a specific example, in the encoding device, it is determined whether or not the current block belongs to a region in which luminance changes have occurred, and in the case of belonging to a region in which luminance changes have occurred, a value of 1 is set as lic_flag, and encoding is performed using LIC processing, and in the case of not belonging to a region in which luminance changes have occurred, a value of 0 is set as lic_flag, and encoding is performed without using LIC processing. On the other hand, in the decoding device, lic_flag described in the stream can be decoded, and whether or not to adopt LIC processing is switched according to the value, and decoding is performed.

[0440] Other methods for determining whether to use LIC processing include methods based on whether LIC processing was used in surrounding blocks. As a specific example, when the current block is in merge mode, it is determined whether the surrounding encoded blocks selected during the export of the MV in merge mode processing were encoded using LIC processing. Based on the result, encoding is switched between using LIC processing and other methods. Furthermore, in this example, the same processing also applies to the decoding device.

[0441] use Figure 39 The form of LIC processing (luminance correction processing) has been described. The following is a detailed explanation of its content.

[0442] First, the inter-frame prediction unit 126 derives motion vectors from a reference image, which is a coded image, to obtain a reference image corresponding to the coded object block.

[0443] Next, the inter-frame prediction unit 126 uses the luminance pixel values ​​of the coded target block from the left and top adjacent encoded peripheral reference regions and the luminance pixel values ​​at the same position in the reference image specified by the motion vector to extract information indicating how the luminance values ​​change in the reference image and the coded target image, and calculates luminance correction parameters. For example, the luminance pixel value of a pixel in the peripheral reference region of the coded target image is set to p0, and the luminance pixel value of a pixel in the peripheral reference region of the reference image at the same position as that pixel is set to p1. The inter-frame prediction unit 126 calculates coefficients A and B for optimizing A×p1+B=p0 as luminance correction parameters for multiple pixels in the peripheral reference region.

[0444] Next, the inter-frame prediction unit 126 performs brightness correction processing on the reference image within the reference image specified by the motion vector using brightness correction parameters, generating a prediction image for the coded object block. For example, the brightness pixel value in the reference image is set as p2, and the brightness pixel value of the brightness-corrected prediction image is set as p3. The inter-frame prediction unit 126 generates the brightness-corrected prediction image by calculating A×p2+B=p3 for each pixel in the reference image.

[0445] also, Figure 39 The shape of the surrounding reference area in the image is one example; other shapes can also be used. Additionally, you can also use... Figure 39 The peripheral reference region shown is a portion of the surrounding reference region. For example, a region containing a predetermined number of pixels that have been removed at intervals from the upper adjacent pixel and the left adjacent pixel can also be used as the peripheral reference region. Furthermore, the peripheral reference region is not limited to the region adjacent to the coded object block, but can also be a region that is not adjacent to the coded object block. The predetermined number associated with the pixels can also be determined in advance.

[0446] In addition,Figure 39 In the example shown, the peripheral reference region in the reference picture is a region specified by a motion vector of the peripheral reference region in the coding target picture, but can also be a region specified by another motion vector. For example, the other motion vector can also be a motion vector of the peripheral reference region in the coding target picture.

[0447] Also, here, the operation in the encoding apparatus 100 is explained, but typically, the operation in the decoding apparatus 200 is also the same.

[0448] Further, the LIC process can be applied not only to the luminance but also to the color difference. At this time, the correction parameter can be derived individually for each of Y, Cb, and Cr, or a common correction parameter can be used for any of them.

[0449] Further, the LIC process can also be applied in sub-block units. For example, the correction parameter can be derived using the peripheral reference region of the current sub-block and the peripheral reference region of the reference sub-block in the reference picture specified by the MV of the current sub-block.

[0450] [Prediction control section]

[0451] The prediction control section 128 selects either the intra prediction signal (the signal output from the intra prediction section 124) or the inter prediction signal (the signal output from the inter prediction section 126), and outputs the selected signal as a prediction signal to the subtraction section 104 and the addition section 116.

[0452] As shown in Figure 1 In various encoding apparatus examples, the prediction control section 128 can also output a prediction parameter input to the entropy encoding section 110. The entropy encoding section 110 can generate an encoded bitstream (or sequence) based on the prediction parameter input from the prediction control section 128 and the quantization coefficients input from the quantization section 108. The prediction parameter can also be used in a decoding apparatus. The decoding apparatus can receive an encoded bitstream and decode it, and perform the same processing as the prediction processing performed in the intra prediction section 124, the inter prediction section 126, and the prediction control section 128. The prediction parameter can include a selected prediction signal (e.g., a motion vector, a prediction type, or a prediction mode used by the intra prediction section 124 or the inter prediction section 126), or any index, flag, or value based on or representing the prediction processing performed in the intra prediction section 124, the inter prediction section 126, and the prediction control section 128.

[0453] [Installation example of encoding apparatus]

[0454] Figure 40 is a block diagram showing an installation example of the encoding apparatus 100. The encoding apparatus 100 is provided with a processor a1 and a memory a2. For example, Figure 1The illustrated plurality of components of the encoding apparatus 100 are implemented by Figure 40 the processor a1 and the memory a2.

[0455] The processor a1 is a circuit that performs information processing, and is a circuit that can access the memory a2. For example, the processor a1 is a dedicated or general electronic circuit that encodes a moving image. The processor a1 can also be a processor like a CPU. In addition, the processor a1 can also be a collection of a plurality of electronic circuits. In addition, for example, the processor a1 can also function as a plurality of components of the encoding apparatus 100 illustrated in Figure 1 and the like.

[0456] The memory a2 is a dedicated or general memory that stores information used by the processor a1 to encode a moving image. The memory a2 can be an electronic circuit, or can be connected to the processor a1. In addition, the memory a2 can be included in the processor a1. In addition, the memory a2 can also be a collection of a plurality of electronic circuits. In addition, the memory a2 can be a magnetic disk or an optical disk, or can be represented as a storage or a recording medium. In addition, the memory a2 can be a non-volatile memory, or can be a volatile memory.

[0457] For example, the memory a2 can store an encoded moving image, or can store a bit string corresponding to the encoded moving image. In addition, a program used by the processor a1 to encode a moving image can also be stored in the memory a2.

[0458] In addition, for example, the memory a2 can function as a component for storing information in the plurality of components of the encoding apparatus 100 illustrated in Figure 1 and the like. For example, the memory a2 can function as the block memory 118 and the frame memory 122 illustrated in Figure 1 More specifically, the memory a2 can store a reconstructed block and a reconstructed picture, and the like.

[0459] In addition, in the encoding apparatus 100, all of the plurality of components illustrated in Figure 1 and the like can not be installed, and all of the plurality of processes described above can not be performed. Figure 1 In addition, a part of the plurality of components illustrated in

[0460] [Decoding Apparatus]

[0461] Next, a decoding apparatus that can decode an encoded signal (encoded bit stream) output from, for example, the above-described encoding apparatus 100 will be described. Figure 41is a block diagram showing a functional configuration of a decoding apparatus 200 according to an embodiment. The decoding apparatus 200 is a moving image decoding apparatus that decodes a moving image in a block unit.

[0462] As shown in Figure 41 , the decoding apparatus 200 includes an entropy decoding section 202, an inverse quantization section 204, an inverse transform section 206, an addition section 208, a block memory 210, a loop filter section 212, a frame memory 214, an intra prediction section 216, an inter prediction section 218, and a prediction control section 220.

[0463] The decoding apparatus 200 is realized by, for example, a general-purpose processor and a memory. In this case, when a software program stored in the memory is executed by the processor, the processor functions as the entropy decoding section 202, the inverse quantization section 204, the inverse transform section 206, the addition section 208, the loop filter section 212, the intra prediction section 216, the inter prediction section 218, and the prediction control section 220. Alternatively, the decoding apparatus 200 can be realized by one or more electronic circuits dedicated to the entropy decoding section 202, the inverse quantization section 204, the inverse transform section 206, the addition section 208, the loop filter section 212, the intra prediction section 216, the inter prediction section 218, and the prediction control section 220.

[0464] Hereinafter, after the flow of the overall processing of the decoding apparatus 200 is described, each constituent element included in the decoding apparatus 200 is described.

[0465] [Overall flow of decoding processing]

[0466] Figure 42 is a flowchart showing an example of the overall decoding processing performed by the decoding apparatus 200.

[0467] First, the entropy decoding section 202 of the decoding apparatus 200 determines a partition pattern of a fixed-size block (for example, 128 x 128 pixels) (step Sp_1). The partition pattern is a partition pattern selected by the encoding apparatus 100. Then, the decoding apparatus 200 performs the processing of steps Sp_2 to Sp_6 on each of the blocks constituting the partition pattern.

[0468] That is, the entropy decoding section 202 decodes (specifically, entropy decodes) the encoded quantization coefficients and the prediction parameters of a block to be decoded (also referred to as a current block) (step Sp_2).

[0469] Next, the inverse quantization section 204 and the inverse transform section 206 reproduce a plurality of prediction residuals (that is, difference blocks) by performing inverse quantization and inverse transform on the plurality of quantization coefficients (step Sp_3).

[0470] Next, a prediction processing section constituted by all or a part of the intra prediction section 216, the inter prediction section 218, and the prediction control section 220 generates a prediction signal (also referred to as a prediction block) of the current block (Step Sp_4).

[0471] Next, the addition section 208 reconstructs the current block into a reconstructed image (also referred to as a decoded image block) by adding the prediction block to the difference block (Step Sp_5).

[0472] Also, when the reconstructed image is generated, the loop filter 212 filters the reconstructed image (Step Sp_6).

[0473] Then, the decoding apparatus 200 determines whether or not the decoding of the picture as a whole has been completed (Step Sp_7), and in the case where it is determined that the decoding has not been completed (NO in Step Sp_7), the processing from Step Sp_1 is repeatedly executed.

[0474] As illustrated, the processing of Steps Sp_1 to Sp_7 is sequentially performed by the decoding apparatus 200, or a plurality of processes of a part of these processes can be performed in parallel, or the order of the processes can be changed, and the like.

[0475] [Entropy Decoding Section]

[0476] The entropy decoding section 202 entropy-decodes the encoded bitstream. Specifically, the entropy decoding section 202, for example, arithmetic-decodes the encoded bitstream into a binary signal. Next, the entropy decoding section 202 debinarizes the binary signal. Thereby, the entropy decoding section 202 outputs the quantized coefficients to the inverse quantization section 204 in a block unit. The entropy decoding section 202 can also output the prediction parameters included in the encoded bitstream (refer to Figure 1 ) to the intra prediction section 216, the inter prediction section 218, and the prediction control section 220 in the embodiment. The intra prediction section 216, the inter prediction section 218, and the prediction control section 220 can perform the same prediction processing as the processing performed by the intra prediction section 124, the inter prediction section 126, and the prediction control section 128 on the encoding apparatus side.

[0477] [Inverse Quantization Section]

[0478] The inverse quantization section 204 inverse-quantizes the quantized coefficients of a decoded target block (hereinafter referred to as a current block) that is input from the entropy decoding section 202. Specifically, the inverse quantization section 204 inverse-quantizes the quantized coefficients of the current block on a per-quantized coefficient basis based on the quantization parameters corresponding to the quantized coefficients. Also, the inverse quantization section 204 outputs the inverse-quantized quantized coefficients (i.e., transform coefficients) of the current block to the inverse transform section 206.

[0479] [Inverse Transform Section]

[0480] The inverse transform section 206 restores the prediction error by performing inverse transform on the transform coefficient input from the inverse quantization section 204.

[0481] For example, in a case where the information read from the coded bitstream indicates that EMT or AMT is employed (e.g., the AMT flag is true), the inverse transform section 206 performs inverse transform on the transform coefficient of the current block based on the information read from the coded bitstream indicating the transform type.

[0482] Further, for example, in a case where the information read from the coded bitstream indicates that NSST is employed, the inverse transform section 206 applies inverse retransform on the transform coefficient.

[0483] [Addition section]

[0484] The addition section 208 reconstructs the current block by adding the prediction error input from the inverse transform section 206 to the prediction sample input from the prediction control section 220. Further, the addition section 208 outputs the reconstructed block to the block memory 210 and the loop filter 212.

[0485] [Block memory]

[0486] The block memory 210 is a storage section for storing a block within a decoded target picture (hereinafter referred to as a current picture) referred to in intra prediction. Specifically, the block memory 210 stores the reconstructed block output from the addition section 208.

[0487] [Loop filter]

[0488] The loop filter 212 applies loop filtering to the block reconstructed by the addition section 208, and outputs the filtered reconstructed block to the frame memory 214 and a display device or the like.

[0489] In a case where the information read from the coded bitstream indicating the on / off of ALF indicates the on of ALF, one filter is selected from among a plurality of filters based on the direction and activity of the gradient of locality, and the selected filter is applied to the reconstructed block.

[0490] [Frame memory]

[0491] The frame memory 214 is a storage section for storing a reference picture used in inter prediction, and is also referred to as a frame buffer. Specifically, the frame memory 214 stores the reconstructed block filtered by the loop filter 212.

[0492] [Prediction processing section (intra prediction section / inter prediction section / prediction control section)]

[0493] Figure 43is a flowchart showing an example of the processing performed by the prediction processing section of the decoding device 200. Further, the prediction processing section is constituted by all or a part of the constituent elements of the intra prediction section 216, the inter prediction section 218, and the prediction control section 220.

[0494] The prediction processing section generates a prediction image of the current block (step Sq_1). The prediction image is also referred to as a prediction signal or a prediction block. Further, in the prediction signal, there are, for example, an intra prediction signal or an inter prediction signal. Specifically, the prediction processing section generates the prediction image of the current block using a reconstructed image that has been obtained by performing the generation of a prediction block, the generation of a difference block, the generation of a coefficient block, the restoration of a difference block, and the generation of a decoded image block.

[0495] The reconstructed image can be, for example, an image of a reference picture or an image of a decoded block within a current picture that contains the current block. The decoded block within the current picture is, for example, a neighboring block of the current block.

[0496] Figure 44 is a flowchart showing another example of the processing performed by the prediction processing section of the decoding device 200.

[0497] The prediction processing section determines a manner or mode for generating a prediction image (step Sr_1). The manner or mode can be determined, for example, on the basis of, for example, a prediction parameter or the like.

[0498] In a case where it is determined that the first manner is the mode for generating a prediction image, the prediction processing section generates a prediction image in accordance with the first manner (step Sr_2a). Further, in a case where it is determined that the second manner is the mode for generating a prediction image, the prediction processing section generates a prediction image in accordance with the second manner (step Sr_2b). Further, in a case where it is determined that the third manner is the mode for generating a prediction image, the prediction processing section generates a prediction image in accordance with the third manner (step Sr_2c).

[0499] The first manner, the second manner, and the third manner are mutually different manners for generating a prediction image, and can be, for example, an inter prediction manner, an intra prediction manner, and another prediction manner. In such a prediction manner, the reconstructed image described above can also be used.

[0500] [Intra Prediction Section]

[0501] The intra prediction section 216 performs intra prediction with reference to the blocks within the current picture saved in the block memory 210 on the basis of the intra prediction mode read out from the coded bitstream, thereby generating a prediction signal (intra prediction signal). Specifically, the intra prediction section 216 performs intra prediction with reference to the samples (e.g., luminance values, color difference values) of the blocks neighboring the current block, thereby generating an intra prediction signal, which is output to the prediction control section 220.

[0502] In addition, in a case where the intra prediction mode of the reference luminance block is selected in the intra prediction of the color difference block, the intra prediction section 216 can also predict the color difference component of the current block based on the luminance component of the current block.

[0503] Further, in a case where the information read from the coded bitstream indicates that PDPC is employed, the intra prediction section 216 corrects the pixel values after the intra prediction based on the gradient of the reference pixels in the horizontal / vertical direction.

[0504] [Inter prediction section]

[0505] The inter prediction section 218 refers to the reference picture stored in the frame memory 214 to predict the current block. The prediction is performed in units of the current block or sub-blocks (e.g., 4x4 blocks) within the current block. For example, the inter prediction section 218 performs motion compensation using the motion information (e.g., motion vector) read from the coded bitstream (e.g., the prediction parameters output from the entropy decoding section 202), thereby generating an inter prediction signal of the current block or sub-block, and outputs the inter prediction signal to the prediction control section 220.

[0506] In a case where the information read from the coded bitstream indicates that the OBMC mode is employed, the inter prediction section 218 generates the inter prediction signal using not only the motion information of the current block obtained by the motion estimation but also the motion information of the neighboring blocks.

[0507] Further, in a case where the information read from the coded bitstream indicates that the FRUC mode is employed, the inter prediction section 218 performs the motion estimation according to the pattern matching method (bi-directional matching or template matching) read from the coded stream, thereby deriving the motion information. Also, the inter prediction section 218 performs the motion compensation (prediction) using the derived motion information.

[0508] Further, in a case where the BIO mode is employed, the inter prediction section 218 derives the motion vector based on a model assuming constant velocity straight line motion. Further, in a case where the information read from the coded bitstream indicates that the affine motion compensation prediction mode is employed, the inter prediction section 218 derives the motion vector in sub-block units based on the motion vectors of a plurality of neighboring blocks.

[0509] [MV derivation > normal inter mode]

[0510] In a case where the information read from the coded bitstream indicates that the normal inter mode is applied, the inter prediction section 218 derives the MV based on the information read from the coded bitstream, and performs the motion compensation (prediction) using the MV.

[0511] Figure 45 is a flowchart showing an example of the inter prediction based on the normal inter mode in the decoding apparatus 200.

[0512] The inter prediction section 218 of the decoding apparatus 200 performs motion compensation for each block. The inter prediction section 218 acquires a plurality of candidate MVs for a current block based on information such as MVs of a plurality of decoded blocks that are temporally or spatially located around the current block (step Ss_1). That is, the inter prediction section 218 creates a list of candidate MVs.

[0513] Next, the inter prediction section 218 extracts N (N is an integer of 2 or more) candidate MVs as prediction motion vector candidates (also referred to as prediction MV candidates) in a prescribed priority order from the plurality of candidate MVs acquired in step Ss_1 (step Ss_2). Note that the priority order can be determined in advance for each of the N prediction MV candidates.

[0514] Next, the inter prediction section 218 decodes prediction motion vector selection information from the input stream (i.e., the encoded bitstream), and uses the decoded prediction motion vector selection information to select one of the N prediction MV candidates as a prediction MV (also referred to as a prediction MV) for the current block (step Ss_3).

[0515] Next, the inter prediction section 218 decodes the differential MV from the input stream, and derives the MV for the current block by adding the differential value of the decoded differential MV to the selected prediction motion vector (step Ss_4).

[0516] Finally, the inter prediction section 218 generates a prediction image for the current block by performing motion compensation on the current block using the derived MV and the decoded reference picture (step Ss_5).

[0517] [Prediction control section]

[0518] The prediction control section 220 selects one of the intra prediction signal and the inter prediction signal, and outputs the selected signal as a prediction signal to the addition section 208. In general, the structure, functions, and processing of the prediction control section 220, the intra prediction section 216, and the inter prediction section 218 on the decoding apparatus side can correspond to the structure, functions, and processing of the prediction control section 128, the intra prediction section 124, and the inter prediction section 126 on the encoding apparatus side.

[0519] [Installation example of decoding apparatus]

[0520] Figure 46 is a block diagram showing an installation example of the decoding apparatus 200. The decoding apparatus 200 includes a processor b1 and a memory b2. For example, Figure 41 The plurality of constituent elements of the decoding apparatus 200 shown in Figure 46The illustrated processor b1 and memory b2 are installed.

[0521] The processor b1 is a circuit that performs information processing, and is a circuit that can access the memory b2. For example, the processor b1 is a dedicated or general electronic circuit that decodes an encoded moving image (i.e., an encoded bitstream). The processor b1 can also be a CPU-like processor. In addition, the processor b1 can also be a collection of multiple electronic circuits. In addition, for example, the processor b1 can also function as Figure 41 The illustrated processor b1 and memory b2 are installed.

[0522] The memory b2 is a dedicated or general memory that stores information used by the processor b1 to decode an encoded bitstream. The memory b2 can be an electronic circuit, and can be connected to the processor b1. In addition, the memory b2 can be included in the processor b1. In addition, the memory b2 can also be a collection of multiple electronic circuits. In addition, the memory b2 can be a magnetic disk or an optical disk, or can be represented as a storage or a recording medium. In addition, the memory b2 can be a non-volatile memory, or can be a volatile memory.

[0523] For example, the memory b2 can store a moving image, or can store an encoded bitstream. In addition, a program used by the processor b1 to decode an encoded bitstream can also be stored in the memory b2.

[0524] In addition, for example, the memory b2 can function as Figure 41 The illustrated processor b1 and memory b2 are installed. Figure 41 In particular, the memory b2 can function as the illustrated block memory 210 and frame memory 214. More specifically, the memory b2 can store a reconstructed block and a reconstructed picture, or the like.

[0525] In addition, in the decoding apparatus 200, all of the illustrated multiple components can not be installed, and all of the illustrated multiple processes can not be performed. Figure 41 In addition, in the decoding apparatus 200, all of the illustrated multiple components can not be installed, and all of the illustrated multiple processes can not be performed. Figure 41 In addition, a part of the illustrated multiple components can be included in another apparatus, and a part of the illustrated multiple processes can be performed by another apparatus.

[0526] [Definitions of Each Term]

[0527] Each term can also be defined as follows, as an example.

[0528] A picture is an arrangement of multiple luma samples in monochrome format, or an arrangement of multiple luma samples and two corresponding arrangements of multiple chroma samples in color formats of 4:2:0, 4:2:2, and 4:4:4. A picture can be a frame or a field.

[0529] A frame is a combination of a top field generating multiple sample rows 0, 2, 4,... and a bottom field generating multiple sample rows 1, 3, 5,....

[0530] A slice contains an integer number of coding tree units for one independent slice segment, and, if any, all subsequent dependent slice segments before the next independent slice segment, if any, within the same access unit.

[0531] A tile is a rectangular region of multiple coding tree blocks within a particular tile column and a particular tile row in a picture. A tile can still apply a loop filter across tile edges, but can also be a rectangular region of a frame intended to be decodable and encodable independently.

[0532] A block is an MxN (N rows by M columns) arrangement of multiple samples, or an MxN arrangement of multiple transform coefficients. A block can also be a square or rectangular region of multiple pixels consisting of one luma and two chroma matrices.

[0533] A CTU (coding tree unit) can be a coding tree block of multiple luma samples of a picture having three sample arrangements, or two corresponding coding tree blocks of multiple chroma samples. Alternatively, a CTU can be a coding tree block of any multiple samples in a monochrome picture, and a picture encoded using syntax constructs used in encoding of three separate color planes and multiple samples.

[0534] A superblock constitutes one or two mode information blocks, or can also be recursively split into four 32x32 blocks, and further into a 64x64 pixel square block that can be split.

[0535] [1st mode of coefficient encoding]

[0536] Figure 47 is a flowchart showing a basic coefficient encoding method of the 1st mode. Specifically, Figure 47 is a coefficient encoding method of a region for which a prediction residual is obtained by intra-frame encoding or inter-frame encoding. In the following description, actions performed by the encoding device 100 are shown. The decoding device 200 can perform actions corresponding to the actions performed by the encoding device 100. For example, the decoding device 200 can perform inverse orthogonal transformation and decoding corresponding to orthogonal transformation and encoding performed by the encoding device 100.

[0537] In Figure 47In the present embodiment, last_sig_coeff, subblock_flag, thres, and CCB are shown. The last_sig_coeff is a parameter indicating a coordinate position at which a coefficient that is not zero (a non-zero coefficient) first appears when scanning within a block. The subblock_flag is a flag indicating whether there is a coefficient that is not zero in a 4x4 subblock (also referred to as a 16 transform coefficient level). The subblock_flag can also be expressed as coded_sub_block_flag or a subblock flag.

[0538] The thres is a constant determined in units of blocks. The thres can be determined in advance. The thres can be a value that differs depending on the size of a block, or can be the same value regardless of the size of a block. The thres can take different values in a case where a orthogonal transform is applied and in a case where the orthogonal transform is not applied. The thres can be determined depending on a coordinate position determined in a block by the last_sig_coeff.

[0539] The CCB indicates the number of bins that are encoded in a context mode of CABAC (Context-Adaptive Binary Arithmetic Coding). That is, the CCB indicates the number of times of processing of encoding based on the context mode of CABAC. The context mode is also referred to as a regular mode. Here, the encoding based on the context mode of CABAC is referred to as CABAC encoding or context-adaptive encoding. In addition, the encoding based on a bypass mode of CABAC is referred to as bypass encoding. The processing of bypass encoding is simpler than that of CABAC encoding.

[0540] The CABAC encoding is a process of transforming a string of bins obtained by binarizing a signal of an encoding target into a string of encoding bits, based on a probability of occurrence of 0 and 1 for each bin. In addition, the CCB can count the number of all flags used in residual coefficient encoding, or can count the number of a part of flags used in residual coefficient encoding. The bypass encoding is a process of encoding 1 bin in a string of bins as 1 bit of an encoding bit string, without using a variable probability of occurrence of 0 and 1 for each bin (in other words, using a fixed probability).

[0541] For example, the encoding apparatus 100 compares the CCB value and the thres value to determine the manner of coefficient encoding.

[0542] Specifically, in a case where the CCB value is smaller than the thres value, the encoding apparatus 100 determines to perform CABAC encoding. Figure 47In the present embodiment, first, the CCB is initialized to 0 (S101). Then, it is determined whether or not the orthogonal transform is applied to the block (S102). In the case where the orthogonal transform is applied to the block (Yes in S102), the encoding device 100 encodes the last_sig_coeff (S131). Then, the encoding device 100 performs the loop processing for each subblock (S141 to S148).

[0543] In the loop processing for each subblock (S141 to S148), the encoding device 100 encodes the subblock_flag related to the subblock. Then, in the case where the subblock_flag is different from 0 (Yes in S146), the encoding device 100 encodes the 16 coefficients in the subblock by the 1st encoding method described later (S147).

[0544] In addition, in the case where the orthogonal transform is not applied to the block (No in S102), the encoding device 100 performs the loop processing for each subblock (S121 to S128).

[0545] In the loop processing for each subblock (S121 to S128), the encoding device 100 determines whether or not the CCB is thres or less (S122). If the CCB is thres or less (Yes in S122), the encoding device 100 encodes the subblock_flag by CABAC encoding (S123). Then, the encoding device 100 counts up the CCB (S124). Otherwise (No in S122), the encoding device 100 encodes the subblock_flag by bypass encoding (S125).

[0546] Then, in the case where the subblock_flag is different from 0 (Yes in S126), the encoding device 100 encodes the 16 coefficients in the subblock by the 2nd encoding method described later (S127).

[0547] The case where the orthogonal transform is not applied to the block can be, for example, the case where the orthogonal transform is skipped. The CCB is used in both the 1st encoding method and the 2nd encoding method. The CCB can be initialized in the unit of subblock. In this case, thres can not be a fixed value in the block, but a value that varies for each subblock.

[0548] In addition, here, the CCB is counted up from 0 and it is determined whether or not thres is reached, but the CCB can be counted down from thres (or a specific value) and it is determined whether or not 0 is reached.

[0549] Figure 48 is a flag indicating Figure 47A flowchart showing details of the first encoding method. In the first encoding method, a plurality of coefficients within a sub-block are encoded. At this time, a first loop process (S151-S156) is performed for each coefficient information flag of each coefficient within the sub-block, and a second loop process (S161-S165) is performed for each coefficient within the sub-block.

[0550] In the first loop process (S151-S156), one or more coefficient information flags each indicating one or more attributes of a coefficient are sequentially encoded. The one or more coefficient information flags can include sig_flag, gt1_flag, parity_flag, and gt3_flag described later. Then, within a range where the CCB does not exceed thres, the one or more coefficient information flags are sequentially encoded by CABAC encoding, and the CCB is incrementally counted by one each time of encoding. After the CCB exceeds thres, the coefficient information flags are not encoded.

[0551] That is, in the first loop process (S151-S156), the encoding device 100 determines whether the CCB is equal to or less than thres (S152). Then, in a case where the CCB is equal to or less than thres (Yes in S152), the encoding device 100 encodes the coefficient information flags by CABAC encoding (S153). Then, the encoding device 100 incrementally counts the CCB (S154). In a case where the CCB is not equal to or less than thres (No in S152), the encoding device 100 ends the first loop process (S151-S156).

[0552] In the second loop process (S161-S165), for a coefficient for which the coefficient information flag is encoded, a remainder (remainder) that is a remaining value (i.e., a remaining value of a value used for reconstructing the coefficient using the coefficient information flag) that is not expressed by the coefficient information flag is encoded by Golomb encoding. A coefficient for which the coefficient information flag is not encoded is directly encoded by Golomb encoding. In addition, the remainder can be encoded using another encoding method instead of Golomb encoding.

[0553] That is, in the second loop process (S161-S165), the encoding device 100 determines whether the coefficient information flag corresponding to the coefficient of the processing target has been encoded (S162). Then, in a case where the coefficient information flag is encoded (Yes in S162), the encoding device 100 encodes the remainder by Golomb encoding (S163). In a case where the coefficient information flag is not encoded (No in S162), the encoding device 100 encodes the value of the coefficient by Golomb encoding (S164).

[0554] In addition, the number of the loop processes here is 2, but the number of the loop processes can be different from 2.

[0555] The sig_flag described above is a flag indicating whether or not the AbsLevel is non-zero. The AbsLevel is the value of the coefficient, more specifically, the absolute value of the coefficient. The gt1_flag is a flag indicating whether or not the AbsLevel is greater than 1. The parity_flag is a flag of the 1st bit of the AbsLevel, and is a flag indicating whether or not the AbsLevel is odd or even. The gt3_flag is a flag indicating whether or not the AbsLevel is greater than 3.

[0556] The gt1_flag and the gt3_flag are sometimes expressed as abs_gt1_flag and abs_gt3_flag, respectively. In addition, for example, as the remainder described above, the value of (Abslevel - 4) / 2 can be encoded by Golomb coding.

[0557] Other one or more coefficient information flags different from the one or more coefficient information flags described above can also be encoded. For example, a part of the coefficient information flags can not be encoded. The coefficient information flags included in the one or more coefficient information flags described above can be replaced with coefficient information flags or parameters having other meanings.

[0558] Figure 49 is a flag indicating Figure 47 A flowchart showing details of the second encoding mode. In the second encoding mode, a plurality of coefficients within a sub-block are encoded. At this time, a first loop process is performed for each coefficient information flag of each coefficient within the sub-block (S171 to S176), and a second loop process is performed for each coefficient within the sub-block (S181 to S185).

[0559] In the first loop process (S171 to S176), one or more coefficient information flags respectively indicating one or more attributes of the coefficient are sequentially encoded. The one or more coefficient information flags can include a sig_flag, a sign_flag, a gt1_flag, a parity_flag, a gt3_flag, a gt5_flag, a gt7_flag, and a gt9_flag.

[0560] Here, the sign_flag is a flag indicating the sign of the coefficient. The gt5_flag is a flag indicating whether the AbsLevel is greater than 5. The gt7_flag is a flag indicating whether the AbsLevel is greater than 7. The gt9_flag is a flag indicating whether the AbsLevel is greater than 9. The gt5_flag, the gt7_flag, and the gt9_flag are sometimes expressed as abs_gt5_flag, abs_gt7_flag, and abs_gt9_flag, respectively. In addition, a flag indicating whether the AbsLevel is greater than x (x is an integer of 1 or more) can be collectively expressed as gtx_flag or abs_gtx_flag. The AbsLevel is, for example, the absolute value of the transform coefficient level.

[0561] In addition, other one or more coefficient information flags different from the above-described one or more coefficient information flags can be encoded. For example, a part of the coefficient information flags can not be encoded. The coefficient information flags included in the above-described one or more coefficient information flags can be replaced with coefficient information flags or parameters having other meanings.

[0562] The above-described one or more coefficient information flags are sequentially encoded by CABAC encoding. Then, the CCB is incrementally counted by 1 each time of encoding. After the CCB exceeds the thres, the coefficient information flags are encoded by bypass encoding.

[0563] That is, in the first loop process (S171 to S176), the encoding apparatus 100 determines whether the CCB is equal to or less than the thres (S172). Then, in a case where the CCB is equal to or less than the thres (Yes in S172), the encoding apparatus 100 encodes the coefficient information flags by CABAC encoding (S173). Then, the encoding apparatus 100 incrementally counts the CCB (S174). In a case where the CCB is not equal to or less than the thres (No in S172), the encoding apparatus 100 encodes the coefficient information flags by bypass encoding (S175).

[0564] Figure 49 The syntax of the second loop process in the above-described (1) is not changed before and after the CCB exceeds the thres. That is, whether the coefficient information flags are encoded by CABAC encoding or the coefficient information flags are encoded by bypass encoding, the following same process is performed in the second loop process (S181 to S185).

[0565] Specifically, in the 2nd loop processing (S181-S185), the encoding device 100 encodes remainder, which is a residual value that cannot be represented by the coefficient information flag (i.e., a residual value for reconstructing the coefficient value using the coefficient information flag), by the Golomb encoding (S183). In addition, remainder can be encoded by another encoding method instead of the Golomb encoding.

[0566] In addition, the number of loop processes is 2 here, but the number of loop processes can be different from 2.

[0567] As shown in Figure 47 , Figure 48 and Figure 49 , in the basic operation of the present aspect, there are different flags included in the number of times of CABAC encoding depending on whether or not the orthogonal transform is applied. In addition, the syntax of the coefficient encoding is different between the case where the orthogonal transform is applied and the case where the orthogonal transform is not applied. Thus, it can be necessary to prepare circuits separately. Therefore, the circuit structure can become complicated.

[0568] [1st example of 1st aspect of coefficient encoding]

[0569] Figure 50 is a flowchart showing the coefficient encoding method of the 1st example of the 1st aspect. In Figure 50 , the post-processing of last_sig_coeff (S132) and the processing of subblock_flag (S142-S145) are different from those of Figure 47 .

[0570] In Figure 47 , in the case where the orthogonal transform is applied, CCB counts sig_flag, parity_flag, and gtX_flag (X=1, 3) incrementally. In Figure 50 , CCB counts last_sig_coeff and subblock_flag incrementally as well. On the other hand, the processing flow in the case where the orthogonal transform is not applied is the same as that of Figure 47 .

[0571] That is, in the example of Figure 50 , the encoding device 100 encodes last_sig_coeff (S131), and then adds the number of times of CABAC encoding in the encoding of last_sig_coeff to CCB (S132).

[0572] Further, the encoding device 100 determines whether the CCB is equal to or lower than the thres before encoding the subblock_flag (S142). Then, in a case where the CCB is equal to or lower than the thres (Yes in S142), the encoding device 100 encodes the subblock_flag by CABAC encoding (S143). Then, the encoding device 100 adds 1 to the CCB (S144). On the other hand, in a case where the CCB is not equal to or lower than the thres (No in S142), the encoding device 100 encodes the subblock_flag by bypass encoding (S145).

[0573] [Effects of the first example of the first mode of coefficient encoding]

[0574] According to Figure 50 , in a case where the orthogonal transform is performed and in a case where the orthogonal transform is not performed, it is sometimes possible to commonize and unify the processing flow of the encoding of the subblock_flag. Therefore, in the case where the orthogonal transform is performed and in the case where the orthogonal transform is not performed, it is possible to share a part of the circuit, and it is possible to reduce the circuit size. As a result, in addition to the last_sig_coeff, the processing flows divided according to the presence or absence of the orthogonal transform can become the same.

[0575] For example, even in a case where the number of times of CABAC encoding is limited at the block level, in the example of Figure 47 , the subblock_flag is encoded by CABAC encoding after the CCB reaches the thres. On the other hand, in the case of Figure 50 , the subblock_flag is not encoded by CABAC encoding after the CCB reaches the thres. Thus, the number of times of CABAC encoding can be appropriately limited to the thres.

[0576] Further, the number of times of CABAC encoding in the last_sig_coeff can not be included in the CCB. Further, the thres can be determined depending on the coordinate position determined in the block by the last_sig_coeff.

[0577] Further, the encoding device 100 can also encode the subblock_flag with the value always determined as 1 after the CCB exceeds the thres. In a case where the value of the subblock_flag is always determined as 1 after the CCB exceeds the thres, the encoding device 100 can not encode the subblock_flag after the CCB exceeds the thres.

[0578] Further, even in the case where the orthogonal transform is not applied, the encoding apparatus 100 can always determine the value of the subblock_flag as 1 and encode it after the CCB exceeds the thres. Further, in this case, in the case where the value of the subblock_flag is always determined as 1 after the CCB exceeds the thres, the encoding apparatus 100 can not encode the subblock_flag after the CCB exceeds the thres.

[0579] [Example 2 of the 1st Mode of the Coefficient Encoding]

[0580] Figure 51 is a flowchart showing the coefficient encoding method of Example 2 of the 1st Mode. In Figure 51 , the processing of the subblock_flag (S123) is different from that of Figure 47 .

[0581] In Figure 47 , in the case where the orthogonal transform is not applied, the CCB has already been incremented for the sig_flag, the parity_flag, the gtX_flag (X = 1, 3, 5, 7, 9), and the subblock_flag. In Figure 51 , the CCB is not incremented for the subblock_flag. On the other hand, the processing flow in the case where the orthogonal transform is applied is the same as that of Figure 47 .

[0582] That is, in the example of Figure 51 , the encoding apparatus 100 always encodes the subblock_flag by the CABAC encoding regardless of whether the CCB exceeds the thres or not, without incrementing the CCB (S123).

[0583] [Effects of Example 2 of the 1st Mode of the Coefficient Encoding]

[0584] According to the example of Figure 51 , in the case where the orthogonal transform is performed and in the case where the orthogonal transform is not performed, it is sometimes possible to commonize and unify the processing of the encoding of the subblock_flag. Therefore, in the case where the orthogonal transform is performed and in the case where the orthogonal transform is not performed, it is possible that a part of the circuit is commonized, and it is possible to reduce the circuit size. As a result, except for the last_sig_coeff, it is possible that the plurality of processing flows divided according to the presence or absence of the orthogonal transform become the same.

[0585] Further, in Figure 50 , compared to Figure 51The processing is simplified in this way. Therefore, it is possible to reduce the circuit size. Furthermore, it is assumed that the frequency of occurrence of 0s or 1s associated with the subblock_flag is prone to deviation depending on the surrounding conditions. Therefore, in the CABAC encoding of the subblock_flag, it is assumed that the reduction in the amount of code is greater than the increase in processing latency. Therefore, it is useful to limit the number of times the CABAC encoding of the subblock_flag is excluded from the CABAC encoding process.

[0586] Furthermore, the number of times the CABAC encoding of last_sig_coeff is processed can be included in the CCB. Additionally, the thresholds can depend on the coordinates determined by last_sig_coeff within the block.

[0587] [Second form of coefficient encoding]

[0588] [Example 1 of the second form of coefficient encoding]

[0589] Figure 52 This is a flowchart illustrating the coefficient encoding method for the first example of the second form. Figure 52 In the example, even when no orthogonal transformation is applied to the block, the case of encoding the 16 coefficients within the sub-block using the first encoding method (S127a) is also similar to... Figure 47 The examples are different.

[0590] That is, in Figure 52 In the example, when no orthogonal transformation is applied to the block, the encoding device 100 uses... Figure 48 The first encoding method shown is not Figure 49 The second encoding method shown encodes the 16 coefficients within the sub-block (S127a). That is, whether or not an orthogonal transformation is applied, the encoding device 100 does not use... Figure 49 The second encoding method shown is instead used through... Figure 48 The first encoding method shown encodes the 16 coefficients within the sub-block.

[0591] More specifically, regardless of whether there is an orthogonal transformation, the encoding device 100 follows... Figure 48 In the first encoding method shown, during the first loop processing, if the CCB exceeds the threshold, the encoding of the coefficient information flag is skipped without using bypass encoding. Then, during the second loop processing, if the coefficient information flag corresponding to the coefficient of the processing object is not encoded, the encoding device 100 encodes the value of the coefficient using Columbus encoding instead of using the coefficient information flag.

[0592] In addition, Figure 48In the 1st cycle processing of the 1st embodiment, the syntax for encoding the coefficient information flag can be different between the case where the orthogonal transform is applied and the case where the orthogonal transform is not applied. For example, between the one or more coefficient information flags in the case where the orthogonal transform is applied and the one or more coefficient information flags in the case where the orthogonal transform is not applied, some or all of the coefficient information flags can also be different.

[0593] [Effect of the 1st example of the 2nd form of coefficient encoding]

[0594] According to Figure 52 the example, even in the case where the encoding syntax of the coefficient information flag is different depending on the presence or absence of the orthogonal transform, the encoding syntax of the 16 coefficients within the sub-block after the CCB exceeds the thres can be commonized regardless of the presence or absence of the orthogonal transform. Therefore, in the case where the orthogonal transform is applied and in the case where the orthogonal transform is not applied, it is possible that some of the circuits are commonized, and it is possible to reduce the circuit size.

[0595] Further, after the CCB exceeds the thres, the coefficients are no longer encoded into the coefficient information flag encoded by the bypass encoding and the residual value information encoded by the Golomb encoding. Therefore, it is possible to suppress the increase in the amount of information, and it is possible to suppress the increase in the amount of encoding.

[0596] [2nd example of the 2nd form of coefficient encoding]

[0597] Figure 53 is a flowchart showing the coefficient encoding method of the 2nd example of the 2nd form. In Figure 53 the example, even in the case where the orthogonal transform is applied to the block, the case where the 16 coefficients within the sub-block are encoded by the 2nd encoding method (S147a) is different from Figure 47 the example.

[0598] That is, in Figure 53 the example, in the case where the orthogonal transform is applied to the block, the encoding device 100 does not encode the 16 coefficients within the sub-block using the 1st encoding method shown in Figure 48 but uses the 2nd encoding method shown in Figure 49 (S147a). That is, in the case where the orthogonal transform is applied and in the case where the orthogonal transform is not applied, the encoding device 100 does not encode the 16 coefficients within the sub-block using the 1st encoding method shown in Figure 48 but uses the 2nd encoding method shown in Figure 49 .

[0599] More specifically, the encoding device 100, regardless of the presence or absence of the orthogonal transform, encodes the 16 coefficients within the sub-block using the 2nd encoding method shown in Figure 49In the second coding mode shown, in the first loop process, in a case where the CCB exceeds the thres, the coefficient information flag is not skipped from coding but is coded by the bypass coding. Then, in the second loop process, the coding device 100 codes the remainder dependent on the coefficient information flag by the Golomb coding.

[0600] Further, in Figure 49 The syntax for coding the coefficient information flag in the first loop process of the first coding mode can be different between a case where the orthogonal transform is applied and a case where the orthogonal transform is not applied. For example, one or more of the coefficient information flags in the case where the orthogonal transform is applied and one or more of the coefficient information flags in the case where the orthogonal transform is not applied can also be different.

[0601] [Effects of the second example of the second form of coefficient coding]

[0602] According to Figure 53 In the example of the first coding mode, even in a case where the syntax of coding the coefficient information flag is different depending on the presence or absence of the orthogonal transform, the syntax of coding the 16 coefficients within the sub-block after the CCB exceeds the thres can be commonized regardless of the presence or absence of the orthogonal transform. Therefore, in the case where the orthogonal transform is applied and in the case where the orthogonal transform is not applied, a part of the circuit can be commonized, and the circuit size can be reduced.

[0603] [Third form of coefficient coding]

[0604] Figure 54 is a syntax diagram showing the first coding mode of the third form. Figure 54 The syntax shown corresponds to Figure 47 is an example of the syntax of the first coding mode shown. Basically, the first coding mode is used in the case where the orthogonal transform is applied.

[0605] Here, the coefficient information flags and the parameters are the same as those shown in the first form. Further, the plurality of coefficient information flags shown here is an example, and other plurality of coefficient information flags can also be coded. For example, a part of the coefficient information flags can not be coded. Further, the coefficient information flags shown here can be replaced with coefficient information flags or parameters having other meanings.

[0606] Figure 54 The initial for loop in the example of the first coding mode corresponds to Figure 48The first loop process in the example of the first mode. In the first for loop, if the CCB remains, i.e., if the CCB does not exceed the threshold, the coefficient information flag such as sig_flag is encoded by CABAC encoding. If the CCB does not remain, the coefficient information flag is not encoded. In addition, in the present example, the second example of the second mode can also be applied. That is, if the CCB does not remain, the coefficient information flag can be encoded by bypass encoding.

[0607] The second for loop from the top and the third for loop from the top correspond to Figure 48 The second loop process in the example of the first mode. In the second for loop from the top, for the coefficients for which the coefficient information flag has been encoded, the residual value is encoded by Golomb encoding. In the third for loop from the top, for the coefficients for which the coefficient information flag has not been encoded, the coefficient is encoded by Golomb encoding. In addition, by applying the second example of the second mode to the present example, the residual value can always be encoded by Golomb encoding.

[0608] In the fourth for loop from the top, sign_flag is encoded by bypass encoding.

[0609] The syntax explained in the present mode can be applied to Figure 47 , Figure 48 , Figure 50 , Figure 51 and Figure 52 each example.

[0610] Figure 55 is a syntax diagram indicating a basic second encoding mode of the third mode. Figure 55 The syntax shown in FIG. 7 corresponds to an example of the syntax of the second encoding mode of Figure 47 . Basically, the second encoding mode is used without applying orthogonal transformation.

[0611] Furthermore, the plurality of coefficient information flags shown here is an example, and other plurality of coefficient information flags can be encoded. For example, a part of the coefficient information flags can not be encoded. Furthermore, the coefficient information flags shown here can be replaced with coefficient information flags or parameters having other meanings.

[0612] Figure 55 The first five for loops in the example of the first mode correspond to Figure 49The first loop in the example processes the coefficient information flags, such as sig_flag, if CCB remains after the first 5 for loops (i.e., CCB has not exceeded the threshold). If no CCB remains, the coefficient information flags are encoded using bypass encoding. Alternatively, in this example, the first example of the second form can also be applied. That is, if no CCB remains, the coefficient information flags may not be encoded.

[0613] The 6th for loop from the top corresponds to Figure 49 The second loop in the example. In the sixth for loop from above, the remaining value is encoded using Columbus coding. Alternatively, in this example, the first example of the second form can also be applied. That is, for coefficients whose coefficient information flags are encoded, the remaining value can be encoded using Columbus coding. Then, for coefficients whose coefficient information flags are not encoded, the coefficients can be encoded using Columbus coding.

[0614] The syntax described in this form can be applied to Figure 47 , Figure 49 , Figure 50 , Figure 51 and Figure 53 Examples from the text.

[0615] Without applying orthogonal transformations ( Figure 55 The number of loops used to encode coefficient information flags is greater than that used when applying orthogonal transformations. Figure 54 Therefore, the hardware processing load may increase compared to the case where orthogonal transformation is applied, even without it. Furthermore, since the syntax of coefficient encoding differs depending on whether orthogonal transformation is performed, separate circuits may need to be prepared. Thus, the circuitry can become more complex.

[0616] [Example 1 of the third form of coefficient encoding]

[0617] Figure 56 It is a syntactic diagram representing the second encoding method of the first example of the third form. Figure 56 The syntax shown corresponds to Figure 47 An example of the second encoding method. In Figure 47 It can also be used in the first encoding method Figure 54 The syntax is shown. Furthermore, this example can be combined with other examples of the third form, or with other forms.

[0618] Figure 56 The initial for loop in the example corresponds to Figure 49The example shows the first loop processing. If, in this initial for loop, the CCB has more than 8 remaining (i.e., if adding 8 to the CCB does not exceed the threshold), then up to 8 coefficient information flags are encoded using CABAC encoding according to the coefficients, and the CCB is incremented up to 8 times. If the CCB does not have more than 8 remaining (i.e., if adding 8 to the CCB exceeds the threshold), then the 8 coefficient information flags are encoded using bypass encoding according to the coefficients.

[0619] In other words, before encoding the eight coefficient information markers, it is comprehensively determined whether it is possible to encode the eight coefficient information markers using CABAC encoding. Then, if it is possible to encode the eight coefficient information markers using CABAC encoding, at most eight coefficient information markers are encoded using CABAC encoding.

[0620] Alternatively, in this example, the first example of the second form can also be applied. That is, if there are no more than 8 CCBs remaining, then the 8 coefficient information flags do not need to be encoded. In other words, in this case, instead of encoding the 8 coefficient information flags through bypass encoding, the encoding of the 8 coefficient information flags can be skipped.

[0621] In addition, such as Figure 56 As shown, the encoding of more than one of the eight coefficient information flags can also be omitted based on the value of the coefficient. For example, when sig_flag is 0, the encoding of the remaining seven coefficient information flags can be omitted.

[0622] The second for loop from the top corresponds to Figure 49 The second loop in the example. In the second for loop from the top, the remaining value is encoded using Columbus coding. Alternatively, in this example, the first example of the second form can also be applied. That is, for coefficients whose 8 coefficient information flags are encoded, the remaining value can be encoded using Columbus coding; for coefficients whose 8 coefficient information flags are not encoded, the coefficients can be encoded using Columbus coding.

[0623] The multiple coefficient information markers shown here are one example; other coefficient information markers can also be encoded. For example, some coefficient information markers may not be encoded. Furthermore, the coefficient information markers shown here can be replaced with coefficient information markers or parameters that have other meanings.

[0624] Alternatively, combinations are also possible. Figure 55 Examples and Figure 56 For example, in Figure 55In the example, before the four coefficient information flags, such as sig_flag and sign_flag, are encoded, it can be generally determined whether it is possible to encode the four coefficient information flags using CABAC encoding.

[0625] [The effect of the first example of the third form of coefficient encoding]

[0626] exist Figure 56 In the example, all coefficient information flags encoded via CABAC encoding are encoded in a single loop process. That is, compared to... Figure 55 Compared to the examples, in Figure 56 In the example, the number of cycles processed is relatively small. Therefore, the processing volume can potentially be reduced.

[0627] In addition, Figure 54 Examples and Figure 56 In the example, the number of loops used to encode multiple coefficient information flags via CABAC encoding is consistent. Therefore, with Figure 54 Examples and Figure 55 Compared to combinations of examples, in Figure 54 Examples and Figure 56 In combinations of examples, the number of circuit modifications may be reduced.

[0628] In addition, before encoding multiple coefficient information flags, a comprehensive determination is made as to whether multiple coefficient information flags can be encoded using CABAC encoding, thus simplifying the process and potentially reducing processing latency.

[0629] Furthermore, in both cases where orthogonal transformation is applied and those where it is not applied, it can be comprehensively determined whether multiple coefficient information flags can be encoded using CABAC encoding before they are encoded. Therefore, the difference between the encoding methods used in blocks applying orthogonal transformation and those not applying orthogonal transformation is further reduced, and the circuit size can potentially be further reduced.

[0630] In addition, Figure 56 In the example, sig_flag to abs_gt9_flag are contained within a single loop, but the encoding method is not limited to this. Multiple loops can be used (e.g., two loops), or it can be determined collectively whether it is possible to encode multiple coefficient information flags for each loop using CABAC encoding. Compared to one loop, the processing increases, but compared to... Figure 55 Compared to the previous example, the same reduction effect can be achieved.

[0631] [Example 2 of the third form of coefficient encoding]

[0632] Figure 57is a syntax diagram of the second encoding method of the second example of the third modality. Figure 57 The syntax shown corresponds to Figure 47 an example of the second encoding method of Figure 47 The syntax shown can also be used in the first encoding method of Figure 54 In addition, the present example can be combined with other examples of the third modality, or can be combined with other modalities.

[0633] Figure 57 The initial for loop in the example of Figure 49 The first loop process in the example of If the CCB remains 7 or more in the initial for loop, that is, if the CCB added to 7 does not exceed the threshold value, the coefficient information flags for up to 7 coefficients are encoded by CABAC encoding, and the CCB is incremented by up to 7. If the CCB does not remain 7 or more, that is, if the CCB added to 7 exceeds the threshold value, the 7 coefficient information flags are encoded by bypass encoding.

[0634] In other words, before the 7 coefficient information flags are encoded, it is determined in general whether it is possible to encode the 7 coefficient information flags by CABAC encoding. Then, in the case where it is possible to encode the 7 coefficient information flags by CABAC encoding, the 7 coefficient information flags are encoded by CABAC encoding.

[0635] In addition, in the present example, the first example of the second modality can also be applied. That is, if the CCB remains 7 or more, the 7 coefficient information flags can also not be encoded. That is, in this case, the 7 coefficient information flags can also be skipped without being encoded by bypass encoding.

[0636] In addition, as shown in Figure 57 the encoding of one or more of the 7 coefficient information flags can also be omitted according to the values of the coefficients. For example, in the case where the sig_flag is 0, the encoding of the remaining 6 coefficient information flags can be omitted.

[0637] The second for loop from the top corresponds to the second loop process in the example of Figure 49 In the second for loop from the top, the remaining values are encoded by Golomb-Rice encoding. In addition, in the present example, the first example of the second modality can also be applied. That is, for the coefficients for which the 7 coefficient information flags are encoded, the remaining values can be encoded by Golomb-Rice encoding, and for the coefficients for which the 7 coefficient information flags are not encoded, the coefficients can be encoded by Golomb-Rice encoding.

[0638] In the 3rd for loop from above, if CCB remains, that is, CCB does not exceed the threshold, sign_flag is encoded by CABAC encoding, and CCB is incremented. If CCB does not remain, sign_flag is encoded by bypass encoding. Further, as in the example of Figure 54 , sign_flag can be always encoded by bypass encoding.

[0639] The plurality of coefficient information flags shown here is an example, and other plurality of coefficient information flags can be encoded. For example, a part of the coefficient information flags can not be encoded. Further, the coefficient information flags shown here can be replaced with coefficient information flags or parameters having other meanings.

[0640] In addition, the example of Figure 55 and the example of Figure 57 may be combined. For example, in the example of Figure 55 , before the 4 coefficient information flags of sig_flag and sign_flag and the like are encoded, it can be generally determined whether it is possible to encode the 4 coefficient information flags by CABAC encoding.

[0641] [Effects of the 2nd example of the 3rd form of coefficient encoding]

[0642] As in the example of Figure 56 , in the example of Figure 57 , the plurality of coefficient information flags (specifically, abs_gt3_flag and abs_gt5_flag and the like) for representing the size of the coefficient by comparing with the threshold are encoded in 1 loop processing. Therefore, compared with the example of Figure 55 , in the example of Figure 57 , the number of loop processing is small. Therefore, the processing amount can be reduced.

[0643] Compared with the example of Figure 56 , in the example of Figure 57 , the number of loop processing for encoding the plurality of coefficient information flags increases. However, compared with the example of Figure 56 , in the example of Figure 57 , there is a similar part to the example of Figure 54 . For example, sign_flag has been finally encoded. Therefore, compared with the combination of the example of Figure 54 and the example of Figure 56 , in the combination of the example of Figure 54 and the example of Figure 57 , the changed part of the circuit can be less.

[0644] In addition, before the plurality of coefficient information flags are encoded, it is determined in general whether it is possible to encode the plurality of coefficient information flags by CABAC encoding, so the processing is simplified, and the processing delay can be reduced.

[0645] In both cases of the case where the orthogonal transform is applied and the case where the orthogonal transform is not applied, it is also possible to determine in general whether it is possible to encode the plurality of coefficient information flags by CABAC encoding before the plurality of coefficient information flags are encoded. Thus, the difference between the encoding method used in the block where the orthogonal transform is applied and the encoding method used in the block where the orthogonal transform is not applied is further reduced, and the circuit size can be further reduced.

[0646] Further, in the example of Figure 57 From sig_flag to abs_gt9_flag is included in 1 loop, but the encoding method is not limited to this. A plurality of loops (for example, 2 loops) can be used, and it is also possible to determine in general whether it is possible to encode the plurality of coefficient information flags of each loop by CABAC encoding. Compared with 1 loop, the processing is increased, but compared with the example of Figure 55 The effect of processing reduction can also be obtained.

[0647] [Variation of coefficient encoding]

[0648] Any of the plurality of modes and the plurality of examples related to the above-described coefficient encoding can be combined. Further, any of the plurality of modes, the plurality of examples, and any of the plurality of combinations thereof related to the above-described coefficient encoding can be applied to the block of the luminance or the block of the color difference. At this time, different thres can be used in the block of the luminance and the block of the color difference.

[0649] In addition, any of the plurality of modes, the plurality of examples, and any of the plurality of combinations thereof related to the above-described coefficient encoding can be used for a block that is a block where the orthogonal transform is not applied and is a block where BDPCM (Block-based Delta Pulse Code Modulation) is applied. In the block where the BDPCM is applied, each residual signal within the block is reduced in the amount of information by subtracting a residual signal vertically or horizontally adjacent to the residual signal from the residual signal.

[0650] Further, any of the plurality of modes, the plurality of examples, and any of the plurality of combinations thereof related to the above-described coefficient encoding can also be used for a block that is a block where the BDPCM is applied and is a block of the color difference.

[0651] Furthermore, for a block to which ISP (Intra Sub-Partitions) is applied, one of the plurality of modes, the plurality of examples, and the arbitrary combination of them related to the coefficient coding described above can be used. In ISP, an intra block is divided vertically or horizontally, and the pixel values of the sub-blocks adjacent to each sub-block are used for intra prediction of the sub-block.

[0652] Furthermore, for a block to which ISP is applied and which is a color difference block, one of the plurality of modes, the plurality of examples, and the arbitrary combination of them related to the coefficient coding described above can be used.

[0653] Furthermore, in a case where Chroma Joint Coding is used as an encoding mode of a color difference block, one of the plurality of modes, the plurality of examples, and the arbitrary combination of them related to the coefficient coding described above can be used. Here, Chroma Joint Coding is an encoding method in which the value of Cr is derived from the value of Cb.

[0654] Furthermore, the value of thres in a case where the orthogonal transform is applied can be twice the value of thres in a case where the orthogonal transform is not applied. Alternatively, the value of thres in a case where the orthogonal transform is not applied can be twice the value of thres in a case where the orthogonal transform is applied.

[0655] Furthermore, the value of thres of CCB in a case where the orthogonal transform is applied can be twice the value of thres of CCB in a case where the orthogonal transform is not applied, only in a case where Chroma Joint Coding is used. Alternatively, the value of thres of CCB in a case where the orthogonal transform is not applied can be twice the value of thres of CCB in a case where the orthogonal transform is applied, only in a case where Chroma Joint Coding is used.

[0656] In addition, among the plurality of modes and the plurality of examples related to the coefficient coding described above, the scan order of the plurality of coefficients in a block in which the orthogonal transform is not applied can be the same as the scan order of the plurality of coefficients in a block in which the orthogonal transform is applied.

[0657] Furthermore, although several examples of syntax are shown in the 3rd mode and the plurality of examples of the 3rd mode, the syntax to be applied is not limited to these examples. For example, in a plurality of modes different from the 3rd mode and in a plurality of examples of them, a syntax different from any one of the plurality of syntaxes shown in the 3rd mode and the plurality of examples thereof can be used. Various syntaxes for encoding 16 coefficients can be applied.

[0658] In addition, in the multiple modes and multiple examples of coefficient coding, the processing flow of encoding is shown, but the processing flow of decoding is substantially the same as that of encoding except for the difference between transmitting and receiving a bit stream. For example, the decoding device 200 can perform inverse orthogonal transformation and decoding corresponding to orthogonal transformation and encoding performed by the encoding device 100.

[0659] Furthermore, each flowchart related to the multiple modes and multiple examples of coefficient coding is an example. Each flowchart can be newly added with a condition or processing, or a condition or processing can be deleted, or a condition or processing can be changed.

[0660] In addition, here, a coefficient is a value of an image constituting a block or a sub-block, or the like. Specifically, a plurality of coefficients constituting an image can be obtained from a plurality of pixel values of the image via orthogonal transformation. In addition, a plurality of coefficients constituting an image can be obtained from a plurality of pixel values of the image without via orthogonal transformation. That is, a plurality of coefficients constituting an image can be a plurality of pixel values of the image itself. In addition, each pixel value can be a pixel value of an original image, or can be a value of a prediction residual. Furthermore, a coefficient can be quantized.

[0661] [Representative Example of Structure and Processing]

[0662] The following indicates a representative example of the structure and processing of the encoding device 100 and the decoding device 200 shown above.

[0663] Figure 58 is a flowchart indicating an action of the encoding device 100. For example, the encoding device 100 has a circuit and a memory connected to the circuit. The circuit and the memory possessed by the encoding device 100 can correspond to the processor a1 and the memory a2 shown in Figure 40 . The circuit of the encoding device 100 performs the action shown in Figure 58 . Specifically, the circuit of the encoding device 100 encodes a block of an image in the action (S211).

[0664] In an example, the circuit of the encoding device 100 can encode the block of the image by limiting the number of times of processing of context adaptive coding. Then, it can be that, in both a case where orthogonal transformation is applied to the block and a case where orthogonal transformation is not applied to the block, the sub-block flag coding processing is performed without being included in the number of times of processing of the limitation.

[0665] Here, the sub-block flag coding processing is processing of encoding a sub-block flag by context adaptive coding, the sub-block flag indicating whether a sub-block included in the block includes a non-zero coefficient.

[0666] Thus, the sub-block flag can be encoded by the context adaptive coding regardless of whether or not the orthogonal transform is applied and regardless of whether or not the number of times of the processing is limited. Therefore, the amount of encoding can be reduced. Further, a difference between the encoding method used in the block in which the orthogonal transform is applied and the encoding method used in the block in which the orthogonal transform is not applied can be made small, and the circuit size can be made small.

[0667] Further, the circuit of the encoding apparatus 100 can perform the position parameter encoding processing without including the position parameter encoding processing in the number of times of the processing to be limited, in a case where the orthogonal transform is applied to the block. Here, the position parameter encoding processing is processing of encoding a parameter indicating a position of a first non-zero coefficient in the block in the scanning order by the context adaptive coding.

[0668] Thus, in the case where the orthogonal transform is applied, the parameter indicating the position of the first non-zero coefficient can be encoded by the context adaptive coding regardless of whether or not the number of times of the processing is limited. Therefore, the amount of encoding can be reduced.

[0669] Further, the circuit of the encoding apparatus 100 can determine the range of the number of times of the processing to be limited in accordance with the position of the first non-zero coefficient, in the case where the orthogonal transform is applied to the block. Thus, in the case where the orthogonal transform is applied, it is possible to appropriately determine the number of times of the processing to be limited. Therefore, it is possible to appropriately adjust a balance between reduction of the amount of encoding and reduction of the processing delay.

[0670] In another example, the circuit of the encoding apparatus 100 can act as follows in both a case where the orthogonal transform is applied to the block of the encoding target image and a case where the orthogonal transform is not applied to the block.

[0671] Specifically, in both the cases, in a case where the number of times of the context adaptive coding is within the range of the number of times of the processing to be limited, the circuit of the encoding apparatus 100 can encode the coefficient information flag by the context adaptive coding. Further, in both the cases, in a case where the number of times of the context adaptive coding is not within the range of the number of times of the processing to be limited, the circuit of the encoding apparatus 100 can also skip the encoding of the coefficient information flag. Here, the coefficient information flag indicates a property of the coefficient included in the block.

[0672] Then, in a case where the coefficient information flag is encoded, the circuit of the encoding apparatus 100 can encode the residual value information by the Golomb coding. Here, the residual value information is information for reconstructing a value of the coefficient using the coefficient information flag. Further, in a case where the encoding of the coefficient information flag is skipped, the circuit of the encoding apparatus 100 can encode the value of the coefficient by the Golomb coding.

[0673] Thus, it is possible to skip the encoding of the coefficient information flag, regardless of whether or not the orthogonal transform is applied, in accordance with the limit on the number of times of processing of context adaptive encoding. Therefore, it is possible to suppress an increase in processing delay and an increase in the amount of encoding. Furthermore, it is possible to make the difference between the encoding method used in the block in which the orthogonal transform is applied and the encoding method used in the block in which the orthogonal transform is not applied smaller, and to make the circuit size smaller.

[0674] In addition, the coefficient information flag can be a flag indicating whether or not the value of the coefficient is greater than 1. Thus, it is possible to skip the encoding of the coefficient information flag indicating whether or not the value of the coefficient is greater than 1, regardless of whether or not the orthogonal transform is applied, in accordance with the limit on the number of times of processing of context adaptive encoding. Therefore, it is possible to suppress an increase in processing delay and an increase in the amount of encoding.

[0675] In yet another example, the circuit of the encoding apparatus 100 can limit the number of times of processing of context adaptive encoding, and encode the block of the image. Also, in a case where the orthogonal transform is not applied to the block, it can be determined whether or not a plurality of coefficient information flags respectively indicating a plurality of properties of coefficients included in the block satisfy a processing condition. Then, in a case where it is determined that the processing condition is satisfied, the plurality of coefficient information flags can be encoded by context adaptive encoding.

[0676] Here, the processing condition is a condition in which the number of times of processing to which the number of the plurality of coefficient information flags is added is within the limit on the number of times of processing.

[0677] Thus, in a case where the orthogonal transform is not applied, it is possible to collectively determine whether or not it is possible to use context adaptive encoding for the plurality of coefficient information flags. Therefore, it is possible to simplify the processing, and to reduce the processing delay. Furthermore, in a case where similar processing is performed on the block in which the orthogonal transform is applied, it is possible to make the difference between the encoding method used in the block in which the orthogonal transform is applied and the encoding method used in the block in which the orthogonal transform is not applied smaller, and to make the circuit size smaller.

[0678] Furthermore, the plurality of coefficient information flags can include a coefficient information flag indicating whether or not the value of the coefficient is greater than 3 and a coefficient information flag indicating whether or not the value of the coefficient is greater than 5. Thus, it is possible to make a collective determination for the plurality of coefficient information flags including the coefficient information flag indicating whether or not the value of the coefficient is greater than 3 and the coefficient information flag indicating whether or not the value of the coefficient is greater than 5. Therefore, it is possible to simplify the processing, and to reduce the processing delay.

[0679] Furthermore, multiple coefficient information flags can also include a coefficient information flag indicating whether the coefficient value is greater than 7 and a coefficient information flag indicating whether the coefficient value is greater than 9. Therefore, it is possible to perform a comprehensive determination on multiple coefficient information flags that include four coefficient information flags: whether the coefficient value is greater than 3, whether the coefficient value is greater than 5, whether the coefficient value is greater than 7, and whether the coefficient value is greater than 9. Thus, the processing can potentially be simplified, and processing latency can potentially be reduced.

[0680] In addition, the aforementioned actions performed by the circuitry of the encoding device 100 can also be performed by the entropy encoding unit 110 of the encoding device 100.

[0681] Figure 59 This is a flowchart illustrating the operation of the decoding device 200. For example, the decoding device 200 includes circuitry and a memory connected to the circuitry. The circuitry and memory included in the decoding device 200 can correspond to... Figure 46 The processor b1 and memory b2 are shown. The circuitry of the decoding device 200 is as follows. Figure 59 The operation is shown. Specifically, the circuitry of the decoding device 200 decodes blocks of the image during the operation (S221).

[0682] In one example, the circuitry of the decoding device 200 can limit the number of processing steps for context-adaptive decoding and decode the image block. Alternatively, in both cases where an inverse orthogonal transform is applied to the block and where no inverse orthogonal transform is applied to the block, sub-block flag decoding processing can be performed without including it in the limited number of target processing steps.

[0683] Here, the sub-block flag decoding process is a process of decoding the sub-block flag, which indicates whether the sub-block contained in the block contains non-zero coefficients, through context-adaptive decoding.

[0684] Therefore, regardless of whether inverse orthogonal transform is applied or the number of processing steps in context-adaptive decoding is limited, it is possible to decode sub-block flags using context-adaptive decoding. Consequently, the amount of coding can be reduced. Furthermore, the difference between the decoding method used in blocks applying inverse orthogonal transform and that used in blocks without inverse orthogonal transform can be reduced, and the circuit size can be decreased.

[0685] Alternatively, the circuitry of the decoding device 200 may perform position parameter decoding processing when applying an inverse orthogonal transform to the block, without including it in the number of processing iterations for the restricted object. Here, the position parameter decoding processing is a process of decoding parameters representing the positions of the first non-zero coefficients in the block according to the scan order through context-adaptive decoding.

[0686] Thus, in the case where the inverse orthogonal transform is applied, the parameter indicating the position of the first nonzero coefficient is likely to be decoded by the context-adaptive decoding regardless of whether the number of times of processing of the context-adaptive decoding is limited. Therefore, the amount of coding is likely to be reduced.

[0687] Further, the circuit of the decoding device 200 can determine the range of limitation of the number of times of processing according to the position of the first nonzero coefficient in the case where the inverse orthogonal transform is applied to the block. Thus, in the case where the inverse orthogonal transform is applied, the number of times of limitation of processing is likely to be appropriately determined. Therefore, the balance between the reduction of the amount of coding and the reduction of the processing delay is likely to be appropriately adjusted.

[0688] In another example, the circuit of the decoding device 200 can act as follows in both the case where the inverse orthogonal transform is applied to the block of the decoded image and the case where the inverse orthogonal transform is not applied to the block.

[0689] Specifically, in both the cases, in the case where the number of times of processing of the context-adaptive decoding is within the range of limitation of the number of times of processing, the circuit of the decoding device 200 can also decode the coefficient information flag by the context-adaptive decoding. Further, in both the cases, in the case where the number of times of processing of the context-adaptive decoding is not within the range of limitation of the number of times of processing, the circuit of the decoding device 200 can also skip the decoding of the coefficient information flag. Here, the coefficient information flag indicates the attribute of the coefficient included in the block.

[0690] Then, in the case where the coefficient information flag is decoded, the circuit of the decoding device 200 can decode the residual value information by the Golomb decoding. Here, the residual value information is information for reconstructing the value of the coefficient using the coefficient information flag. Further, in the case where the decoding of the coefficient information flag is skipped, the circuit of the decoding device 200 can decode the value of the coefficient by the Golomb decoding.

[0691] Thus, in both the case where the inverse orthogonal transform is applied and the case where the inverse orthogonal transform is not applied, the decoding of the coefficient information flag is likely to be skipped according to the limitation of the number of times of processing of the context-adaptive decoding. Therefore, the increase in the processing delay and the increase in the amount of coding are likely to be suppressed. Further, the difference between the decoding method used in the block to which the inverse orthogonal transform is applied and the decoding method used in the block to which the inverse orthogonal transform is not applied is likely to be small, and the circuit scale is likely to be small.

[0692] In addition, the coefficient information flag can also be a flag indicating whether the value of the coefficient is greater than 1. Thus, in both the case where the inverse orthogonal transform is applied and the case where the inverse orthogonal transform is not applied, the decoding of the coefficient information flag indicating whether the value of the coefficient is greater than 1 is likely to be skipped according to the limitation of the number of times of processing of the context-adaptive decoding. Therefore, the increase in the processing delay and the increase in the amount of coding are likely to be suppressed.

[0693] In yet another example, the circuit of the decoding device 200 can decode the block of the image while limiting the number of times of processing of context-adaptive decoding. Then, in a case where the inverse orthogonal transform is not applied to the block, it can be determined whether a plurality of coefficient information flags respectively indicating a plurality of properties of the coefficients included in the block satisfy a processing condition. Then, in a case where it is determined that the processing condition is satisfied, the plurality of coefficient information flags can also be decoded by the context-adaptive decoding.

[0694] Here, the processing condition is a condition in which the number of times of processing to which the number of the plurality of coefficient information flags is added is within the limit of the number of times of processing.

[0695] Thus, in a case where the inverse orthogonal transform is not applied, it is possible to generally determine whether it is possible to use the context-adaptive decoding for the plurality of coefficient information flags. Therefore, the processing can be simplified, and the processing delay can be reduced. Further, in a case where similar processing is performed on a block to which the inverse orthogonal transform is applied, the difference between the decoding method used in the block to which the inverse orthogonal transform is applied and the decoding method used in the block to which the inverse orthogonal transform is not applied can be small, and the circuit size can be small.

[0696] Further, the plurality of coefficient information flags can include a coefficient information flag indicating whether the value of the coefficient is greater than 3 and a coefficient information flag indicating whether the value of the coefficient is greater than 5. Thus, it is possible to generally determine the plurality of coefficient information flags including the coefficient information flag indicating whether the value of the coefficient is greater than 3 and the coefficient information flag indicating whether the value of the coefficient is greater than 5. Therefore, the processing can be simplified, and the processing delay can be reduced.

[0697] Further, the plurality of coefficient information flags can include a coefficient information flag indicating whether the value of the coefficient is greater than 7 and a coefficient information flag indicating whether the value of the coefficient is greater than 9. Thus, it is possible to generally determine the plurality of coefficient information flags including the four coefficient information flags of whether the value of the coefficient is greater than 3, whether the value of the coefficient is greater than 5, whether the value of the coefficient is greater than 7, and whether the value of the coefficient is greater than 9. Therefore, the processing can be simplified, and the processing delay can be reduced.

[0698] In addition, the above-described operations performed by the circuit of the decoding device 200 can be performed by the entropy decoding section 202 of the decoding device 200.

[0699] [Other Examples]

[0700] The encoding device 100 and the decoding device 200 in the above-described examples can each be used as an image encoding device and an image decoding device, respectively, or as a moving image encoding device and a moving image decoding device, respectively.

[0701] Moreover, the encoding apparatus 100 and the decoding apparatus 200 can perform only a part of the above-described operations, and other apparatuses can perform other operations. Furthermore, the encoding apparatus 100 and the decoding apparatus 200 can have only a part of the above-described constituent elements, and other apparatuses can have other constituent elements.

[0702] Furthermore, at least a part of each of the above-described examples can be utilized as an encoding method or a decoding method, or can be utilized as other methods.

[0703] Each of the constituent elements can be configured by a dedicated hardware or can be realized by executing a software program suitable for each of the constituent elements. Each of the constituent elements can be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory.

[0704] Specifically, the encoding apparatus 100 and the decoding apparatus 200 can each have a processing circuitry and a storage device capable of being accessed from the processing circuitry, which is electrically connected to the processing circuitry. For example, the processing circuitry corresponds to the processor a1 or b1, and the storage device corresponds to the memory a2 or b2.

[0705] The processing circuitry includes at least one of a dedicated hardware and a program execution unit, and performs processing using the storage device. In addition, the storage device stores a software program executed by the program execution unit in a case where the processing circuitry includes the program execution unit.

[0706] Here, a software realizing the above-described encoding apparatus 100 or decoding apparatus 200 or the like is a program as follows.

[0707] For example, the program can cause a computer to execute an encoding method of encoding a block of an image while limiting a number of times of processing of context adaptive coding, and in encoding of the block, performing sub-block flag coding processing by context adaptive coding to encode a sub-block flag indicating whether a sub-block included in the block includes a non-zero coefficient without including the sub-block flag coding processing in the number of times of processing in both a case where an orthogonal transform is applied to the block and a case where the orthogonal transform is not applied to the block.

[0708] Further, for example, the program can cause the computer to execute a decoding method of decoding a block of an image while limiting a number of times of processing of context-adaptive decoding, in decoding of the block, in both a case where an inverse orthogonal transform is applied to the block and a case where the inverse orthogonal transform is not applied to the block, performing sub-block flag decoding processing of decoding, by context-adaptive decoding, a sub-block flag indicating whether or not a sub-block included in the block includes a non-zero coefficient without including the sub-block flag decoding processing in the number of times of processing.

[0709] Further, for example, the program can cause the computer to execute an encoding method of, in both a case where an orthogonal transform is applied to a block of an encoding target image and a case where the orthogonal transform is not applied to the block, in a case where a number of times of processing of context-adaptive encoding is within a limit range of the number of times of processing, encoding, by context-adaptive encoding, a coefficient information flag indicating a property of a coefficient included in the block, in a case where the number of times of processing is not within the limit range of the number of times of processing, skipping encoding of the coefficient information flag, in a case where the coefficient information flag is encoded, encoding, by Golomb encoding, residual value information for reconstructing a value of the coefficient using the coefficient information flag, in a case where encoding of the coefficient information flag is skipped, encoding, by Golomb encoding, the value of the coefficient.

[0710] Further, for example, the program can cause the computer to execute a decoding method of, in both a case where an inverse orthogonal transform is applied to a block of a decoding target image and a case where the inverse orthogonal transform is not applied to the block, in a case where a number of times of processing of context-adaptive decoding is within a limit range of the number of times of processing, decoding, by context-adaptive decoding, a coefficient information flag indicating a property of a coefficient included in the block, in a case where the number of times of processing is not within the limit range of the number of times of processing, skipping decoding of the coefficient information flag, in a case where the coefficient information flag is decoded, decoding, by Golomb decoding, residual value information for reconstructing a value of the coefficient using the coefficient information flag, in a case where decoding of the coefficient information flag is skipped, decoding, by Golomb decoding, the value of the coefficient.

[0711] Further, for example, the program can cause the computer to execute an encoding method of encoding a block of an image while limiting the number of times of processing of context-adaptive encoding, in the encoding of the block, in a case where an orthogonal transform is not applied to the block, determining whether a plurality of coefficient information flags respectively indicating a plurality of attributes of coefficients included in the block satisfy a processing condition, in a case where it is determined that the processing condition is satisfied, encoding the plurality of coefficient information flags by context-adaptive encoding, the processing condition being a condition in which the number of times of processing, to which the number of the plurality of coefficient information flags is added, is within a limit range of the number of times of processing.

[0712] Further, for example, the program can cause the computer to execute an encoding method of encoding a block of an image while limiting the number of times of processing of context-adaptive encoding, in the encoding of the block, in a case where an orthogonal transform is not applied to the block, determining whether a plurality of coefficient information flags respectively indicating a plurality of attributes of coefficients included in the block satisfy a processing condition, in a case where it is determined that the processing condition is satisfied, encoding the plurality of coefficient information flags by context-adaptive encoding, the processing condition being a condition in which the number of times of processing, to which the number of the plurality of coefficient information flags is added, is within a limit range of the number of times of processing.

[0713] Further, as described above, each of the constituent elements can be a circuit. These circuits can be a single circuit or different circuits. Further, each of the constituent elements can be implemented by a general-purpose processor or a dedicated processor.

[0714] Further, the processing performed by a specific constituent element can be performed by another constituent element. Further, the order of the processing can be changed, and a plurality of processes can be performed simultaneously. Further, the coding and decoding device can include the encoding device 100 and the decoding device 200.

[0715] Further, the first and second ordinal numbers and the like used in the description can be appropriately exchanged. Further, ordinal numbers can be newly assigned to the constituent elements, or the ordinal numbers can be removed.

[0716] The above describes the configurations of the encoding device 100 and the decoding device 200 based on a plurality of examples, but the configurations of the encoding device 100 and the decoding device 200 are not limited to these examples. As long as the gist of the present application is not deviated from, various modified configurations that can be thought of by those skilled in the art can be implemented for each example, and configurations in which constituent elements in different examples are combined can also be included in the range of the configurations of the encoding device 100 and the decoding device 200.

[0717] One or more of the aspects disclosed herein can be implemented in combination with at least a portion of another aspect of the present disclosure. In addition, a portion of the processing, a portion of the structure of the device, a portion of the syntax, and the like described in the flowchart of one or more of the aspects disclosed herein can be implemented in combination with another aspect.

[0718] [Implementation and Use]

[0719] In each of the above embodiments, each functional block or a block that functions can be realized by an MPU (micro processing unit) and a memory, and the like. In addition, it can be that the processing of each functional block is realized by a program execution section such as a processor that reads and executes software (program) recorded in a recording medium such as a ROM. The software can be distributed. The software can also be recorded in various recording media such as a semiconductor memory. In addition, each functional block can be realized by hardware (a dedicated circuit). Various combinations of hardware and software can be adopted.

[0720] The processing described in each of the embodiments can be realized by centralized processing using a single device (system), or can be realized by distributed processing using a plurality of devices. In addition, the processor that executes the above-described program can be a single one, or can be a plurality of processors. That is, centralized processing or distributed processing can be performed.

[0721] The aspects of the present disclosure are not limited to the above-described embodiments, and various modifications can be made thereto. Such modifications are also within the scope of the aspects of the present disclosure.

[0722] Further, application examples of the moving image encoding method (image encoding method) or the moving image decoding method (image decoding method) represented in each of the above-described embodiments, and various systems that implement the application examples are described herein. Such a system can be characterized by having an image encoding device that uses the image encoding method, an image decoding device that uses the image decoding method, or an image encoding / decoding device that has both. Other structures of such a system can be appropriately changed depending on the situation.

[0723] [Use Example]

[0724] Figure 60 is a diagram that shows the overall structure of an appropriate content supply system ex100 that implements a content distribution service. The provision area of the communication service is divided into desired sizes, and a base station ex106, ex107, ex108, ex109, ex110, which is a fixed wireless station in the illustrated example, is provided in each unit.

[0725] In the content supply system ex100, each of the devices such as the computer ex111, the game machine ex112, the camera ex113, the home appliance ex114, and the smart phone ex115 is connected via the Internet service provider ex102 or the communication network ex104, and the base stations ex106 to ex110 on the Internet ex101. The content supply system ex100 can also be connected by combining some of the above-described devices. In various implementations, the devices can also be directly or indirectly connected to each other via a telephone network or close proximity wireless, or the like, without passing through the base stations ex106 to ex110. Also, the streaming server ex103 can be connected to each of the devices such as the computer ex111, the game machine ex112, the camera ex113, the home appliance ex114, and the smart phone ex115 via the Internet ex101 or the like. Further, the streaming server ex103 can be connected to a terminal or the like in a hotspot in the airplane ex117 via the satellite ex116.

[0726] In addition, a wireless access point or a hotspot, or the like can be used instead of the base stations ex106 to ex110. Further, the streaming server ex103 can be directly connected to the communication network ex104 without passing through the Internet ex101 or the Internet service provider ex102, or can be directly connected to the airplane ex117 without passing through the satellite ex116.

[0727] The camera ex113 is a device such as a digital camera that can take still images and moving images. Further, the smart phone ex115 is a smart phone, a portable telephone, or a PHS (Personal Handy-phone System), or the like, which corresponds to a mode of a mobile communication system called 2G, 3G, 3.9G, 4G, and 5G to be called in future.

[0728] The home appliance ex114 is a device such as a refrigerator or a device included in a household fuel cell cogeneration system.

[0729] In the content supply system ex100, a terminal having a photographing function is connected to the streaming server ex103 via the base station ex106 or the like, whereby live distribution or the like can be performed. In the live distribution, the terminal (the computer ex111, the game machine ex112, the camera ex113, the home appliance ex114, the smart phone ex115, and a terminal in the airplane ex117, or the like) can perform the encoding processing described in each of the above-described embodiments on still image or moving image content taken by a user using the terminal, can multiplex video data obtained by the encoding and audio data obtained by encoding sound corresponding to the video, and can transmit the obtained data to the streaming server ex103. That is, each of the terminals functions as an image encoding device according to one aspect of the present application.

[0730] On the other hand, the streaming server ex103 performs streaming distribution of content data transmitted from a client having a request. The client is a computer ex111, a game machine ex112, a camera ex113, a household electrical appliance ex114, a smart phone ex115, or a terminal in an airplane ex117, which is capable of decoding the data subjected to the above-described encoding process. Each device that receives the distributed data can also perform decoding process on the received data and reproduce it. That is, each device can also function as an image decoding apparatus of one aspect of the present application.

[0731] [Decentralized processing]

[0732] Further, the streaming server ex103 can be a plurality of servers or a plurality of computers, which perform decentralized processing or recording of data and distribute it. For example, the streaming server ex103 can be realized by a CDN (Contents Delivery Network), which performs content distribution by connecting a plurality of edge servers distributed in the world and a network between the edge servers. In the CDN, a physically closer edge server can be dynamically assigned according to a client. Further, by caching and distributing content to the edge server, it is possible to reduce delay. Further, in the case where several types of errors occur or the communication state changes due to an increase in traffic or the like, it is possible to decentralize processing by a plurality of edge servers, or switch the distribution subject to another edge server, or continue distribution by bypassing a part of the network in which a failure has occurred, so that high-speed and stable distribution can be realized.

[0733] Further, not limited to decentralized processing of distribution itself, the encoding process of the captured data can be performed by each terminal, on the server side, or shared by them. As an example, two processing cycles are generally performed in the encoding process. In the first cycle, the complexity or the amount of encoding of an image of a frame or a scene unit is detected. Further, in the second cycle, processing that maintains the quality and improves the encoding efficiency is performed. For example, by performing the first encoding process by a terminal and the second encoding process by the server side that receives content, it is possible to reduce the processing load in each terminal while improving the quality and efficiency of content. In this case, if there is a request to receive and decode almost in real time, the data completed by the first encoding by the terminal can also be received and reproduced by another terminal, so that more flexible real-time distribution can also be performed.

[0734] As other examples, the camera ex113 or the like extracts a feature amount (feature or amount of feature) from an image, compresses and transmits data on the feature amount as metadata to the server. The server switches quantization accuracy or the like, for example, according to the importance of the target judged from the feature amount, and performs compression corresponding to the meaning (or importance of content) of the image. The feature amount data is particularly effective for improving the accuracy and efficiency of motion vector prediction at the time of re-compression in the server. In addition, it is also possible to perform simple encoding such as VLC (Variable Length Coding) by the terminal, and perform processing with a large load such as CABAC (Context Adaptive Binary Arithmetic Coding) by the server.

[0735] As other examples, in a stadium, a shopping center, or a factory, or the like, there are cases where a plurality of image data obtained by a plurality of terminals photographing substantially the same scene. In this case, using a plurality of terminals that have performed photographing, and using other terminals that have not performed photographing and a server as necessary, for example, respectively distribute and perform distributed processing of encoding processing in units of GOP (Group of Picture), units of a picture, or units of a tile obtained by dividing a picture, or the like. Thereby, it is possible to reduce delay and more preferably realize real-time performance.

[0736] Since a plurality of image data is substantially the same scene, it is also possible to manage and / or instruct by the server to refer to image data photographed by each terminal to each other. In addition, it is also possible for the server to receive encoded data from each terminal and change the reference relationship between a plurality of data, or to correct or replace a picture itself and re-encode. Thereby, it is possible to generate a stream that improves the quality and efficiency of one data.

[0737] Furthermore, the server can also perform transcoding to change the encoding method of the image data and distribute the image data. For example, the server can also change the encoding method of the MPEG type to the VP type (for example, VP9), and can also change H.264 to H.265 or the like.

[0738] In this way, the encoding processing can be performed by the terminal or one or more servers. Therefore, the following description uses "server" or "terminal" or the like as the subject of the processing, but it is also possible to perform part or all of the processing by the server with the terminal, and it is also possible to perform part or all of the processing by the terminal with the server. In addition, as for these, the same is true for the decoding processing.

[0739] [3D, multi-angle]

[0740] The number of cases in which images or videos taken by a plurality of terminals such as cameras ex113 and / or smartphones ex115 and the like, which are roughly synchronized with each other, of different scenes or of the same scene from different angles are merged and utilized is increasing. The videos taken by the respective terminals can be merged based on the relative positional relationship between the terminals obtained separately, or a region in which feature points coincide in the videos, and the like.

[0741] The server not only encodes two-dimensional moving images, but also automatically or at a time specified by the user encodes still images based on scene analysis of the moving images and the like and transmits them to the receiving terminal. The server also generates a three-dimensional shape of a scene based on videos of the same scene taken from different angles, not only two-dimensional moving images, when the relative positional relationship between the terminals that take the videos can be obtained. The server can also separately encode three-dimensional data generated by a point cloud and the like, and can select or reconstruct videos to be transmitted to the receiving terminal from the videos taken by the plurality of terminals based on the results of recognizing or tracking a person or a target using the three-dimensional data.

[0742] In this way, the user can not only arbitrarily select each video corresponding to each terminal to enjoy the scene, but also enjoy the content of a video in which a viewpoint is selected from three-dimensional data reconstructed using a plurality of images or videos. Furthermore, sound can also be collected from a plurality of different angles together with the video, and the server multiplexes sound from a specific angle or space with the corresponding video and transmits the multiplexed video and sound.

[0743] In addition, in recent years, contents that correspond the real world and the virtual world such as Virtual Reality (VR) and Augmented Reality (AR) are becoming popular. In the case of a VR image, the server creates viewpoint images for the right eye and the left eye, respectively, and can either encode them to allow reference between the respective viewpoint images by Multi-View Coding (MVC) or the like, or encode them as different streams without reference to each other. At the time of decoding of the different streams, they can be reproduced in synchronization with each other according to the viewpoint of the user to reproduce a virtual three-dimensional space.

[0744] In the case of the AR image, the server can also superimpose the virtual object information on the virtual space on the camera information of the real space based on the three-dimensional position or movement of the viewpoint of the user. The decoding device acquires or holds the virtual object information and the three-dimensional data, generates a two-dimensional image according to the movement of the viewpoint of the user, and creates superimposition data by smoothly connecting them. Alternatively, the decoding device can transmit the movement of the viewpoint of the user to the server in addition to the commission of the virtual object information. The server can create superimposition data according to the three-dimensional data held in the server, match the received movement of the viewpoint, encode the superimposition data, and distribute it to the decoding device. Typically, the superimposition data has an alpha value indicating the degree of transparency in addition to RGB, and the server sets the alpha value of a portion other than the target created according to the three-dimensional data to 0 or the like and encodes it in a state of being transparent at that portion. Alternatively, the server can set the RGB value of a predetermined value as the background, as in chroma keying, and generate data in which the portion other than the target is set to the background color. The RGB value of the predetermined value can be determined in advance.

[0745] Likewise, the decoding process of the distributed data can be performed by the client (for example, the terminal), on the server side, or shared with each other. As an example, a certain terminal can first transmit a reception request to the server, another terminal can receive and decode the content corresponding to the request, and the decoded signal can be transmitted to the device having a display. By dispersing the process regardless of the performance of the communicable terminal itself and selecting appropriate content, it is possible to reproduce data with better image quality. Further, as another example, a large-size image data can be received by a TV or the like, and a part of the region such as a tile after the picture is divided can be decoded and displayed by the personal terminal of the viewer. Thus, it is possible to confirm one's own responsible area or an area that one wants to confirm in more detail while sharing the entire image.

[0746] In a situation where a plurality of wireless communications of short, medium, or long distances indoors and outdoors can be used, it is possible to seamlessly receive the content using the distribution system standards such as MPEG-DASH. The user can also switch in real time while freely selecting the user's terminal, the decoding device or the display device such as a display arranged indoors and outdoors. Further, it is possible to switch and decode the terminal of the decoding and the terminal of the display using one's own position information or the like. Thus, it is also possible to map and display information on a part of the wall surface or the floor of a building next to a device that can display during the user's movement to the destination. Further, it is also possible to switch the bit rate of the received data based on the ease of access to the encoded data on the network, which is cached in a server that can be accessed from the receiving terminal in a short time, or duplicated in an edge server of a content distribution service, or the like.

[0747] [Hyper-level coding]

[0748] Regarding content switching, use Figure 61 The scalable stream, which is compressed using the motion picture coding method described in the above embodiments, will be explained. For the server, a single stream can also consist of multiple streams with the same content but different qualities, or the content structure can be switched using the characteristics of a temporally / spatially scalable stream achieved through layered coding, as shown in the illustration. That is, by having the decoding side determine which layer to decode based on intrinsic factors such as performance and extrinsic factors such as the state of the communication band, the decoding side can freely switch between decoding low-resolution and high-resolution content. For example, if the user wants to watch a video viewed on a smartphone (ex115) while on the move, and then watch it again at home using an internet TV device, the device only needs to decode the same stream to a different layer, thus reducing the burden on the server side.

[0749] Furthermore, in addition to the hierarchical structure described above, where images are encoded layer by layer and enhancement layers are implemented above the base layers, the enhancement layer can also contain metadata such as image statistics. Alternatively, the decoding side can generate high-quality content by performing super-resolution on the base layer images based on the metadata. Super-resolution can improve the S / N ratio while maintaining and / or increasing the resolution. The metadata includes information used to determine linear or nonlinear filter coefficients used in the super-resolution process, or information determining parameter values ​​in filtering, machine learning, or least-squares operations used in the super-resolution process.

[0750] Alternatively, a structure can be provided to segment the image into tiles based on the meaning of objects, etc., within the image. The decoding side decodes only a portion of the region by selecting tiles to decode. Furthermore, by storing the attributes of the target (people, cars, balls, etc.) and its position within the image (coordinates within the same image, etc.) as metadata, the decoding side can determine the desired target's location based on this metadata and decide which tiles include that target. For example, as... Figure 62 As shown, metadata can also be stored using data storage structures different from pixel data, such as the SEI (supplementary enhancement information) message in HEVC. This metadata can represent, for example, the position, size, or color of the main target.

[0751] The meta information can also be saved in units of a plurality of pictures such as a stream, a sequence, or a random access unit. The decoding side can acquire the time when a specific person appears in the video, and by matching the information of the picture unit and the time information, the picture in which the target exists can be determined, and the position of the target in the picture can be decided.

[0752] [Optimization of Web Page]

[0753] Figure 63 Fig. 1 is a diagram showing an example of a display screen of a web page in a computer ex111 or the like. Figure 64 Fig. 2 is a diagram showing an example of a display screen of a web page in a smart phone ex115 or the like. As shown in Figs. 1 and 2, a web page can include a plurality of link images as links to image contents. Figure 63 Figure 64 As shown in Figs. 1 and 2, a web page can include a plurality of link images as links to image contents. The visibility of the link images can differ depending on the device on which the web page is viewed. In a case where a plurality of link images are visible on the screen, before the user explicitly selects a link image, or before the link image approaches the vicinity of the center of the screen or the entirety of the link image enters the screen, the display device (decoding device) can display a still image or an I picture possessed by each content as a link image, can display a moving image such as a gif animation using a plurality of still images or I pictures, or can decode and display the moving image using only the base layer.

[0754] In a case where a link image is selected by the user, the display device decodes the base layer with the highest priority, for example. If there is information indicating that the content is scalable in the HTML constituting the web page, the display device can decode up to the enhancement layer. Further, in a case where the selection is made in order to ensure real-time performance or the communication band is very tight, the display device can reduce the delay between the decoding time and the display time of the initial picture (the delay from the start of decoding to the start of display) by decoding and displaying only the pictures referred to in the front (I pictures, P pictures, and B pictures that refer only to the front). Furthermore, the display device can forcibly ignore the reference relationship of the pictures and decode all the B pictures and P pictures as referring to the front, and as the number of pictures received increases with the passage of time, normal decoding can be performed.

[0755] [Automatic Travel]

[0756] Further, in a case where still images or moving image data such as two-dimensional or three-dimensional map information are transmitted and received for the automatic travel or travel assistance of a vehicle, the receiving terminal can receive information such as weather or construction information as meta information in addition to the image data belonging to one or more layers, and decode them in correspondence. The meta information can belong to a layer or can be multiplexed with the image data.

[0757] ​In this case, since the vehicle, the drone, or the airplane, etc. including the reception terminal is moving, the reception terminal can perform seamless reception and decoding at the time of switching the base stations ex106 to ex110 by transmitting the position information of the reception terminal. Further, the reception terminal can dynamically switch the degree of reception of the meta information or the degree of update of the map information according to the selection of the user, the situation of the user, and / or the state of the communication band.

[0758] In the content supply system ex100, the client can receive and decode the encoded information transmitted by the user in real time, and reproduce it.

[0759] [Distribution of Personal Contents]

[0760] Further, in the content supply system ex100, not only the high-quality, long-time contents provided by the video distribution business operator, but also the low-quality, short-time contents provided by the individual can be distributed by unicast or multicast. It is conceivable that such personal contents will increase in the future. In order to make the personal contents better contents, the server can perform the encoding process after performing the editing process. This can be realized, for example, by the following structure.

[0761] After the shooting, the server performs the recognition process of the shooting error, the scene search, the meaning analysis, and the target detection, etc. based on the original image data or the encoded data in real time or cumulatively. Further, the server performs the editing of the correction of the focus deviation or the hand shake, etc. manually or automatically, or the deletion of the scene which is less important than the other pictures or the scene in which the focus is not aligned, or the edge emphasis of the target, or the change of the color tone, etc. based on the recognition result. The server encodes the edited data based on the editing result. Further, it is known that the viewing rate decreases if the shooting time is too long, so the server can automatically limit the scene which is less important than the above, or the scene in which the motion is less, etc. based on the image processing result to be the contents within a specific time range according to the shooting time. Alternatively, the server can generate and encode the summary based on the result of the meaning analysis of the scene.

[0762] Personal content may contain elements that infringe on copyrights, the author's moral rights, or portrait rights, or it may be shared beyond the intended scope, causing inconvenience to the individual. Therefore, servers can encode images such as faces in the periphery of a scene or a home scene by forcibly changing them to out-of-focus images. Furthermore, servers can identify whether a face different from a pre-registered person is captured within the image to be encoded, and if so, apply processing such as mosaic to the face. Alternatively, as pre- or post-processing for encoding, from a copyright perspective, users can specify the people or background areas they want to process in the image. The server can then replace the specified area with another image or blur the focus. If it involves a person, the server can track the person in a moving image and replace the image of their face.

[0763] For personal content with small data volumes, real-time audiovisual requirements are more stringent. Therefore, although bandwidth also plays a role, the decoding device can prioritize receiving, decoding, and reproducing the base layer. The decoding device can also receive enhancement layers during this process, including them in high-quality image reproduction if the playback is looped or repeated more than twice. In this way, if the stream is scalably encoded, it can provide an experience where the motion images are initially coarse, but gradually become smoother and the image quality improves. Besides scalable encoding, the same experience can be provided when the first, coarser stream and a second stream encoded based on the first motion image are combined into a single stream.

[0764] [Other Implementation Examples]

[0765] Furthermore, these encoding or decoding processes are typically handled within the LSI ex500 integrated circuit on each terminal. (LSI (large-scale integration circuitry) ex500 (reference)) Figure 60 The structure can be either a single chip or composed of multiple chips. Furthermore, software for motion image encoding or decoding can be installed on a recording medium (CD-ROM, floppy disk, hard disk, etc.) that can be read by a computer such as the ex111, and the software can be used for encoding and decoding. Moreover, when the ex115 smartphone has a camera, motion image data acquired by that camera can also be transmitted. This motion image data can also be encoded using the LSIex500 chip found in the ex115 smartphone.

[0766] Alternatively, the LSIex500 can also be a structure that downloads and activates application software. In this case, the terminal first determines whether it corresponds to the content's encoding method or whether it has the capability to execute a specific service. If the terminal does not correspond to the content's encoding method or does not have the capability to execute a specific service, the terminal can also download a codec or application software and then retrieve and reproduce the content.

[0767] Furthermore, not limited to the content delivery system ex100 via the Internet ex101, at least one of the moving image encoding device (image encoding device) or moving image decoding device (image decoding device) of the above embodiments can also be assembled in a digital broadcasting system. Since multiplexed data that multiplexes images and sound is carried and transmitted and received using radio waves for broadcasting via satellites or the like, it is more suitable for multicasting than the unicast-friendly structure of the content delivery system ex100, but the same applications can be performed for encoding and decoding processing.

[0768] [Hardware Structure]

[0769] Figure 65 This is a further detailed explanation Figure 60 The image shown is of the EX115 smartphone. Additionally... Figure 66 This diagram illustrates a structural example of a smartphone ex115. The smartphone ex115 includes an antenna ex450 for transmitting and receiving radio waves with a base station ex110, a camera unit ex465 for capturing images and still images, and a display unit ex458 for displaying decoded data such as images captured by the camera unit ex465 and images received by the antenna ex450. The smartphone ex115 also includes an operation unit ex466, such as a touch panel; a sound output unit ex457, such as a speaker, for outputting sound or audio; a sound input unit ex456, such as a microphone, for inputting sound; a memory unit ex467 capable of storing encoded or decoded data such as captured images or still images, recorded audio, received images or still images, and emails; and a slot unit ex464 serving as an interface with a SIM ex468 used to identify the user and authenticate access to various data sources, such as the network. Alternatively, an external memory may be used instead of the memory unit ex467.

[0770] The main control section ex460 capable of comprehensively controlling the display section ex458, the operation section ex466, and the like, and the power supply circuit section ex461, the operation input control section ex462, the image signal processing section ex455, the camera interface section ex463, the display control section ex459, the modulation / demodulation section ex452, the multiplexing / demultiplexing section ex453, the sound signal processing section ex454, the slot section ex464, and the memory section ex467 are connected to each other via a bus ex470 in synchronization.

[0771] The power supply circuit section ex461 activates the smartphone ex115 to an operable state if the power key is brought to an on state by the user's operation, and supplies electric power from a battery pack to each section.

[0772] The smartphone ex115 performs processing of a call and data communication and the like based on the control of the main control section ex460 having a CPU, a ROM, a RAM, and the like. At the time of a call, a sound signal of a sound collected by the sound input section ex456 is converted into a digital sound signal by the sound signal processing section ex454, a spectrum spread process is performed by the modulation / demodulation section ex452, a digital-analog conversion process and a frequency conversion process are performed by the transmission / reception section ex451, and the resultant signal is transmitted via the antenna ex450. Further, received data is amplified and subjected to a frequency conversion process and an analog-digital conversion process, a spectrum inverse spread process is performed by the modulation / demodulation section ex452, and an analog sound signal is converted by the sound signal processing section ex454, after which it is output from the sound output section ex457. At the time of data communication, a text, a still image, or image data can be sent out under the control of the main control section ex460 via the operation input control section ex462 based on the operation of the operation section ex466 of the main body section and the like. The same transmission and reception processing is performed. At the time of data communication, in the case of transmitting an image, a still image, or image and sound, the image signal processing section ex455 compressively encodes an image signal saved in the memory section ex467 or an image signal input from the camera section ex465 by the moving image encoding method indicated in each of the embodiments described above, and sends the encoded image data to the multiplexing / demultiplexing section ex453. The sound signal processing section ex454 encodes a sound signal collected by the sound input section ex456 in the process of photographing an image, a still image, by the camera section ex465, and sends the encoded sound data to the multiplexing / demultiplexing section ex453. The multiplexing / demultiplexing section ex453 multiplexes the encoded image data and the encoded sound data in a prescribed manner, performs a modulation process and a conversion process by the modulation / demodulation section (modulation / demodulation circuit section) ex452 and the transmission / reception section ex451, and transmits them via the antenna ex450. The prescribed manner can be determined in advance.

[0773] In a case where an image attached to an e-mail or a chat tool or an image linked on a Web page is received, in order to decode multiplexed data received via the antenna ex450, the multiplexing / demultiplexing section ex453 separates the multiplexed data into a bit stream of image data and a bit stream of sound data by demultiplexing the multiplexed data, supplies the encoded image data to the image signal processing section ex455 via the synchronous bus ex470, and supplies the encoded sound data to the sound signal processing section ex454. The image signal processing section ex455 decodes the image signal by a moving image decoding method corresponding to the moving image encoding method indicated in each of the above-described embodiments, and displays an image or a still image included in a linked moving image file from the display section ex458 via the display control section ex459. The sound signal processing section ex454 decodes the sound signal, and outputs the sound from the sound output section ex457. Since real-time streaming media is becoming more popular, depending on the situation of the user, there can be a case where reproduction of sound is not socially appropriate. Therefore, it is also possible that, as an initial value, a structure in which the sound signal is not reproduced and only the image data is reproduced is preferable, and the sound is reproduced in synchronization only when the user performs a click or the like on the image data.

[0774] Furthermore, although the smart phone ex115 is described here as an example, as a terminal, in addition to a transceiving type terminal having both an encoder and a decoder, a transmission terminal having only an encoder, a reception terminal having only a decoder, and the like can be considered. In a system for digital broadcasting, a case where multiplexed data in which sound data is multiplexed with image data is received and transmitted is described. However, in addition to sound data, character data associated with an image or the like can be multiplexed in the multiplexed data. In addition, instead of multiplexed data, the image data itself can be received or transmitted.

[0775] In addition, although a case where the main control section ex460 including a CPU controls the encoding or decoding process is described, a case where various terminals are provided with a GPU is also common. Therefore, a structure in which a larger region is processed together using the performance of a GPU by a memory shared by the CPU and the GPU or a memory in which addresses are managed in a manner that can be commonly used can be made. Thereby, the encoding time can be shortened, real-time performance can be ensured, and low latency can be achieved. In particular, if the processes of motion estimation, deblocking filtering, SAO (Sample Adaptive Offset), and transform / quantization are not performed by the CPU but are performed by the GPU together in units of pictures or the like, the processes are more efficient.

[0776] Industrial Applicability

[0777] The present application can be used in, for example, a television receiver, a digital video recorder, a car navigation system, a mobile phone, a digital camera, a digital video camera, a video conference system, or an electronic mirror.

[0778] Reference Signs List

[0779] 100 encoding device

[0780] 102 division section

[0781] 104 subtraction section

[0782] 106 conversion section

[0783] 108 quantization section

[0784] 110 entropy encoding section

[0785] 112, 204 inverse quantization section

[0786] 114, 206 inverse conversion section

[0787] 116, 208 addition section

[0788] 118, 210 block memory

[0789] 120, 212 loop filter section

[0790] 122, 214 frame memory

[0791] 124, 216 intra prediction section

[0792] 126, 218 inter prediction section

[0793] 128, 220 prediction control section

[0794] 200 decoding device

[0795] 202 entropy decoding section

[0796] 1201 boundary determination section

[0797] 1202, 1204, 1206 switch

[0798] 1203 filter determination section

[0799] 1205 filter processing section

[0800] 1207 filter characteristic decision section

[0801] 1208 processing determination section

[0802] a1, b1 processor

[0803] a2, b2 memory

Claims

1. An encoding apparatus, comprising: Possessing: circuitry; and a memory connected to the circuitry, in residual encoding of a current block, in both a case where an orthogonal transform is applied using mutually different syntaxes and a case where the orthogonal transform is skipped, in a case where a processing number limit of context adaptive encoding allows context adaptive encoding of a plurality of coefficient information flags related to a coefficient included in the current block, the circuitry encodes the plurality of coefficient information flags by the context adaptive encoding, and encodes a residual value of the coefficient by Golomb encoding, the residual value of the coefficient being a value used to reconstruct a value of the coefficient using the plurality of coefficient information flags, in a case where the processing number limit does not allow the context adaptive encoding of the plurality of coefficient information flags, the circuitry skips encoding of the plurality of coefficient information flags, and encodes a value of the coefficient by the Golomb encoding, in the case where the orthogonal transform is skipped, the circuitry encodes a plurality of absolute value flags by the context adaptive encoding after encoding coefficient information flags other than the plurality of absolute value flags among the plurality of coefficient information flags and before encoding a residual value of the coefficient, the plurality of absolute value flags being flags related to whether an absolute value of the coefficient is larger than a prescribed value, and the prescribed value being an integer larger than 1, the plurality of coefficient information flags include a flag indicating whether the value of the coefficient is zero or nonzero, and a flag indicating whether the coefficient is odd or even, in the case where the orthogonal transform is skipped, the plurality of coefficient information flags further include a flag indicating whether the coefficient is a positive value or a negative value.

2. A decoding apparatus, wherein, Possessing: circuitry; and a memory connected to the circuitry, in residual decoding of a current block, in both a case where an inverse orthogonal transform is applied using mutually different syntaxes and a case where the inverse orthogonal transform is skipped, in a case where a processing number limit of context adaptive decoding allows context adaptive decoding of a plurality of coefficient information flags related to a coefficient included in the current block, the circuitry decodes the plurality of coefficient information flags by the context adaptive decoding, and decodes a residual value of the coefficient by Golomb decoding, the residual value of the coefficient being a value used to reconstruct a value of the coefficient using the plurality of coefficient information flags, in a case where the processing number limit does not allow the context adaptive decoding of the plurality of coefficient information flags, the circuitry skips decoding of the plurality of coefficient information flags, and decodes a value of the coefficient by the Golomb decoding, In a case where the inverse orthogonal transform is skipped, the plurality of absolute value flags are decoded by the context adaptive decoding after the decoding of the coefficient information flags other than the plurality of absolute value flags among the plurality of coefficient information flags and before the decoding of the residual value of the coefficient, the plurality of absolute value flags are flags related to whether the absolute value of the coefficient is larger than a prescribed value, and the prescribed value is an integer larger than 1, the plurality of coefficient information flags include a flag indicating whether the value of the coefficient is zero or nonzero and a flag indicating whether the coefficient is odd or even, in a case where the inverse orthogonal transform is skipped, the plurality of coefficient information flags further include a flag indicating whether the coefficient is a positive value or a negative value.

3. A non-transitory computer-readable storage medium storing a bitstream, wherein the bitstream includes information for causing a decoding device to perform residual decoding processing, in the residual decoding of a current block, in a case where an inverse orthogonal transform is applied using mutually different syntaxes and in a case where the inverse orthogonal transform is skipped, in a case where a number-of-processes limit of the context adaptive decoding allows the plurality of coefficient information flags related to a coefficient included in the current block to be subjected to the context adaptive decoding, the plurality of coefficient information flags are decoded by the context adaptive decoding, and a residual value of the coefficient, which is a value used to reconstruct the value of the coefficient using the plurality of coefficient information flags, is decoded by a Golomb-Rice decoding, in a case where the number-of-processes limit does not allow the plurality of coefficient information flags to be subjected to the context adaptive decoding, the decoding of the plurality of coefficient information flags is skipped, and the value of the coefficient is decoded by the Golomb-Rice decoding, in a case where the inverse orthogonal transform is skipped, the plurality of absolute value flags are decoded by the context adaptive decoding after the decoding of the coefficient information flags other than the plurality of absolute value flags among the plurality of coefficient information flags and before the decoding of the residual value of the coefficient, the plurality of absolute value flags are flags related to whether the absolute value of the coefficient is larger than a prescribed value, and the prescribed value is an integer larger than 1, the plurality of coefficient information flags include a flag indicating whether the value of the coefficient is zero or nonzero and a flag indicating whether the coefficient is odd or even, in a case where the inverse orthogonal transform is skipped, the plurality of coefficient information flags further include a flag indicating whether the coefficient is a positive value or a negative value.

Images