Encoding device, decoding device and non-transitory memory medium

Abstract

The encoding device includes: a circuit; and a memory connected to the circuit. During operation, the circuit: derives a reference movement vector for predicting the target block; derives a first movement vector different from the reference movement vector; derives a differential movement vector based on the difference between the reference movement vector and the first movement vector; and determines whether the differential movement vector is greater than a threshold. If the differential movement vector is greater than the threshold, the first movement vector is modified; if the differential movement vector is not greater than the threshold, the first movement vector is not modified, and the target block is encoded using either the modified or unchanged first movement vector.

Description

Technical Field

[0001] The present disclosure relates to an encoding device and the like for encoding dynamic images. Prior Art

[0002] Background Art So far, as a standard for encoding dynamic images, there is H.265, which is also known as HEVC (High Efficiency Video Coding) (Non-Patent Document 1).

[0003] Prior Art Documents Non-Patent Documents Non-Patent Document 1: H.265 (ISO / IEC 23008-2 HEVC) / HEVC (High Efficiency Video Coding) Summary of the Invention

[0004] Problems to be Solved by the Invention In such encoding and decoding methods, in order to improve processing efficiency, image quality, and reduce circuit scale, etc., it is expected to propose a new method.

[0005] Each of the configurations or methods disclosed in the embodiments or a part thereof in the present disclosure can contribute to at least any one of, for example, improvement of encoding efficiency, reduction of encoding / decoding processing volume, reduction of circuit scale, improvement of encoding / decoding speed, and appropriate selection of components / actions such as filters, blocks, sizes, motion vectors, reference pictures, and reference blocks in encoding and decoding.

[0006] In addition, the present disclosure also includes the disclosure of configurations or methods that can provide benefits other than the above. For example, configurations or methods that improve encoding efficiency while suppressing an increase in processing volume.

[0007] Means for Solving the Problems An encoding device according to an aspect of the present disclosure is an encoding device that encodes a dynamic image, and includes a circuit and a memory connected to the circuit. During operation, the circuit: derives a reference motion vector for predicting a target block; derives a first motion vector different from the reference motion vector; derives a differential motion vector based on the difference between the reference motion vector and the first motion vector; and determines whether the differential motion vector is greater than a threshold value. When it is determined that the differential motion vector is greater than the threshold value, the first motion vector is changed. When it is determined that the differential motion vector is not greater than the threshold value, the first motion vector is not changed. The target block is encoded using the changed first motion vector or the unchanged first motion vector.

[0008] A decoding device according to an aspect of the present disclosure is a decoding device that decodes a dynamic image, and includes a circuit and a memory connected to the circuit. During operation, the circuit: derives a reference motion vector for predicting a target block; derives a first motion vector different from the reference motion vector; derives a motion vector based on the difference between the reference motion vector and the first motion vector; and determines whether the differential motion vector is greater than a threshold value. When it is determined that the differential motion vector is greater than the threshold value, the first motion vector is changed. When it is determined that the differential motion vector is not greater than the threshold value, the first motion vector is not changed. The target block is decoded using the changed first motion vector or the unchanged first motion vector.

[0009] In addition, these general or specific aspects can be implemented by a system, a device, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or can be implemented by any combination of a system, a device, a method, an integrated circuit, a computer program, and a recording medium.

[0010] Further benefits and advantages provided by the disclosed embodiments can be understood from the specification and the drawings. These benefits and advantages can sometimes be obtained by the features of various embodiments or the specification and the drawings respectively. In order to obtain one or more benefits and advantages, it is not necessary to provide all of them.

[0011] Effects of the Invention The present disclosure can provide an encoding device, a decoding device, an encoding method, or a decoding method that can improve processing efficiency. Brief Description of the Drawings

[0012] FIG. 1 is a block diagram showing the functional configuration of the encoding device according to the embodiment.

[0013] FIG. 2 is a flowchart showing an example of the overall encoding process performed by the encoding device.

[0014] FIG. 3 is a diagram showing an example of block division.

[0015] FIG. 4A is a diagram showing an example of the configuration of a slice.

[0016] FIG. 4B is a diagram showing an example of the configuration of a block.

[0017] FIG. 5A is a table showing the transform basis functions corresponding to the respective transform types.

[0018] FIG. 5B is a diagram showing SVT (Spatially Varying Transform).

[0019] FIG. 6A is a diagram showing an example of the shape of the filter used by the ALF (adaptive loop filter).

[0020] FIG. 6B is a diagram showing another example of the shape of the filter used by the ALF.

[0021] FIG. 6C is a diagram showing another example of the shape of the filter used by the ALF.

[0022] FIG. 7 is a block diagram showing an example of the detailed configuration of the loop filter section that functions as a DBF.

[0023] FIG. 8 is a diagram showing an example of a deblocking filter having filter characteristics symmetric with respect to the block boundary.

[0024] FIG. 9 is a diagram for explaining the block boundary where deblocking filtering processing is performed.

[0025] FIG. 10 is a diagram showing an example of the Bs value.

[0026] FIG. 11 is a diagram showing an example of the processing performed by the prediction processing section of the encoding device.

[0027] FIG. 12 is a diagram showing another example of the processing performed by the prediction processing unit of the encoding device.

[0028] FIG. 13 is a diagram showing another example of the processing performed by the prediction processing unit of the encoding device.

[0029] FIG. 14 is a diagram showing an example of 67 intra prediction modes in intra prediction.

[0030] FIG. 15 is a flowchart showing the flow of the basic processing of inter prediction.

[0031] FIG. 16 is a flowchart showing an example of deriving a motion vector.

[0032] FIG. 17 is a flowchart showing another example of deriving a motion vector.

[0033] FIG. 18 is a flowchart showing another example of deriving a motion vector.

[0034] FIG. 19 is a flowchart showing an example of inter prediction performed in the normal inter mode.

[0035] FIG. 20 is a flowchart showing an example of inter prediction performed in the merge mode.

[0036] FIG. 21 is a flowchart for explaining an example of the motion vector derivation process performed in the merge mode.

[0037] FIG. 22 is a flowchart showing an example of FRUC (frame rate up conversion).

[0038] FIG. 23 is a diagram for explaining an example of pattern matching (bidirectional matching) between two blocks along a motion trajectory.

[0039] FIG. 24 is a diagram for explaining an example of pattern matching (template matching) between a template in the current picture and a block in the reference picture.

[0040] FIG. 25A is a diagram for explaining an example of deriving a motion vector in units of sub-blocks based on motion vectors of a plurality of adjacent blocks.

[0041] FIG. 25B is a diagram for explaining an example of a motion vector of a derived sub-block unit in an affine mode having three control points.

[0042] FIG. 26A is a conceptual diagram for explaining an affine merge mode.

[0043] FIG. 26B is a conceptual diagram for explaining an affine merge mode having two control points.

[0044] FIG. 26C is a conceptual diagram for explaining an affine merge mode having three control points.

[0045] FIG. 27 is a flowchart showing an example of processing of an affine merge mode.

[0046] FIG. 28A is a diagram for explaining an affine inter mode having two control points.

[0047] FIG. 28B is a diagram for explaining an affine inter mode having three control points.

[0048] FIG. 29 is a flowchart showing an example of processing of an affine inter mode.

[0049] FIG. 30A is a diagram for explaining an affine inter mode in which a current block has three control points and an adjacent block has two control points.

[0050] FIG. 30B is a diagram for explaining an affine inter mode in which a current block has two control points and an adjacent block has three control points.

[0051] FIG. 31A is a diagram showing the relationship between a merge mode and DMVR (dynamic motion vector refreshing).

[0052] FIG. 31B is a conceptual diagram for explaining an example of DMVR processing.

[0053] FIG. 32 is a flowchart showing an example of generating a predicted image.

[0054] FIG. 33 is a flowchart showing another example of generating a predicted image.

[0055] FIG. 34 is a flowchart showing still another example of generating a predicted image.

[0056] FIG. 35 is a flowchart for explaining an example of a predictive image correction process performed according to overlapped block motion compensation (OBMC).

[0057] FIG. 36 is a flowchart for explaining an example of a predictive image correction process performed according to OBMC.

[0058] FIG. 37 is a diagram for explaining the generation of a predictive image with two triangles.

[0059] FIG. 38 is a diagram for explaining a model assuming uniform linear motion.

[0060] FIG. 39 is a diagram for explaining an example of a method for generating a predictive image using a luminance correction process performed according to local illumination compensation (LIC).

[0061] FIG. 40 is a block diagram showing an installation example of an encoding device.

[0062] FIG. 41 is a block diagram showing the functional configuration of a decoding device according to an embodiment.

[0063] FIG. 42 is a flowchart showing an example of an overall decoding process performed by a decoding device.

[0064] FIG. 43 is a diagram showing an example of a process performed by a prediction processing unit of a decoding device.

[0065] FIG. 44 is a diagram showing another example of a process performed by a prediction processing unit of a decoding device.

[0066] FIG. 45 is a flowchart showing an example of inter prediction performed in a normal inter mode in a decoding device.

[0067] FIG. 46 is a block diagram showing an installation example of a decoding device.

[0068] FIG. 47 is a flowchart showing an example of inter prediction processing in a first aspect.

[0069] FIG. 48 is a diagram showing an example of a processing target block.

[0070] FIG. 49 is a diagram showing an example of a calculated threshold value.

[0071] FIG. 50 is an overall configuration diagram of a content supply system that implements a content distribution service.

[0072] FIG. 51 is a diagram showing an example of an encoding structure during adaptive encoding.

[0073] FIG. 52 is a diagram showing an example of an encoding structure during adaptive encoding.

[0074] FIG. 53 is a diagram showing an example of a display screen of a web page.

[0075] FIG. 54 is a diagram showing an example of a display screen of a web page.

[0076] FIG. 55 is a diagram showing an example of a smartphone.

[0077] FIG. 56 is a block diagram showing an example of the configuration of a smartphone. Embodiment

[0078] Mode for Carrying Out the Invention (Insight Underlying the Present Disclosure) For example, in an encoding device that encodes a dynamic image, during encoding of the dynamic image, etc., when encoding a dynamic image that performs more detailed prediction processing while suppressing an increase in the processing amount, a prediction error is derived by subtracting a predicted image from the image constituting the dynamic image. Next, the encoding device performs frequency conversion and quantization on the prediction error and encodes the result as image data. At this time, when performing motion prediction processing on the movement of an encoding target unit such as a block included in the dynamic image in units of blocks or sub-blocks constituting the block, there is a possibility of improving the encoding efficiency by controlling the deviation of the motion vector.

[0079] However, in encoding of blocks, etc. included in a dynamic image, etc., if the deviation of the motion vector is not controlled, it will be associated with an increase in the processing amount and the encoding efficiency will be reduced.

[0080] Here, an encoding device according to an aspect of the present disclosure is an encoding device that encodes a dynamic image, and includes: a circuit; and a memory connected to the aforementioned circuit. During operation, the aforementioned circuit: derives a reference motion vector for predicting a target block; derives a first motion vector different from the aforementioned reference motion vector; derives a differential motion vector based on the difference between the aforementioned reference motion vector and the aforementioned first motion vector; and determines whether the aforementioned differential motion vector is greater than a threshold value. When it is determined that the aforementioned differential motion vector is greater than the aforementioned threshold value, the aforementioned first motion vector is changed. When it is determined that the aforementioned differential motion vector is not greater than the aforementioned threshold value, the aforementioned first motion vector is not changed. The aforementioned target block is encoded using the changed aforementioned first motion vector or the unchanged aforementioned first motion vector.

[0081] Thereby, the encoding device can adjust the motion vector so that the deviation of a plurality of motion vectors within a target block divided into a plurality of sub-blocks falls within a predetermined range. Therefore, compared with the case where the deviation of the motion vector is greater than the predetermined range, the number of pixels to be referred to is reduced, so the amount of data transferred from the memory can be suppressed within a predetermined range. Therefore, the encoding device can reduce the memory access amount in the inter prediction process, so the encoding efficiency can be improved.

[0082] For example, it may also be: the aforementioned reference motion vector corresponds to a first pixel combination within the aforementioned target block, and the aforementioned first motion vector corresponds to a second pixel combination within the aforementioned target block that is different from the aforementioned first pixel combination.

[0083] Thereby, the encoding device can appropriately control the deviation of a plurality of motion vectors within the target block using the reference motion vector determined based on the first pixel combination and the first motion vector determined based on the second pixel combination.

[0084] For example, it may also be: when it is determined that the aforementioned differential motion vector is greater than the aforementioned threshold value, the aforementioned circuit changes the aforementioned first motion vector using a value obtained by clipping the aforementioned differential motion vector.

[0085] Thereby, the encoding device can change the first motion vector so that the deviation of the first motion vector falls within a predetermined range. Therefore, the encoding device can reduce the memory access amount in the inter prediction process, so the encoding efficiency can be improved.

[0086] For example, the prediction mode of the aforementioned target block may also be an affine mode.

[0087] Accordingly, the encoding device can perform prediction on the processing target block in sub-block units, thereby improving the prediction accuracy.

[0088] For example, it can also be: the memory access amount in the worst case when performing prediction processing on the aforementioned processing target block by using the aforementioned prediction mode is equal to or less than the memory access amount in the worst case when performing prediction processing by using a prediction mode other than the aforementioned prediction mode, to determine the aforementioned threshold value.

[0089] Accordingly, the encoding device limits the memory access amount for transferring data from the memory area within a predetermined range, thereby smoothly performing the transfer process and improving the processing efficiency.

[0090] For example, it can also be: the memory access amount in the worst case when performing prediction processing by using a prediction mode other than the aforementioned prediction mode is the memory access amount when performing bidirectional prediction processing on the aforementioned processing target block in units of 8×8 pixels by using a prediction mode other than the aforementioned prediction mode.

[0091] At this time, when the encoding device performs prediction processing without using the affine mode, for example, it generates a prediction image from past pictures in block units. When performing bidirectional prediction in units of 8×8 pixel blocks by using a prediction mode other than the affine mode, the amount of memory accessed is the largest. Therefore, the threshold value of the affine mode is determined so that it can also fall within the memory access amount. In this way, the encoding device limits the memory access amount for transferring data from the memory area within a predetermined range, thereby smoothly performing the transfer process and improving the processing efficiency.

[0092] For example, the aforementioned threshold value can also be different depending on whether the aforementioned processing target block performs unidirectional prediction or bidirectional prediction.

[0093] Accordingly, the number of accesses when the encoding device performs bidirectional prediction on the processing target block is more than that when performing unidirectional prediction. Therefore, the threshold value can also be set to be smaller in the case of bidirectional prediction than in the case of unidirectional prediction. In this way, the more the number of accesses, the smaller the threshold value is set, thereby improving the encoding efficiency.

[0094] For example, it can also be: the foregoing circuit determines whether the foregoing differential motion vector is greater than the foregoing threshold value. When the absolute value of the horizontal component of the foregoing differential motion vector is greater than the first value of the foregoing threshold value, or the absolute value of the vertical component of the foregoing differential motion vector is greater than the second value of the foregoing threshold value, it is determined that the foregoing differential motion vector is greater than the foregoing threshold value. When the absolute value of the foregoing horizontal component of the foregoing differential motion vector is not greater than the first value of the foregoing threshold value and the absolute value of the foregoing vertical component of the foregoing differential motion vector is not greater than the foregoing second value of the foregoing threshold value, it is determined that the foregoing differential motion vector is not greater than the foregoing threshold value.

[0095] Thereby, the threshold value can be represented by the horizontal component, i.e., the first value (hereinafter also referred to as the first threshold value), and the vertical component, i.e., the second value (hereinafter also referred to as the second threshold value). Therefore, the encoding device can determine the deviation of the first motion vector in a two-dimensional manner. Furthermore, the encoding device determines whether the deviation of the first motion vector within the processing target block is within a predetermined range, so that the first motion vector within the processing target block can be adjusted to an appropriate value. Therefore, the encoding efficiency can be improved.

[0096] For example, it can also be: when it is determined that the foregoing differential motion vector is greater than the foregoing threshold value, the foregoing circuit uses the value obtained by clipping the foregoing differential motion vector and the foregoing reference motion vector to change the foregoing first motion vector, and the differential motion vector between the changed foregoing first motion vector and the foregoing reference motion vector is not greater than the foregoing threshold value.

[0097] At this time, when the deviation of the first motion vector in the encoding device exceeds the predetermined range, for example, the absolute value of the component that exceeds the threshold value among the horizontal component and the vertical component of the differential motion vector is clipped to the value of the threshold value. Then, for example, the encoding device can also add the value obtained by clipping the differential motion vector to the reference motion vector as the changed first motion vector. Thereby, the deviation of the first motion vector within the processing target block can be made to fall within the predetermined range. Therefore, the encoding efficiency can be improved.

[0098] For example, the foregoing reference motion vector can also be the average of a plurality of the foregoing first motion vectors of the foregoing processing target block.

[0099] Thereby, the encoding device can derive the difference from the reference for each of the plurality of first motion vectors within the processing target block, using the average of all the first motion vectors within the processing target block as the reference.

[0100] For example, the foregoing reference motion vector can also be one of a plurality of the foregoing first motion vectors within the foregoing processing target block.

[0101] Thus, for each of a plurality of first motion vectors in a processing target block, the encoding device can derive a difference from a reference by using one first motion vector in the processing target block as the reference.

[0102] For example, the aforementioned threshold value may also correspond to the size of the aforementioned processing target block.

[0103] Thus, since the memory access amount varies depending on the size of the processing target block, the encoding device can also make the threshold value smaller when the memory access amount is larger. For example, the larger the size of the processing target block, the smaller the threshold value. Therefore, even if the size of the processing target block increases, the encoding device can keep the memory access amount within a predetermined range, thereby improving the encoding efficiency.

[0104] For example, the aforementioned threshold value may also be determined in advance and not encoded into the stream.

[0105] Thus, the encoding device can use the threshold value determined in advance, so there is no need to encode the threshold value every time prediction processing is performed. Therefore, the encoding efficiency can be improved.

[0106] Furthermore, a decoding device according to an aspect of the present disclosure is a decoding device that decodes a dynamic image, including: a circuit; and a memory connected to the aforementioned circuit. During operation, the aforementioned circuit: derives a reference motion vector for predicting a processing target block; derives a first motion vector different from the aforementioned reference motion vector; derives a motion vector based on the difference between the aforementioned reference motion vector and the aforementioned first motion vector; and determines whether the difference motion vector is greater than a threshold value. When it is determined that the difference motion vector is greater than the aforementioned threshold value, the aforementioned first motion vector is changed. When it is determined that the difference motion vector is not greater than the aforementioned threshold value, the aforementioned first motion vector is not changed, and the aforementioned processing target block is decoded using the changed aforementioned first motion vector or the unchanged aforementioned first motion vector.

[0107] Thus, the decoding device can adjust the motion vector so that the deviation of a plurality of motion vectors in a processing target block divided into a plurality of sub-blocks falls within a predetermined range. Therefore, compared with the case where the deviation of the motion vector is greater than the predetermined range, the number of pixels to be referred to is reduced, so the amount of data transferred from the memory can be suppressed within a predetermined range. Therefore, the decoding device can reduce the memory access amount during the inter prediction processing, thereby improving the processing efficiency.

[0108] For example, it may also be that: the foregoing reference motion vector corresponds to a first pixel combination within the foregoing processing target block, and the foregoing first motion vector corresponds to a second pixel combination within the foregoing processing target block that is different from the foregoing first pixel combination.

[0109] Thereby, the decoding device can appropriately control the deviation of a plurality of motion vectors within the processing target block by using the reference motion vector defined based on the first pixel combination and the first motion vector defined based on the second pixel combination.

[0110] For example, it may also be that: when it is determined that the foregoing differential motion vector is greater than the foregoing threshold value, the foregoing circuit uses the value obtained by clipping the foregoing differential motion vector to change the foregoing first motion vector.

[0111] Thereby, the decoding device can change the first motion vector so that the deviation of the first motion vector falls within a predetermined range. Therefore, the decoding device can reduce the memory access amount in the inter prediction process, so the processing efficiency can be improved.

[0112] For example, the prediction mode of the foregoing processing target block may also be an affine mode.

[0113] Thereby, the decoding device can perform prediction on the processing target block in units of sub-blocks, so the prediction accuracy can be improved.

[0114] For example, it may also be that: the threshold value is determined in such a way that the memory access amount in the worst case when performing prediction processing on the foregoing processing target block in the foregoing prediction mode is less than or equal to the memory access amount in the worst case when performing prediction processing in a prediction mode other than the foregoing prediction mode.

[0115] Thereby, the decoding device will limit the memory access amount for transferring data from the memory area within a predetermined range, and thus can smoothly perform the transfer process, so the processing efficiency can be improved.

[0116] For example, it may also be that: the memory access amount in the worst case when performing prediction processing in a prediction mode other than the foregoing prediction mode is the memory access amount when performing bidirectional prediction processing on the foregoing processing target block in units of 8×8 pixels in a prediction mode other than the foregoing prediction mode.

[0117] At this time, when the decoding device performs prediction processing without using the affine mode, for example, it generates a prediction image from past pictures in block units. When performing bidirectional prediction in block units of 8×8 pixels using a prediction mode other than the affine mode, the amount of memory accessed becomes the largest. For this reason, a threshold value of the affine mode is determined so that it can also fall within the memory access amount in the affine mode. In this way, the decoding device limits the memory access amount for transferring data from the memory area within a predetermined range, thereby enabling the transfer process to be smoothly executed, and thus improving the processing efficiency.

[0118] For example, the aforementioned threshold value may also be different depending on the case of performing unidirectional prediction and the case of performing bidirectional prediction on the aforementioned processing target block.

[0119] Thereby, the number of accesses when the decoding device performs bidirectional prediction on the processing target block is more than that when performing unidirectional prediction. Therefore, the threshold value can also be set to be smaller in the case of bidirectional prediction than in the case of unidirectional prediction. In this way, the more the number of accesses, the smaller the threshold value is set, thereby improving the processing efficiency.

[0120] For example, it may also be: the aforementioned circuit determines whether the aforementioned differential motion vector is greater than the aforementioned threshold value. When the absolute value of the horizontal component of the aforementioned differential motion vector is greater than the first value of the aforementioned threshold value, or the absolute value of the vertical component of the aforementioned differential motion vector is greater than the second value of the aforementioned threshold value, it is determined that the aforementioned differential motion vector is greater than the aforementioned threshold value. When the aforementioned absolute value of the aforementioned horizontal component of the aforementioned differential motion vector is not greater than the aforementioned first value of the aforementioned threshold value, and the aforementioned absolute value of the aforementioned vertical component of the aforementioned differential motion vector is not greater than the aforementioned second value of the aforementioned threshold value, it is determined that the aforementioned differential motion vector is not greater than the aforementioned threshold value.

[0121] Thereby, the threshold value can be represented by the horizontal component, that is, the first value (hereinafter also referred to as the first threshold value), and the vertical component, that is, the second value (hereinafter also referred to as the second threshold value). Therefore, the decoding device can determine the deviation of the first motion vector in a two-dimensional manner. Furthermore, the decoding device determines whether the deviation of the first motion vector within the processing target block is within a predetermined range, so that the first motion vector within the processing target block can be adjusted to an appropriate value. Therefore, the processing efficiency can be improved.

[0122] For example, it may also be: when it is determined that the aforementioned differential motion vector is greater than the aforementioned threshold value, the aforementioned circuit uses the value obtained by clipping the aforementioned differential motion vector and the aforementioned reference motion vector to change the aforementioned first motion vector, and the differential motion vector between the changed aforementioned first motion vector and the aforementioned reference motion vector is not greater than the aforementioned threshold value.

[0123] At this time, when the deviation of the first motion vector exceeds a predetermined range, for example, the absolute value of the component that exceeds the threshold among the horizontal component and the vertical component of the differential motion vector is clipped to the threshold value. Then, for example, the decoding device can also add the value obtained by clipping the differential motion vector to the value after the reference motion vector as the changed first motion vector. Thereby, the deviation of the first motion vector within the processing target block can be made to fall within the predetermined range. Therefore, the processing efficiency can be improved.

[0124] For example, the aforementioned reference motion vector may also be the average of a plurality of the aforementioned first motion vectors of the aforementioned processing target block.

[0125] Thereby, the decoding device can, for each of a plurality of first motion vectors within the processing target block, use the average of all the first motion vectors within the processing target block as a reference to derive the difference from this reference.

[0126] For example, the aforementioned reference motion vector may also be one of a plurality of the aforementioned first motion vectors within the aforementioned processing target block.

[0127] Thereby, the decoding device can, for each of a plurality of first motion vectors within the processing target block, use one of the first motion vectors within the processing target block as a reference to derive the difference from this reference.

[0128] For example, the aforementioned threshold value may also correspond to the size of the aforementioned processing target block.

[0129] Thereby, since the memory access amount of the decoding device varies depending on the size of the processing target block, the threshold value can also be made smaller when the memory access amount is larger. For example, the larger the size of the processing target block, the smaller the threshold value. Therefore, the decoding device can keep the memory access amount within the predetermined range even when the size of the processing target block becomes larger, so the processing efficiency can be improved.

[0130] For example, the aforementioned threshold value may also be determined in advance and not decoded from the stream.

[0131] Thereby, the decoding device can use the threshold value determined in advance, so there is no need to decode the threshold value every time prediction processing is performed. Therefore, the processing efficiency can be improved.

[0132] Furthermore, an encoding method according to an aspect of the present disclosure is an encoding method for encoding a dynamic image, including: deriving a reference motion vector for predicting a target block; deriving a first motion vector different from the reference motion vector; deriving a differential motion vector based on the difference between the reference motion vector and the first motion vector; and determining whether the differential motion vector is greater than a threshold value. When it is determined that the differential motion vector is greater than the threshold value, the first motion vector is changed. When it is determined that the differential motion vector is not greater than the threshold value, the first motion vector is not changed. The target block is encoded using the changed first motion vector or the unchanged first motion vector.

[0133] Thereby, the encoding method can adjust the motion vector so that the deviation of a plurality of motion vectors in a target block divided into a plurality of sub-blocks falls within a predetermined range. Therefore, compared with the case where the deviation of the motion vector is greater than the predetermined range, the number of pixels to be referred to is reduced, so the amount of data transferred from the memory can be suppressed within a predetermined range. Therefore, the encoding method can reduce the memory access amount in the inter prediction process, so the encoding efficiency can be improved.

[0134] Furthermore, a decoding method according to an aspect of the present disclosure is a decoding method for decoding a dynamic image, including: deriving a reference motion vector for predicting a target block; deriving a first motion vector different from the reference motion vector; deriving a differential motion vector based on the difference between the reference motion vector and the first motion vector; and determining whether the differential motion vector is greater than a threshold value. When it is determined that the differential motion vector is greater than the threshold value, the first motion vector is changed. When it is determined that the differential motion vector is not greater than the threshold value, the first motion vector is not changed. The target block is decoded using the changed first motion vector or the unchanged first motion vector.

[0135] Thereby, the decoding method can adjust the motion vector so that the deviation of a plurality of motion vectors in a target block divided into a plurality of sub-blocks falls within a predetermined range. Therefore, compared with the case where the deviation of the motion vector is greater than the predetermined range, the number of pixels to be referred to is reduced, so the amount of data transferred from the memory can be suppressed within a predetermined range. Therefore, the decoding method can reduce the memory access amount in the inter prediction process, so the processing efficiency can be improved.

[0136] Furthermore, these general or specific aspects can be implemented through a system, a device, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or can be implemented by any combination of a system, a device, a method, an integrated circuit, a computer program, and a recording medium.

[0137] Hereinafter, embodiments will be specifically described with reference to the drawings. Additionally, each of the embodiments described below shows general or specific examples. The numerical values, shapes, materials, components, the arrangement positions and connection forms of the components, steps, the relationship and order of the steps, etc. shown in the following embodiments are merely examples, and their purport is not to limit the scope of the claims.

[0138] Hereinafter, embodiments of an encoding device and a decoding device will be described. The embodiments are examples of an encoding device and a decoding device to which the processes and / or configurations described in various aspects of the present disclosure are applicable. The processes and / or configurations are also implemented in encoding devices and decoding devices different from the embodiments. For example, regarding the processes and / or configurations applicable to the embodiments, any of the following methods can be implemented, for example.

[0139] (1) Any one of the plurality of components of the encoding device or the decoding device of the embodiments described in various aspects of the present disclosure can be replaced with or combined with other components described in any one of the various aspects of the present disclosure.

[0140] (2) In the encoding device or the decoding device of the embodiment, for the functions or processes performed by a part of the plurality of components of the encoding device or the decoding device, any changes such as addition, replacement, deletion, etc. of the functions or processes can also be applied. For example, any function or process can be replaced with or combined with other functions or processes described in any one of the various aspects of the present disclosure.

[0141] (3) In the method implemented by the encoding device or the decoding device of the embodiment, for a part of the plurality of processes included in the method, any changes such as addition, replacement, and deletion can also be applied. For example, any process in the method can be replaced with or combined with other processes described in any one of the various aspects of the present disclosure.

[0142] (4) Among a plurality of constituent elements of the encoding device or the decoding device according to the embodiment, a part of the constituent elements may be combined with the constituent elements described in any one of the aspects of the present disclosure, may be combined with the constituent elements having a part of the functions described in any one of the aspects of the present disclosure, or may be combined with the constituent elements that implement a part of the processes implemented by the constituent elements described in any one of the aspects of the present disclosure.

[0143] (5) A constituent element having a part of the functions of the encoding device or the decoding device according to the embodiment, or a constituent element that implements a part of the processes of the encoding device or the decoding device according to the embodiment may be combined with or replaced by the constituent elements described in any one of the aspects of the present disclosure, the constituent elements having a part of the functions described in any one of the aspects of the present disclosure, or the constituent elements that implement a part of the processes described in any one of the aspects of the present disclosure.

[0144] (6) In the method implemented by the encoding device or the decoding device according to the embodiment, any one of the plurality of processes included in the method may be replaced with or combined with the processes described in any one of the aspects of the present disclosure, or the processes of any one of the same.

[0145] (7) A part of the plurality of processes included in the method implemented by the encoding device or the decoding device according to the embodiment may also be combined with the processes described in any one of the aspects of the present disclosure.

[0146] (8) The implementation modes of the processes and / or configurations described in each aspect of the present disclosure are not limited to the encoding device or the decoding device according to the embodiment. For example, the processes and / or configurations may also be implemented in a device used for a purpose different from the dynamic image encoding or the dynamic image decoding disclosed in the embodiment. (Embodiment 1) [Encoding Device]

[0147] First, the encoding device according to the present embodiment will be described. FIG. 1 is a block diagram showing the functional configuration of the encoding device 100 according to the present embodiment. The encoding device 100 is a dynamic image encoding device that encodes a dynamic image in block units.

[0148] As shown in FIG. 1, the encoding device 100 is a device that encodes an image in block units, and includes: a splitting unit 102, a subtraction unit 104, a conversion unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse conversion unit 114, an addition unit 116, a block memory 118, a loop filtering unit 120, a frame memory 122, an intra prediction unit 124, an inter prediction unit 126, and a prediction control unit 128.

[0149] The encoding device 100 is implemented, for example, by a general-purpose processor and a memory. At this time, when the software program stored in the memory is executed by the processor, the processor functions as the splitting unit 102, the subtraction unit 104, the conversion unit 106, the quantization unit 108, the entropy encoding unit 110, the inverse quantization unit 112, the inverse conversion unit 114, the addition unit 116, the loop filtering unit 120, the intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128. Also, the encoding device 100 can also be implemented as one or more dedicated electronic circuits corresponding to the splitting unit 102, the subtraction unit 104, the conversion unit 106, the quantization unit 108, the entropy encoding unit 110, the inverse quantization unit 112, the inverse conversion unit 114, the addition unit 116, the loop filtering unit 120, the intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128.

[0150] Hereinafter, after explaining the overall processing flow of the encoding device 100, each component included in the encoding device 100 will be described. [Overall Flow of Encoding Processing]

[0151] FIG. 2 is a flowchart showing an example of the overall encoding processing of the encoding device 100.

[0152] First, the splitting unit 102 of the encoding device 100 splits each picture included in the input image as a moving image into a plurality of blocks of a fixed size (128×128 pixels) (step Sa_1). Next, the splitting unit 102 selects a splitting pattern (also referred to as a block shape) for the block of the fixed size (step Sa_2). That is, the splitting unit 102 further splits the block of the fixed size into a plurality of blocks, and the plurality of blocks constitute the selected splitting pattern. Then, the encoding device 100 performs the processing of steps Sa_3 to Sa_9 for each of the plurality of blocks (i.e., the encoding target blocks).

[0153] That is, a prediction processing unit formed by all or part of the intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128 generates a prediction signal (also referred to as a prediction block) for a block to be encoded (also referred to as a current block) (step Sa_3).

[0154] Next, the subtraction unit 104 generates a difference between the block to be encoded and the prediction block as a prediction residual (also referred to as a difference block) (step Sa_4).

[0155] Next, the conversion unit 106 and the quantization unit 108 perform conversion and quantization on the difference block, thereby generating a plurality of quantization coefficients (step Sa_5). In addition, the block composed of the plurality of quantization coefficients is also referred to as a coefficient block.

[0156] Next, the entropy encoding unit 110 encodes the coefficient block and the prediction parameters for the generation of the prediction signal (specifically, entropy encoding), thereby generating an encoded signal (step Sa_6). In addition, the encoded signal is also referred to as an encoded bitstream, a compressed bitstream, or a stream.

[0157] Next, the inverse quantization unit 112 and the inverse conversion unit 114 perform inverse quantization and inverse conversion on the coefficient block, thereby restoring the plurality of prediction residuals (i.e., difference blocks) (step Sa_7).

[0158] Next, the addition unit 116 adds the restored difference block to the prediction block, thereby reconstructing the current block into a reconstructed image (also referred to as a reconstructed block or a decoded image block) (step Sa_8). Thereby, a reconstructed image is generated.

[0159] Once the reconstructed image is generated, the loop filter unit 120 filters the reconstructed image as needed (step Sa_9).

[0160] Then, the encoding device 100 determines whether the encoding of the entire picture has been completed (step Sa_10). When it is determined that the encoding has not been completed (No in step Sa_10), the processing starting from step Sa_2 is repeatedly executed.

[0161] In addition, in the above example, the encoding device 100 selects one splitting pattern for a block of a fixed size and encodes each block according to the splitting pattern. However, each block can also be encoded according to each of a plurality of splitting patterns. At this time, the encoding device 100 can also evaluate the cost for each of the plurality of splitting patterns. For example, the encoding signal obtained by encoding according to the splitting pattern with the minimum cost is selected as the encoding signal to be finally output.

[0162] Furthermore, the processes of the steps Sa_1 to Sa_10 can also be sequentially performed by the encoding device 100, a plurality of processes among them can be performed in parallel, or the order can be reversed. [Splitting Unit]

[0163] The splitting unit 102 splits each picture included in the input dynamic image into a plurality of blocks and outputs each block to the subtraction unit 104. For example, the splitting unit 102 first splits the picture into blocks of a fixed size (for example, 128×128). These blocks of a fixed size are sometimes referred to as coding tree units (CTUs). Next, the splitting unit 102 splits each block of a fixed size into blocks of a variable size (for example, 64×64 or less) according to, for example, recursive quadtree and / or binary tree block splitting. That is, the splitting unit 102 selects a splitting pattern. These blocks of a variable size are sometimes referred to as coding units (CUs), prediction units (PUs), or transform units (TUs). In addition, in various installation examples, there is no need to distinguish between CUs, PUs, and TUs, and a part or all of the blocks in the picture can also be the processing units of CUs, PUs, and TUs.

[0164] FIG. 3 is a diagram showing an example of block splitting in the present embodiment. In FIG. 3, the solid line represents the block boundary obtained by quadtree block splitting, and the dashed line represents the block boundary obtained by binary tree block splitting.

[0165] Here, the block 10 is a square block of 128×128 pixels (128×128 block). This 128×128 block 10 is first split into 4 square 64×64 blocks (quadtree block splitting).

[0166] The upper left 64×64 block is further vertically divided into two rectangular 32×64 blocks, and the left 32×64 block is further vertically divided into two rectangular 16×64 blocks (binary tree block division). As a result, the upper left 64×64 block is divided into two 16×64 blocks 11, 12, and a 32×64 block 13.

[0167] The upper right 64×64 block is horizontally divided into two rectangular 64×32 blocks 14, 15 (binary tree block division).

[0168] The lower left 64×64 block is divided into four square 32×32 blocks (quadtree block division). Among the four 32×32 blocks, the upper left block and the lower right block are further divided. The upper left 32×32 block is vertically divided into two rectangular 16×32 blocks, and the right 16×32 block is further horizontally divided into two 16×16 blocks (binary tree block division). The lower right 32×32 block is horizontally divided into two 32×16 blocks (binary tree block division). As a result, the lower left 64×64 block is divided into a 16×32 block 16; two 16×16 blocks 17, 18; two 32×32 blocks 19, 20; and two 32×16 blocks 21, 22.

[0169] The lower right 64×64 block 23 is not divided.

[0170] As described above, in FIG. 3, block 10 is divided into 13 variable-size blocks 11 to 23 according to recursive quadtree and binary tree block division. Such division is sometimes referred to as QTBT (quad-tree plus binary tree) division.

[0171] In addition, in FIG. 3, one block has been divided into four or two blocks (quadtree or binary tree block division), but the division is not limited to this. For example, one block can also be divided into three blocks (ternary tree block division). Such division including ternary tree block division is sometimes referred to as MBT (multi type tree) division. [Composition of Picture Slice / Block]

[0172] In order to decode pictures in parallel, pictures are sometimes composed in units of slices or blocks. Pictures composed of units of slices or blocks can also be composed by the dividing unit 102.

[0173] A slice is a basic encoding unit that makes up an image. An image is composed of, for example, more than one slice. Also, a slice is composed of more than one consecutive CTU (Coding Tree Unit).

[0174] Figure 4A is a diagram showing an example of the composition of a slice. For example, an image contains 11×8 CTUs and is divided into 4 slices (slice 1 to slice 4). Slice 1 is composed of 16 CTUs, slice 2 is composed of 21 CTUs, slice 3 is composed of 29 CTUs, and slice 4 is composed of 22 CTUs. Here, each CTU in the image belongs to any one of the slices. The shape of the slice is formed to divide the image in the horizontal direction. The boundary of the slice does not have to be at the edge of the screen, and it can be anywhere as long as it is at the boundary of the CTUs within the screen. The processing order (encoding order or decoding order) of the CTUs in a slice is, for example, the raster scan order. Also, a slice includes header information and encoded data. In the header information, features of the slice such as the address of the first CTU of the slice and the slice type can also be described.

[0175] A block is a unit of a rectangular area that makes up an image. For each block, a number called TileId can also be assigned in the raster scan order.

[0176] Figure 4B is a diagram showing an example of the composition of a block. For example, an image contains 11×8 CTUs and is divided into 4 rectangular area blocks (block 1 to block 4). When using blocks, the processing order of CTUs is changed compared to when not using blocks. When not using blocks, multiple CTUs in the image are processed in the raster scan order. When using blocks, at least one CTU in each of the multiple blocks is processed in the raster scan order. As shown in Figure 4B for example, the processing order of the multiple CTUs contained in block 1 means: from the left end of the first column of block 1 to the right end of the first column of block 1, and then from the left end of the second column of block 1 to the right end of the second column of block 1.

[0177] In addition, when one block includes more than one slice, and when one slice includes more than one block. [Subtraction unit]

[0178] The subtraction unit 104 subtracts the predicted signal (predicted samples input from the prediction control unit 128 described below) from the original signal (original samples) in block units input from the division unit 102 and divided by the division unit 102. That is, the subtraction unit 104 calculates the prediction error (also referred to as the residual) of the encoding target block (hereinafter, referred to as the current block). Then, the subtraction unit 104 outputs the calculated prediction error to the conversion unit 106.

[0179] The original signal is the input signal of the encoding device 100 and is a signal representing the images of each picture constituting the moving image (for example, a luma signal and two chroma signals). In the following content, the signal representing the image is sometimes also referred to as a sample. [Conversion Unit]

[0180] The conversion unit 106 converts the prediction error in the spatial domain into conversion coefficients in the frequency domain and outputs the conversion coefficients to the quantization unit 108. Specifically, the conversion unit 106 performs a predetermined discrete cosine transform (DCT) or discrete sine transform (DST) on the prediction error in the spatial domain, for example.

[0181] In addition, the conversion unit 106 can also adaptively select a conversion type from a plurality of conversion types, and use a transform basis function corresponding to the selected conversion type to convert the prediction error into conversion coefficients. Such a conversion is sometimes referred to as EMT (explicit multiple core transform) or AMT (adaptive multiple transform).

[0182] The plurality of conversion types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII. FIG. 5A is a table showing the conversion basis functions corresponding to each conversion type. In FIG. 5A, N shows the number of input pixels. The selection of the conversion type from among the plurality of conversion types can be based on, for example, the type of prediction (intra prediction and inter prediction), or can also be based on the intra prediction mode.

[0183] Information indicating whether such EMT or AMT is applicable (e.g., referred to as an EMT flag or an AMT flag) and information indicating the selected conversion type are usually signaled at the CU level. Additionally, the signaling of such information need not be limited to the CU level and may also be at other levels (e.g., sequence level, picture level, slice level, tile level, or CTU level).

[0184] Furthermore, the conversion unit 106 may also re-convert the conversion coefficients (conversion results). Such re-conversion is sometimes referred to as AST (adaptive secondary transform) or NSST (non-separable secondary transform). For example, the conversion unit 106 re-converts each sub-block (e.g., 4×4 sub-block) included in the block of conversion coefficients corresponding to the intra-prediction error. Information indicating whether NSST is applicable and information regarding the conversion matrix used in NSST are usually signaled at the CU level. Additionally, the signaling of such information need not be limited to the CU level and may also be at other levels (e.g., sequence level, picture level, slice level, tile level, or CTU level).

[0185] In the conversion unit 106, separable and non-separable conversions may also be applied. Separable conversion refers to a method of performing multiple conversions by separating the dimensions of the input according to direction, and non-separable conversion refers to a method of treating two or more dimensions as one dimension when the input is multi-dimensional and then performing the conversion together.

[0186] For example, as an example of non-separable conversion, when the input is a 4×4 block, the block is regarded as an array having 16 elements, and the array is subjected to a conversion process using a 16×16 conversion matrix.

[0187] Furthermore, in another example of non-separable conversion, a 4×4 input block may also be regarded as an array having 16 elements, and then a composition of multiple Givens rotations (Hypercube Givens Transform) is performed on the array.

[0188] In the conversion in the conversion unit 106, the form of the basis converted into the frequency domain can also be switched according to the region in the CU. For example, there is SVT (Spatially Varying Transform). In SVT, as shown in FIG. 5B, the CU is divided into two equal parts along the horizontal or vertical direction, and the conversion into the frequency domain is performed only on the region of either side. The form of the conversion basis can be set for each region. For example, DST7 and DCT8 are used. In this example, among the two regions in the CU, the conversion is performed only on one side, and the other side is not converted, but the conversion can also be performed on both regions. Also, the splitting method is not limited to two equal parts, and can also be four equal parts, or other more flexible methods such as encoding the information indicating the display split and performing signalization in the same way as the CU split. In addition, SVT is sometimes also called SBT (Sub-block Transform). [Quantization Unit]

[0189] The quantization unit 108 quantizes the conversion coefficients output from the conversion unit 106. Specifically, the quantization unit 108 scans the conversion coefficients of the current block in a predetermined scanning order, and quantizes the conversion coefficients according to the quantization parameter (QP) corresponding to the scanned conversion coefficients. Then, the quantization unit 108 outputs the quantized conversion coefficients (hereinafter referred to as quantization coefficients) of the current block to the entropy encoding unit 110 and the inverse quantization unit 112.

[0190] The predetermined scanning order is the order for quantization / inverse quantization of the conversion coefficients. For example, the predetermined scanning order is defined by ascending power of frequency (order from low frequency to high frequency) or descending power of frequency (order from high frequency to low frequency).

[0191] The quantization parameter (QP) is a parameter that specifies the quantization step (quantization width). For example, if the value of the quantization parameter increases, the quantization step also increases. That is, if the value of the quantization parameter increases, the quantization error also becomes larger.

[0192] Also, in quantization, there is a case where a quantization matrix is used. For example, there are cases where several quantization matrices are used corresponding to frequency conversion sizes such as 4×4 and 8×8, prediction modes such as intra prediction and inter prediction, and pixel components such as luminance and chrominance. In addition, quantization means digitizing the values sampled at a predetermined interval corresponding to a predetermined level, and in this technical field, representations such as approximation, rounding, and scaling are sometimes used.

[0193] In the method using a quantization matrix, there are a method of using a quantization matrix directly set on the encoding device side and a method of using a default matrix. On the encoding device side, by directly setting the quantization matrix, a quantization matrix corresponding to the characteristics of the image can be set. However, at this time, there is a disadvantage that the encoding amount increases due to the encoding of the quantization matrix.

[0194] On the other hand, there is also a method of quantizing without using a quantization matrix, where the coefficients of the high-frequency components and the low-frequency components are the same. In addition, this method is equivalent to using a quantization matrix (flat matrix) with the same value for all coefficients.

[0195] The quantization matrix can be specified by, for example, SPS (Sequence Parameter Set) or PPS (Picture Parameter Set). SPS includes parameters used for a sequence, and PPS includes parameters used for a picture. SPS and PPS are sometimes abbreviated as parameter sets. [Entropy Encoding Unit]

[0196] The entropy encoding unit 110 generates an encoded signal (encoded bitstream) based on the quantized coefficients input from the quantization unit 108. Specifically, the entropy encoding unit 110, for example, binarizes the quantized coefficients, performs arithmetic encoding on the binary signal, and outputs the compressed bitstream or sequence. [Inverse Quantization Unit]

[0197] The inverse quantization unit 112 inverse-quantizes the quantized coefficients input from the quantization unit 108. Specifically, the inverse quantization unit 112 inverse-quantizes the quantized coefficients of the current block in a predetermined scanning order. Then, the inverse quantization unit 112 outputs the inverse-quantized transform coefficients of the current block to the inverse transform unit 114. [Inverse Transform Unit]

[0198] The inverse transform unit 114 inverse-transforms the transform coefficients input from the inverse quantization unit 112, thereby restoring the prediction error (residual). Specifically, the inverse transform unit 114 performs an inverse transform on the transform coefficients corresponding to the transform performed by the transform unit 106, thereby restoring the prediction error of the current block. Then, the inverse transform unit 114 outputs the restored prediction error to the addition unit 116.

[0199] In addition, the restored prediction error usually loses information due to quantization, and thus is inconsistent with the prediction error calculated by the subtraction unit 104. That is, the restored prediction error usually contains quantization error. [Addition Unit]

[0200] The addition unit 116 adds the prediction error input from the inverse transformation unit 114 and the prediction sample input from the prediction control unit 128 to thereby reconstitute the current block. Then, the addition unit 116 outputs the reconstituted block to the block memory 118 and the loop filter unit 120. The reconstituted block is sometimes also referred to as a partial decoded block. [Block Memory]

[0201] The block memory 118 is, for example, used to store the block referred to in intra prediction and is a block within the coding target picture (hereinafter referred to as the current picture). Specifically, the block memory 118 stores the reconstituted block output from the addition unit 116. [Frame Memory]

[0202] The frame memory 122 is, for example, a memory unit used to store the reference picture used in inter prediction and is sometimes also referred to as a frame buffer. Specifically, the frame memory 122 stores the reconstituted block that has been filtered by the loop filter unit 120. [Loop Filter Unit]

[0203] The loop filter unit 120 performs loop filtering on the block reconstituted by the addition unit 116 and outputs the filtered reconstituted block to the frame memory 122. Loop filtering refers to the filter (in-loop filter) used within the coding loop and includes, for example, a deblocking filter (DF or DBF), sample adaptive offset (SAO), and adaptive loop filter (ALF).

[0204] In the ALF, a least squares error filter for removing coding distortion is applied. For example, for each 2×2 sub-block within the current block, one filter selected from a plurality of filters is applied according to the direction and activity of the local gradient.

[0205] Specifically, first, sub-blocks (e.g., 2×2 sub-blocks) are classified into a plurality of categories (e.g., 15 or 25 categories). The classification of sub-blocks is performed according to the direction and activity of the gradient. For example, using the direction value D of the gradient (e.g., 0 to 2 or 0 to 4) and the activity value A of the gradient (e.g., 0 to 4), the classification value C is calculated (e.g., C = 5D + A). Then, according to the classification value C, the sub-blocks are classified into a plurality of categories.

[0206] The direction value D of the gradient is derived, for example, by comparing the gradients in a plurality of directions (e.g., horizontal, vertical, and two diagonal directions). Also, the activity value A of the gradient is derived, for example, by adding the gradients in a plurality of directions and quantifying the addition result.

[0207] Based on the result of such classification, a filter for the sub-block is determined from among a plurality of filters.

[0208] Regarding the shape of the filter used in ALF, for example, a circularly symmetric shape is used. FIGS. 6A to 6C are diagrams showing a plurality of examples of the shape of the filter used in ALF. FIG. 6A shows a 5×5 diamond-shaped filter, FIG. 6B shows a 7×7 diamond-shaped filter, and FIG. 6C shows a 9×9 diamond-shaped filter. The information showing the shape of the filter is signaled at the picture level. In addition, the signaling of the information showing the shape of the filter does not have to be limited to the picture level and can also be other levels (e.g., sequence level, slice level, block level, CTU level, or CU level).

[0209] The on / off of ALF can also be determined, for example, at the picture level or the CU level. For example, for luminance, it can be determined at the CU level whether ALF is applicable, and for chrominance difference, it can be determined at the picture level whether ALF is applicable. The information showing the on / off of ALF is usually signaled at the picture level or the CU level. In addition, the signaling of the information showing the on / off of ALF does not have to be limited to the picture level or the CU level and can also be other levels (e.g., sequence level, slice level, block level, or CTU level).

[0210] The coefficient combinations of a plurality of selectable filters (e.g., filters up to 15 or 25) are usually signaled at the picture level. In addition, the signaling of the coefficient combinations does not have to be limited to the picture level and can also be other levels (e.g., sequence level, slice level, block level, CTU level, CU level, or sub-block level). [Loop Filter Section>Deblocking Filter]

[0211] In the de-blocking filter, the loop filter unit 120 performs filtering processing on the block boundaries of the reconstructed image, thereby reducing the distortion generated at the block boundaries.

[0212] FIG. 7 is a block diagram showing an example of the detailed configuration of the loop filter unit 120 that functions as a de-blocking filter.

[0213] The loop filter unit 120 includes a boundary determination unit 1201, a filtering determination unit 1203, a filtering processing unit 1205, a processing determination unit 1208, a filter characteristic determination unit 1207, and switches 1202, 1204, and 1206.

[0214] The boundary determination unit 1201 determines whether there is a block boundary near the pixel (i.e., the target pixel) for which de-blocking filtering processing is to be performed. Then, the boundary determination unit 1201 outputs the determination result to the switches 1202 and the processing determination unit 1208.

[0215] When it has been determined by the boundary determination unit 1201 that the target pixel exists near the block boundary, the switch 1202 outputs the image before filtering processing to the switch 1204. Conversely, when it has been determined by the boundary determination unit 1201 that the target pixel does not exist near the block boundary, the switch 1202 outputs the image before filtering processing to the switch 1206.

[0216] The filtering determination unit 1203 determines whether to perform de-blocking filtering processing on the target pixel based on the pixel values of at least one peripheral pixel located around the target pixel. Then, the filtering determination unit 1203 outputs the determination result to the switches 1204 and the processing determination unit 1208.

[0217] When it has been determined by the filtering determination unit 1203 that de-blocking filtering processing is to be performed on the target pixel, the switch 1204 outputs the image before filtering processing obtained from the switch 1202 to the filtering processing unit 1205. Conversely, when it has been determined by the filtering determination unit 1203 that de-blocking filtering processing is not to be performed on the target pixel, the switch 1204 outputs the image before filtering processing obtained from the switch 1202 to the switch 1206.

[0218] When an image before filtering is acquired by switches 1202 and 1204, the deblocking filter processing unit 1205 performs deblocking filter processing on the target image, and the deblocking filter processing has filter characteristics determined by the filter characteristic determination unit 1207. Then, the filtering processing unit 1205 outputs the pixels after its filtering processing to the switch 1206.

[0219] The switch 1206 selectively outputs the pixels that have not undergone deblocking filter processing and the pixels that have undergone deblocking filter processing by the filtering processing unit 1205 in response to the control of the processing determination unit 1208.

[0220] The processing determination unit 1208 controls the switch 1206 based on the determination results of the boundary determination unit 1201 and the filtering determination unit 1203 respectively. That is, when it is determined by the boundary determination unit 1201 that the target pixel exists near the block boundary and it is determined by the filtering determination unit 1203 that deblocking filter processing is to be performed on the target pixel, the processing determination unit 1208 outputs the pixels that have undergone deblocking filter processing from the switch 1206. Also, other than the above cases, the processing determination unit 1208 outputs the pixels that have not undergone deblocking filter processing from the switch 1206. By repeatedly outputting such pixels, the filtered image is output from the switch 1206.

[0221] FIG. 8 is a diagram showing an example of a deblocking filter having filter characteristics symmetric with respect to the block boundary.

[0222] In the deblocking filter processing, for example, using the pixel values and quantization parameters, two deblocking filters with different characteristics are selected, that is, either the strong filter or the weak filter is selected. In the strong filter, as shown in FIG. 8, when there are pixels p0 to p2 and pixels q0 to q2 across the block boundary, the pixel values of pixels q0 to q2 are changed to pixel values q'0 to q'2 by the operations shown in the following equations. q’0=(p1 + 2×p0 + 2×q0 + 2×q1 + q2 + 4) / 8 q’1=(p0 + q0 + q1 + q2 + 2) / 4 q’2=(p0 + q0 + q1 + 3×q2 + 2×q3 + 4) / 8

[0223] Further, in the above equations, p0 to p2 and q0 to q2 are the pixel values of pixels p0 to p2 and pixels q0 to q2, respectively. Also, q3 is the pixel value of pixel q3 adjacent to pixel q2 and on the opposite side of the block boundary. Further, in the right side of each of the above equations, the coefficient multiplied by the pixel value of each pixel used in the deblocking filter process is the filter coefficient.

[0224] Furthermore, in the deblocking filter process, in order not to let the pixel value after the operation change by more than the threshold value, a clipping process may also be performed. In this clipping process, the pixel value after the operation performed by the above equations is clipped to "the pixel value before the operation ± 2 × the threshold value" using the threshold value determined from the quantization parameter. Thereby, excessive smoothing can be prevented.

[0225] FIG. 9 is a diagram for explaining the block boundary where the deblocking filter process is performed. FIG. 10 is a diagram showing an example of the Bs value.

[0226] The block boundary where the deblocking filter process is performed is, for example, the boundary of a PU (Prediction Unit) or a TU (Transform Unit) of an 8×8 pixel block as shown in FIG. 9. The deblocking filter process is performed in units of 4 columns or 4 rows. First, as shown in FIG. 10, for block P and block Q shown in FIG. 9, a Bs (Boundary Strength) value is determined.

[0227] According to the Bs value in FIG. 10, even for block boundaries belonging to the same image, it is possible to determine whether to perform the deblocking filter process with different strengths. The deblocking filter process for the chrominance signal is performed when the Bs value is 2. The deblocking filter process for the luminance signal is performed when the Bs value is 1 or more and satisfies a predetermined condition. Further, the determination condition of the Bs value is not limited to that shown in FIG. 10, and may be determined according to other parameters. [Prediction processing unit (intra prediction unit, inter prediction unit, prediction control unit)]

[0228] FIG. 11 is a diagram showing an example of the process performed by the prediction processing unit of the encoding device 100. Further, the prediction processing unit is composed of all or part of the constituent elements of the intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128.

[0229] The prediction processing unit generates a prediction image of the current block (step Sb_1). This prediction image is also referred to as a prediction signal or a prediction block. Additionally, in the prediction signal, there are, for example, an intra prediction signal or an inter prediction signal. Specifically, the prediction processing unit uses a reconstructed image to generate a prediction image of the current block, where the reconstructed image is an image obtained by performing the generation of a prediction block, the generation of a differential block, the generation of a coefficient block, the restoration of the differential block, and the generation of a decoded image block.

[0230] The reconstructed image can, for example, also be an image of a reference picture, or an image of a picture containing the current block, that is, an image of the encoded blocks within the current picture. The encoded blocks within the current picture are, for example, adjacent blocks of the current block.

[0231] Figure 12 is a diagram showing another example of the processing performed by the prediction processing unit of the encoding device 100.

[0232] The prediction processing unit generates a prediction image in a first manner (step Sc_1a), generates a prediction image in a second manner (step Sc_1b), and generates a prediction image in a third manner (step Sc_1c). The first manner, the second manner, and the third manner are different manners for generating a prediction image, and are, for example, an inter prediction manner, an intra prediction manner, and a prediction manner other than these, respectively. In these prediction manners, the above-mentioned reconstructed image can also be used.

[0233] Next, the prediction processing unit selects any one of the plurality of prediction images generated in steps Sc_1a, Sc_1b, and Sc_1c (step Sc_2). The selection of this prediction image, that is, the selection of the manner or mode for obtaining the final prediction image, can also be calculated based on the cost for each generated prediction image and performed according to the cost. Alternatively, the selection of the prediction image can also be performed according to the parameters used in the encoding process. The encoding device 100 can also signalize the information for specifying the selected prediction image, manner, or mode, and make it an encoded signal (also referred to as an encoded bitstream). This information can be, for example, a flag or the like. Thereby, the decoding device generates a prediction image according to this information and in accordance with the manner or mode selected in the encoding device 100. Additionally, in the example shown in Figure 12, the prediction processing unit selects any one prediction image after generating the prediction images in each manner. However, the prediction processing unit can also select a manner or mode according to the parameters used in the above-mentioned encoding process before generating these prediction images, and generate the prediction image in accordance with the manner or mode.

[0234] For example, the first method and the second method may also be intra prediction and inter prediction respectively, and the prediction processing unit selects the final prediction image for the current block from the prediction images generated according to these prediction methods.

[0235] FIG. 13 is a diagram showing another example of the processing performed by the prediction processing unit of the encoding device 100.

[0236] First, the prediction processing unit generates a prediction image by intra prediction (step Sd_1a), and generates a prediction image by inter prediction (step Sd_1b). In addition, the prediction image generated by the intra prediction image is also called an intra prediction image, and the prediction image generated by the inter prediction is also called an inter prediction image.

[0237] Next, the prediction processing unit evaluates each of the intra prediction image and the inter prediction image (step Sd_2). Cost can also be used in this evaluation. That is, the prediction processing unit calculates the cost C of each of the intra prediction image and the inter prediction image. This cost C is calculated by an equation of the R-D optimization model, for example, C = D + λ × R. In this equation, D is the encoding distortion of the prediction image, which is represented by, for example, the sum of the absolute differences between the pixel values of the current block and the pixel values of the prediction image. Also, R is the generation encoding amount of the prediction image. Specifically, it is the encoding amount required for encoding the motion information or the like used to generate the prediction image. Also, λ is, for example, the undetermined multiplier of Lagrange.

[0238] Then, the prediction processing unit selects the prediction image with the minimum calculated cost C from the intra prediction image and the inter prediction image as the final prediction image for the current block (step Sd_3). That is, the prediction method or mode for generating the prediction image of the current block is selected. [Intra Prediction Unit]

[0239] The intra prediction unit 124 performs intra prediction (also called intra-frame prediction) of the current block by referring to the blocks in the current picture stored in the block memory 118 to generate a prediction signal (intra prediction signal). Specifically, the intra prediction unit 124 performs intra prediction by referring to the samples (for example, luminance values, chrominance difference values) of the blocks adjacent to the current block to generate an intra prediction signal, and outputs the intra prediction signal to the prediction control unit 128.

[0240] For example, the intra prediction unit 124 performs intra prediction using one of a plurality of pre-defined intra prediction modes. The plurality of intra prediction modes usually include one or more non-directional prediction modes and a plurality of directional prediction modes.

[0241] One or more non-directional prediction modes, for example, including the Planar prediction mode and the DC prediction mode defined by the H.265 / HEVC standard.

[0242] A plurality of directional prediction modes, for example, including 33-direction prediction modes defined by the H.265 / HEVC standard. In addition, the plurality of directional prediction modes may further include 32-direction prediction modes (a total of 65 directional prediction modes) in addition to the 33 directions. FIG. 14 is a diagram showing 67 intra prediction modes (2 non-directional prediction modes and 65 directional prediction modes) in intra prediction. The solid arrow symbol represents 33 directions defined by the H.265 / HEVC standard, and the dashed arrow symbol represents the additional 32 directions (the 2 non-directional prediction modes are not shown in FIG. 14).

[0243] In various installation examples, in the intra prediction of the chrominance block, the luminance block may also be referred to. That is to say, according to the luminance component of the current block, the chrominance component of the current block can also be predicted. Such intra prediction is sometimes referred to as CCLM (cross-component linear model) prediction. The intra prediction mode of the chrominance block that refers to the luminance block like this (for example, referred to as the CCLM mode) can also be added as one kind of intra prediction mode of the chrominance block.

[0244] The intra prediction unit 124 may also correct the pixel value after intra prediction according to the gradient of the reference pixels in the horizontal / vertical directions. Such intra prediction accompanied by correction is sometimes referred to as PDPC (position dependent intra prediction combination). Information indicating whether PDPC is applied (for example, referred to as the PDPC flag) is usually signaled at the CU level. In addition, the signaling of this information does not have to be limited to the CU level and may also be at other levels (such as the sequence level, the picture level, the slice level, the block level, or the CTU level). [Inter prediction unit]

[0245] The inter prediction unit 126 performs inter prediction (also referred to as inter - picture prediction) of the current block by referring to a reference picture stored in the frame memory 122 and different from the current picture, thereby generating a prediction signal (inter prediction signal). The inter prediction is performed in units of the current block or a current sub - block (e.g., 4×4 block) within the current block. For example, the inter prediction unit 126 performs motion estimation within the reference picture for the current block or the current sub - block to find a reference block or sub - block that most closely matches the current block or the current sub - block. Next, the inter prediction unit 126 obtains motion information (e.g., a motion vector) for compensating for the movement or change of the current block or sub - block from the reference block or sub - block. The inter prediction unit 126 performs motion compensation (or motion prediction) based on the motion information to generate an inter prediction signal for the current block or sub - block. The inter prediction unit 126 outputs the generated inter prediction signal to the prediction control unit 128.

[0246] The motion information used for motion compensation can also be signaled in various forms as an inter prediction signal. For example, the motion vector can also be signaled. In another example, the difference between the motion vector and the motion vector predictor can also be signaled. [Basic process of inter prediction]

[0247] FIG. 15 is a flowchart showing the basic process of inter prediction.

[0248] First, the inter prediction unit 126 generates a prediction image (steps Se_1 to Se_3). Next, the subtraction unit 104 generates the difference between the current block and the prediction image as a prediction residual (step Se_4).

[0249] Here, the inter prediction unit 126 determines the motion vector (MV) of the current block (steps Se_1 and Se_2) and performs motion compensation (step Se_3) in the generation of the predicted image, thereby generating its predicted image. Also, the inter prediction unit 126 selects a candidate motion vector (candidate MV) (step Se_1) and derives the MV (step Se_2) in the determination of the MV, thereby determining its MV. The selection of the candidate MV is performed, for example, by selecting at least one candidate MV from the candidate MV list. Also, in the derivation of the MV, the inter prediction unit 126 may further select at least one candidate MV from at least one candidate MV, thereby determining the at least one selected candidate MV as the MV of the current block. Alternatively, the inter prediction unit 126 may estimate the region of the reference picture indicated by the candidate MV for each of the at least one selected candidate MV to determine the MV of the current block. In addition, the act of estimating the region of the reference picture may also be referred to as motion estimation.

[0250] Also, in the above example, steps Se_1 to Se_3 are performed by the inter prediction unit 126, but the processing such as step Se_1 or step Se_2 may be performed by other components included in the encoding device 100, for example. [Flow of Deriving Motion Vector]

[0251] FIG. 16 is a flowchart showing an example of the flow of deriving a motion vector.

[0252] The inter prediction unit 126 derives the MV of the current block in a mode of encoding the motion information (such as MV). At this time, for example, the motion information is encoded as a prediction parameter and signalized. That is, the encoded motion information is included in the encoded signal (also referred to as the encoded bitstream).

[0253] Alternatively, the inter prediction unit 126 derives the MV in a mode of not encoding the motion information. At this time, the motion information is not included in the encoded signal.

[0254] Here, in the mode of deriving the MV, there are the following normal inter mode, merge mode, FRUC mode, affine mode, etc. Among these modes, the modes that encode the motion information include the normal inter mode, merge mode, and affine mode (specifically, affine inter mode and affine merge mode), etc. In addition, in the motion information, not only the MV but also the predicted motion vector selection information described later can be included. Also, in the mode that does not encode the motion information, there is the FRUC mode, etc. The inter prediction unit 126 selects a mode for deriving the MV of the current block from these plural modes, and uses the selected mode to derive the MV of the current block.

[0255] FIG. 17 is a flowchart showing another example of motion vector derivation.

[0256] The inter prediction unit 126 derives the MV of the current block in a mode that encodes the differential MV. At this time, for example, the differential MV is encoded as a prediction parameter and signaled. That is, the encoded differential MV is included in the encoded signal. The differential MV is the difference between the MV of the current block and its predicted MV.

[0257] Alternatively, the inter prediction unit 126 derives the MV in a mode that does not encode the differential MV. At this time, the encoded differential MV is not included in the encoded signal.

[0258] Here, as described above, in the mode of deriving the MV, there are the following normal inter mode, merge mode, FRUC mode, affine mode, etc. Among these modes, the modes that encode the differential MV include the normal inter mode and affine mode (specifically, affine inter mode), etc. Also, in the mode that does not encode the differential MV, there are the FRUC mode, merge mode, and affine mode (specifically, affine merge mode), etc. The inter prediction unit 126 selects a mode for deriving the MV of the current block from these plural modes, and uses the selected mode to derive the MV of the current block. [Flow of Deriving Motion Vector]

[0259] FIG. 18 is a flowchart showing another example of deriving a motion vector. In the mode of deriving an MV, i.e., the inter prediction mode, there are a plurality of modes, which can be roughly divided into: a mode of encoding a differential MV, and a mode of not encoding a differential motion vector. In the mode of not encoding a differential MV, there are a merge mode, a FRUC mode, and an affine mode (specifically, an affine merge mode). These modes will be described in detail later. Briefly speaking, the merge mode is a mode of deriving the MV of the current block by selecting a motion vector from the encoded blocks around, and the FRUC mode is a mode of estimating between the encoded regions to derive the MV of the current block. Also, the affine mode is a mode of assuming an affine transformation and deriving the motion vectors of the respective plural sub-blocks constituting the current block as the MV of the current block.

[0260] Specifically, when the inter prediction mode information indicates 0 (at Sf_1,0), the inter prediction unit 126 derives a motion vector by the merge mode (Sf_2). Also, when the inter prediction mode information indicates 1 (at Sf_1,1), the inter prediction unit 126 derives a motion vector by the FRUC mode (Sf_3). Also, when the inter prediction mode information indicates 2 (at Sf_1,2), the inter prediction unit 126 derives a motion vector by the affine mode (specifically, the affine merge mode) (Sf_4). Also, when the inter prediction mode information indicates 3 (at Sf_1,3), the inter prediction unit 126 derives a motion vector by the mode of encoding a differential MV (for example, the normal inter mode) (Sf_5). [Derive MV>Normal inter mode]

[0261] The normal inter mode is an inter prediction mode of deriving the MV of the current block by finding a block in the region of the reference picture shown by the candidate MV that is similar to the image of the current block. Also, in this normal inter mode, the differential MV is encoded.

[0262] FIG. 19 is a flowchart showing an example of inter prediction performed according to the normal inter mode.

[0263] The inter prediction unit 126 first obtains a plurality of candidate MVs for the current block based on the information such as the MVs of a plurality of encoded blocks temporally or spatially around the current block (step Sg_1). That is, the inter prediction unit 126 creates a candidate MV list.

[0264] Next, the inter prediction unit 126 extracts N (N is an integer greater than or equal to 2) candidate MVs from among the plurality of candidate MVs obtained in step Sg_1, according to a previously determined priority order, and uses each of them as a prediction motion vector candidate (also referred to as a prediction MV candidate) (step Sg_2). Further, the priority order is previously determined for each of the N candidate MVs.

[0265] Next, the inter prediction unit 126 selects one prediction motion vector candidate from among the N prediction motion vector candidates as the prediction motion vector (also referred to as the prediction MV) for the current block (step Sg_3). At this time, the inter prediction unit 126 encodes the prediction motion vector selection information for identifying the selected prediction motion vector into a bitstream. Further, the bitstream is the above-described encoded signal or encoded bitstream.

[0266] Next, the inter prediction unit 126 refers to the encoded reference picture and derives the MV for the current block (step Sg_4). At this time, the inter prediction unit 126 further encodes the difference value between the derived MV and the prediction motion vector as a differential MV into a bitstream. Further, the encoded reference picture is a picture composed of a plurality of blocks that are reconstructed after encoding.

[0267] Finally, the inter prediction unit 126 performs motion compensation on the current block using the derived MV and the encoded reference picture, thereby generating a predicted picture for the current block (step Sg_5). Further, the predicted picture is the above-described inter prediction signal.

[0268] Further, information on the inter prediction mode (in the above example, the normal inter mode) used for generating the predicted picture included in the encoded signal is encoded as a prediction parameter, for example.

[0269] Further, the candidate MV list may be used in common with the lists used in other modes. Further, the processing related to the candidate MV list may be applied to the processing related to the lists used in other modes. The processing related to the candidate MV list is, for example, extraction or selection of candidate MVs from the candidate MV list, rearrangement of candidate MVs, or deletion of candidate MVs. [MV Derivation > Merge Mode]

[0270] The merge mode is an inter prediction mode in which a candidate MV is selected from the candidate MV list as the MV for the current block, thereby deriving the MV.

[0271] FIG. 20 is a flowchart showing an example of inter prediction performed in the merge mode.

[0272] The inter prediction unit 126 first obtains a plurality of candidate MVs for the current block based on information such as the MVs of a plurality of coded blocks temporally or spatially around the current block (step Sh_1). That is, the inter prediction unit 126 creates a candidate MV list.

[0273] Next, the inter prediction unit 126 selects one candidate MV from the plurality of candidate MVs obtained in step Sh_1, thereby deriving the MV of the current block (step Sh_2). At this time, the inter prediction unit 126 encodes the MV selection information for identifying the selected candidate MV into the stream.

[0274] Finally, the inter prediction unit 126 performs motion compensation on the current block using the derived MV and the coded reference picture, thereby generating a predicted picture of the current block (step Sh_3).

[0275] Also, information indicating the inter prediction mode (merge mode in the above example) used in the generation of the predicted picture included in the coded signal is encoded as a prediction parameter, for example.

[0276] FIG. 21 is a diagram for explaining an example of the motion vector derivation process of the current picture in the merge mode.

[0277] First, a predicted MV list registering candidates of predicted MVs is generated. Examples of candidates of predicted MVs include the following MVs: a spatial neighboring predicted MV, an MV that a plurality of coded blocks spatially adjacent to the target block have; a temporal neighboring predicted MV, an MV that a block near the position of the target block in the projected coded reference picture has; a combined predicted MV, an MV generated by combining the MV values of the spatial neighboring predicted MV and the temporal neighboring predicted MV; and a zero predicted MV, an MV with a value of zero.

[0278] Next, one predicted MV is selected from the plurality of predicted MVs registered in the predicted MV list, and this is determined as the MV of the target block.

[0279] Furthermore, in the variable length coding unit, the signal indicating which predicted MV has been selected, i.e., merge_idx, is described in the stream and encoded.

[0280] Incidentally, taking the predicted MV registered in the predicted MV list described in FIG. 21 as an example, the number of predicted MVs may be different from the number in the figure, or may be a configuration including a part of the types of the predicted MVs in the figure, or may be a configuration with predicted MVs other than the types of the predicted MVs in the figure added.

[0281] The MV of the target block derived by the merge mode may also be used to perform the subsequent DMVR (dynamic motion vector refreshing) process to determine the final MV.

[0282] Incidentally, the candidate for the predicted MV is the above-mentioned candidate MV, and the predicted MV list is the above-mentioned candidate MV list. Also, the candidate MV list may be referred to as the candidate list. Also, merge_idx is the MV selection information. [MV Derivation>FRUC Mode]

[0283] The motion information may not be signaled from the encoding device side, but may be derived on the decoding device side. Incidentally, as described above, the merge mode defined in the H.265 / HEVC standard may also be used. Also, for example, motion estimation may be performed on the decoding device side to derive the motion information. In this case, the pixel value of the current block is not used for motion estimation on the decoding device side.

[0284] Here, the mode of performing motion estimation on the decoding device side will be described. The mode of performing motion estimation on the decoding device side is sometimes referred to as the PMMVD (pattern matched motion vector derivation) mode or the FRUC (frame rate up-conversion) mode.

[0285] An example of FRUC processing is shown in FIG. 22. First, referring to the motion vectors of the encoded blocks that are spatially or temporally adjacent to the current block, a list of a plurality of candidates each having a motion vector (MV) is generated (i.e., a candidate MV list, which may also be common to the merge list) (step Si_1). Next, the best candidate MV is selected from among the plurality of candidate MVs registered in the candidate MV list (step Si_2). For example, the evaluation value of each candidate MV included in the candidate MV list is calculated, and based on the evaluation value, one candidate MV is selected. Next, based on the motion vector of the selected candidate, the motion vector for the current block is derived (step Si_4). Specifically, for example, the motion vector of the selected candidate (the best candidate MV) is directly derived as the motion vector for the current block. Also, for example, pattern matching may be performed in the peripheral region of the position in the reference picture corresponding to the motion vector of the selected candidate, thereby deriving the motion vector for the current block. That is, pattern matching in the reference picture and estimation using the evaluation value may be performed for the region around the best candidate MV, and when an MV with a good evaluation value is obtained, the best candidate MV is updated to the aforementioned MV, and this MV is used as the final MV for the current block. A configuration may also be made in which the process of updating to an MV with a better evaluation value is not performed.

[0286] Finally, the inter prediction unit 126 uses the derived MV and the encoded reference picture to perform motion compensation on the current block, thereby generating a predicted image of the current block (step Si_5).

[0287] When processing is performed in units of sub-blocks, the configuration may be exactly the same.

[0288] The evaluation value can also be calculated by various methods. For example, the reconstructed image of the region in the reference picture corresponding to the motion vector is compared with the reconstructed image of a predetermined region (this region, for example, as shown below, may also be a region of another reference picture or a region of an adjacent block of the current picture). Then, the difference in pixel values of the two reconstructed images can be calculated and used in the evaluation value of the motion vector. In addition, in addition to the difference value, other information may also be used to calculate the evaluation value.

[0289] Next, pattern matching will be described in detail. First, one candidate MV included in the candidate MV list (for example, the merged list) is selected as the starting point for the estimation performed by pattern matching. For pattern matching, the first pattern matching or the second pattern matching is used. The first pattern matching and the second pattern matching are sometimes referred to as bilateral matching and template matching, respectively. [MV Derivation>FRUC>Bilateral Matching]

[0290] In the first pattern matching, pattern matching is performed between two blocks in two different reference pictures and two blocks along the motion trajectory of the current block. Therefore, in the first pattern matching, the area in other reference pictures along the motion trajectory of the current block is used as the predetermined area for calculating the evaluation value of the above-mentioned candidate.

[0291] FIG. 23 is a diagram for explaining an example of the first pattern matching (bilateral matching) between two blocks in two reference pictures along the motion trajectory. As shown in FIG. 23, in the first pattern matching, among the pairs of two blocks in two different reference pictures (Ref0, Ref1) along the motion trajectory of the current block (Cur block), the most matching pair is estimated, and thereby two motion vectors (MV0, MV1) are derived. Specifically, for the current block, the difference between the reconstructed image at the specified position in the first encoded reference picture (Ref0) specified by the candidate MV and the reconstructed image at the specified position in the second encoded reference picture (Ref1) specified by the symmetric MV obtained by scaling the candidate MV by the display time interval is derived, and the evaluation value is calculated using the obtained difference value. Among the plurality of candidate MVs, the candidate MV with the best evaluation value is selected as the final MV.

[0292] Under the assumption of a continuous motion trajectory, the motion vectors (MV0, MV1) indicating the two reference blocks are proportional to the temporal distances (TD0, TD1) between the current picture (Cur Pic) and the two reference pictures (Ref0, Ref1). For example, when the current picture is temporally located between the two reference pictures and the temporal distances from the current picture to the two reference pictures are equal, in the first pattern matching, mirror-symmetric bidirectional motion vectors can be derived. [MV Derivation>FRUC>Template Matching]

[0293] In the second pattern matching (template matching), pattern matching is performed between a template within the current picture (a block adjacent to the current block within the current picture, such as an upper and / or left adjacent block) and a block within the reference picture. Therefore, in the second pattern matching, a block adjacent to the current block within the current picture is used as a predetermined area for calculating the above-mentioned candidate evaluation value.

[0294] FIG. 24 is a diagram for explaining an example of pattern matching (template matching) between a template within the current picture and a block within the reference picture. As shown in FIG. 24, in the second pattern matching, it is estimated within the reference picture (Ref0) which block within the current picture (Cur Pic) most matches the block adjacent to the current block (Cur block), thereby deriving the motion vector of the current block. Specifically, for the current block, the difference between the reconstructed image of the left adjacent and upper adjacent sides or the reconstructed image of either side's encoded area and the reconstructed image at the same position within the encoded reference picture (Ref0) specified by the candidate MV is derived, and using the obtained difference value, the evaluation value is calculated, and among a plurality of candidate MVs, the candidate MV with the best evaluation value is selected as the best candidate MV.

[0295] Information indicating whether such display applies the FRUC mode (such as what is called the FRUC flag) can also be signaled at the CU level. Also, when the FRUC mode is applied (for example, when the FRUC flag is true), information indicating the applicable pattern matching method (the first pattern matching or the second pattern matching) can also be signaled at the CU level. Additionally, the signaling of such information is not limited to the CU level and can also be at other levels (such as the sequence level, picture level, slice level, block level, CTU level, or sub-block level). [MV Derivation > Affine Mode]

[0296] Next, an affine mode for deriving a motion vector in sub-block units based on the motion vectors of a plurality of adjacent blocks is described. This mode is sometimes called the affine motion compensation prediction mode.

[0297] FIG. 25A is a diagram for explaining the derivation of the motion vector of a sub-block unit based on the motion vectors of a plurality of adjacent blocks. In FIG. 25A, the current block contains 16 4×4 sub-blocks. Here, based on the motion vectors of the adjacent blocks, the motion vector v0 of the upper left control point of the current block is derived. Similarly, based on the motion vectors of the adjacent blocks, the motion vector v1 of the upper right control point of the current block is derived. Then, through the following formula (1A), the two motion vectors v0 and v1 are projected to derive the motion vectors (vx, vy) of each sub-block within the current block. (Equation 1)

[0298] Here, x and y respectively represent the horizontal position and vertical position of the sub-block, and w represents a preset weight coefficient.

[0299] The information indicating such an affine mode (e.g., referred to as an affine flag) can also be signaled at the CU level. In addition, the signaling of the information indicating the affine mode is not limited to the CU level and can also be at other levels (e.g., sequence level, picture level, slice level, block level, CTU level, or sub-block level).

[0300] Furthermore, in such an affine mode, several modes with different methods for deriving the motion vectors of the upper left and upper right control points may also be included. For example, in the affine mode, there are two modes: the affine intermediate (also referred to as affine normal intermediate) mode and the affine merge mode. [MV Derivation>Affine Mode]

[0301] FIG. 25B is a diagram for explaining an example of the derivation of the motion vector of a sub-block unit in an affine mode with three control points. In FIG. 25B, the current block contains 16 4×4 sub-blocks. Here, based on the motion vectors of the adjacent blocks, the motion vector v0 of the upper left control point of the current block is derived. Similarly, based on the motion vectors of the adjacent blocks, the motion vector v1 of the upper right control point of the current block is derived, and based on the motion vectors of the adjacent blocks, the motion vector v2 of the lower left control point of the current block is derived. Then, through the following formula (1B), the three motion vectors v0, v1, and v2 are projected to derive the motion vectors (vx, vy) of each sub-block within the current block. (Equation 2)

[0302] Here, x and y respectively represent the horizontal position and the vertical position of the sub-block center, w represents the width of the current block, and h represents the height of the current block.

[0303] Affine modes with different numbers of control points (e.g., 2 and 3) can also be signaled by CU-level switching. Additionally, the number of control points of the affine mode used at the CU level can also be signaled at other levels (e.g., sequence level, picture level, slice level, block level, CTU level, or sub-block level).

[0304] Furthermore, in such an affine mode with 3 control points, several modes with different methods for deriving the motion vectors of the upper-left, upper-right, and lower-left control points can also be included. For example, in the affine mode, there are 2 modes: the affine-inter (also referred to as affine-normal-inter) mode and the affine-combined mode. [MV Derivation > Affine-Combined Mode]

[0305] Figures 26A, 26B, and 26C are conceptual diagrams for explaining the affine-combined mode.

[0306] In the affine-combined mode, as shown in Figure 26A, for example, among the encoded blocks A (left), B (upper), C (upper-right), D (lower-left), and E (upper-left) adjacent to the current block, according to a plurality of motion vectors corresponding to the block encoded by the affine mode, the predicted motion vectors of the respective control points of the current block are calculated. Specifically, the encoded blocks A (left), B (upper), C (upper-right), D (lower-left), and E (upper-left) are checked in this order to identify the first valid block encoded by the affine mode. According to the plurality of motion vectors corresponding to the identified block, the predicted motion vectors of the control points of the current block are calculated.

[0307] For example, as shown in Figure 26B, when the block A adjacent to the left side of the current block is encoded in the affine mode with 2 control points, the motion vectors v3 and v4 projected onto the positions of the upper-left and upper-right corners of the encoded block containing block A are derived. Then, from the derived motion vectors v3 and v4, the predicted motion vector v0 of the upper-left control point of the current block and the predicted motion vector v1 of the upper-right control point of the current block are calculated.

[0308] For example, as shown in FIG. 26C, when block A adjacent to the left side of the current block is encoded in an affine mode with three control points, motion vectors v3, v4, and v5 projected onto the positions of the upper left corner, upper right corner, and lower left corner of the encoded block containing block A are derived. Then, from the derived motion vectors v3, v4, and v5, the predicted motion vectors v0 of the control point at the upper left corner of the current block, the predicted motion vector v1 of the control point at the upper right corner, and the predicted motion vector v2 of the control point at the lower left corner of the current block are calculated.

[0309] In addition, for the derivation of the predicted motion vectors of the respective control points of the current block in step Sj_1 of FIG. 29 described later, this predicted motion vector derivation method can also be used.

[0310] FIG. 27 is a flowchart showing an example of the affine merge mode.

[0311] In the affine merge mode, first, the inter prediction unit 126 derives the predicted MV for each control point of the current block (step Sk_1). The control points are, as shown in FIG. 25A, the points at the upper left corner and upper right corner of the current block, or, as shown in FIG. 25B, the points at the upper left corner, upper right corner, and lower left corner of the current block.

[0312] That is to say, as shown in FIG. 26A, the inter prediction unit 126 checks these blocks in the order of the encoded block A (left), block B (upper), block C (upper right), block D (lower left), and block E (upper left) to identify the first valid block encoded in the affine mode.

[0313] Next, when block A has been identified and block A has two control points, as shown in FIG. 26B, the inter prediction unit 126 calculates the motion vector v0 of the control point at the upper left corner of the current block and the motion vector v1 of the control point at the upper right corner from the motion vectors v3 and v4 at the upper left corner and upper right corner of the encoded block containing block A. For example, the inter prediction unit 126 projects the motion vectors v3 and v4 at the upper left corner and upper right corner of the encoded block onto the current block, thereby calculating the predicted motion vector v0 of the control point at the upper left corner of the current block and the predicted motion vector v1 of the control point at the upper right corner of the current block.

[0314] Alternatively, when block A has been specified and block A has three control points, as shown in FIG. 26C, the inter prediction unit 126 calculates the motion vectors v0, v1, and v2 of the control points at the upper left corner, upper right corner, and lower left corner of the current block from the motion vectors v3, v4, and v5 at the upper left corner, upper right corner, and lower left corner of the encoded block containing block A. For example, the inter prediction unit 126 projects the motion vectors v3, v4, and v5 at the upper left corner, upper right corner, and lower left corner of the encoded block onto the current block, thereby calculating the predicted motion vectors v0, v1, and v2 of the control points at the upper left corner, upper right corner, and lower left corner of the current block.

[0315] Next, the inter prediction unit 126 performs motion compensation on each of the plurality of sub-blocks included in the current block. That is, the inter prediction unit 126 calculates the motion vectors of its plurality of sub-blocks using two predicted motion vectors v0 and v1 and the above formula (1A), or three predicted motion vectors v0, v1, and v2 and the above formula (1B) as affine MVs (step Sk_2). Then, the inter prediction unit 126 performs motion compensation on its sub-blocks using these affine MVs and the encoded picture (step Sk_3). As a result, motion compensation can be performed on the current block to generate a predicted image of the current block. [MV Derivation>Affine Inter Mode]

[0316] FIG. 28A is a diagram for explaining the affine inter mode with two control points.

[0317] In this affine inter mode, as shown in FIG. 28A, the motion vector selected from the motion vectors of the encoded blocks A, B, and C adjacent to the current block is used as the predicted motion vector v0 of the control point at the upper left corner of the current block. Similarly, the motion vector selected from the motion vectors of the encoded blocks D and E adjacent to the current block is used as the predicted motion vector v1 of the control point at the upper right corner of the current block.

[0318] FIG. 28B is a diagram for explaining the affine inter mode with three control points.

[0319] In the affine inter mode, as shown in FIG. 28B, the motion vector selected from the motion vectors of the encoded blocks A, B, and C adjacent to the current block is used as the predicted motion vector v0 of the control point at the upper left corner of the current block. Similarly, the motion vector selected from the motion vectors of the encoded blocks D and E adjacent to the current block is used as the predicted motion vector v1 of the control point at the upper right corner of the current block. Further, the motion vector selected from the motion vectors of the encoded blocks F and G adjacent to the current block is used as the predicted motion vector v2 of the control point at the lower left corner of the current block.

[0320] FIG. 29 is a flowchart showing an example of the affine inter mode.

[0321] In the affine inter mode, first, the inter prediction unit 126 derives the respective predicted MVs (v0, v1) or (v0, v1, v2) of two or three control points of the current block (step Sj_1). As shown in FIG. 25A or FIG. 25B, the control points are the points at the upper left corner, upper right corner, or lower left corner of the current block.

[0322] That is, the inter prediction unit 126 selects the motion vector of any one of the encoded blocks near each control point of the current block shown in FIG. 28A or FIG. 28B, thereby deriving the predicted motion vectors (v0, v1) or (v0, v1, v2) of the control points of the current block. At this time, the inter prediction unit 126 encodes the predicted motion vector selection information for identifying the two selected motion vectors into the stream.

[0323] For example, the inter prediction unit 126 can also use cost evaluation or the like to determine which block's motion vector to select from the encoded blocks adjacent to the current block as the predicted motion vector of the control point, and describe the flag indicating which predicted motion vector has been selected in the bitstream.

[0324] Next, the inter-prediction unit 126 performs motion estimation (Steps Sj_3 and Sj_4) while updating the predicted motion vectors selected or derived in Step Sj_1, respectively. That is, the inter-prediction unit 126 uses the above formula (1A) or formula (1B) to calculate the motion vectors of the respective sub-blocks corresponding to the updated predicted motion vectors as affine MVs (Step Sj_3). Then, the inter-prediction unit 126 performs motion compensation on each sub-block using these affine MVs and the encoded reference picture (Step Sj_4). As a result, the inter-prediction unit 126 determines, for example, the predicted motion vector with the minimum cost as the motion vector of the control point in the motion estimation loop (Step Sj_5). At this time, the inter-prediction unit 126 further encodes the respective difference values between the determined MV and the predicted motion vector as differential MVs into the stream.

[0325] Finally, the inter-prediction unit 126 performs motion compensation on the current block using the determined MV and the encoded reference picture, thereby generating a predicted image of the current block (Step Sj_6). [MV Derivation>Affine Inter Mode]

[0326] When switching between affine modes with different numbers of control points (e.g., 2 and 3) at the CU level, there are cases where the number of control points on the encoded block and the current block is different. FIGS. 30A and 30B are conceptual diagrams for explaining a method of deriving the predicted vectors of the control points when the number of control points on the encoded block and the current block is different.

[0327] For example, as shown in FIG. 30A, when the current block is in an affine mode with three control points at the upper left corner, upper right corner, and lower left corner, and the block A adjacent to the left side of the current block is encoded in an affine mode with two control points, the motion vectors v3 and v4 projected onto the positions of the upper left corner and upper right corner of the encoded block containing block A are derived. Then, from the derived motion vectors v3 and v4, the predicted motion vector v0 of the control point at the upper left corner of the current block and the predicted motion vector v1 of the control point at the upper right corner are calculated. Further, from the derived motion vectors v0 and v1, the predicted motion vector v2 of the control point at the lower left corner is calculated.

[0328] For example, as shown in FIG. 30B, when the current block is in an affine mode with two control points having a top left corner and a top right corner, and the block A adjacent to the left side of the current block is encoded in an affine mode with three control points, motion vectors v3, v4, and v5 projected onto the positions of the top left corner, top right corner, and bottom left corner of the encoded block containing block A are derived. Then, from the derived motion vectors v3, v4, and v5, the predicted motion vector v0 of the control point at the top left corner of the current block and the predicted motion vector v1 of the control point at the top right corner are calculated.

[0329] For the derivation of the predicted motion vector for each control point of the current block in step Sj_1 of FIG. 29, this predicted motion vector derivation method can also be used. [MV Derivation>DMVR]

[0330] FIG. 31A is a diagram showing the relationship between the merge mode and DMVR.

[0331] The inter prediction unit 126 derives the motion vector of the current block in the merge mode (step Sl_1). Next, the inter prediction unit 126 determines whether to perform motion vector estimation, that is, determines whether to perform motion estimation (step Sl_2). Here, when the inter prediction unit 126 determines not to perform motion estimation (No in step Sl_2), the motion vector derived in step Sl_1 is determined as the final motion vector for the current block (step Sl_4). That is, at this time, the motion vector of the current block is determined in the merge mode.

[0332] On the other hand, when it is determined in step Sl_1 that motion estimation is to be performed (Yes in step Sl_2), the inter prediction unit 126 estimates the peripheral area of the reference picture indicated by the motion vector derived in step Sl_1, thereby deriving the final motion vector for the current block (step Sl_3). That is, at this time, the motion vector of the current block is determined by DMVR.

[0333] FIG. 31B is a conceptual diagram for explaining an example of the DMVR process for determining the MV.

[0334] First, in the merging mode for example, the most suitable MVP set in the current block is used as the candidate MV. Next, with reference to the candidate MV (L0), reference pixels are specified from the coded picture in the L0 direction, i.e., the first reference picture (L0). Similarly, with reference to the candidate MV (L1), reference pixels are specified from the coded picture in the L1 direction, i.e., the second reference picture (L1). The average of these reference pixels is taken to generate a template.

[0335] Second, using the aforementioned template, the peripheral areas of the candidate MVs of the first reference picture (L0) and the second reference picture (L1) are respectively estimated, and the MV with the minimum cost is determined as the final MV. Note that the cost value can be calculated, for example, using the difference values between the pixel values of the template and the pixel values of the estimated area, as well as the candidate MV values.

[0336] Note that in the encoding device and the decoding device, the outline of the processing described herein is basically common.

[0337] Even if it is not the processing content described herein, any processing can be used as long as it can estimate the periphery of the candidate MV and derive the final MV. [Motion Compensation>BIO / OBMC]

[0338] In motion compensation, there is a mode of generating a predicted image and correcting the predicted image. Examples of such a mode are BIO and OBMC described later.

[0339] FIG. 32 is a flowchart showing an example of generating a predicted image.

[0340] The inter prediction unit 126 generates a predicted image (step Sm_1) and corrects the predicted image by any of the above-mentioned modes (step Sm_2).

[0341] FIG. 33 is a flowchart showing another example of generating a predicted image.

[0342] The inter prediction unit 126 determines the motion vector of the current block (step Sn_1). Next, the inter prediction unit 126 generates a prediction image (step Sn_2), and determines whether to perform correction processing (step Sn_3). Here, when the inter prediction unit 126 determines that correction processing is to be performed (Yes in step Sn_3), it corrects its prediction image, thereby generating a final prediction image (step Sn_4). On the other hand, when the inter prediction unit 126 determines that correction processing is not to be performed (No in step Sn_3), it does not correct its prediction image and outputs it as the final prediction image (step Sn_5).

[0343] Also, in motion compensation, there is a mode of correcting luminance when generating a prediction image. Such a mode is, for example, LIC described later.

[0344] FIG. 34 is a flowchart showing still another example of the generation of a prediction image.

[0345] The inter prediction unit 126 derives the motion vector of the current block (step So_1). Next, the inter prediction unit 126 determines whether to perform luminance correction processing (step So_2). Here, when the inter prediction unit 126 determines that luminance correction processing is to be performed (Yes in step So_2), it generates a prediction image while performing luminance correction (step So_3). That is, a prediction image is generated by LIC. On the other hand, when the inter prediction unit 126 determines that luminance correction processing is not to be performed (No in step So_2), it does not perform luminance correction, and generates a prediction image by ordinary motion compensation (step So_4). [Motion Compensation>OBMC]

[0346] It is also possible to generate an inter prediction signal not only using the motion information of the current block obtained through motion estimation, but also using the motion information of adjacent blocks. Specifically, it is also possible to perform weighted summation of the prediction signal based on the motion information obtained through motion estimation (within the reference picture) and the prediction signal based on the motion information of adjacent blocks (within the current picture), thereby generating an inter prediction signal in units of sub-blocks within the current block. Such inter prediction (motion compensation) is sometimes referred to as OBMC (overlapped block motion compensation).

[0347] In the OBMC mode, information indicating the size of the sub-blocks for OBMC (e.g., referred to as the OBMC block size) can also be signaled at the slice level. Furthermore, information indicating whether the OBMC mode is applicable (e.g., referred to as the OBMC flag) can also be signaled at the CU level. Additionally, the level at which such information is signaled need not be limited to the slice level and the CU level, and can also be other levels (e.g., picture level, slice level, block level, CTU level, or sub-block level).

[0348] Regarding the OBMC mode, a more specific description will be given. FIGS. 35 and 36 are a flowchart and a conceptual diagram for explaining the outline of the prediction image correction process performed by the OBMC process.

[0349] First, as shown in FIG. 36, using the motion vector (MV) assigned to the processing target (current) block, a prediction image (Pred) obtained by normal motion compensation is acquired. In FIG. 36, the "MV" arrow symbol refers to the reference picture, indicating which picture the current block of the current picture is referring to for the prediction image.

[0350] Next, the motion vector (MV_L) derived for the already encoded left adjacent block is applied (reused) to the encoding target block to obtain a prediction image (Pred_L). The motion vector (MV_L) is represented by the "MV_L" arrow symbol indicating the reference picture from the current block. Then, the two prediction images Pred and Pred_L are superimposed to perform the first correction of the prediction image. This has the effect of blending the boundaries between adjacent blocks.

[0351] Similarly, the motion vector (MV_U) derived for the already encoded upper adjacent block is applied (reused) to the encoding target block to obtain a prediction image (Pred_U). The motion vector (MV_U) is represented by the "MV_U" arrow symbol indicating the reference picture from the current block. Then, the prediction image Pred_U is superimposed on the prediction image that has undergone the first correction (e.g., Pred and Pred_L) to perform the second correction of the prediction image. This has the effect of blending the boundaries between adjacent blocks. The prediction image after the second correction is the final prediction image of the current block with the boundaries (smoothed) with adjacent blocks blended.

[0352] Additionally, the above example illustrates a two-stage correction method using the left and upper adjacent blocks, but the correction method can also be a three-stage or more-stage correction method that uses even the right and / or lower adjacent blocks.

[0353] In addition, the area for superposition can also be an area that is only a part near the block boundary, rather than the pixel area of the entire block.

[0354] In addition, although it is described here as the prediction image correction process of OBMC that superimposes and adds the prediction images Pred_L and Pred_U from one reference picture to obtain one prediction image Pred. However, when correcting the prediction image based on a plurality of reference pictures, the same process also applies to each of the plurality of reference pictures. In such a case, image correction of OBMC based on a plurality of reference pictures is performed, thereby obtaining corrected prediction images from each reference picture, and then the obtained plurality of corrected prediction images are further superimposed to obtain the final prediction image.

[0355] In addition, in OBMC, the unit of the target block can also be the prediction block unit or the sub-block unit obtained by further dividing the prediction block.

[0356] As a method for determining whether to apply the OBMC process, for example, there is a method of using the obmc_flag that indicates whether to apply the OBMC process. Taking a specific example, the encoding device can also determine whether the target block belongs to a region with complex movement. When it belongs to a region with complex movement, the encoding device sets the value to 1 as the obmc_flag and applies the OBMC process for encoding. When it does not belong to a region with complex movement, the value is set to 0 as the obmc_flag and the block is encoded without applying the OBMC process. On the other hand, in the decoding device, the obmc_flag described in the stream (such as a compressed sequence) is decoded, and whether to apply the OBMC process is switched according to this value for decoding.

[0357] In the example above, the inter prediction unit 126 generates one rectangular prediction image for the rectangular current block. However, the inter prediction unit 126 can also generate a plurality of prediction images with shapes different from the rectangle for the rectangular current block, and combine these plurality of prediction images to generate the final rectangular prediction image. Shapes different from the rectangle can be, for example, triangles.

[0358] Figure 37 is a diagram for explaining the generation of two triangular prediction images.

[0359] The inter prediction unit 126 performs motion compensation on the first triangular partition within the current block using the first MV of the first partition, thereby generating a triangular prediction image. Similarly, the inter prediction unit 126 performs motion compensation on the second triangular partition within the current block using the second MV of the second partition, thereby generating a triangular prediction image. Then, the inter prediction unit 126 combines these prediction images to generate a rectangular prediction image identical to the current block.

[0360] In addition, in the example shown in FIG. 37, the first partition and the second partition are each a triangle, but they may also be trapezoids, or they may be different shapes respectively. Furthermore, in the example shown in FIG. 37, the current block is composed of two partitions, but it may also be composed of three or more partitions.

[0361] Also, the first partition and the second partition may overlap. That is, the first partition and the second partition may contain the same pixel area. In this case, the prediction image in the first partition and the prediction image in the second partition may also be used to generate the prediction image of the current block.

[0362] Also, in this example, an example is shown in which prediction images are generated for both partitions by inter prediction, but prediction images may also be generated for at least one partition by intra prediction. [Motion Compensation>BIO]

[0363] Next, a method for deriving a motion vector will be described. First, a mode for deriving a motion vector based on a model assuming uniform linear motion will be described. This mode is sometimes referred to as the BIO (bi - directional optical flow) mode.

[0364] FIG. 38 is a diagram for explaining a model assuming uniform linear motion. In FIG. 38, (v x, v y) represents the velocity vector, and τ 0 and τ 1 respectively represent the temporal distances between the current picture (Cur Pic) and two reference pictures (Ref 0, Ref 1). (MVx 0, MVy 0) represents the motion vector corresponding to the reference picture Ref 0, and (MVx 1, MVy 1) represents the motion vector corresponding to the reference picture Ref 1.

[0365] At this time, under the assumption of uniform linear motion, the velocity vectors (vx, vy) are represented by (MVx0, MVy0) and (MVx1, MVy1) as (vxτ0, vyτ0) and (-vxτ1, -vyτ1), respectively, and the following optical flow equation (2) holds. (Equation 3)

[0366] Here, I(k) represents the luminance value of the reference image k (k = 0, 1) after motion compensation. This optical flow equation shows that the sum of (i) the temporal differential of the luminance value, (ii) the product of the horizontal velocity and the horizontal component of the spatial gradient of the reference image, and (iii) the product of the vertical velocity and the vertical component of the spatial gradient of the reference image is equal to zero. Also, based on the combination of this optical flow equation and Hermite interpolation, the motion vectors in block units obtained from a merge list or the like can be corrected in pixel units.

[0367] In addition, a method different from the derivation of motion vectors based on a model assuming uniform linear motion can be used to derive motion vectors on the decoder side. For example, motion vectors can be derived in sub-block units based on the motion vectors of a plurality of adjacent blocks. [Motion Compensation > LIC]

[0368] Next, an example of a mode for generating a predicted image (prediction) using LIC (local illumination compensation) processing will be described.

[0369] FIG. 39 is a diagram for explaining an example of a method for generating a predicted image using luminance correction processing performed by LIC processing.

[0370] First, an MV is derived from the encoded reference picture, and the reference image corresponding to the current block is obtained.

[0371] Next, for the current block, information indicating how the display luminance changes between the reference picture and the current picture is extracted. This extraction is performed based on the luminance pixel values of the encoded left adjacent reference area (peripheral reference area) and the encoded upper adjacent reference area (peripheral reference area) in the current picture, and the luminance pixel values at the same positions in the reference picture specified by the derived MV. Then, using the information indicating how the display luminance value changes, a luminance correction parameter is calculated.

[0372] Perform a brightness correction process using the aforementioned brightness correction parameters on the reference image within the reference picture specified by the MV, thereby generating a predicted image for the current block.

[0373] In addition, the shape of the aforementioned peripheral reference area in FIG. 39 is merely an example, and other shapes may also be used.

[0374] Furthermore, the process of generating a predicted image from one reference picture has been described herein, but the same applies to the case of generating a predicted image from a plurality of reference pictures. It is also possible to first perform a brightness correction process on the reference images obtained from each reference picture in the same manner and then generate a predicted image.

[0375] Regarding the method of determining whether to apply the LIC process, for example, there is a method of using the lic_flag that indicates whether the LIC process is applicable. As a specific example, in an encoding device, it is determined whether the current block belongs to an area where a brightness change has occurred. If it belongs to an area where a brightness change has occurred, the value of the lic_flag is set to 1, and the LIC process is applied for encoding. If it does not belong to an area where a brightness change has occurred, the value of the lic_flag is set to 0, and the LIC process is not applied for encoding. On the other hand, in a decoding device, the lic_flag described in the stream can also be decoded, and decoding is performed by switching whether to apply the LIC process according to its value.

[0376] Regarding another method of determining whether to apply the LIC process, for example, there is also a method of determining according to whether the LIC process has been applied to the peripheral blocks. As a specific example, when the current block is in the merge mode, it is determined whether the LIC process has been applied to the peripheral encoded blocks selected during the derivation of the MV in the merge mode process for encoding. According to the result, whether to apply the LIC process for encoding is switched. In addition, in the case of this example, the same process also applies to the processing on the decoding device side.

[0377] FIG. 39 has been used to illustrate the LIC process (brightness correction process), and the following provides a detailed description.

[0378] First, the inter prediction unit 126 derives a motion vector from the encoded picture, i.e., the reference picture, for obtaining a reference image corresponding to the encoding target block.

[0379] Next, the inter prediction unit 126 extracts information on how the display luminance values change between the reference picture and the picture to be coded for the block to be coded, using the luminance pixel values of the coded peripheral reference areas adjacent to the left and above, and the luminance pixel values at the same positions in the reference picture specified by the motion vector, to calculate the luminance correction parameter. For example, let the luminance pixel value of a certain pixel in the peripheral reference area within the picture to be coded be p0, and the luminance pixel value of the pixel in the peripheral reference area within the reference picture at the same position as this pixel be p1. The inter prediction unit 126 calculates the coefficients A and B that optimize A×p1 + B = p0 for a plurality of pixels in the peripheral reference area, as the luminance correction parameter.

[0380] Next, the inter prediction unit 126 performs a luminance correction process on the reference image in the reference picture specified by the motion vector using the luminance correction parameter, thereby generating a prediction image for the block to be coded. For example, let the luminance pixel value in the reference image be p2, and the luminance pixel value of the prediction image after the luminance correction process be p3. The inter prediction unit 126 calculates A×p2 + B = p3 for each pixel in the reference image, thereby generating the prediction image after the luminance correction process.

[0381] In addition, the shape of the peripheral reference area in FIG. 39 is only an example, and other shapes may also be used. Also, a part of the peripheral reference area shown in FIG. 39 may be used. For example, a region containing pixels obtained by subtracting a predetermined number from each of the upper adjacent pixel and the left adjacent pixel may be used as the peripheral reference area. Also, the peripheral reference area is not limited to the area adjacent to the block to be coded, and may be an area not adjacent to the block to be coded. Also, in the example shown in FIG. 39, the peripheral reference area in the reference picture is the area specified by the motion vector of the picture to be coded from the peripheral reference area in the picture to be coded, but it may also be the area specified by other motion vectors. For example, the other motion vector may also be the motion vector of the peripheral reference area in the picture to be coded.

[0382] In addition, the operation of the coding device 100 has been described here, but the operation of the decoding device 200 is the same.

[0383] In addition, the LIC process can be applied not only to luminance but also to color difference. In this case, correction parameters may be individually derived for Y, Cb, and Cr, respectively, or a common correction parameter may be used for any one of them.

[0384] Also, the LIC process can also be applied in sub-block units. For example, the correction parameters are derived using the peripheral reference area of the current sub-block and the peripheral reference area of the reference sub-block in the reference picture specified by the MV of the current sub-block. [Prediction control unit]

[0385] The prediction control unit 128 selects either an intra prediction signal (the signal output from the intra prediction unit 124) or an inter prediction signal (the signal output from the inter prediction unit 126), and outputs the selected signal as a prediction signal to the subtraction unit 104 and the addition unit 116.

[0386] As shown in FIG. 1, in various installation examples, the prediction control unit 128 can also output the prediction parameters input to the entropy encoding unit 110. The entropy encoding unit 110 can also generate an encoded bitstream (or sequence) based on the prediction parameters input from the prediction control unit 128 and the quantization coefficient input from the quantization unit 108. The prediction parameters can also be used in the decoding device. The decoding device can also receive the encoded bitstream, perform decoding, and perform the same processing as the prediction processing performed in the intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128. The prediction parameters can also include a selection prediction signal (such as a motion vector, a prediction type, or a prediction mode used in the intra prediction unit 124 or the inter prediction unit 126), or an arbitrary index, flag, or value according to the prediction processing performed in the intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128, or indicating the prediction processing. [Installation example of the encoding device]

[0387] FIG. 40 is a block diagram showing an installation example of the encoding device 100. The encoding device 100 includes a processor a1 and a memory a2. For example, a plurality of components of the encoding device 100 shown in FIG. 1 are installed by the processor a1 and the memory a2 shown in FIG. 40.

[0388] The processor a1 is a circuit that performs information processing and is a circuit that can access the memory a2. For example, the processor a1 is a dedicated or general-purpose electronic circuit for encoding dynamic images. The processor a1 can also be a processor such as a CPU. Also, the processor a1 can also be an aggregate of a plurality of electronic circuits. Also, for example, the processor a1 can also implement the functions of a plurality of components of the encoding device 100 shown in FIG. 1 and the like, except for the components used to store information.

[0389] The memory a2 is a general-purpose or dedicated memory for storing information that enables the processor a1 to encode dynamic images. The memory a2 can also be an electronic circuit and can be connected to the processor a1. Additionally, the memory a2 can be included in the processor a1. Moreover, the memory a2 can be a collection of multiple electronic circuits. Furthermore, the memory a2 can be a magnetic disk or an optical disk and can be represented as a storage or a recording medium, etc. Also, the memory a2 can be a non-volatile memory or a volatile memory.

[0390] For example, the memory a2 can store the dynamic image to be encoded and can also store the bit stream corresponding to the encoded dynamic image. Also, the memory a2 can store the program that enables the processor a1 to encode the dynamic image.

[0391] Also, for example, the memory a2 can perform the function of the component for storing information among the multiple components of the encoding device 100 shown in FIG. 1 etc. Specifically, the memory a2 can perform the functions of the block memory 118 and the frame memory 122 shown in FIG. 1 etc. More specifically, the memory a2 can store the reconstructed blocks and the reconstructed pictures, etc.

[0392] In addition, in the encoding device 100, not all of the multiple components shown in FIG. 1 etc. may be installed, and not all of the above-mentioned multiple processes may be performed. A part of the multiple components shown in FIG. 1 etc. may be included in other devices, and a part of the above-mentioned multiple processes may be executed by other devices. [Decoding Device]

[0393] Next, a decoding device that can decode the encoded signal (encoded bit stream) output from the above-mentioned encoding device 100 will be described. FIG. 41 is a block diagram showing the functional configuration of the decoding device 200 of the present embodiment. The decoding device 200 is a dynamic image decoding device that decodes dynamic images in block units.

[0394] As shown in FIG. 41, the decoding device 200 includes an entropy decoding unit 202, an inverse quantization unit 204, an inverse transformation unit 206, an addition unit 208, a block memory 210, a loop filter 212, a frame memory 214, an intra prediction unit 216, an inter prediction unit 218, and a prediction control unit 220.

[0395] The decoding device 200 can be implemented, for example, by a general-purpose processor and a memory. At this time, when the software program stored in the memory is executed by the processor, the processor operates as an entropy decoding unit 202, an inverse quantization unit 204, an inverse transform unit 206, an addition unit 208, a loop filtering unit 212, an intra prediction unit 216, an inter prediction unit 218, and a prediction control unit 220. Also, the decoding device 200 can be implemented as one or more dedicated electronic circuits corresponding to the entropy decoding unit 202, the inverse quantization unit 204, the inverse transform unit 206, the addition unit 208, the loop filtering unit 212, the intra prediction unit 216, the inter prediction unit 218, and the prediction control unit 220.

[0396] Hereinafter, after explaining the overall processing flow of the decoding device 200, each component included in the decoding device 200 will be described. [Overall Flow of Decoding Processing]

[0397] FIG. 42 is a flowchart showing an example of the overall decoding process of the decoding device 200.

[0398] First, the entropy decoding unit 202 of the decoding device 200 specifies a segmentation pattern (step Sp_1) of a block of a fixed size (128×128 pixels). This segmentation pattern is the segmentation pattern selected by the encoding device 100. Then, the decoding device 200 performs the processing of steps Sp_2 to Sp_6 for each of the plurality of blocks constituting the segmentation pattern.

[0399] That is, the entropy decoding unit 202 decodes (specifically, entropy decodes) the encoded quantization coefficients and prediction parameters of the decoding target block (also referred to as the current block) (step Sp_2).

[0400] Next, the inverse quantization unit 204 and the inverse transform unit 206 perform inverse quantization and inverse transform on the plurality of quantization coefficients, thereby restoring the plurality of prediction residuals (i.e., differential blocks) (step Sp_3).

[0401] Next, a prediction processing unit composed of all or a part of the intra prediction unit 216, the inter prediction unit 218, and the prediction control unit 220 generates a prediction signal (also referred to as a prediction block) of the current block (step Sp_4).

[0402] Next, the addition unit 208 adds the prediction block to the differential block, thereby reconstructing the current block into a reconstructed image (also referred to as a decoded image block) (step Sp_5).

[0403] Next, a reconstructed image is generated, and the loop filter unit 212 filters the reconstructed image (step Sp_6).

[0404] Then, the decoding device 200 determines whether the decoding of the entire picture has been completed (step Sp_7). When it is determined that the decoding has not been completed (No in step Sp_7), the processing starting from step Sp_1 is repeatedly executed.

[0405] In addition, the processing of steps Sp_1 to Sp_7 can also be sequentially performed by the decoding device 200, a plurality of processes among them can be performed in parallel, or the order can be reversed. [Entropy decoding unit]

[0406] The entropy decoding unit 202 performs entropy decoding on the encoded bitstream. Specifically, the entropy decoding unit 202, for example, performs arithmetic decoding on the encoded bitstream into a binary signal. Next, the entropy decoding unit 202 de-binarizes the binary signal. The entropy decoding unit 202 outputs the quantization coefficients to the inverse quantization unit 204 in block units. The entropy decoding unit 202 can also output the prediction parameters included in the encoded bitstream (refer to FIG. 1) to the intra prediction unit 216, the inter prediction unit 218, and the prediction control unit 220. The intra prediction unit 216, the inter prediction unit 218, and the prediction control unit 220 can perform the same prediction processing as that performed by the intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128 on the encoding device side. [Inverse quantization unit]

[0407] The inverse quantization unit 204 inverse quantizes the quantization coefficients of the decoding target block (hereinafter referred to as the current block) of the input from the entropy decoding unit 202. Specifically, the inverse quantization unit 204 inverse quantizes each of the quantization coefficients of the current block according to the quantization parameter corresponding to the quantization coefficient. Then, the inverse quantization unit 204 outputs the inverse quantized quantization coefficients (i.e., transform coefficients) of the current block to the inverse transform unit 206. [Inverse transform unit]

[0408] The inverse transform unit 206 inverse transforms the transform coefficients of the input from the inverse quantization unit 204, thereby restoring the prediction error.

[0409] For example, when the information decoded from the encoded bitstream indicates that EMT or AMT is applicable (e.g., the AMT flag is true), the inverse transform unit 206 inverse-transforms the transform coefficients of the current block according to the information on the transform type decoded for display.

[0410] Also, for example, when the information decoded from the encoded bitstream indicates that NSST is applicable, the inverse transform unit 206 applies an inverse transform to the transform coefficients. [Addition unit]

[0411] The addition unit 208 adds the prediction error of the input from the inverse converter 206 and the prediction sample of the input from the prediction control unit 220, thereby reconstructing the current block. Then, the addition unit 208 outputs the reconstructed block to the block memory 210 and the loop filter unit 212. [Block memory]

[0412] The block memory 210 is a memory unit for storing the blocks referred to in intra prediction and being blocks within the decoding target picture (hereinafter referred to as the current picture). Specifically, the block memory 210 stores the reconstructed block output from the addition unit 208. [Loop filter unit]

[0413] The loop filter unit 212 performs loop filtering on the block reconstructed by the addition unit 208, and outputs the filtered reconstructed block to the frame memory 214, the display device, etc.

[0414] When the information on the ON / OFF of ALF decoded from the encoded bitstream indicates that ALF is ON, one filter is selected from a plurality of filters according to the direction and activity of a part of the gradients, and the selected filter is applied to the reconstructed block. [Frame memory]

[0415] The frame memory 214 is a memory unit for storing the reference pictures used in inter prediction, and is sometimes referred to as a frame buffer. Specifically, the frame memory 214 stores the reconstructed block filtered by the loop filter unit 212. [Prediction processing unit (intra prediction unit · inter prediction unit · prediction control unit)]

[0416] FIG. 43 is a diagram showing an example of the processing performed by the prediction processing unit of the decoding device 200. Further, the prediction processing unit is composed of all or some of the constituent elements of the intra prediction unit 216, the inter prediction unit 218, and the prediction control unit 220.

[0417] The prediction processing unit generates a prediction image of the current block (step Sq_1). This prediction image is also referred to as a prediction signal or a prediction block. Further, in the prediction signal, for example, there are an intra prediction signal or an inter prediction signal. Specifically, the prediction processing unit uses the reconstructed image obtained by generating a prediction block, a difference block, a coefficient block, restoring the difference block, and generating a decoded image block to generate a prediction image of the current block.

[0418] The reconstructed image can be, for example, an image of a reference picture, or an image of a decoded block in the current picture containing the current block, that is, the current picture. The decoded block in the current picture is, for example, an adjacent block of the current block.

[0419] FIG. 44 is a diagram showing another example of the processing performed by the prediction processing unit of the decoding device 200.

[0420] The prediction processing unit determines the method or mode for generating the prediction image (step Sr_1). For example, this method or mode can be determined based on prediction parameters, etc.

[0421] When the prediction processing unit determines that the first method is used as the mode for generating the prediction image, it generates the prediction image according to the first method (step Sr_2a). Further, when the prediction processing unit determines that the second method is used as the mode for generating the prediction image, it generates the prediction image according to the second method (step Sr_2b). Further, when the prediction processing unit determines that the third method is used as the mode for generating the prediction image, it generates the prediction image according to the third method (step Sr_2c).

[0422] The first method, the second method, and the third method are different methods for generating the prediction image, and can be, for example, an inter prediction method, an intra prediction method, and a prediction method other than these, respectively. Among these prediction methods, the above-mentioned reconstructed image can be used. [Intra Prediction Unit]

[0423] The intra prediction unit 216 performs intra prediction with reference to the blocks within the current picture stored in the reference block memory 210 according to the intra prediction mode decoded from the encoded bitstream, so as to generate a prediction signal (intra prediction signal). Specifically, the intra prediction unit 216 performs intra prediction with reference to the samples (such as luminance values, chrominance differences) of the blocks adjacent to the current block, thereby generating an intra prediction signal, and outputs the intra prediction signal to the prediction control unit 220.

[0424] In addition, when the intra prediction mode of the reference luminance block has been selected in the intra prediction of the chrominance difference block, the intra prediction unit 216 can also predict the chrominance difference component of the current block according to the luminance component of the current block.

[0425] Furthermore, when the information decoded from the encoded bitstream indicates the application of PDPC, the intra prediction unit 216 corrects the pixel value after intra prediction according to the gradients of the reference pixels in the horizontal / vertical directions. [Inter prediction unit]

[0426] The inter prediction unit 218 predicts the current block with reference to the reference pictures stored in the frame memory 214. The prediction is performed in units of the current block or sub-blocks (such as 4×4 blocks) within the current block. For example, the inter prediction unit 218 performs motion compensation using the motion information (such as motion vectors) decoded from the encoded bitstream (for example, the prediction parameters output from the entropy decoding unit 202), thereby generating an inter prediction signal for the current block or sub-block, and outputs the inter prediction signal to the prediction control unit 220.

[0427] In addition, when the information decoded from the encoded bitstream indicates the application of the OBMC mode, the inter prediction unit 218 generates an inter prediction signal by using not only the motion information of the current block obtained through motion estimation but also the motion information of the adjacent blocks.

[0428] Furthermore, when the information decoded from the encoded bitstream indicates the application of the FRUC mode, the inter prediction unit 218 performs motion estimation according to the pattern matching method (bidirectional matching or template matching) decoded from the encoded stream, thereby deriving the motion information. Then, the inter prediction unit 218 uses the derived motion information to perform motion compensation (prediction).

[0429] Also, the inter prediction unit 218 derives a motion vector based on a model assuming uniform linear motion when the BIO mode is applied. Also, when the information decoded from the coded bitstream indicates that the affine motion compensation prediction mode is applied, the inter prediction unit 218 derives a motion vector in units of sub-blocks based on the motion vectors of a plurality of adjacent blocks. [MV Derivation>Normal Inter Mode]

[0430] When the information decoded from the coded bitstream indicates that the normal inter mode is applied, the inter prediction unit 218 derives an MV based on the information decoded from the coded stream, and performs motion compensation (prediction) using the MV.

[0431] FIG. 45 is a flowchart showing an example of inter prediction performed in the normal inter mode in the decoding device 200.

[0432] The inter prediction unit 218 of the decoding device 200 performs motion compensation on each block. At this time, the inter prediction unit 218 first obtains a plurality of candidate MVs (step Ss_1) for the current block based on information such as the MVs of a plurality of decoded blocks temporally or spatially surrounding the current block. That is, the inter prediction unit 218 creates a candidate MV list.

[0433] Next, the inter prediction unit 218 extracts N (N is an integer of 2 or more) candidate MVs from the plurality of candidate MVs obtained in step Ss_1 as prediction motion vector candidates (also referred to as prediction MV candidates) in accordance with a previously determined priority order. In addition, the priority order has been determined in advance for each of the N prediction MV candidates.

[0434] Next, the inter prediction unit 218 decodes the prediction motion vector selection information from the input stream (i.e., the coded bitstream), and uses the decoded prediction motion vector selection information to select one prediction MV candidate from the N prediction MV candidates as the prediction motion vector (also referred to as the prediction MV) of the current block (step Ss_3).

[0435] Next, the inter prediction unit 218 decodes the differential MV from the input stream, and adds the decoded differential MV, i.e., the difference value, and the selected prediction motion vector to derive the MV of the current block (step Ss_4).

[0436] Finally, the inter prediction unit 218 uses the MV it has derived and the decoded reference pictures to perform motion compensation on the current block, thereby generating a predicted picture of the current block (step Ss_5). [Prediction control unit]

[0437] The prediction control unit 220 selects either the intra prediction signal or the inter prediction signal, and outputs the selected signal as the prediction signal to the adder unit 208. Overall, the configurations, functions, and processes of the prediction control unit 220, the intra prediction unit 216, and the inter prediction unit 218 on the decoder side can also correspond to the configurations, functions, and processes of the prediction control unit 128, the intra prediction unit 124, and the inter prediction unit 126 on the encoder side. [Installation example of decoder]

[0438] FIG. 46 is a block diagram showing an installation example of the decoder 200. The decoder 200 includes a processor b1 and a memory b2. For example, a plurality of components of the decoder 200 shown in FIG. 41 are installed by the processor b1 and the memory b2 shown in FIG. 46.

[0439] The processor b1 is a circuit that performs information processing and is a circuit that can access the memory b2. For example, the processor b1 is a dedicated or general-purpose electronic circuit that decodes an encoded dynamic image (i.e., an encoded bitstream). The processor b1 can also be a processor such as a CPU. Also, the processor b1 can be an aggregate of a plurality of electronic circuits. Also, for example, the processor b1 can also implement the functions of a plurality of components of the decoder 200 shown in FIG. 41 and the like, except for the components used to store information.

[0440] The memory b2 is a general-purpose or dedicated memory that stores information for the processor b1 to decode the encoded bitstream. The memory b2 can also be an electronic circuit and can be connected to the processor b1. Also, the memory b2 can be included in the processor b1. Also, the memory b2 can be an aggregate of a plurality of electronic circuits. Also, the memory b2 can be a magnetic disk or an optical disk, and can also be represented as a storage or a recording medium, etc. Also, the memory b2 can be a non-volatile memory or a volatile memory.

[0441] For example, the memory b2 can also store a dynamic image or an encoded bitstream. Also, the memory b2 can store a program for the processor b1 to decode the encoded bitstream.

[0442] Also, for example, the memory b2 can also function as a component for storing information among a plurality of components of the decoding device 200 shown in FIG. 41 or the like. Specifically, the memory b2 can also function as the block memory 210 and the frame memory 214 shown in FIG. 41 or the like. More specifically, the memory b2 can store the reconstructed blocks, the reconstructed pictures, and the like.

[0443] In addition, in the decoding device 200, not all of the plurality of components shown in FIG. 41 or the like need to be installed, and not all of the plurality of processes described above need to be performed. A part of the plurality of components shown in FIG. 41 or the like may be included in other devices, and a part of the plurality of processes described above may be executed by other devices. [Definitions of Terms]

[0444] For example, each term may be defined as follows.

[0445] A picture is an array of a plurality of measurement samples in a monochromatic format, or an array of a plurality of luminance samples and two corresponding arrays of a plurality of chrominance samples in a color format of 4:2:0, 4:2:2, or 4:4:4. A picture may also be a frame or a field.

[0446] A frame is a composition of a top field generated by a plurality of sample rows 0, 2, 4,... and a bottom field generated by a plurality of sample rows 1, 3, 5,....

[0447] A slice is an integer number of coding tree units including one independent slice segment and all subsequent dependent slice segments (if any) before the next independent slice segment (if any) in the same access unit.

[0448] A block is a rectangular area of a plurality of coding tree blocks within a specific block row and a specific block column in a picture. Although a loop filter spanning the edge of the block may still be applied, a block may also be a rectangular area of a frame intended for independent decoding and encoding.

[0449] A block is an M×N (N rows and M columns) array of a plurality of samples, or an M×N array of a plurality of transform coefficients. A block may also be a square or rectangular area of a plurality of pixels composed of a plurality of matrices of one luminance and two chrominances.

[0450] A CTU (Coding Tree Unit) may also be a coding tree block for a plurality of luminance samples of a picture having 3 sample arrays, or may be two corresponding coding tree blocks for a plurality of chroma samples. Alternatively, a CTU may also be a coding tree unit for a plurality of samples in any one of a monochrome picture, a picture encoded using 3 separate color planes and a syntax structure used for encoding a plurality of samples.

[0451] A super block may also constitute one or two mode information blocks, or may be recursively divided into 4 32×32 blocks, or may be a 64×64 pixel square block that can be further divided. [First aspect]

[0452] The encoding device 100, decoding device 200, encoding method, and decoding method of the first aspect of the present disclosure will be described below.

[0453] FIG. 47 is a flowchart showing an example of the inter prediction process between the first aspects. An example of the inter prediction process of the decoding device 200 will be described below.

[0454] The inter prediction unit 218 in the decoding device 200 derives a reference motion vector for predicting the target block; derives a first motion vector different from the reference motion vector; derives a motion vector based on the difference between the reference motion vector and the first motion vector; determines whether the differential motion vector is greater than a threshold value. When it is determined that the differential motion vector is greater than the threshold value, the first motion vector is changed. When it is determined that the differential motion vector is not greater than the threshold value, the first motion vector is not changed. The target block is decoded using the changed first motion vector or the unchanged first motion vector. For example, the reference motion vector corresponds to the first pixel combination in the target block, and the first motion vector corresponds to the second pixel combination in the target block that is different from the first pixel combination. Hereinafter, the inter prediction process will be described in more detail with reference to the drawings. In addition, the reference motion vector and the first motion vector described below are merely examples and are not limited thereto.

[0455] First, in step S1001, the inter prediction unit 218 derives a reference motion vector for the first pixel combination in the target block. The process of step S1001 will be described in more detail below with reference to FIG. 48.

[0456] FIG. 48 is a diagram showing an example of a processing target block. As shown in FIG. 48, the processing target block may also be composed of a plurality of sub-blocks (sub-block 0 to sub-block 5). Also, each sub-block may respectively have different first motion vectors (MV 0 to MV 5). An example of the first pixel combination may be sub-block 0. At this time, the reference motion vector is one of the plurality of first motion vectors within the processing target block. In this example, the reference motion vector is MV 0.

[0457] Another example of the first pixel combination may be the entire processing target block. In this example, the reference motion vector is the average of the first motion vectors of all the sub-blocks within the processing target block, that is, the average from MV 0 to MV 5.

[0458] Next, in step S1002, the inter prediction unit 218 derives the first motion vector of the second pixel combination within the processing target block. The second pixel combination is different from the first pixel combination. An example of the second pixel combination may be sub-block 2. In this example, the first motion vector of the second pixel combination is MV 2.

[0459] Also, another example of the second pixel combination may be sub-block 1. In this example, the first motion vector is MV 1.

[0460] Next, in step S1003, the inter prediction unit 218 derives a difference vector based on the difference between the reference motion vector of the first pixel combination and the first motion vector of the second pixel combination. An example of the value of the reference motion vector may be (-3, 4). An example of the first motion vector may be (16, 5). Therefore, the differential motion vector between the first motion amount and the reference motion vector becomes (-19, -1).

[0461] Next, in step S1004, the inter prediction unit 218 determines whether the differential motion vector derived in step S1003 is greater than a threshold value. The threshold value is a first value (hereinafter referred to as the first threshold value) and a second value (hereinafter referred to as the second threshold value) of one combination. An example of the threshold value may be (10, 20). The following will explain this determination process.

[0462] When the absolute value of the horizontal component of the differential motion vector derived in step S1003 is greater than the first threshold, or when the absolute value of the vertical component of the differential motion vector is greater than the second threshold, the inter prediction unit 218 determines that the differential motion vector is greater than the threshold (in step S1004, yes). In other words, if either the absolute value of the horizontal component or the absolute value of the vertical component of the differential motion vector is greater than the threshold, the differential motion vector is determined to be greater than the threshold. For example, comparing the absolute value of the horizontal component of the differential motion vector (-19, -1) shown as an example in step S1003 with the first threshold, since |-19| > 10, the differential motion vector is determined to be greater than the threshold. At this time, the inter prediction unit 218 changes the first motion vector. The details of the change are described in step S1006.

[0463] On the other hand, when the absolute value of the horizontal component of the differential motion vector is not greater than the first threshold and the absolute value of the vertical component of the differential motion vector is not greater than the second threshold, the inter prediction unit 218 determines that the differential motion vector is not greater than the threshold (in step S1004, no). In other words, when either the absolute value of the horizontal component or the absolute value of the vertical component of the differential motion vector is below the threshold, the differential motion vector is determined to be below the threshold. When it is determined that the differential motion vector is not greater than the threshold (in step S1004, no), the inter prediction unit 218 does not change the first motion vector (step S1005).

[0464] Next, in step S1006, when it is determined that the differential motion vector is greater than the threshold (in step S1004, yes), the inter prediction unit 218 changes the first motion vector using the value obtained by clipping the differential motion vector. At this time, the differential motion vector (-19, -1) is clipped to (-10, -1). As described above, since the absolute value of the horizontal component of the differential motion vector |-19| is greater than the first threshold (10, 20) which is 10, it is clipped so that the absolute value of the horizontal component of the differential motion vector is the same as the first threshold. Therefore, the clipped differential motion vector becomes (-10, -1).

[0465] Next, the inter prediction unit 218 changes the first motion vector using the clipped differential motion vector and the reference motion vector. More specifically, the differential motion vector can be added to the reference motion vector to change the first motion vector. For example, the changed first motion vector: (-3, 4) - (-10, -1) = (7, 5).

[0466] In step S1007, the intermediate prediction unit 218 decodes the second pixel combination using the changed first motion vector or the unchanged first motion vector. For example, sub-block 2 decodes using the changed first motion vector (7, 5).

[0467] In addition, the process 1000 shown in FIG. 47 may also be a process of the encoding device.

[0468] The process 1000 can be applied to all sub-blocks within the processing target block. When the process 1000 is applied to all sub-blocks within the processing target block, the prediction mode of the processing target block may also be an affine mode. Also, when the process 1000 is applied to all sub-blocks within the processing target block, the prediction mode of the processing target block may also be an ATMVP (Alternative Temporal Motion Vector Prediction) mode.

[0469] In addition, the ATMVP mode is an example of a sub-block mode classified into the merge mode. For example, in the encoded reference picture specified by the MV (MV0) of the block adjacent to the lower left corner of the current block, the temporal MV reference block corresponding to the current block is specified, and for each sub-block within the current block, the MV used during the encoding of the area corresponding to the sub-block within the temporal MV reference block is specified.

[0470] Also, the first motion vector of other sub-blocks of the processing target block can be updated using the first motion vector changed for a specific sub-block. Hereinafter, a processing example will be described.

[0471] For example, the MV 0 of sub-block 0 is determined as the reference motion vector, the MV 2 of sub-block 2 is determined as the first motion vector, and the process 1000 is used to determine the changed first motion vector. The changed first motion vector of sub-block 2 can be represented as MV 2’ = (V 2x’, V 2y’). Using the reference motion vector MV 0 and the changed first motion vector MV 2’, the changed first motion vector MV i’ = (V ix’, V iy’) of sub-blocks i (i = 1, 3, 4, 5) other than sub-block 2 within the processing target block is calculated by the following formula. V ix’=(V 2x’-V 0x)*POS ix / W-(V 2y’-V 0y)*POS iy / W+V 0x V iy’=(V 2y’-V 0y)*POS ix / W+(V 2x’-V 0x)*POS iy / W+V 0y

[0472] Next, when MV i’ is not greater than the threshold value, update MV i to be MV i’, and use MV i to decode sub-block i. POS ix and POS iy are the horizontal position and vertical position of sub-block i. In the example of FIG. 48, POS 1x=W / 2,POS 1y=0 POS 3x=0,POS 3y=H POS 4x=W / 2,POS 4y=H POS 5x=W,POS 5y=H

[0473] W and H are the horizontal position and vertical position of sub-block 2 (e.g., the X coordinate and Y coordinate of the lower left corner).

[0474] The threshold value may also correspond to the size of the block to be processed. For example, the larger the size of the block to be processed, the larger the threshold value becomes.

[0475] Also, the threshold value may correspond to the number of reference pictures. For example, the more the number of reference pictures, the smaller the threshold value becomes.

[0476] The threshold value may also be encoded in a header area such as an SPS header, a PPS header, or a slice header.

[0477] The threshold value may also be determined in advance and not encoded into the stream.

[0478] The threshold value may also be determined in the following manner, that is, in order to limit the memory access amount in the worst case when performing motion compensation processing (prediction processing) on the block to be processed in a predetermined prediction mode to be less than the memory access amount when performing bidirectional motion compensation processing (prediction processing) on the block to be processed in a prediction mode other than the predetermined prediction mode and for every 8×8 pixels. The predetermined prediction mode is, for example, an affine mode. Hereinafter, an example of the limitation will be described.

[0479] "Mem_base" means the memory access amount of the worst case when performing bidirectional prediction motion compensation processing (luminance value) on a processing target block in a prediction mode other than a predetermined prediction mode and for each 8×8 size, and taking one 8×8 size block as one unit.

[0480] When performing motion compensation processing (luminance value) on a processing target block (size: M×N) in a predetermined prediction mode, the memory access amount of the worst case is represented by "Mem_CU", and the threshold value "Mem_th" for this is expressed using "Mem_base" as follows. Mem_th = M×N / (8×8)*Mem_base

[0481] Mem_CU is restricted to not be larger than Mem_th. Hereinafter, an example of the calculation process of the threshold value will be described.

[0482] Assuming that the size of each pixel of the luminance signal is 1 byte and the number of taps of the filter for motion compensation processing is 8 taps, it becomes: Mem_base = (8 + 7)*(8 + 7)*2 = 450 (in byte units).

[0483] Assuming that the prediction mode of the processing target block is the affine mode and the size of the processing target block is 64×64 pixels, it becomes: Mem_th = 64 / 8*64 / 8*450 = 28800 (in byte units).

[0484] "H" and "V" are respectively assumed to represent the threshold value of the first component (the first threshold value) and the threshold value of the second component (the second threshold value) of the threshold value.

[0485] When performing prediction processing in a normal mode other than the affine mode, the worst condition is that the processing target block is divided into 8×8 pixel blocks, and all 8×8 pixel blocks are subjected to bidirectional prediction motion compensation processing. Under the state where the memory access amount does not exceed this worst condition, the range of the memory that can be referred to when performing the prediction mode in the affine mode is determined. Assume that the range of the memory that can be referred to in the affine mode is calculated by the following formula [1]. Make the range of the memory access in the affine mode be below Mem_th (for example, 28800), and calculate H and V. Here, an example of performing bidirectional prediction in this affine mode is used for explanation. Mem_CU = (64 + 7 + 2*H)(64 + 7 + 2*V)*2 ≦ 28800 [1] Assume H = V to solve Equation [1], then H = V = 24 is obtained.

[0486] In addition, the above-described calculation process of the threshold value is also applicable to processing target blocks with different sizes and different numbers of reference pictures. FIG. 49 is a diagram showing an example of the calculated threshold value. In addition, although an example of H = V is shown in FIG. 49, it may also be H > V or H < V.

[0487] As shown in FIG. 49, the threshold value is different in the case of performing unidirectional prediction on the processing target block (number of reference frames = 1) and in the case of performing bidirectional prediction (number of reference frames = 2). In the example of Equation [1] above, the size of the processing target block predicted in the affine mode is 64×64. When bidirectional prediction is performed, that is, when two reference pictures are referred to, both the first threshold value H and the second threshold value V become 24. For example, when unidirectional prediction is performed on a 64×64-sized processing target block predicted in the affine mode, that is, when only one reference picture is referred to, both the first threshold value H and the second threshold value V become 49. In FIG. 49, using the above Equation [1], the threshold values for all block sizes that can be predicted in the affine mode are calculated and tabulated. [Technical Advantages of the First Aspect]

[0488] In the first aspect of the present disclosure, a process for deriving a first motion vector is introduced in the inter prediction process. As described above, the process of selecting an appropriate first motion vector so that the deviation of a plurality of first motion vectors in the processing target block falls within a predetermined range can reduce the memory bandwidth of the inter prediction process. [Supplementary]

[0489] The encoding device 100 and the decoding device 200 in the present embodiment can be used as an image encoding device and an image decoding device, respectively, and can also be used as a dynamic image encoding device and a dynamic image decoding device.

[0490] Alternatively, the encoding device 100 and the decoding device 200 can also be used as an entropy prediction device and an entropy decoding device, respectively. That is, the encoding device 100 and the decoding device 200 can respectively only correspond to the entropy encoding unit 110 and the entropy decoding unit 202. Then, other components can also be included in other devices.

[0491] Furthermore, at least a part of the present embodiment can be used as an encoding method, can be used as a decoding method, can be used as an entropy decoding method, and can be used as other methods.

[0492] Also, in the present embodiment, each component can be configured by dedicated hardware or implemented by executing a software program suitable for each component. Each component can also be implemented by having a program execution unit such as a CPU or a processor read and execute a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

[0493] Specifically, each of the encoding device 100 and the decoding device 200 may also include a processing circuitry and a storage device. The storage device is electrically connected to the processing circuitry and can be accessed by the processing circuitry. For example, the processing circuitry corresponds to processors a1 or b1, and the storage device corresponds to memories a2 or b2.

[0494] The processing circuitry includes at least one of dedicated hardware and a program execution unit, and uses the storage device to execute processing. Also, when the processing circuitry includes a program execution unit, the storage device stores a software program executed by the program execution unit.

[0495] Here, the software for implementing the encoding device 100 or the decoding device 200 of the present embodiment is as follows.

[0496] For example, this program can also cause a computer to execute an encoding method for encoding a dynamic image, including: deriving a reference motion vector for predicting a processing target block; deriving a first motion vector different from the reference motion vector; deriving a differential motion vector based on the difference between the reference motion vector and the first motion vector; and determining whether the differential motion vector is greater than a threshold value. When it is determined that the differential motion vector is greater than the threshold value, the first motion vector is changed. When it is determined that the differential motion vector is not greater than the threshold value, the first motion vector is not changed, and the processing target block is encoded using the changed first motion vector or the unchanged first motion vector.

[0497] Also, for example, the present program can also cause a computer to execute a decoding method for decoding a dynamic image, including: deriving a reference motion vector for predicting a target block; deriving a first motion vector different from the reference motion vector; deriving a differential motion vector based on the difference between the reference motion vector and the first motion vector; and determining whether the differential motion vector is greater than a threshold value. When it is determined that the differential motion vector is greater than the threshold value, the first motion vector is changed. When it is determined that the differential motion vector is not greater than the threshold value, the first motion vector is not changed. The target block is decoded using the changed first motion vector or the unchanged first motion vector.

[0498] Also, as described above, each component may also be a circuit. These circuits may be configured as a single circuit as a whole, or may be individual circuits respectively. Also, each component may be implemented by a general-purpose processor or a dedicated processor.

[0499] Also, other components may be caused to execute the processing to be executed by a specific component. Also, the order of executing the processing may be changed, and a plurality of processes may be executed in parallel. Also, the encoding / decoding device may include an encoding device 100 and a decoding device 200.

[0500] The first, second, etc. ordinal numbers used in the description may be appropriately replaced. Also, ordinal numbers may be reattached to components, etc., or removed.

[0501] As described above, the aspects of the encoding device 100 and the decoding device 200 have been described based on the embodiments. However, the aspects of the encoding device 100 and the decoding device 200 are not limited to the embodiments. As long as it does not deviate from the gist of the present disclosure, various modifications conceivable by those skilled in the art, or forms constructed by combining the components of different embodiments, may also be included in the scope of the aspects of the encoding device 100 and the decoding device 200.

[0502] This aspect may also be implemented by combining at least a part of this aspect with other aspects in the present disclosure. Also, a part of the processing described in the flowchart of this aspect, a part of the configuration of the device, a part of the syntax, etc. may be combined with other aspects for implementation. (Embodiment 2) [Implementation and Application]

[0503] In the above embodiments, each of the functional or operative blocks can generally be implemented by an MPU (micro processing unit), a memory, and the like. Also, the processing performed by each of the functional blocks can be implemented as a program execution unit of a processor or the like that executes a software (program) read from a recording medium such as a ROM. This software can also be distributed. This software can be recorded on various recording media such as a semiconductor memory. Of course, each functional block can also be implemented by hardware (a dedicated circuit).

[0504] Also, the processing described in each of the embodiments can be implemented by centralized processing using a single device (system), or can be implemented by distributed processing using a plurality of devices. Also, the processor that executes the above program can be singular or plural. That is, centralized processing can be performed, or distributed processing can also be performed.

[0505] The aspects of the present disclosure are not limited to the above embodiments, and various modifications can be made, and these are also included in the scope of the aspects of the present disclosure.

[0506] Furthermore, here, application examples of the moving image encoding method (image encoding method) or the moving image decoding method (image decoding method) shown in the above embodiments and various systems for implementing such application examples will be described. Such a system can also be characterized by having an image encoding device using the image encoding method, an image decoding device using the image decoding method, or an image encoding / decoding device having both. For other configurations of such a system, appropriate changes can be made according to the needs of the situation. [Usage Example]

[0507] FIG. 50 is a diagram showing the overall configuration of a content supply system ex100 that realizes a content distribution service. The area for providing the communication service is divided into a desired size, and in each cell, fixed radio stations, namely base stations ex106, ex107, ex108, ex109, ex110 in the illustrated example, are respectively provided.

[0508] In the content supply system ex100, machines such as computer ex111, game console ex112, camera ex113, home appliance ex114, and smartphone ex115 are connected to the Internet ex101 via Internet service provider ex102 or communication network ex104, and base stations ex106 to ex110. The content supply system ex100 can also be configured to connect by combining any of the above elements. In various implementations, the machines can also be directly or indirectly connected to each other via a telephone network or short-range wireless, etc., without passing through base stations ex106 to ex110 which are fixed wireless stations. Furthermore, the streaming server ex103 can also be connected to machines such as computer ex111, game console ex112, camera ex113, home appliance ex114, and smartphone ex115 via the Internet ex101, etc. Also, the streaming server ex103 can be connected to terminals, etc. within a hotspot in an airplane ex117 via a satellite ex116.

[0509] In addition, wireless access points or hotspots, etc. can be used to replace base stations ex106 to ex110. Also, the streaming server ex103 can be directly connected to the communication network ex104 without passing through the Internet ex101 or Internet service provider ex102, and can be directly connected to the airplane ex117 without passing through the satellite ex116.

[0510] The camera ex113 is a machine such as a digital camera that can perform still image photography and moving image photography. Also, the smartphone ex115 generally refers to a smart phone, mobile phone, or PHS (Personal Handyphone System), etc. that corresponds to mobile communication systems such as 2G, 3G, 3.9G, 4G, and what will be called 5G in the future.

[0511] The home appliance ex114 includes machines such as a refrigerator or a home fuel cell cogeneration system.

[0512] In the content supply system ex100, a terminal with a photographing function is connected to a streaming server ex103 via a base station ex106 or the like, so that live broadcasting or the like can be performed. In live broadcasting, the terminal (such as the computer ex111, the game machine ex112, the camera ex113, the home appliance ex114, the smart phone ex115, and the terminal in the airplane ex117) can also perform the encoding process described in the above embodiments on the static image or dynamic image content photographed by the user using the terminal, multiplex the video data obtained by encoding and the audio data obtained by encoding the sound corresponding to the video, and send the obtained data to the streaming server ex103. That is, each terminal functions as an image encoding device according to an aspect of the present disclosure.

[0513] On the other hand, the streaming server ex103 performs streaming distribution of the content data to be sent to the requesting client. The client refers to a computer ex111, a game machine ex112, a camera ex113, a home appliance ex114, a smart phone ex115, or a terminal in the airplane ex117 that can decode the data processed by the above encoding. Each machine that has received the distributed data performs decoding processing on the received data and then plays it. That is, each machine can also function as an image decoding device according to an aspect of the present disclosure. [Distributed processing]

[0514] Furthermore, the streaming server ex103 can also be a plurality of servers or a plurality of computers, and perform distributed processing or record distribution of data. For example, the streaming server ex103 can be implemented by a CDN (Content Delivery Network), and content distribution is achieved by connecting the networks between a majority of edge servers distributed around the world. In the CDN, a physically close edge server is dynamically allocated according to the client. Then, the content is cached and distributed by the edge server, thereby reducing latency. Also, when several types of errors occur or the communication state changes due to increased traffic or the like, distributed processing can be performed by a plurality of edge servers, or the distribution main body can be switched to another edge server to bypass the network part with a failure and continue the distribution, so that high-speed and stable distribution can be achieved.

[0515] Also, not only the distributed processing of its own distribution, but also the encoding process of the photographed data can be performed on each terminal, on the server side, or shared among each other. For example, generally in the encoding process, the processing loop is performed twice. In the first loop, the complexity or encoding amount of the image in units of frames or scenes is detected. Also, in the second loop, processing for maintaining image quality and improving encoding efficiency is performed. For example, the terminal performs the first encoding process, and the server side that has received the content performs the second encoding process. In this way, the processing burden on each terminal can be reduced while improving the quality and efficiency of the content. At this time, when there is a request to decode almost in real time, the data that has completed the first encoding by the terminal can also be received and played on other terminals, so more flexible real-time distribution can be achieved.

[0516] For another example, a camera ex113, etc., extracts feature amounts from an image, compresses the data related to the feature amounts as meta data, and sends it to the server. The server, for example, judges the importance of an object from the feature amounts, and switches the quantization accuracy, etc., and performs compression according to the meaning of the image (or the importance of the content). Feature amount data is particularly effective for improving the accuracy and efficiency of motion vector prediction during re-compression on the server. Also, simple encoding such as VLC (Variable Length Coding) can be performed on the terminal, and encoding with a large processing load such as CABAC (Context Adaptive Binary Arithmetic Coding) can be performed on the server.

[0517] Furthermore, as another example, in a stadium, a shopping mall, or a factory, etc., there may be a situation where there are a plurality of image data of almost the same scene photographed via a plurality of terminals. In this case, using a plurality of terminals that have performed shooting, and other terminals and servers that have not performed shooting as needed, encoding processes are respectively allocated in units such as GOP (Group of Picture), picture units, or block units obtained by dividing the picture, to perform distributed processing. Thereby, the delay can be reduced and better real-time performance can be achieved.

[0518] Also, since the plurality of image data are of almost the same scene, management and / or instruction can also be performed on the server so that the image data photographed by each terminal can be mutually referenced. Also, the server can receive the encoded data from each terminal, change the reference relationship between the plurality of data, or correct or replace the picture itself, and re-encode. Thereby, a stream that improves the quality and efficiency of each data can be generated.

[0519] Furthermore, the server can also perform transcoding to change the encoding method of the video data and then distribute the video data. For example, the server can also convert the encoding method of the MPEG system to the VP system (such as VP9), or convert H.264 to H.265.

[0520] In this way, the encoding process can be performed by the terminal or one or more servers. Therefore, hereinafter, the description uses "server" or "terminal" as the main body of the processing, but part or all of the processing performed by the server can also be performed by the terminal, and part or all of the processing performed by the terminal can also be performed by the server. Also, regarding these parts, the same applies to the decoding process. [3D, Multi-view]

[0521] In recent years, there have been more cases of integrating images or videos of different scenes photographed by terminals such as a plurality of cameras ex113 and / or smartphones ex115 that are almost synchronized, or the same scene photographed from different perspectives. The images photographed by each terminal are integrated based on the relative positional relationship between the terminals obtained separately, or the regions where the feature points contained in the images match.

[0522] The server not only encodes 2D moving images, but also, based on scene analysis of the moving images, etc., automatically or at a time specified by the user, encodes still images and sends them to the receiving terminal. Furthermore, when the server can obtain the relative positional relationship between the photographing terminals, it can generate the 3D shape of the scene based on images of the same scene photographed from different perspectives, not just 2D moving images. In addition, the server can also encode 3D data generated through point cloud, etc., and based on the results of identifying or tracking people or objects using the 3D data, select or reconstruct from the images photographed by a plurality of terminals to generate the images to be sent to the receiving terminal.

[0523] After performing this, the user can also arbitrarily select each image corresponding to each photographing terminal to view the scene, or view the content of the image cut from the 3D data reconstructed from a plurality of images or videos from any viewpoint. Furthermore, together with the images, sounds can be collected from a plurality of different perspectives, and the server can multiplex the sound from a specific perspective or space and the corresponding image, and send the multiplexed image and sound.

[0524] In recent years, content that establishes a correspondence between the real world and the virtual world, such as Virtual Reality (VR / virtual reality) and Augmented Reality (AR / augmented reality), has also gradually become popular. In the case of VR images, the server can also create viewpoint images for the right eye and the left eye respectively, and through Multi-View Coding (MVC / multi-view coding), etc., perform coding that allows reference between images at each viewpoint, or perform coding as different streams without mutual reference. When decoding different streams, it is also possible to synchronize and play them by reproducing the virtual three-dimensional space according to the user's viewpoint.

[0525] In the case of AR images, the server overlays the virtual object information in the virtual space on the camera information in the real space according to the movement of the three-dimensional position or the user's viewpoint. The decoding device can also obtain or hold the virtual object information and three-dimensional data, and generate two-dimensional images according to the movement of the user's viewpoint, and smoothly continue to create the overlay data. Alternatively, the decoding device can also send the movement of the user's viewpoint to the server in addition to the request for the virtual object information. The server creates the overlay data from the three-dimensional data held in the server according to the received movement of the viewpoint, encodes the overlay data, and distributes it to the decoding device. Additionally, it can also be that: the overlay data has an α value indicating the display transparency in addition to RGB, and the server sets the α value of the part other than the object created from the three-dimensional data to 0, etc., and encodes it in a state where this part is in a transparent state. Or, the server can use the RGB value set with a predetermined value like a chroma key as the background, and create data with the part other than the object set as the background color.

[0526] Similarly, the decoding process of the distributed data can be performed on each terminal device on the client side, or can also be performed on the server side, or can be shared mutually. For example, a certain terminal device can first send a reception request to the server, and another terminal device receives the content corresponding to this request, performs the decoding process, and sends the decoded signal to the device with a display. It is possible to disperse the processing and select appropriate content without relying on the performance of the terminal device itself that can communicate, thereby playing high-quality video data. Also, for another example, it is also possible to receive large-sized image data on a TV, etc., and decode and display a part of the area such as a block after dividing the picture on the personal terminal of the viewer. Thereby, the overall image can be shared, and the area of one's own responsibility or the area that one wants to confirm in more detail can be confirmed nearby.

[0527] Also, in several wireless communication situations at short, medium, or long distances indoors and outdoors, a distribution system specification such as MPEG-DASH can be used to seamlessly receive content. The user can also freely select their own terminal, a decoding device or a display device such as a monitor installed indoors or outdoors, and switch in real time. Also, based on their own location information, etc., the decoding terminal and the display terminal can be switched while decoding. Thereby, while the user is moving towards the destination, map information is mapped and displayed on a part of the wall surface or the ground of the building next to the device capable of displaying, and the user moves while viewing the map information. Also, based on the ease of access to the encoded data on the network, such as the encoded data being cached in a server that can be accessed from the receiving terminal in a short time, or being replicated to an edge server in a Content Delivery Service, etc., the bit rate of the received data can be switched. [Adaptive Coding]

[0528] Regarding the switching of content, an adaptive stream shown in FIG. 51 is used for explanation. The aforementioned adaptive stream is a stream that has been compression-encoded by applying the dynamic image encoding method shown in each of the above embodiments. The server may have multiple streams with the same content but different qualities for each individual stream, or it may be configured as follows: By making flexible use of the characteristics of the temporal / spatial adaptive stream, which is realized by hierarchical encoding as shown in the figure, to switch content. That is, the decoding side determines which layer to decode according to internal factors such as performance and external factors such as the state of the communication bandwidth. In this way, the decoding side can freely switch between low-resolution content and high-resolution content for decoding. For example, when the user wants to watch the subsequent part of the video that was watched on the smartphone ex115 during movement at home on a device such as an Internet TV, the device only needs to decode the same stream to different layers, thus reducing the burden on the server side.

[0529] Furthermore, as described above, in addition to encoding the image layer by layer and implementing scalability with the enhancement layer above the base layer, the enhancement layer may also contain meta-information such as image-based statistical information. On the decoding side, based on the meta-information, the image of the base layer can be super-resolved to generate high-quality content. Super-resolution can also mean maintaining and / or increasing the resolution and improving the signal-to-noise ratio. The meta-information includes information such as linear or non-linear filtering coefficients used to specify the super-resolution process, or information on parameter values in filtering processes, machine learning, or least-squares operations used to specify the super-resolution process.

[0530] Alternatively, according to the meaning of objects in the image, etc., the image can be segmented into blocks. The decoding side selects the blocks to be decoded, and only decodes a part of the area. Furthermore, the attributes of the object (person, car, ball, etc.) and the position in the image (coordinate position in the same image, etc.) are stored as meta-information, so that the decoding side can specify the position of the desired object based on the meta-information and determine the block containing the object. For example, as shown in FIG. 52, the meta-information is stored using a data storage structure different from pixel data, such as the SEI (supplemental enhancement information) message in HEVC. The meta-information shows, for example, the position, size, or color of the main object.

[0531] The meta-information can also be stored in units composed of a plurality of images, such as streams, sequences, or random access units. The decoding side can obtain the moment when a specific person appears in the image, etc., and combine the information of the image unit and the time information to specify the image where the object exists and determine the position of the object in the image. [Optimization of Web Pages]

[0532] FIG. 53 is a diagram showing an example of a display screen of a web page in a computer ex111 or the like. FIG. 54 is a diagram showing an example of a display screen of a web page in a smartphone ex115 or the like. As shown in FIGS. 53 and 54, when a web page includes a plurality of linked images including links to complex image content, the way of viewing these linked images varies depending on the device for viewing. When a plurality of linked images are visible on the screen, until the user clearly indicates a selection of a linked image, or until the linked image approaches near the center of the screen or the entire linked image enters the screen, the display device (decoding device) may also display a static image or an I picture included in each content as a linked image, or may display an image like a gif animation with a plurality of static images or I pictures, etc., or may only receive the base layer and decode and display the image.

[0533] When a linked image is selected by the user, the display device gives the highest priority to the base layer for decoding. Also, if there is information in the HTML constituting the web page that shows adjustable content, the display device may also decode up to the enhancement layer. Also, in order to ensure real-time performance, before being selected or when the communication bandwidth is extremely narrow, the display device may only decode and display the pictures referring to the front (I pictures, P pictures, B pictures referring only to the front), thereby reducing the delay between the decoding time and the display time of the leading picture (the delay from the start of content decoding to the start of display). Also, the display device may forcibly ignore the reference relationship of the pictures and make all B pictures and P pictures refer to the front, first roughly decode, and as time passes and the received pictures increase, perform normal decoding. [Autopilot]

[0534] Also, when sending and receiving static images or video data such as two-dimensional or three-dimensional map information for the autopilot or assisted driving of an automobile, in addition to the image data of a level above the first layer, the receiving terminal may also receive weather or construction information, etc. as meta-information, and establish a correspondence relationship with such information for decoding. Also, the meta-information may belong to a layer, or may simply be multiplexed with the image data.

[0535] At this time, since vehicles, drones, airplanes, etc. containing the receiving terminal move, the receiving terminal sends the position information of the receiving terminal, thereby enabling seamless reception and decoding execution while switching base stations ex106 to ex110. Also, the receiving terminal can dynamically switch to what extent to receive meta-information or update the map information according to the user's selection, the user's situation, and / or the state of the communication bandwidth.

[0536] In the content supply system ex100, the client can receive and decode the encoded information sent by the user in real time and play it. [Distribution of Personal Content]

[0537] Also, in the content supply system ex100, not only high-quality and long-duration content distributed by video distributors, but also short-duration content of low quality through individuals can be distributed by unicast or multicast. It is considered that such personal content will increase in the future. In order to make personal content into more excellent content, the server can also perform editing processing and then encoding processing. This can be achieved, for example, by the following configuration.

[0538] During shooting in real time or after storing first and then after shooting, the server performs recognition processing such as shooting error, scene search, meaning analysis, and object detection on the original image or encoded data. Then, based on the recognition result, the server manually or automatically corrects out-of-focus or camera shake, etc., or deletes scenes of low importance such as scenes with lower brightness or out-of-focus than other pictures, or emphasizes the edges of objects, or changes the color tone, etc. for editing. Based on the editing result, the server encodes the edited data. Also, it is known that when the shooting time is too long, the viewing rate will decrease. The server can also automatically edit not only scenes of low importance as described above but also scenes with little movement based on the image processing result so that the content is within a specific time range according to the shooting time. Or, the server can also generate a digest based on the result of the meaning analysis of the scene and perform encoding.

[0539] In addition, in personal content, if left unchanged, there are cases where things that may infringe on copyright, moral rights of the author, or portrait rights are captured, and there are also cases where the sharing range exceeds the intended range, etc., which are not suitable for individuals. Therefore, for example, the server can deliberately change the faces or the inside of the house, etc. at the periphery of the screen into out-of-focus images for encoding. Also, the server can recognize whether a face of a person different from the pre-registered person is captured in the image to be encoded. If so, perform processing such as adding a mosaic to the face part. Or, from the perspective of copyright, etc., the user can also specify the person or background area to be processed in the pre-processing or post-processing of encoding. The server can replace the specified area with another image or perform processing such as blurring the focus. If it is a person, in a moving image, the face part image can be replaced while tracking the person.

[0540] Since the viewing of personal content with a small amount of data has a high requirement for real-time performance, although there are differences depending on the bandwidth, the decoding device first receives the base layer with the highest priority and decodes and plays it. The decoding device can also receive the enhancement layer during this period. When there is loop playback or when playing more than twice, it plays the high-quality video including the enhancement layer. If the scalable encoded stream is processed in this way, the following experience can be provided. That is, although it is rough animation at the stage of not being selected or just starting to watch, the stream will gradually become more refined and the image will improve. In addition to scalable encoding, a stream composed of a rough stream played for the first time and a second stream encoded with reference to the first animation can also provide the same experience. [Other implementation application examples]

[0541] Also, these encoding or decoding processes are generally processed by the LSIex500 in each terminal. The LSI (large scale integration circuitry) ex500 (refer to Figure 50) can be a single chip or composed of multiple chips. In addition, software for dynamic image encoding or decoding can be loaded into some recording media (such as CD-ROM, floppy disk, or hard disk) that can be read by a computer ex111, etc., and the encoding or decoding process can be performed using this software. Furthermore, when a smartphone ex115 is equipped with a camera, it can also transmit the video data obtained by the camera. The video data at this time is the data that has been encoded by the LSIex500 in the smartphone ex115.

[0542] In addition, the LSIex500 can also be configured to be activated by downloading an application software program. At this time, first, the terminal needs to determine whether the terminal supports the encoding method of the content or has the execution ability of a specific service. When the terminal does not support the encoding method of the content or does not have the execution ability of a specific service, the terminal needs to download an encoder / decoder or an application software program, and then obtain and play the content.

[0543] Furthermore, not limited to the content supply system ex100 via the Internet ex101, at least either the dynamic image encoding device (image encoding device) or the dynamic image decoding device (image decoding device) of the above-described embodiments can be incorporated into a digital broadcast system. Since multiplexed data in which video and audio are multiplexed is carried on radio waves for broadcasting using a satellite or the like for transmission and reception, with respect to the configuration of the content supply system ex100 that is easy to perform unicast, although the digital broadcast system is advantageous for multicast in terms of differences, the same applications can still be implemented for encoding processing and decoding processing. [Hardware Configuration]

[0544] FIG. 55 is a diagram showing the smartphone ex115 shown in FIG. 50. Further, FIG. 56 is a diagram showing a configuration example of the smartphone ex115. The smartphone ex115 includes: an antenna ex450 for transmitting and receiving radio waves to and from the base station ex110; a camera unit ex465 capable of capturing video and still images; and a display unit ex458 for displaying data obtained by decoding the video captured by the camera unit ex465 and the video received by the antenna ex450. The smartphone ex115 further includes: an operation unit ex466 such as a touch panel; a sound output unit ex457 such as a speaker for outputting sound or audio; a sound input unit ex456 such as a microphone for inputting sound; a memory unit ex467 capable of storing the captured video or still images, the recorded sound, the received video or still images, the encoded data such as emails, or the decoded data; and a slot unit ex464 which is an interface portion with the SIM ex468, and the SIM ex468 is used to identify the user and perform authentication for accessing various data mainly via the network. In addition, an external memory can be used instead of the memory unit ex467.

[0545] Furthermore, a main control unit ex460 that integrally controls the display unit ex458 and the operation unit ex466, etc., is connected to a power supply circuit unit ex461, an operation input control unit ex462, an image signal processing unit ex455, a camera interface unit ex463, a display control unit ex459, a modulation / demodulation unit ex452, a multiplexing / demultiplexing unit ex453, a sound signal processing unit ex454, a slot unit ex464, and a memory unit ex467 via a bus ex470.

[0546] When the power switch is turned on by the user's operation, the power supply circuit unit ex461 activates the smartphone ex115 to a workable state and supplies power to each unit from the battery pack.

[0547] The smartphone ex115 processes calls and data communications based on the control of the main control unit ex460 having a CPU, ROM, RAM, etc. During a call, the voice signal received by the voice input unit ex456 is converted into a digital voice signal by the voice signal processing unit ex454, the signal is subjected to spectrum spreading processing in the modulation / demodulation unit ex452, digital-to-analog conversion processing and frequency conversion processing are performed in the transmission / reception unit ex451, and the resulting signal is transmitted via the antenna ex450. Also, the received data is amplified, frequency conversion processing and analog-to-digital conversion processing are performed, spectrum despreading processing is performed in the modulation / demodulation unit ex452, it is converted into an analog voice signal by the voice signal processing unit ex454, and the signal is output from the voice output unit ex457. In the data communication mode, through operations of the operation unit ex466 of the main body unit, etc., text, still images, or video data are sent to the main control unit ex460 via the operation input control unit ex462 and are similarly subjected to transceiver processing. In the data communication mode, when transmitting images, still images, or video and sound, the video signal processing unit ex455 compresses and encodes the video signal stored in the memory unit ex467 or the video signal input from the camera unit ex465 by the moving image encoding method shown in the above embodiments, and sends the encoded video data to the multiplexing / demultiplexing unit ex453. Also, the voice signal processing unit ex454 encodes the voice signal received by the voice input unit ex456 during photographing of images or still images, etc. by the camera unit ex465, and sends the encoded voice data to the multiplexing / demultiplexing unit ex453. The multiplexing / demultiplexing unit ex453 multiplexes the encoded video data and the encoded voice data in a predetermined manner, performs modulation processing and conversion processing in the modulation / demodulation unit (modulation / demodulation circuit unit) ex452 and the transmission / reception unit ex451, and transmits it via the antenna ex450.

[0548] When receiving an image attached to an email or a chat, or an image linked to a web page, etc., in order to decode the multiplexed data received via the antenna ex450, the multiplexing / demultiplexing unit ex453 separates the multiplexed data, thereby dividing the multiplexed data into a bit stream of video data and a bit stream of audio data. Via the synchronous bus ex470, the encoded video data is supplied to the video signal processing unit ex455, and the encoded audio data is supplied to the audio signal processing unit ex454. The video signal processing unit ex455 decodes the video signal through a video decoding method corresponding to the dynamic image encoding method shown in each of the above embodiments, and through the display control unit ex459, displays the image or still image contained in the linked dynamic image file from the display unit ex458. Also, the audio signal processing unit ex454 decodes the audio signal and outputs the sound from the audio output unit ex457. In addition, since real-time streaming has become popular, depending on the user's situation, there may be scenes that are inappropriate for society during the playback of sound. For this reason, as a first value, it is more desirable to have a configuration where the audio signal is not played and only the video data is played. It is also possible to play the sound synchronously only when the user performs an operation such as selecting the video data.

[0549] Also, here, the smartphone ex115 is used as an example for explanation, but for a terminal, the following three installation forms can also be considered. In addition to the signal transceiver type terminal having both an encoder and a decoder, there are also a transmitting terminal having only an encoder and a receiving terminal having only a decoder. Furthermore, in a digital broadcast system, the case of receiving or transmitting multiplexed data in which audio data, etc. is multiplexed on video data is described, but in addition to audio data, text data related to the video, etc. can also be multiplexed on the multiplexed data, and it is also possible to receive or transmit the video data itself instead of the multiplexed data.

[0550] In addition, although the case where the main control unit ex460 including a CPU controls encoding processing or decoding processing has been described, most of various terminals have a GPU. Therefore, the following configuration is also possible, that is, by using a memory shared by the CPU and the GPU, or a memory that manages addresses to form a state where it can be commonly used, the performance of the GPU is flexibly utilized, and a large area is aggregated and processed together. Thereby, the encoding time can be shortened, real-time performance can be ensured, and low latency can be achieved. In particular, it is efficient to perform motion estimation, deblocking filter, SAO (Sample Adaptive Offset), and transform and quantization processing in units of pictures, etc., through the GPU instead of the CPU. Industrial applicability

[0551] The present disclosure can be used in, for example, a television set, a digital video recorder, a car navigation system, a mobile phone, a digital camera, a digital video camera, a video conferencing system, or an electronic mirror.

[0552] 10: Block 100: Encoding device 102: Splitting unit 104: Subtraction unit 106: Transformation unit 108: Quantization unit 110: Entropy encoding unit 112, 204: Inverse quantization unit 114, 206: Inverse transformation unit 116, 208: Addition unit 118, 210: Block memory 120, 212: Loop filtering unit 122, 214: Frame memory 124, 216: Intra prediction unit 126, 218: Inter prediction unit 128, 220: Prediction control unit 200: Decoding device 202: Entropy decoding unit 1000: Processing 1201: Boundary determination unit 1202, 1204, 1206: Switch 1203: Filtering determination unit 1205: Filtering processing unit 1207: Filter characteristic determination unit 1208: Processing determination unit a1, b1: Processors a2, b2: Memories ex100: Content supply system ex101: Internet ex102: Internet service provider ex103: Streaming server ex104: Communication network ex106 to ex110: Base stations ex111: Computers ex112: Game consoles ex113: Cameras ex114: Home appliances ex115: Smartphones ex116: Satellites ex117: Aircraft ex450: Antennas ex451: Transmission / reception unit ex452: Modulation / demodulation unit ex453: Multiplexing / demultiplexing unit ex454: Audio signal processing unit ex455: Video signal processing unit ex456: Audio input unit ex457: Audio output unit ex458: Display unit ex459: Display control unit ex460: Main control unit ex461: Power supply circuit unit ex462: Operation input control unit ex463: Camera interface unit ex464: Slot unit ex465: Camera unit ex466: Operation unit ex467: Memory unit ex468: SIM ex470: Bus ex500: LSI Sa_1 to Sa_10, Sb_1, Sc_1a to Sc_1c, Sc_2, Sd_1a to Sd_1b, Sd_2 to Sd_3, Se_1 to Se_4, Sf_1 to Sf_5, Sg_1 to Sg_5, Sh_1 to Sh_3, Si_1 to Si_5, Sk_1 to Sk_3, Sj_1 to Sj_6, Sl_1 to Sl_4, Sm_1 to Sm_2, Sn_1 to Sn_5, So_1 to So_4, Sp_1 to Sp_7, Sq_1, Sr_1, Sr_2a to Sr_2c, Ss_1 to Ss_5, S1001 to S1007: Steps

Claims

1. An encoding device, which is an encoding device for encoding a dynamic image, includes: a circuit; and a memory connected to the aforementioned circuit. During operation, the prediction mode of the processing target block of the aforementioned circuit is the affine mode, and the aforementioned circuit: derives a reference motion vector for predicting the aforementioned processing target block; derives a first motion vector different from the aforementioned reference motion vector; derives a differential motion vector based on the difference between the aforementioned reference motion vector and the aforementioned first motion vector; and determines whether the aforementioned differential motion vector is greater than a threshold value. When it is determined that the aforementioned differential motion vector is greater than the aforementioned threshold value, a second motion vector different from the aforementioned reference motion vector and the aforementioned first motion vector is set to a first value. When it is determined that the aforementioned differential motion vector is not greater than the aforementioned threshold value, the aforementioned second motion vector is set to a second value different from the aforementioned first value. The aforementioned processing target block is encoded using the aforementioned second motion vector, and the aforementioned threshold value is different depending on whether the aforementioned processing target block is unidirectionally predicted or bidirectionally predicted.

2. A decoding device, which is a decoding device for decoding a dynamic image, includes: a circuit; and a memory connected to the aforementioned circuit. During operation, the prediction mode of the processing target block of the aforementioned circuit is the affine mode, and the aforementioned circuit: derives a reference motion vector for predicting the aforementioned processing target block; derives a first motion vector different from the aforementioned reference motion vector; derives a differential motion vector based on the difference between the aforementioned reference motion vector and the aforementioned first motion vector; and determines whether the aforementioned differential motion vector is greater than a threshold value. When it is determined that the aforementioned differential motion vector is greater than the aforementioned threshold value, a second motion vector different from the aforementioned reference motion vector and the aforementioned first motion vector is set to a first value. When it is determined that the aforementioned differential motion vector is not greater than the aforementioned threshold value, the aforementioned second motion vector is set to a second value different from the aforementioned first value. The aforementioned processing target block is decoded using the aforementioned second motion vector, and the aforementioned threshold value is different depending on whether the aforementioned processing target block is unidirectionally predicted or bidirectionally predicted.

3. A non-transitory memory medium that stores a bitstream and is a computer-readable non-transitory memory medium. The bitstream includes: coding information indicating that the prediction mode of the processing target block is an affine mode, and syntax for causing the computer to perform decoding processing. The decoding processing includes: deriving a reference motion vector for predicting the processing target block; deriving a first motion vector different from the reference motion vector; deriving a differential motion vector based on the difference between the reference motion vector and the first motion vector; and determining whether the differential motion vector is greater than a threshold. When it is determined that the differential motion vector is greater than the threshold, a second motion vector different from the reference motion vector and the first motion vector is set to a first value. When it is determined that the differential motion vector is not greater than the threshold, the second motion vector is set to a second value different from the first value. The processing target block is decoded using the second motion vector. The threshold is different depending on whether the processing target block is unidirectionally predicted or bidirectionally predicted.