Decoding device, encoding device, storage medium, decoding method and encoding method

By deriving motion vectors and performing motion compensation processing in prediction block units during the encoding and decoding methods, and generating prediction images using the spatial gradient of brightness, the problem of insufficient processing volume in existing technologies is solved, thereby reducing processing volume and improving encoding efficiency.

CN117097890BActive Publication Date: 2026-11-17PANASONIC INTELLECTUAL PROPERTY CORP OF AMERICA
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202311199423.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2017-06-05
Filing Date
2018-05-30
Publication Date
2026-11-17
Estimated Expiration
2038-05-30

AI Technical Summary

Technical Problem

Existing encoding and decoding methods are insufficient in terms of processing power, and it is necessary to reduce the processing power to improve efficiency.

Method used

The method employs motion vector derivation and motion compensation processing at the prediction block level. It generates prediction images by referencing the spatial gradient of brightness, thereby reducing the amount of processing. At the same time, it does not refer to the spatial gradient of brightness when processing at the sub-block level, thus improving coding efficiency.

Benefits of technology

While reducing the amount of processing, it avoids reducing encoding efficiency and improves the efficiency of encoding and decoding processes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117097890B_ABST
    Figure CN117097890B_ABST
Patent Text Reader

Abstract

This disclosure relates to a decoding apparatus, an encoding apparatus, a storage medium, a decoding method, and an encoding method. A decoding apparatus includes circuitry and a memory. The circuitry uses the memory to perform the following processing: in a first operating mode, using a candidate list containing motion vector candidates, derives a first motion vector from a first block obtained by segmenting an image; and generates a predicted image corresponding to the first block by referring to a spatial gradient of brightness generated based on the first motion vector. In a second operating mode, derives a second motion vector from a sub-block obtained by segmenting a second image, wherein the predicted image corresponding to the sub-block is generated without referring to a spatial gradient of brightness.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of the invention patent application filed on May 30, 2018, with application number 201880036730.3 and entitled "Encoding device, decoding device, encoding method and decoding method". Technical Field

[0002] This invention relates to encoding apparatus, decoding apparatus, encoding method, and decoding method. Background Technology

[0003] Previously, H.265 existed as a specification for encoding moving images. H.265 is also known as HEVC (High Efficiency Video Coding).

[0004] Existing technical documents Non-patent literature Non-patent literature 1: H.265 (ISO / IEC 23008-2 HEVC (High Efficiency Video Coding)) Summary of the Invention

[0005] The problem that the invention aims to solve In such encoding and decoding methods, it is hoped that the amount of processing can be reduced.

[0006] The purpose of this invention is to provide a decoding device, encoding device, decoding method, or encoding method that can reduce the amount of processing required.

[0007] Methods used to solve problems A decoding apparatus according to a technical solution of the present invention includes a circuit and a memory. The circuit uses the memory to perform the following processing: in a first operation mode, a first motion vector is derived in units of prediction blocks obtained by segmenting an image using a candidate list containing motion vector candidates; a prediction image corresponding to the prediction block is generated by referring to the spatial gradient of brightness generated based on the first motion vector; in a second operation mode, a second motion vector is derived in units of sub-blocks obtained by segmenting the prediction block; a prediction image corresponding to the sub-block is generated without referring to the spatial gradient of brightness.

[0008] An encoding apparatus according to a technical solution of the present invention includes a circuit and a memory. The circuit uses the memory to perform the following processing: in a first operation mode, a first motion vector is derived in units of prediction blocks obtained by segmenting an image using a candidate list containing motion vector candidates; a prediction image corresponding to the prediction block is generated by referring to the spatial gradient of brightness generated based on the first motion vector; in a second operation mode, a second motion vector is derived in units of sub-blocks obtained by segmenting the prediction block; a prediction image corresponding to the sub-block is generated without referring to the spatial gradient of brightness.

[0009] In addition, these inclusive or specific technical solutions can also be implemented by non-transitory recording media such as systems, devices, methods, integrated circuits, computer programs, or computer-readable CD-ROMs, or by any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0010] Invention Effects The present invention can provide a decoding device, encoding device, decoding method or encoding method that can reduce the amount of processing. Attached Figure Description

[0011] Figure 1 This is a block diagram showing the functional structure of the encoding device according to Embodiment 1.

[0012] Figure 2 This is a diagram illustrating an example of block segmentation in Implementation Method 1.

[0013] Figure 3 It is a table representing the transformation basis functions corresponding to each transformation type.

[0014] Figure 4A This is a diagram showing an example of the shape of the filter used in ALF.

[0015] Figure 4B This is another example of the shape of the filter used in ALF.

[0016] Figure 4C This is another example of the shape of the filter used in ALF.

[0017] Figure 5A This is a diagram representing the 67 intra-prediction modes of intra-frame prediction.

[0018] Figure 5B This is a flowchart illustrating the outline of predictive image correction processing based on OBMC processing.

[0019] Figure 5C This is a conceptual diagram used to illustrate the outline of predictive image correction processing based on OBMC processing.

[0020] Figure 5D This is a diagram representing an example of FRUC.

[0021] Figure 6 It is a diagram used to illustrate pattern matching (bidirectional matching) between two blocks along a motion trajectory.

[0022] Figure 7 It is a diagram used to illustrate pattern matching (template matching) between a template in the current image and a block in a reference image.

[0023] Figure 8 It is a diagram used to illustrate a model that assumes uniform linear motion.

[0024] Figure 9A It is a diagram used to illustrate the derivation of the motion vectors of sub-block units based on the motion vectors of multiple adjacent blocks.

[0025] Figure 9B This is a diagram used to illustrate the overview of motion vector derivation processing based on the merging mode.

[0026] Figure 9C This is a conceptual diagram used to illustrate the outline of DMVR processing.

[0027] Figure 9D This is a diagram used to illustrate an overview of a predictive image generation method that employs LIC-based brightness correction processing.

[0028] Figure 10 This is a block diagram illustrating the functional structure of the decoding device according to Embodiment 1.

[0029] Figure 11 This is a flowchart of the inter-screen prediction processing for Comparative Example 1.

[0030] Figure 12 This is a flowchart of the inter-screen prediction processing for Comparative Example 2.

[0031] Figure 13 This is a flowchart of the inter-screen prediction processing for Implementation Method 1.

[0032] Figure 14 This is a flowchart of the inter-screen prediction processing for a variation of Implementation 1.

[0033] Figure 15 This is a flowchart of the encoding and decoding processes for a variation of Implementation 1.

[0034] Figure 16 This is a conceptual diagram representing the template FRUC method of implementation method 1.

[0035] Figure 17This is a conceptual diagram representing the bidirectional FRUC method of implementation method 1.

[0036] Figure 18 This is a flowchart illustrating the action of deriving motion vectors in the FRUC method of implementation method 1.

[0037] Figure 19 This is a conceptual diagram representing the BIO processing of Implementation Method 1.

[0038] Figure 20 This is a flowchart illustrating the BIO processing of Implementation Method 1.

[0039] Figure 21 This is a block diagram showing an installation example of the encoding device according to Embodiment 1.

[0040] Figure 22 This is a block diagram showing an installation example of the decoding device according to Embodiment 1.

[0041] Figure 23 This is a diagram showing the overall structure of a content supply system that enables content distribution services.

[0042] Figure 24 This is a diagram illustrating an example of encoding construction in the case of hierarchical encoding.

[0043] Figure 25 This is a diagram illustrating an example of encoding construction in the case of hierarchical encoding.

[0044] Figure 26 This is an example of a web page display.

[0045] Figure 27 This is an example of a web page display.

[0046] Figure 28 This is a diagram illustrating an example of a smartphone.

[0047] Figure 29 This is a block diagram representing a structural example of a smartphone. Detailed Implementation

[0048] An encoding apparatus according to a technical solution of the present invention includes a circuit and a memory; the circuit uses the memory to divide the image contained in the moving image into prediction block units, derives a first motion vector through a first inter-frame prediction method, and performs a first motion compensation process using the prediction block units. The first inter-frame prediction method uses the fit of two reconstructed images of two regions in two different images. The first motion compensation process generates a prediction image with reference to the spatial gradient of brightness in the image generated by motion compensation using the derived first motion vector.

[0049] Therefore, this encoding apparatus reduces processing power by performing motion vector derivation and first motion compensation processing based on the first inter-frame prediction method on a prediction block basis, compared to, for example, performing these processes on a sub-block basis. Furthermore, since the first motion compensation processing, which generates the prediction image based on the spatial gradient of brightness, can be implemented in units smaller than the prediction block unit, the reduction in encoding efficiency that would occur without processing on a sub-block basis can be suppressed. Thus, this encoding apparatus can reduce processing power while suppressing a decrease in encoding efficiency.

[0050] For example, the circuit may also use the memory to derive a second motion vector in the form of a second prediction block using a second-frame prediction method, and perform a second motion compensation process in the form of a second prediction block to generate an encoded bitstream containing information for determining the second motion vector. The second motion vector is used to determine the fit of the reconstructed image of the region contained in the object prediction block and the reference image. The second motion compensation process generates a prediction image with reference to the spatial gradient of brightness in the image generated by motion compensation using the second motion vector.

[0051] Therefore, whether using the first-frame prediction method or the second-frame prediction method, the processing unit for motion compensation processing can be the same. This simplifies the installation of motion compensation processing.

[0052] For example, the two regions mentioned above in the first frame prediction method can also be (1) the region in the object image adjacent to the object prediction block and the region in the reference image, or (2) two regions in two different reference images.

[0053] A decoding apparatus according to a technical solution of the present invention includes a circuit and a memory; the circuit uses the memory to divide the image contained in the moving image into prediction block units, derives a first motion vector through a first inter-frame prediction method, and performs a first motion compensation process using the prediction block units. The first inter-frame prediction method uses the fit of two reconstructed images of two regions in two different images. The first motion compensation process generates a prediction image with reference to the spatial gradient of brightness in the image generated by motion compensation using the derived first motion vector.

[0054] Therefore, this decoding apparatus reduces processing power by performing motion vector derivation and first motion compensation processing based on the first inter-frame prediction method on a prediction block basis, compared to, for example, performing these processes on a sub-block basis. Furthermore, since the first motion compensation processing, which generates the predicted image based on the spatial gradient of brightness, can be implemented in units smaller than the prediction block unit, the reduction in coding efficiency that would occur without processing on a sub-block basis can be suppressed. Thus, this decoding apparatus can reduce processing power while suppressing a reduction in coding efficiency.

[0055] For example, the circuit may also use the memory to obtain information from the encoded bitstream for determining the second motion vector in the prediction block unit, derive the second motion vector in the prediction block unit using the second inter-frame prediction method using the information, and perform a second motion compensation process in the prediction block unit, which generates a predicted image with reference to the spatial gradient of brightness in the image generated by motion compensation using the derived second motion vector.

[0056] Therefore, whether using the first-frame prediction method or the second-frame prediction method, the processing unit for motion compensation processing can be the same. This simplifies the installation of motion compensation processing.

[0057] For example, the two regions mentioned above in the first frame prediction method can also be (1) the region in the object image adjacent to the object prediction block and the region in the reference image, or (2) two regions in two different reference images.

[0058] An encoding method for a technical solution of the present invention involves segmenting a motion image into prediction block units, deriving a first motion vector through a first-frame prediction method, and performing motion compensation processing using the aforementioned prediction block units. The first-frame prediction method uses the fit of two reconstructed images of two regions within two different images. The motion compensation processing generates a prediction image with reference to the spatial gradient of brightness in the image generated by motion compensation using the aforementioned derived first motion vector.

[0059] Therefore, this encoding method reduces processing complexity by performing motion vector derivation based on the first inter-frame prediction method and first motion compensation processing on a prediction block basis, compared to, for example, performing these processes on a sub-block basis. Furthermore, since the first motion compensation processing, which generates the prediction image based on the spatial gradient of brightness, can be implemented in units smaller than the prediction block unit, the reduction in encoding efficiency that would occur without processing on a sub-block basis can be suppressed. Thus, this encoding method reduces processing complexity while suppressing a decrease in encoding efficiency.

[0060] A decoding method of the present invention involves dividing a moving image into prediction block units, deriving a first motion vector through a first-frame prediction method, and performing motion compensation processing using the prediction block units. The first-frame prediction method uses the fit of two reconstructed images of two regions within two different images. The motion compensation processing generates a prediction image with reference to the spatial gradient of brightness in the image generated by motion compensation using the derived first motion vector.

[0061] Therefore, this decoding method reduces processing complexity by performing motion vector derivation based on the first inter-frame prediction method and first motion compensation processing on a prediction block basis, compared to, for example, performing these processes on a sub-block basis. Furthermore, since the first motion compensation processing, which generates the prediction image based on the spatial gradient of brightness, can be implemented in units smaller than the prediction block unit, the reduction in coding efficiency that would occur without processing on a sub-block basis can be suppressed. Thus, this decoding method reduces processing complexity while suppressing a decrease in coding efficiency.

[0062] An encoding apparatus according to a technical solution of the present invention includes a circuit and a memory. The circuit uses the memory to, in a first operating mode, derive a first motion vector in prediction block units that segment the image contained in the moving image, and perform a first motion compensation process in the prediction block units. This first motion compensation process generates a predicted image with reference to the spatial gradient of brightness in the image generated by motion compensation using the derived first motion vector. In a second operating mode, a second motion vector is derived in sub-block units that segment the prediction block, and a second motion compensation process is performed in the sub-block units. This second motion compensation process generates a predicted image without reference to the spatial gradient of brightness in the image generated by motion compensation using the second motion vector.

[0063] Therefore, in the first operating mode, the encoding apparatus reduces the processing workload by performing motion vector derivation and first motion compensation processing on a prediction block basis, compared to performing these processes on a sub-block basis. Furthermore, since the first motion compensation processing, which generates the prediction image based on the spatial gradient of brightness, can be implemented in units smaller than the prediction block unit, the reduction in encoding efficiency when processing is not performed on a sub-block basis can be suppressed. In addition, in the second operating mode, the encoding apparatus performs motion vector derivation and second motion compensation processing on a sub-block basis. Here, the second motion compensation processing, since it does not refer to the spatial gradient of brightness, requires less processing compared to the first motion compensation processing. Thus, by having these two operating modes, the encoding apparatus can improve encoding efficiency. In this way, the encoding apparatus can reduce the processing workload while suppressing the reduction in encoding efficiency.

[0064] For example, in the first operation mode, the circuit can derive the first motion vector in the prediction block unit using the first inter-frame prediction method, and in the second operation mode, derive the second motion vector in the sub-block unit using the second inter-frame prediction method, which is different from the first inter-frame prediction method.

[0065] For example, the second inter-image prediction method mentioned above can also be an inter-image prediction method that uses the fit of two reconstructed images of two regions within two different images.

[0066] Therefore, this inter-frame prediction method, which enables the calculation of motion vectors based on sub-block units, significantly improves coding efficiency. Thus, it enhances coding efficiency.

[0067] For example, the first inter-frame prediction method described above could be one of (1) the third inter-frame prediction method which uses the fit of the reconstructed image of the region in the object image adjacent to the object prediction block and the reconstructed image of the region in the reference image, and (2) the fourth inter-frame prediction method which uses the fit of the two reconstructed images of two regions in two different reference images, and the second inter-frame prediction method described above could be the other of the third inter-frame prediction method and the fourth inter-frame prediction method described above.

[0068] For example, the prediction method between the first frame described above could be the prediction method between the third frame described above, and the prediction method between the second frame described above could be the prediction method between the fourth frame described above.

[0069] Therefore, this inter-frame prediction method, which enables the calculation of motion vectors based on sub-block units, significantly improves coding efficiency. Thus, it enhances coding efficiency.

[0070] For example, the first inter-frame prediction method described above can also be an inter-frame prediction method that uses the fit of the reconstructed image of the region contained in the object prediction block and the reference image, and an encoded bitstream containing information for determining the first motion vector described above is generated.

[0071] A decoding apparatus according to a technical solution of the present invention includes a circuit and a memory. The circuit uses the memory to, in a first operating mode, derive a first motion vector in prediction block units that segment the image contained in the moving image, and perform a first motion compensation process in the prediction block units. This first motion compensation process generates a predicted image with reference to the spatial gradient of brightness in the image generated by motion compensation using the derived first motion vector. In a second operating mode, a second motion vector is derived in sub-block units that segment the prediction block, and a second motion compensation process is performed in the sub-block units. This second motion compensation process generates a predicted image without reference to the spatial gradient of brightness in the image generated by motion compensation using the second motion vector.

[0072] Therefore, in the first operating mode, the decoding device reduces the processing load by performing motion vector derivation and first motion compensation processing on a prediction block basis, compared to performing these processes on a sub-block basis. Furthermore, since the first motion compensation processing, which generates the prediction image based on the spatial gradient of brightness, can be implemented in units smaller than the prediction block unit, the reduction in coding efficiency when processing is not performed on a sub-block basis can be suppressed. In addition, in the second operating mode, the decoding device performs motion vector derivation and second motion compensation processing on a sub-block basis. Here, the second motion compensation processing, since it does not refer to the spatial gradient of brightness, requires less processing load compared to the first motion compensation processing. Thus, by having these two operating modes, the decoding device can improve coding efficiency. In this way, the encoding device can reduce the processing load while suppressing the reduction in coding efficiency.

[0073] For example, in the first operation mode, the circuit can derive the first motion vector in the prediction block unit using the first inter-frame prediction method, and in the second operation mode, derive the second motion vector in the sub-block unit using the second inter-frame prediction method, which is different from the first inter-frame prediction method.

[0074] For example, the second inter-image prediction method mentioned above can also be an inter-image prediction method that uses the fit of two reconstructed images from two different regions within two images.

[0075] Therefore, this inter-frame prediction method, which enables the calculation of motion vectors based on sub-block units, significantly improves coding efficiency. Thus, it enhances coding efficiency.

[0076] For example, the first inter-frame prediction method described above could be one of (1) the third inter-frame prediction method which uses the fit of the reconstructed image of the region in the object image adjacent to the object prediction block and the reconstructed image of the region in the reference image, and (2) the fourth inter-frame prediction method which uses the fit of the two reconstructed images of two regions in two different reference images, and the second inter-frame prediction method described above could be the other of the third inter-frame prediction method and the fourth inter-frame prediction method described above.

[0077] For example, the prediction method between the first frame described above could be the prediction method between the third frame described above, and the prediction method between the second frame described above could be the prediction method between the fourth frame described above.

[0078] Therefore, this inter-frame prediction method, which enables the calculation of motion vectors based on sub-block units, significantly improves coding efficiency. Thus, it enhances coding efficiency.

[0079] For example, in the first inter-frame prediction method described above, information for determining the first motion vector in the first prediction block unit can be obtained from the encoded bitstream, and the first motion vector can be derived using the information.

[0080] In a coding method relating to a technical solution of the present invention, in a first operation mode, a first motion vector is derived from a prediction block unit comprising a segmented image contained in a moving image, and a first motion compensation process is performed using the prediction block unit. The first motion compensation process generates a predicted image with reference to the spatial gradient of brightness in the image generated by motion compensation using the derived first motion vector. In a second operation mode, a second motion vector is derived from a sub-block unit comprising a segmented prediction block, and a second motion compensation process is performed using the sub-block unit. The second motion compensation process generates a predicted image without reference to the spatial gradient of brightness in the image generated by motion compensation using the second motion vector.

[0081] Therefore, in the first action mode, this encoding method reduces the processing workload by performing motion vector derivation and first motion compensation processing on a prediction block basis, compared to, for example, performing these processes on a sub-block basis. Furthermore, since the first motion compensation processing, which generates the prediction image based on the spatial gradient of brightness, can achieve correction in units smaller than the prediction block unit, it can suppress the reduction in encoding efficiency when processing is not performed on a sub-block basis. In addition, in the second action mode, this encoding method performs motion vector derivation and second motion compensation processing on a sub-block basis. Here, the second motion compensation processing, since it does not refer to the spatial gradient of brightness, requires less processing workload compared to the first motion compensation processing. Thus, by having these two action modes, this encoding method can improve encoding efficiency. In this way, this encoding method can reduce the processing workload while suppressing the reduction in encoding efficiency.

[0082] A decoding method according to a technical solution of the present invention, in a first operation mode, derives a first motion vector in a prediction block unit that segments the image contained in the motion image, and performs a first motion compensation process in the prediction block unit. The first motion compensation process generates a prediction image with reference to the spatial gradient of brightness in the image generated by motion compensation using the derived first motion vector. In a second operation mode, derives a second motion vector in a sub-block unit that segments the prediction block, and performs a second motion compensation process in the sub-block unit. The second motion compensation process generates a prediction image without reference to the spatial gradient of brightness in the image generated by motion compensation using the second motion vector.

[0083] Therefore, in the first action mode, this decoding method reduces the processing workload by performing motion vector derivation and first motion compensation processing on a prediction block basis, compared to, for example, performing these processes on a sub-block basis. Furthermore, since the first motion compensation processing, which generates the prediction image based on the spatial gradient of brightness, can be implemented in units smaller than the prediction block unit, the reduction in coding efficiency when processing is not performed on a sub-block basis can be suppressed. In addition, in the second action mode, this decoding method performs motion vector derivation and second motion compensation processing on a sub-block basis. Here, the second motion compensation processing, since it does not refer to the spatial gradient of brightness, requires less processing workload compared to the first motion compensation processing. Thus, by having these two action modes, this decoding method can improve coding efficiency. In this way, the coding method can reduce the processing workload while suppressing the reduction in coding efficiency.

[0084] In addition, these inclusive or specific technical solutions can also be implemented by non-transitory recording media such as systems, devices, methods, integrated circuits, computer programs, or computer-readable CD-ROMs, or by any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0085] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings.

[0086] Furthermore, the embodiments described below are inclusive or specific examples. The numerical values, shapes, materials, constituent elements, arrangements and connection methods of constituent elements, steps, and sequences of steps shown in the following embodiments are examples and are not intended to limit the scope of the claims. In addition, any constituent elements in the following embodiments that are not described in the independent claim representing the highest-level concept are described as arbitrary constituent elements.

[0087] (Implementation Method 1) First, as an example of an encoding and decoding apparatus for the processing and / or structure described in the various embodiments of the present invention described later, an outline of Embodiment 1 will be described. However, Embodiment 1 is merely an example of an encoding and decoding apparatus for the processing and / or structure described in the various embodiments of the present invention, and the processing and / or structure described in the various embodiments of the present invention can also be implemented in encoding and decoding apparatuses different from Embodiment 1.

[0088] When applying the processing and / or structure described in various aspects of the present invention to Embodiment 1, one of the following may also be performed, for example.

[0089] (1) For the encoding or decoding device of Embodiment 1, the constituent element that corresponds to the constituent element described in each aspect of the present invention is replaced with the constituent element described in each aspect of the present invention. (2) For the encoding or decoding device of Embodiment 1, after any modification such as adding, replacing, or deleting any of the constituent elements of the plurality of constituent elements constituting the encoding or decoding device, the constituent elements corresponding to the constituent elements described in each aspect of the present invention are replaced with the constituent elements described in each aspect of the present invention. (3) After adding processing to the method implemented by the encoding or decoding device of Embodiment 1, and / or replacing or deleting any of the processing of a plurality of processing included in the method, the processing corresponding to the processing described in each aspect of the present invention is replaced with the processing described in each aspect of the present invention. (4) A portion of the constituent elements constituting the encoding or decoding apparatus of Embodiment 1 are combined with constituent elements described in various aspects of the present invention, a portion of constituent elements having the functions of constituent elements described in various aspects of the present invention, or a portion of constituent elements implementing the processing performed by constituent elements described in various aspects of the present invention. (5) A component having a portion of the functions of a portion of the components constituting the encoding or decoding apparatus of embodiment 1, or a component implementing a portion of the processing performed by a portion of the components constituting the encoding or decoding apparatus of embodiment 1, is combined with the components described in various aspects of the present invention, the components having a portion of the functions of the components described in various aspects of the present invention, or the components implementing a portion of the processing performed by the components described in various aspects of the present invention. (6) For the method implemented by the encoding or decoding apparatus of Embodiment 1, the processing that corresponds to the processing described in each aspect of the present invention among the multiple processing included in the method is replaced with the processing described in each aspect of the present invention. (7) A portion of the processing included in the method implemented by the encoding or decoding apparatus of Embodiment 1 is combined with the processing described in the various aspects of the present invention.

[0090] Furthermore, the implementation of the processes and / or structures described in the various embodiments of the present invention is not limited to the examples described above. For example, it may be implemented in an apparatus used for a different purpose than the moving image / image encoding apparatus or moving image / image decoding apparatus disclosed in Embodiment 1, or the processes and / or structures described in each embodiment may be implemented individually. In addition, the processes and / or structures described in different embodiments may be combined and implemented.

[0091] [Overview of the encoding device] First, an overview of the encoding device for Embodiment 1 will be provided. Figure 1 This is a block diagram illustrating the functional structure of the encoding apparatus 100 according to Embodiment 1. The encoding apparatus 100 is a motion picture / image encoding apparatus that encodes motion pictures / images in block units.

[0092] like Figure 1As shown, the encoding device 100 is a device for encoding images in block units, and includes a segmentation unit 102, a subtraction unit 104, a transformation unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse transformation unit 114, an addition unit 116, a block memory 118, a cyclic filtering unit 120, a frame memory 122, an intra-frame prediction unit 124, an inter-frame prediction unit 126, and a prediction control unit 128.

[0093] The encoding device 100 is implemented, for example, by a general-purpose processor and memory. In this case, when the software program stored in the memory is executed by the processor, the processor functions as the segmentation unit 102, subtraction unit 104, transform unit 106, quantization unit 108, entropy coding unit 110, inverse quantization unit 112, inverse transform unit 114, addition unit 116, cyclic filtering unit 120, intra-frame prediction unit 124, inter-frame prediction unit 126, and prediction control unit 128. Alternatively, the encoding device 100 may be implemented as one or more dedicated electronic circuits corresponding to the segmentation unit 102, subtraction unit 104, transform unit 106, quantization unit 108, entropy coding unit 110, inverse quantization unit 112, inverse transform unit 114, addition unit 116, cyclic filtering unit 120, intra-frame prediction unit 124, inter-frame prediction unit 126, and prediction control unit 128.

[0094] The following describes the constituent elements included in the encoding device 100.

[0095] [Divider] The segmentation unit 102 divides each image contained in the input moving image into multiple blocks and outputs each block to the subtraction unit 104. For example, the segmentation unit 102 first segments the image into fixed-size blocks (e.g., 128×128). These fixed-size blocks may be called coding tree units (CTUs). Furthermore, the segmentation unit 102 divides each fixed-size block into variable-size blocks (e.g., 64×64 or less) based on recursive quadtree and / or binary tree block segmentation. These variable-size blocks may be called coding units (CUs), prediction units (PUs), or transform units (TUs). In addition, in this embodiment, it is not necessary to distinguish between CUs, PUs, and TUs, and some or all of the blocks in the image may be used as processing units of CUs, PUs, and TUs.

[0096] Figure 2 This is a diagram illustrating an example of block segmentation in Implementation Method 1. In Figure 2 In the diagram, solid lines represent block boundaries based on quadtree block partitioning, and dashed lines represent block boundaries based on binary tree block partitioning.

[0097] Here, block 10 is a square block of 128×128 pixels (128×128 block). This 128×128 block 10 is first divided into 4 square blocks of 64×64 (quadtree block partitioning).

[0098] The 64×64 block in the upper left corner is then vertically divided into two rectangular 32×64 blocks, and the 32×64 block on the left is then vertically divided into two rectangular 16×64 blocks (binary tree block partitioning). As a result, the 64×64 block in the upper left corner is divided into two 16×64 blocks (11 and 12) and a 32×64 block (13).

[0099] The 64×64 block in the upper right corner is horizontally divided into two rectangular 64×32 blocks, 14 and 15 (binary tree block division).

[0100] The 64×64 block in the lower left corner is divided into four 32×32 square blocks (quadtree block partitioning). The upper left and lower right blocks of these four 32×32 blocks are further partitioned. The upper left 32×32 block is vertically divided into two 16×32 rectangular blocks, and the right 16×32 block is horizontally divided into two 16×16 blocks (binary tree block partitioning). The lower right 32×32 block is horizontally divided into two 32×16 blocks (binary tree block partitioning). As a result, the lower left 64×64 block is divided into 16×32 block 16, two 16×16 blocks 17 and 18, two 32×32 blocks 19 and 20, and two 32×16 blocks 21 and 22.

[0101] The 64×64 block 23 in the lower right corner is not divided.

[0102] As described above, in Figure 2 In the example, block 10 is divided into 13 variable-size blocks 11 to 23 based on recursive quadtree and binary tree block partitioning. Such partitioning is sometimes referred to as QTBT (quadtree plus binary tree) partitioning.

[0103] In addition, Figure 2 In this context, a block can be divided into 2 or 4 blocks (quadtree or binary tree block partitioning), but the partitioning is not limited to these. For example, a block can also be divided into 3 blocks (ternary tree partitioning). Partitioning including such ternary tree partitioning is sometimes referred to as MBT (multi-type tree) partitioning.

[0104] [Subtraction Section] The subtraction unit 104 subtracts the prediction signal (prediction sample) from the original signal (original sample) in block units divided by the segmentation unit 102. That is, the subtraction unit 104 calculates the prediction error (also called residual) of the encoded target block (hereinafter referred to as the current block). Furthermore, the subtraction unit 104 outputs the calculated prediction error to the transformation unit 106.

[0105] The original signal is the input signal of the encoding device 100, which is the signal representing the image of each picture that constitutes the moving image (e.g., luminance signal and two chroma signals). Hereinafter, the signal representing the image may also be referred to as a sample.

[0106] [Transformation Section] The transformation unit 106 transforms the prediction error in the spatial domain into transformation coefficients in the frequency domain, and outputs the transformation coefficients vectorization unit 108. Specifically, the transformation unit 106 performs a preset discrete cosine transform (DCT) or discrete sine transform (DST) on the prediction error in the spatial domain, for example.

[0107] Alternatively, the transform unit 106 can adaptively select a transform type from multiple transform types and use the transform basis function corresponding to the selected transform type to transform the prediction error into transform coefficients. Such a transform is sometimes referred to as EMT (explicit multiple core transform) or AMT (adaptive multiple transform).

[0108] Several transformation types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII. Figure 3 This is a table representing the transformation basis functions corresponding to each transformation type. Figure 3 In this context, N represents the number of input pixels. The choice of transform type from these multiple transform types can depend on the type of prediction (intra-frame prediction and inter-frame prediction) or the intra-frame prediction mode.

[0109] Information indicating whether such EMT or AMT is applied (e.g., referred to as the AMT flag) and information indicating the selected transform type are signaled at the CU level. Furthermore, the signaling of this information is not limited to the CU level and can also be at other levels (e.g., sequence level, image level, slice level, tile level, or CTU level).

[0110] Furthermore, the transform unit 106 can also perform a re-transformation on the transform coefficients (transformation results). Such a re-transformation may be referred to as AST (adaptive secondary transform) or NSST (non-separable secondary transform). For example, the transform unit 106 performs a re-transformation on each sub-block (e.g., a 4×4 sub-block) contained in the block of transform coefficients corresponding to the intra-frame prediction error. Information indicating whether NSST is applied and information related to the transform matrix used in NSST are signaled at the CU level. In addition, the signaling of this information is not limited to the CU level and can also be at other levels (e.g., sequence level, picture level, slice level, tile level, or CTU level).

[0111] Here, a separable transformation refers to a method of performing multiple transformations by separating the input in each direction, which is equivalent to the number of dimensions of the input. A non-separable transformation refers to a method of treating two or more dimensions together as one dimension and transforming them together when the input is multidimensional.

[0112] For example, as one example of a non-separable transformation, one can cite the way that when the input is a 4×4 block, it is treated as a permutation of 16 elements, and the permutation is transformed using a 16×16 transformation matrix.

[0113] Furthermore, the Hypercube Givens Transform, which treats a 4×4 input block as a permutation of 16 elements and then performs multiple Givens rotations on that permutation, is also an example of a non-separable transformation.

[0114] [Quantitative Department] The quantization unit 108 quantizes the transform coefficients output from the transform unit 106. Specifically, the quantization unit 108 scans the transform coefficients of the current block in a predetermined scan order and quantizes the transform coefficients based on the quantization parameters (QP) corresponding to the scanned transform coefficients. Furthermore, the quantization unit 108 outputs the quantized transform coefficients (hereinafter referred to as quantized coefficients) of the current block to the entropy encoding unit 110 and the inverse quantization unit 112.

[0115] The specified order is the order in which the transform coefficients are quantized / dequantized. For example, the specified scan order is defined by ascending frequency (from low frequency to high frequency) or descending frequency (from high frequency to low frequency).

[0116] The quantization parameter is a parameter that defines the quantization step size (quantization width). For example, if the value of the quantization parameter increases, the quantization step size also increases. That is, if the value of the quantization parameter increases, the quantization error increases.

[0117] [Entropy Coding Department] The entropy coding unit 110 generates a coded signal (coded bitstream) by performing variable-length coding on the quantization coefficients, which are input from the quantization unit 108. Specifically, the entropy coding unit 110 performs arithmetic coding on the binary signal, for example, by binarizing the quantization coefficients.

[0118] [Inverse Quantization Department] The inverse quantization unit 112 performs inverse quantization on the quantization coefficients that are input from the quantization unit 108. Specifically, the inverse quantization unit 112 performs inverse quantization on the quantization coefficients of the current block in a predetermined scan order. Furthermore, the inverse quantization unit 112 outputs the inverse quantized transform coefficients of the current block to the inverse transform unit 114.

[0119] [Inverse Transformation Section] The inverse transform unit 114 restores the prediction error by performing an inverse transform on the transform coefficients, which are input from the inverse quantization unit 112. Specifically, the inverse transform unit 114 restores the prediction error of the current block by performing an inverse transform on the transform coefficients corresponding to the transform of the transform unit 106. Furthermore, the inverse transform unit 114 outputs the restored prediction error to the adder unit 116.

[0120] Furthermore, the restored prediction error differs from the prediction error calculated by the subtraction unit 104 because information was lost during quantization. In other words, the restored prediction error includes quantization error.

[0121] [Addition Department] The addition unit 116 reconstructs the current block by adding the prediction error, which is input from the inverse transform unit 114, to the prediction sample, which is input from the prediction control unit 128. Furthermore, the addition unit 116 outputs the reconstructed block to the block memory 118 and the cyclic filtering unit 120. The reconstructed block may be referred to as a local decoding block.

[0122] [Block Memory] Block memory 118 is a storage unit used to save blocks within the encoded object image (hereinafter referred to as the current image) referenced in intra-frame prediction. Specifically, block memory 118 saves the reconstructed blocks output from addition unit 116.

[0123] [Loop Filtering Section] The cyclic filtering unit 120 applies cyclic filtering to the block reconstructed by the addition unit 116 and outputs the filtered reconstructed block to the frame memory 122. Cyclic filtering refers to filtering used within the encoding loop (in-loop filtering), such as deblocking filtering (DF), sample adaptive offset (SAO), and adaptive cyclic filtering (ALF).

[0124] In ALF, a least-squares error filter is used to remove coding distortion. For example, for each 2×2 sub-block within the current block, one filter is selected from multiple filters based on the direction of the gradient and the activity of the locality.

[0125] Specifically, sub-blocks (e.g., 2×2 sub-blocks) are first classified into multiple classes (e.g., 15 or 25 classes). The classification of sub-blocks is based on the direction and activity of the gradient. For example, using the gradient direction value D (e.g., 0–2 or 0–4) and the gradient activity value A (e.g., 0–4), a classification value C (e.g., C = 5D + A) is calculated. Then, based on the classification value C, the sub-blocks are classified into multiple classes (e.g., 15 or 25 classes).

[0126] The gradient direction value D is derived, for example, by comparing gradients in multiple directions (e.g., horizontal, vertical, and two diagonal directions). Furthermore, the gradient activity value A is derived, for example, by summing the gradients in multiple directions and quantizing the sum.

[0127] Based on the results of this classification, the filter used for the sub-block is determined from among multiple filters.

[0128] The shape of the filter used in ALF can be, for example, a circular symmetrical shape. Figures 4A to 4C This is a diagram showing several examples of the shapes of filters used in ALF. Figure 4A This indicates a 5×5 diamond-shaped filter. Figure 4B This indicates a 7×7 diamond-shaped filter. Figure 4C This represents a 9×9 diamond-shaped filter. The information representing the filter's shape is signaled at the image level. However, the signaling of the filter's shape information is not limited to the image level; it can also be at other levels (e.g., sequence level, slice level, tile level, CTU level, or CU level).

[0129] The on / off state of ALF is determined, for example, at the picture level or the CU level. For instance, regarding luminance, the decision to use ALF is made at the CU level, while regarding chromatic aberration, the decision is made at the picture level. Information indicating the on / off state of ALF is signaled at the picture level or the CU level. However, the signaling of information indicating the on / off state of ALF is not limited to the picture level or the CU level; it can also be at other levels (e.g., sequence level, slice level, tile level, or CTU level).

[0130] The coefficient set of a selectable number of filters (e.g., up to 15 or 25 filters) is signaled at the picture level. Furthermore, the signaling of the coefficient set is not limited to the picture level; it can also be at other levels (e.g., sequence level, slice level, tile level, CTU level, CU level, or sub-block level).

[0131] [Frame Memory] The frame memory 122 is a storage unit used to store reference images used in inter-frame prediction, and is also sometimes referred to as a frame buffer. Specifically, the frame memory 122 stores the reconstructed blocks filtered by the cyclic filtering unit 120.

[0132] Intra-frame prediction unit The intra-frame prediction unit 124 performs intra-frame prediction (also called intra-picture prediction) of the current block by referring to the blocks in the current image stored in the block memory 118, thereby generating a prediction signal (intra-frame prediction signal). Specifically, the intra-frame prediction unit 124 generates an intra-frame prediction signal by performing intra-frame prediction by referring to samples (e.g., luminance value, chrominance value) of blocks adjacent to the current block, and outputs the intra-frame prediction signal to the prediction control unit 128.

[0133] For example, the intra-prediction unit 124 performs intra-prediction using one of a plurality of predefined intra-prediction modes. The plurality of intra-prediction modes includes one or more non-directional prediction modes and a plurality of directional prediction modes.

[0134] One or more non-directional prediction modes include, for example, the Planar prediction mode and the DC prediction mode as specified by the H.265 / HEVC (High-Efficiency Video Coding) specification (Non-Patent Document 1).

[0135] Multiple directional prediction modes may include, for example, the 33-directional prediction modes specified in the H.265 / HEVC specification. Alternatively, multiple directional prediction modes may also include 32 additional-directional prediction modes (totaling 65 directional prediction modes). Figure 5A This diagram represents the 67 intra-prediction modes (2 non-directional prediction modes and 65 directional prediction modes) in intra-frame prediction. Solid arrows indicate the 33 directions specified by the H.265 / HEVC specification, while dashed arrows indicate the additional 32 directions.

[0136] Additionally, in intra-frame prediction of chroma blocks, luma blocks can also be referenced. That is, the chroma components of the current block can be predicted based on the luma components of the current block. Such intra-frame prediction is sometimes referred to as CCLM (cross-component linear model) prediction. This intra-frame prediction mode of chroma blocks referencing luma blocks (e.g., called CCLM mode) can also be added as one of the intra-frame prediction modes for chroma blocks.

[0137] The intra-prediction unit 124 can also correct the intra-predicted pixel values ​​based on the gradient of the reference pixels in the horizontal / vertical directions. Intra-prediction accompanied by such correction is sometimes referred to as PDPC (position-dependent intraprediction combination). Information indicating whether PDPC has been used (e.g., a PDPC flag) is signaled, for example, at the CU level. Furthermore, the signaling of this information is not limited to the CU level; it can also be at other levels (e.g., sequence level, image level, slice level, tile level, or CTU level).

[0138] [Inter-frame prediction unit] The inter-frame prediction unit 126 performs inter-frame prediction (also called inter-picture prediction) of the current block by referring to a reference picture stored in the frame memory 122 that is different from the current picture, thereby generating a prediction signal (inter-frame prediction signal). Inter-frame prediction is performed in units of the current block or sub-blocks within the current block (e.g., 4×4 blocks). For example, the inter-frame prediction unit 126 performs motion estimation within the reference picture for the current block or sub-block. Furthermore, the inter-frame prediction unit 126 uses motion information (e.g., motion vectors) obtained through motion estimation to perform motion compensation, thereby generating the inter-frame prediction signal for the current block or sub-block. Finally, the inter-frame prediction unit 126 outputs the generated inter-frame prediction signal to the prediction control unit 128.

[0139] The motion information used in motion compensation is signaled. A motion vector predictor can also be used in the signaling of motion vectors. That is, the difference between the motion vector and the predicted motion vector can also be signaled.

[0140] Alternatively, the inter-frame prediction signal can be generated using not only the motion information of the current block obtained through motion estimation but also the motion information of neighboring blocks. Specifically, the prediction signal based on the motion information obtained through motion estimation can be weighted and added together with the prediction signal based on the motion information of neighboring blocks, thereby generating the inter-frame prediction signal in sub-block units within the current block. Such inter-frame prediction (motion compensation) is sometimes referred to as OBMC (overlapped block motion compensation).

[0141] In this OBMC mode, information indicating the size of the sub-block used for OBMC (e.g., OBMC block size) is signaled at the sequence level. Furthermore, information indicating whether OBMC mode is used (e.g., OBMC flag) is signaled at the CU level. However, the signaling level for this information is not limited to the sequence and CU levels; it can also be other levels (e.g., image level, slice level, tile level, CTU level, or sub-block level).

[0142] The OBMC model will be explained in more detail. Figure 5B and Figure 5C This is a flowchart and concept diagram used to illustrate the outline of predictive image correction processing based on OBMC processing.

[0143] First, the predicted image (Pred) obtained through normal motion compensation is obtained using the motion vectors (MV) assigned to the encoded object block.

[0144] Next, the motion vector (MV_L) of the encoded left adjacent block is used to obtain the prediction image (Pred_L) for the encoded object block. The prediction image is then corrected for the first time by weighted superposition of the prediction image and Pred_L.

[0145] Similarly, the motion vector (MV_U) of the upper adjacent block of the encoded object block is used to obtain the prediction image (Pred_U). The prediction image is then corrected a second time by weighting and superimposing the prediction image after the first correction and Pred_U, and this is used as the final prediction image.

[0146] In addition, this paper describes a two-stage correction method using the left adjacent block and the top adjacent block, but it can also be configured to perform more corrections than two stages using the right adjacent block and the bottom adjacent block.

[0147] In addition, the area to be overlaid may not be the entire pixel area of ​​the block, but only a part of the area near the block boundary.

[0148] Furthermore, the process of correcting the predicted image based on a single reference image is explained here. However, the same principle applies when correcting the predicted image based on multiple reference images. After obtaining the corrected predicted image based on each reference image, the resulting predicted images are further superimposed to obtain the final predicted image.

[0149] In addition, the processing target block mentioned above can be a prediction block unit or a sub-block unit that further divides the prediction block.

[0150] One method for determining whether to use OBMC processing is to use a signal called obmc_flag. Specifically, in an encoding device, it is determined whether the block to be encoded belongs to a motion-complex region. If it does, the obmc_flag is set to 1 and OBMC processing is performed for encoding. If it does not belong to a motion-complex region, the obmc_flag is set to 0, and OBMC processing is not performed for encoding. On the other hand, in a decoding device, decoding is performed by decoding the obmc_flag recorded in the stream and switching between using and not using OBMC processing based on its value.

[0151] Alternatively, motion information can be exported at the decoding device side without being signaled. For example, the merging mode specified by the H.265 / HEVC standard can be used. Furthermore, motion information can also be exported by performing motion estimation at the decoding device side. In this case, motion estimation is performed without using the pixel values ​​of the current block.

[0152] Here, we will explain the motion estimation mode performed on the decoding device side. This motion estimation mode on the decoding device side may be called PMMVD (pattern matched motion vector derivation) mode or FRUC (frame rate up-conversion) mode.

[0153] exist Figure 5D The diagram below illustrates an example of FRUC processing. First, referencing the motion vectors of coded blocks spatially or temporally adjacent to the current block, a list of multiple candidates, each with a predicted motion vector, is generated (this list can also be shared with the merge list). Next, the best candidate MV is selected from the multiple candidate MVs registered in the candidate list. For example, an evaluation value is calculated for each candidate included in the candidate list, and one candidate is selected based on the evaluation value.

[0154] Furthermore, based on the selected candidate motion vectors, motion vectors for the current block are derived. Specifically, for example, the selected candidate motion vector (best candidate MV) can be derived as is, and used as the motion vector for the current block. Alternatively, for example, the motion vector for the current block can be derived by performing pattern matching in the surrounding region of the position within the reference image corresponding to the selected candidate motion vector. That is, the surrounding region of the best candidate MV can be searched using the same method, and if an MV with a better evaluation value is found, the best candidate MV is updated to the aforementioned MV and used as the final MV for the current block. Alternatively, a structure that does not perform this processing can be implemented.

[0155] The exact same processing can also be performed when processing is done in sub-block units.

[0156] Furthermore, the evaluation value is calculated by obtaining the difference value of the reconstructed image through pattern matching between the region within the reference image corresponding to the motion vector and the specified region. Alternatively, information other than the difference value can be used to calculate the evaluation value.

[0157] As a pattern matching, either pattern matching 1 or pattern matching 2 is used. Pattern matching 1 and pattern matching 2 can be referred to as bilateral matching and template matching, respectively.

[0158] In the first pattern matching, pattern matching is performed between two blocks within two different reference images, along the motion trajectory of the current block. Therefore, in the first pattern matching, the regions within other reference images along the motion trajectory of the current block are used as the defined regions for calculating the candidate evaluation values ​​described above.

[0159] Figure 6 This diagram illustrates an example of pattern matching (bidirectional matching) between two blocks along a motion trajectory. For example... Figure 6 As shown, in the first pattern matching, two motion vectors (MV0, MV1) are derived by searching for the best matching pair among two blocks in two different reference images (Ref0, Ref1) along the motion trajectory of the current block. Specifically, for the current block, the difference between the reconstructed image at a specified position in the first encoded reference image (Ref0) specified by the candidate MV and the reconstructed image at a specified position in the second encoded reference image (Ref1) specified by the symmetrical MV scaled by the aforementioned candidate MV over the display time interval is derived, and the obtained difference value is used to calculate an evaluation value. The candidate MV with the best evaluation value can be selected as the final MV from among multiple candidate MVs.

[0160] Under the assumption of continuous motion trajectories, the motion vectors (MV0, MV1) indicating two reference blocks are proportional to the temporal distances (TD0, TD1) between the current image (Cur Pic) and the two reference images (Ref0, Ref1). For example, in the case where the current image is located between the two reference images in time and the temporal distances from the current image to the two reference images are equal, in the first pattern matching, mirror-symmetric bidirectional motion vectors are derived.

[0161] In the second pattern matching, pattern matching is performed between the template in the current image (the block adjacent to the current block in the current image (e.g., the upper and / or left adjacent block)) and the block in the reference image. Therefore, in the second pattern matching, the block adjacent to the current block in the current image is used as the defined area for calculating the candidate evaluation value as described above.

[0162] Figure 7 This is an example of pattern matching (template matching) between a template in the current image and a block in a reference image. For example... Figure 7 As shown, in the second pattern matching, the motion vector of the current block is derived by searching within the reference image (Ref0) for the block that best matches the block adjacent to the current block (Cur block) within the current image (Cur Pic). Specifically, for the current block, the difference between the reconstructed image of the encoded regions of the left and top adjacent regions or one of them and the reconstructed image at the same position within the encoded reference image (Ref0) specified by the candidate MV is derived. The obtained difference value is used to calculate the evaluation value, and the candidate MV with the best evaluation value among multiple candidate MVs is selected as the best candidate MV.

[0163] Information indicating whether FRUC mode is used (e.g., referred to as the FRUC flag) is signaled at the CU level. Furthermore, when FRUC mode is used (e.g., when the FRUC flag is true), information indicating the pattern matching method (first pattern matching or second pattern matching) (e.g., referred to as the FRUC mode flag) is signaled at the CU level. Additionally, the signaling of this information is not limited to the CU level and can also be at other levels (e.g., sequence level, picture level, slice level, tile level, CTU level, or sub-block level).

[0164] This section explains how to derive motion vectors based on a model that assumes uniform linear motion. This model can be termed BIO (bi-directional optical flow).

[0165] Figure 8 This diagram is used to illustrate a model that assumes uniform linear motion. Figure 8 In the middle, (v x v y () represents the velocity vector, and τ0 and τ1 represent the time distances between the current image (Cur Pic) and the two reference images (Ref0, Ref1), respectively. (MVx0, MVy0) represents the motion vector corresponding to the reference image Ref0, and (MVx1, MVy1) represents the motion vector corresponding to the reference image Ref1.

[0166] At this time, in the velocity vector (v x v y Under the assumption of constant linear motion of MVx0, MVy0 and MVx1, MVy1 are expressed as (vxτ0, vyτ0) and (-vxτ1, -vyτ1) respectively, and the following optical flow equation (1) holds.

[0167] [Formula 1] Here, I (k) This represents the luminance value of the reference image k (k=0, 1) after motion compensation. The optical flow equation states that the sum of (i) the temporal derivative of the luminance value, (ii) the product of the horizontal velocity and the horizontal component of the spatial gradient of the reference image, and (iii) the product of the vertical velocity and the vertical component of the spatial gradient of the reference image is equal to zero. Based on this optical flow equation combined with Hermite interpolation, the block-unit motion vector obtained from merge lists, etc., is corrected in pixels.

[0168] Alternatively, motion vectors can be derived on the decoding device side using a different method than deriving motion vectors based on a model assuming constant linear motion. For example, motion vectors can be derived on a sub-block basis based on the motion vectors of multiple adjacent blocks.

[0169] Here, we will explain the mode of deriving motion vectors on a sub-block basis based on the motion vectors of multiple adjacent blocks. This mode is sometimes referred to as the affine motion compensation prediction mode.

[0170] Figure 9A This is a diagram used to illustrate the derivation of sub-block unit motion vectors based on the motion vectors of multiple adjacent blocks. Figure 9AIn this context, the current block comprises 16 4×4 sub-blocks. Here, based on the motion vectors of adjacent blocks, the motion vector v0 of the top-left control point of the current block is derived, and based on the motion vectors of adjacent sub-blocks, the motion vector v1 of the top-right control point of the current block is derived. Furthermore, using the two motion vectors v0 and v1, the motion vectors (v0, v1, v1) of each sub-block within the current block are derived using the following equation (2). x v y ).

[0171] [Formula 2] Here, x and y represent the horizontal and vertical positions of the sub-block, respectively, and w represents the pre-set weight coefficient.

[0172] Such affine motion compensation prediction modes may also include several modes with different methods for deriving the motion vectors of the upper left and upper right control points. Information representing such affine motion compensation prediction modes (e.g., affine flags) is signaled at the CU level. Furthermore, the signaling of information representing such affine motion compensation prediction modes is not limited to the CU level; it can also be at other levels (e.g., sequence level, image level, slice level, tile level, CTU level, or sub-block level).

[0173] [Predictive Control Department] The prediction control unit 128 selects one of the intra-frame prediction signal and the inter-frame prediction signal, and outputs the selected signal as the prediction signal to the subtraction unit 104 and the addition unit 116.

[0174] This section illustrates an example of exporting motion vectors from an encoded object image using a merge mode. Figure 9B This is a diagram used to illustrate the overview of motion vector derivation processing based on the merging mode.

[0175] First, a list of candidate predicted MVs registered with the predicted MVs is generated. Candidate predicted MVs include: spatially adjacent predicted MVs (MVs) belonging to multiple coded blocks spatially surrounding the coded object block; temporally adjacent predicted MVs (MVs) belonging to blocks whose positions in the coded reference image are projected nearby; combined predicted MVs (MVs) generated by combining the MV values ​​of spatially adjacent and temporally adjacent predicted MVs; and zero predicted MVs (MVs with a value of zero).

[0176] Next, the MV for the encoded object block is determined by selecting one predicted MV from the multiple predicted MVs registered in the predicted MV list.

[0177] Furthermore, in the variable-length coding section, merge_idx, which represents the signal that selected which prediction MV was recorded in the stream and encoded.

[0178] In addition, Figure 9B The predicted MVs registered in the predicted MV list described in the figure are one example. They may also be a number different from the number shown in the figure, or a structure that does not include a part of the predicted MVs in the figure, or a structure that adds predicted MVs other than the predicted MVs in the figure.

[0179] Alternatively, the MV of the encoded object block exported through the merge mode can be used for the DMVR processing described later to determine the final MV.

[0180] Here, an example of using DMVR to determine MV is explained.

[0181] Figure 9C This is a conceptual diagram used to illustrate the outline of DMVR processing.

[0182] First, the optimal MVP set for the processing object block is taken as the candidate MV. According to the candidate MV, reference pixels are obtained from the first reference image of the processed image in the L0 direction and the second reference image of the processed image in the L1 direction, respectively. The template is generated by taking the average of each reference pixel.

[0183] Next, using the template described above, the surrounding areas of the candidate music videos (MVs) for the first and second reference images are searched, and the MV with the lowest cost is selected as the final MV. Furthermore, the cost value is calculated using the differences between the pixel values ​​of the template and the pixel values ​​of the search area, as well as the MV value.

[0184] Furthermore, the general outline of the processing described herein is essentially the same in both the encoding and decoding devices.

[0185] In addition, even if it is not the process described here, any other process that can search for the surrounding of candidate MVs and export the final MV can be used.

[0186] Here, the mode of generating predicted images using LIC processing is explained.

[0187] Figure 9D This is a diagram illustrating the outline of a predictive image generation method using LIC-based brightness correction processing.

[0188] First, export the MV used to obtain the reference image corresponding to the encoded object block from the reference image, which is an encoded image.

[0189] Next, for the encoded object block, using the brightness pixel values ​​of the left and top adjacent encoded surrounding reference areas and the brightness pixel values ​​at the same position in the reference image specified by MV, information indicating how the brightness values ​​change in the reference image and the encoded object image is extracted, and brightness correction parameters are calculated.

[0190] By using the aforementioned brightness correction parameters to perform brightness correction processing on the reference image within the reference image specified by MV, a predicted image for the coded object block is generated.

[0191] in addition, Figure 9D The shape of the surrounding reference area mentioned above is one example; other shapes may also be used.

[0192] Furthermore, the process of generating a prediction image based on a single reference image is described here, but the same applies when generating a prediction image based on multiple reference images. The prediction image is generated after performing brightness correction processing on the reference images obtained from each reference image in the same way.

[0193] One method for determining whether to use LIC processing is to use a lic_flag as a signal indicating whether LIC processing is used. Specifically, in an encoding device, it is determined whether the block to be encoded belongs to a region where a brightness change has occurred. If it does, the lic_flag is set to 1, and LIC processing is used for encoding. If it does not belong to a region where a brightness change has occurred, the lic_flag is set to 0, and LIC processing is not used for encoding. On the other hand, in a decoding device, decoding is performed by decoding the lic_flag recorded in the stream and switching between using and not using LIC processing based on its value.

[0194] Other methods for determining whether to use LIC processing include checking whether LIC processing was used in surrounding blocks. As a specific example, when the encoded target block is in merge mode, it is determined whether the surrounding encoded blocks selected during the export of the MV in merge mode processing have been encoded using LIC processing. Based on the result, encoding is switched between using LIC processing and other methods. Furthermore, in this example, the decoding process is exactly the same.

[0195] [Overview of the decoding device] Next, an outline of a decoding apparatus capable of decoding the encoded signal (encoded bit stream) output from the encoding apparatus 100 will be described. Figure 10 This is a block diagram illustrating the functional structure of the decoding device 200 according to Embodiment 1. The decoding device 200 is a motion image / image decoding device that decodes motion images / images in block units.

[0196] like Figure 10As shown, the decoding device 200 includes an entropy decoding unit 202, an inverse quantization unit 204, an inverse transform unit 206, an adder unit 208, a block memory 210, a cyclic filtering unit 212, a frame memory 214, an intra-frame prediction unit 216, an inter-frame prediction unit 218, and a prediction control unit 220.

[0197] The decoding device 200 is implemented, for example, by a general-purpose processor and memory. In this case, when the processor executes the software program stored in the memory, the processor functions as the entropy decoding unit 202, the inverse quantization unit 204, the inverse transform unit 206, the adder 208, the cyclic filter unit 212, the intra-frame prediction unit 216, the inter-frame prediction unit 218, and the prediction control unit 220. Alternatively, the decoding device 200 can also be implemented as one or more dedicated electronic circuits corresponding to the entropy decoding unit 202, the inverse quantization unit 204, the inverse transform unit 206, the adder 208, the cyclic filter unit 212, the intra-frame prediction unit 216, the inter-frame prediction unit 218, and the prediction control unit 220.

[0198] The following describes the constituent elements included in the decoding device 200.

[0199] [Entropy Decoding Department] The entropy decoding unit 202 performs entropy decoding on the encoded bitstream. Specifically, the entropy decoding unit 202, for example, arithmetically decodes the encoded bitstream into a binary signal. Then, the entropy decoding unit 202 debinarizes the binary signal. As a result, the entropy decoding unit 202 outputs the quantization coefficients to the inverse quantization unit 204 in block units.

[0200] [Inverse Quantization Department] The inverse quantization unit 204 performs inverse quantization on the quantization coefficients of the decoded target block (hereinafter referred to as the current block), which is input from the entropy decoding unit 202. Specifically, the inverse quantization unit 204 performs inverse quantization on each quantization coefficient of the current block based on the quantization parameter corresponding to that quantization coefficient. Furthermore, the inverse quantization unit 204 outputs the inverse quantization coefficients (i.e., transform coefficients) of the current block to the inverse transform unit 206.

[0201] [Inverse Transformation Section] The inverse transform unit 206 restores the prediction error by performing an inverse transform on the transform coefficients, which are inputs from the inverse quantization unit 204.

[0202] For example, if the information read from the encoded bitstream represents EMT or AMT (e.g., the AMT flag is true), the inverse transform unit 206 performs an inverse transform on the transform coefficients of the current block based on the information representing the transform type read from the transduced bitstream.

[0203] Furthermore, for example, when the information read from the encoded bitstream is represented using NSST, the inverse transform unit 206 applies an inverse re-transformation to the transform coefficients.

[0204] [Addition Department] The adder 208 reconstructs the current block by adding the prediction error, which is input from the inverse transform 206, to the prediction sample, which is input from the prediction control 220. The adder 208 then outputs the reconstructed block to the block memory 210 and the cyclic filtering 212.

[0205] [Block Memory] Block memory 210 is a storage unit used to store blocks within the decoded target image (hereinafter referred to as the current image) that serve as a reference in intra-frame prediction. Specifically, block memory 210 stores the reconstructed blocks output from adder 208.

[0206] [Loop Filtering Section] The cyclic filtering unit 212 applies cyclic filtering to the block reconstructed by the addition unit 208 and outputs the filtered reconstructed block to the frame memory 214 and the display device, etc.

[0207] Given that the information indicating the on / off state of the ALF is read from the encoded bitstream, and the ALF is on, one filter is selected from multiple filters based on the direction and activity of the gradient of locality, and the selected filter is applied to the reconstructed block.

[0208] [Frame Memory] The frame memory 214 is a storage unit used to store reference images used in inter-frame prediction; it is also sometimes called a frame buffer. Specifically, the frame memory 214 stores the reconstructed blocks filtered by the cyclic filtering unit 212.

[0209] Intra-frame prediction unit The intra-prediction unit 216 performs intra-prediction based on the intra-prediction pattern read from the encoded bitstream, referring to blocks within the current image stored in the block memory 210, thereby generating a prediction signal (intra-prediction signal). Specifically, the intra-prediction unit 216 performs intra-prediction by referring to samples (e.g., luminance values, chrominance values) of blocks adjacent to the current block, thereby generating an intra-prediction signal, and outputs the intra-prediction signal to the prediction control unit 220.

[0210] In addition, if the intra-prediction mode of the reference luma block is selected in the intra-prediction of the chromatic difference block, the intra-prediction unit 216 can also predict the chromatic difference component of the current block based on the luma component of the current block.

[0211] Furthermore, when the information read from the encoded bitstream represents PDPC, the intra-prediction unit 216 corrects the pixel values ​​after intra-prediction based on the gradient of the reference pixel in the horizontal / vertical direction.

[0212] [Inter-frame prediction unit] The inter-frame prediction unit 218 refers to a reference image stored in the frame memory 214 to predict the current block. Prediction is performed in units of the current block or sub-blocks within the current block (e.g., 4×4 blocks). For example, the inter-frame prediction unit 218 uses motion information (e.g., motion vectors) read from the coded bitstream to perform motion compensation, thereby generating an inter-frame prediction signal for the current block or sub-block, and outputs the inter-frame prediction signal to the prediction control unit 220.

[0213] Furthermore, when the information read from the encoded bitstream is represented in OBMC mode, the inter-frame prediction unit 218 uses not only the motion information of the current block obtained through motion estimation, but also the motion information of adjacent blocks to generate the inter-frame prediction signal.

[0214] Furthermore, when the information representation read from the coded bitstream adopts FRUC mode, the inter-frame prediction unit 218 performs motion estimation according to the pattern matching method (bidirectional matching or template matching) read from the coded stream, thereby deriving motion information. The inter-frame prediction unit 218 then uses the derived motion information to perform motion compensation.

[0215] Furthermore, when applying BIO mode, the inter-frame prediction unit 218 derives motion vectors based on a model assuming constant-velocity linear motion. Additionally, when the information representation read from the encoded bitstream employs affine motion compensation prediction mode, the inter-frame prediction unit 218 derives motion vectors on a sub-block basis based on the motion vectors of multiple adjacent blocks.

[0216] [Predictive Control Department] The prediction control unit 220 selects either the intra-frame prediction signal or the inter-frame prediction signal, and outputs the selected signal as the prediction signal to the adder 208.

[0217] [Comparative Example] Before explaining the inter-frame prediction processing of this embodiment, an example of inter-frame prediction processing without using the method of this embodiment will be explained.

[0218] First, let's explain Comparative Example 1. Figure 11 This is a flowchart of the inter-frame prediction processing of the prediction block unit in the motion picture coding method and motion picture decoding method of Comparative Example 1. Figure 11The processing shown is repeated in the form of a prediction block unit, which is the processing unit for inter-frame prediction processing. Furthermore, the operation of the inter-frame prediction unit 126 included in the encoding apparatus 100 will be mainly described below, but the operation of the inter-frame prediction unit 218 included in the decoding apparatus 200 is the same.

[0219] When the FRUC control information is 0 (S101:0), the inter-frame prediction unit 126 derives motion vectors (MV) in prediction block units according to the normal inter-frame prediction method (S102). Here, the normal inter-frame prediction method refers to the conventional method that does not use the FRUC method, such as deriving motion vectors on the encoding side and transmitting information representing the derived motion vectors from the encoding side to the decoding side.

[0220] Next, the inter-frame prediction unit 126 uses the motion vector of the prediction block unit to perform motion compensation at the prediction block unit, thereby obtaining the inter-frame prediction image (S103).

[0221] On the other hand, when FRUC control information is represented as 1 (S101:1), the inter-frame prediction unit 126 derives motion vectors in prediction block units according to the template FRUC method (S104). Then, the inter-frame prediction unit 126 derives motion vectors in sub-block units that divide the prediction blocks according to the template FRUC method (S105).

[0222] On the other hand, in the case of FRUC control information representation 2 (S101:2), the inter-frame prediction unit 126 derives motion vectors in prediction block units according to the bidirectional FRUC method (S106). Then, the inter-frame prediction unit 126 derives motion vectors in the bidirectional FRUC method in sub-block units that divide the prediction blocks (S107).

[0223] Then, the inter-frame prediction unit 126 derives the motion vector of the sub-block unit according to the template FRUC method or the bidirectional FRUC method, and uses the derived motion vector of the sub-block unit to perform motion compensation in the sub-block unit, thereby obtaining the inter-frame prediction image (S108).

[0224] Thus, in FRUC processing, by deriving motion vectors on a block-by-block basis and then correcting them on a sub-block-by-sub-block basis, subtle movements can be tracked. This improves coding efficiency. On the other hand, there is a possibility that deformed blocks cannot be adequately handled on a pixel-by-pixel basis.

[0225] Next, Comparative Example 2 will be explained. Figure 12 This is a flowchart of the inter-frame prediction processing of the prediction block unit in the motion picture coding method and motion picture decoding method of Comparative Example 2. In Comparative Example 2, BIO processing is used as the motion compensation processing. That is, Figure 12 The processing shown is relative to Figure 11 The process shown is modified so that steps S103 and S108 are changed to steps S103A and S108A.

[0226] When the FRUC control information is 0 (S101:0), the inter-frame prediction unit 126 derives motion vectors in prediction block units according to the normal inter-frame prediction method (S102). Next, the inter-frame prediction unit 126 uses the motion vectors in the prediction block units to perform motion compensation based on prediction block unit BIO processing, thereby obtaining the inter-frame prediction image (S103A).

[0227] On the other hand, when FRUC control information is represented as 1 (S101:1), the inter-frame prediction unit 126 derives motion vectors in prediction block units according to the template FRUC method (S104). Then, the inter-frame prediction unit 126 derives motion vectors in sub-block units that divide the prediction blocks according to the template FRUC method (S105).

[0228] On the other hand, in the case of FRUC control information representation 2 (S101:2), the inter-frame prediction unit 126 derives motion vectors in prediction block units according to the bidirectional FRUC method (S106). Then, the inter-frame prediction unit 126 derives motion vectors in the bidirectional FRUC method in sub-block units that divide the prediction blocks (S107).

[0229] Then, the inter-frame prediction unit 126 derives the motion vector of the sub-block unit according to the template FRUC method or the bidirectional FRUC method, and uses the derived motion vector of the sub-block unit to perform motion compensation based on the sub-block unit BIO processing, thereby obtaining the inter-frame prediction image (S108A).

[0230] Thus, in Comparative Example 2, the inter-frame prediction unit 126 can perform BIO processing after FRUC processing, enabling pixel-level correction of the predicted image. Therefore, it is possible to improve coding efficiency even for blocks that have undergone deformation.

[0231] On the other hand, due to the presence of both FRUC and BIO processing, there is an issue of increased processing volume.

[0232] Furthermore, BIO processing is performed at the prediction block level in the normal inter-frame prediction method, while it is performed at the sub-block level in the FRUC method. Therefore, since the units of the motion vectors used as input for BIO processing differ between the normal inter-frame prediction method and the FRUC method, there is a need to install two different BIO processing functionalities.

[0233] [Inter-frame prediction processing] This section describes the inter-frame prediction processing of the inter-frame prediction unit 126 in this embodiment. In the inter-frame prediction processing, the inter-frame prediction unit 126 can perform at least two motion vector derivation methods, namely the normal inter-frame prediction method and the FRUC method. In the normal inter-frame prediction method, information about the motion vectors of the target block is encoded as a stream. In the FRUC method, information about the motion vectors of the target block is not encoded as a stream; instead, motion vectors are derived using a common method on both the encoding and decoding sides, employing the reconstructed image of the processed region and the processed reference image.

[0234] The inter-frame prediction unit 126 further performs motion compensation on each prediction block of the processed reference image using motion vectors to obtain a prediction image, and derives local motion estimates by obtaining brightness gradient values, performing BIO processing to generate a corrected prediction image using the derived local motion estimates. The inter-frame prediction unit 126 processes prediction block units in FRUC mode, deriving motion vectors for each prediction block unit. Furthermore, in BIO processing, regardless of which motion vector deriving method is used, the inter-frame prediction unit 126 always takes the motion vector of the prediction block unit as input and uses common processing for prediction block units to generate the prediction image.

[0235] The following describes the inter-frame prediction processing based on the inter-frame prediction unit 126 in this embodiment. Figure 13 This is a flowchart of the inter-frame prediction processing of the prediction block unit in the motion picture coding method and motion picture decoding method of this embodiment. Figure 13 The processing shown is repeated in the form of a prediction block unit, which is the processing unit for inter-frame prediction processing. Furthermore, the operation of the inter-frame prediction unit 126 included in the encoding apparatus 100 will be mainly described below, but the operation of the inter-frame prediction unit 218 included in the decoding apparatus 200 is the same.

[0236] When the FRUC control information is 0 (S101:0), and Figure 12 Similarly, the inter-frame prediction unit 126 derives motion vectors in prediction block units according to the normal inter-frame prediction method (S102). Next, the inter-frame prediction unit 126 uses the motion vectors in the prediction block units to perform motion compensation based on prediction block unit BIO processing, thereby obtaining the inter-frame prediction image (S103A).

[0237] On the other hand, when FRUC control information is represented as 1 (S101:1), the inter-frame prediction unit 126 derives motion vectors in prediction block units according to the template FRUC method (S104). Furthermore, when FRUC control information is represented as 2 (S101:2), the inter-frame prediction unit 126 derives motion vectors in prediction block units according to the bidirectional FRUC method (S106). Additionally, in Figure 13In the processing shown, with Figure 12 The processing is different; when using the FRUC method, the inter-frame prediction unit 126 does not derive the motion vector of the sub-block unit.

[0238] Then, the inter-frame prediction unit 126 derives the motion vector of the prediction block unit according to the template FRUC method or the bidirectional FRUC method, and uses the derived motion vector of the prediction block unit to perform motion compensation based on the prediction block unit BIO processing, thereby obtaining the inter-frame prediction image (S103A).

[0239] Thus, in this embodiment, when the FRUC control information indicates any of the following modes: normal inter-frame prediction, template FRUC, and bidirectional FRUC, the inter-frame prediction unit 126 derives the motion vector of the prediction block unit. Then, the inter-frame prediction unit 126 performs BIO processing on the prediction block unit. That is, regardless of the situation, the processing unit is the prediction block unit, and the processing unit is the same.

[0240] Furthermore, the FRUC control information number shown above is an example; other numbers can also be used. Additionally, it is possible to use only one of the template FRUC method or the bidirectional FRUC method. Moreover, shared processing can be used during encoding and decoding.

[0241] Here, in Figure 12 In Comparative Example 2, two methods are shown: sub-block FRUC processing, which corrects motion vectors at the sub-block level, and BIO processing, which corrects the predicted image at the pixel level. Both sub-block FRUC and BIO processes correct at units smaller than the predicted block, sharing the same nature and producing similar results. Figure 13 In the processing shown, these processes are aggregated into BIO processing. Additionally, in Figure 13 In the processing shown, the amount of processing can be reduced by avoiding FRUC processing on sub-block units with high processing volume. Furthermore, changing BIO processing from sub-block units to predicted block units when using FRUC also reduces processing volume.

[0242] Thus, with Figure 12 Compared to Comparative Example 2 shown, Figure 13 The processing described in this embodiment may suppress the increase in processing volume and improve coding efficiency for deformed blocks.

[0243] In addition, Figure 13 In the processing of this embodiment shown, BIO processing with motion vectors of prediction block units as input is performed regardless of the FRUC control information value. Therefore, it is consistent with... Figure 12Compared to Comparative Example 2 shown, BIO processing with motion vectors at the sub-block level as input is not required. This simplifies the installation process.

[0244] Furthermore, the BIO processing in step S103A does not need to be exactly the same across multiple motion vector derivation methods. That is, the BIO processing used can differ in any one or each of the following scenarios: using the standard inter-frame prediction method, using the template FRUC method, and using the bidirectional FRUC method.

[0245] Furthermore, regarding at least one of the multiple motion vector derivation methods, the BIO processing in step S103A can be replaced by other processing (including variations of the BIO processing) that generates a prediction image while correcting the prediction image in pixel units or units smaller than the prediction block. In this case, it is also possible to combine two processes with the same properties described above into one process. Therefore, compared to Comparative Example 2, it is possible to improve coding efficiency for deformed blocks while suppressing the increase in processing volume.

[0246] Alternatively, for at least one of the multiple motion vector derivation methods, the BIO processing in step S103A can be replaced with other processing that generates the prediction image while using the motion vectors of the prediction block units as input to correct the prediction image. In this case, the aforementioned processing using motion vectors of the sub-block units as input is not required, thus simplifying the installation process.

[0247] Alternatively, BIO processing can be switched on or off. For example, the inter-frame prediction unit 126 can perform BIO processing while simultaneously performing BIO processing. Figure 13 The processing shown is performed without BIO processing. Figure 11 The processing shown. Alternatively, the inter-frame prediction unit 126 can also be used without performing BIO processing. Figure 13 Similarly, motion vector derivation processing for sub-block units is not performed. That is, the inter-frame prediction unit 126 can also perform... Figure 13 The processing in step S108A shown is replaced by a motion compensation processing for the prediction block unit that does not include BIO processing.

[0248] [A variation of inter-frame predictive processing] The following describes a variation of the inter-screen prediction processing of this embodiment. Figure 14 This is a flowchart of the inter-frame prediction processing in the motion picture coding method and motion picture decoding method of a modified embodiment of this invention. Figure 14 The processing shown is repeated in the form of prediction block units, which are the processing units for inter-frame prediction processing.

[0249] When the FRUC control information is 0 (S101:0), and Figure 13 Similarly, the inter-frame prediction unit 126 derives motion vectors in prediction block units according to the normal inter-frame prediction method (S102). Next, the inter-frame prediction unit 126 uses the motion vectors in the prediction block units to perform motion compensation based on prediction block unit BIO processing, thereby obtaining the inter-frame prediction image (S103A).

[0250] Additionally, in the case of FRUC control information representation 1 (S101:1), with Figure 13 Similarly, the inter-frame prediction unit 126 derives motion vectors in prediction block units according to the template FRUC method (S104). Next, the inter-frame prediction unit 126 uses the motion vectors in the prediction block units to perform motion compensation based on prediction block unit BIO processing, thereby obtaining the inter-frame prediction image (S103A).

[0251] On the other hand, in the case of FRUC control information representation 2 (S101:2), the inter-frame prediction unit 126 derives motion vectors in prediction block units according to the bidirectional FRUC method (S106). Then, the inter-frame prediction unit 126 derives motion vectors in the bidirectional FRUC method in sub-block units that divide the prediction blocks (S107).

[0252] Next, the inter-frame prediction unit 126 uses the derived sub-block unit motion vectors to perform motion compensation on a sub-block unit basis, thereby obtaining an inter-frame prediction image (S108). Furthermore, the motion compensation here is not BIO processing but conventional motion compensation.

[0253] Thus, in the case of using the normal inter-frame prediction method (S101:0) and the case of using the template FRUC method (S101:1), the inter-frame prediction unit 126 uses the derived prediction block unit motion vectors to perform motion compensation based on prediction block unit BIO processing, thereby obtaining the inter-frame prediction image. On the other hand, in the case of using the bidirectional FRUC method (S101:2), the inter-frame prediction unit 126 obtains the inter-frame prediction image by performing normal motion compensation using sub-block unit motion vectors without applying BIO processing.

[0254] Furthermore, the numbering of the FRUC control information is one example; other numbers may also be used. Additionally, the inter-frame prediction unit 218 included in the decoding apparatus 200 performs the same processing as the inter-frame prediction unit 126 included in the encoding apparatus 100.

[0255] Furthermore, here, when using the template FRUC method, the inter-frame prediction unit 126 derives motion vectors at the prediction block unit level and performs BIO processing at the prediction block unit level; when using the bidirectional FRUC method, it derives motion vectors at the sub-block unit level and performs normal motion compensation processing at the sub-block unit level. Alternatively, when using the bidirectional FRUC method, it can derive motion vectors at the prediction block unit level and perform BIO processing at the prediction block unit level; and when using the template FRUC method, it can derive motion vectors at the sub-block unit level and perform normal motion compensation processing at the sub-block unit level.

[0256] Furthermore, in both template FRUC and bidirectional FRUC, the bidirectional FRUC method shows a greater improvement in coding efficiency by deriving motion vectors based on sub-block units. Therefore, as... Figure 14 As shown, it is preferable to derive motion vectors in sub-block units in the bidirectional FRUC mode.

[0257] In addition, Figure 14 Both template-based FRUC and bidirectional FRUC methods are used, but either method can also be used. In this case, when using the FRUC method, the inter-frame prediction unit 126 derives motion vectors on a sub-block basis and performs normal motion compensation processing.

[0258] Furthermore, the BIO processing in step S103A does not need to be exactly the same across multiple motion vector derivation methods. That is, the BIO processing used can differ depending on whether the usual inter-frame prediction method or the template FRUC method is used.

[0259] In addition, for at least one of the multiple motion vector derivation methods, the BIO processing in step S103A can be replaced by other processing (including variations of BIO processing) to correct the predicted image in pixel units or units smaller than the predicted block.

[0260] Alternatively, for at least one of the multiple motion vector derivation methods, the BIO processing in step S103A can be replaced with other processing that uses the motion vector of the prediction block unit as input to correct the prediction image while generating the prediction image.

[0261] Figure 14 The processing shown in the FRUC method applies only one of the following: sub-block FRUC processing, which corrects motion vectors at the sub-block level, and BIO processing, which corrects the predicted image at the pixel level. Therefore, Figure 14 The processing shown is the same as Figure 13 The processing shown is roughly equivalent in terms of processing volume, and methods with greater multiplicative effects can be used for each FRUC method. Therefore, coding efficiency can be improved.

[0262] As described above, the encoding device 100 performs Figure 15 The processing is shown. Similarly, the processing in the decoding device 200 is also the same. In the first operation mode (first operation mode in S111), the encoding device 100 derives a first motion vector in prediction block units that segment the images contained in the moving image using a first inter-frame prediction method (S112), and performs a first motion compensation process using the derived first motion vector in prediction block units (S113). Here, the first motion compensation process is, for example, BIO processing, which is a motion compensation process that generates the predicted image by referring to the spatial gradient of brightness in the image generated by motion compensation using the derived first motion vector.

[0263] Furthermore, in the second operation mode (the second operation mode in S111), the encoding device 100 derives a second motion vector in sub-block units that divide the prediction block using a second inter-frame prediction method (S114), and performs a second motion compensation process using the second motion vector in sub-block units (S115). Here, the second motion compensation process refers to, for example, motion compensation processing that does not apply BIO processing, and motion compensation processing that generates the prediction image without referring to the spatial gradient of brightness in the image generated by motion compensation using the second motion vector.

[0264] Therefore, in the first operating mode, the encoding device 100 reduces the processing workload by performing motion vector derivation processing and first motion compensation processing on a prediction block basis, compared to performing these processes on a sub-block basis. Furthermore, since the first motion compensation processing, which generates the prediction image based on the spatial gradient of brightness, can be implemented in units smaller than the prediction block unit, the reduction in coding efficiency when processing is not performed on a sub-block basis can be suppressed. In the second operating mode, the encoding device 100 performs motion vector derivation processing and second motion compensation processing on a sub-block basis. Here, the second motion compensation processing, since it does not refer to the spatial gradient of brightness, requires less processing compared to the first motion compensation processing. Thus, by having these two operating modes, the encoding device 100 can improve coding efficiency. In this way, the encoding device 100 can reduce the processing workload while suppressing the reduction in coding efficiency.

[0265] For example, the prediction method between the first frame and the prediction method between the second frame are different. Specifically, the prediction method between the second frame is an inter-frame prediction method that uses the fit of two reconstructed images of two different regions within two different images, such as the FRUC method.

[0266] Therefore, this inter-frame prediction method, which enables the calculation of motion vectors based on sub-block units, significantly improves coding efficiency. Thus, it enhances coding efficiency.

[0267] For example, the first inter-frame prediction method is one of (1) the third inter-frame prediction method (e.g., template FRUC method) which uses the fit of the reconstructed image of the region in the object picture adjacent to the object prediction block and the reconstructed image of the region in the reference picture, and (2) the fourth inter-frame prediction method (e.g., bidirectional FRUC method) which uses the fit of the two reconstructed images of two regions in two different reference pictures, and the second inter-frame prediction method is the other of the third and fourth inter-frame prediction methods.

[0268] For example, the prediction method between the first frame is the prediction method between the third frame (e.g., template FRUC method), and the prediction method between the second frame is the prediction method between the fourth frame (e.g., bidirectional FRUC method).

[0269] Therefore, this inter-frame prediction method, which enables the calculation of motion vectors based on sub-block units, significantly improves coding efficiency. Thus, it enhances coding efficiency.

[0270] For example, the first inter-frame prediction method is an inter-frame prediction method that uses the fit of the reconstructed image based on the object prediction block and the region contained in the reference picture (typical inter-frame prediction method). The encoding device 100 generates an encoded bitstream containing information for determining the derived first motion vector. Furthermore, in the first inter-frame prediction method, the decoding device 200 obtains information from the encoded bitstream for determining the first motion vector in units of prediction blocks, and uses this information to derive the first motion vector.

[0271] [Template FRUC method and bidirectional FRUC method] The following describes the methods for exporting motion vectors using either the template FRUC method or the bidirectional FRUC method. The methods for exporting motion vectors at the block unit level and at the sub-block unit level are essentially the same. In the following description, the methods for exporting block motion vectors and sub-block motion vectors will be used as examples of methods for exporting motion vectors for the processing object region.

[0272] Figure 16 This is a conceptual diagram illustrating a template FRUC method used in the encoding device 100 and decoding device 200 to derive motion vectors of the processing target area. In the template FRUC method, the motion vector information of the processing target area is not encoded or decoded; instead, a common method is used between the encoding device 100 and the decoding device 200 to derive the motion vectors.

[0273] In addition, in the template FRUC method, motion vectors are derived using the reconstructed image of the adjacent region of the region adjacent to the region being processed and the reconstructed image of the corresponding adjacent region of the region in the reference image.

[0274] Here, adjacent regions are one or both of the regions that are adjacent to the processing object region on the left and the regions that are adjacent to it above.

[0275] Furthermore, the corresponding adjacent region is a region specified using a candidate motion vector that serves as a candidate for the motion vector of the processing target region. Specifically, the corresponding adjacent region is the region indicated by the candidate motion vector from the adjacent region. Additionally, the relative position of the corresponding adjacent region with respect to the corresponding region indicated by the candidate motion vector from the processing target region is equal to the relative position of the adjacent region with respect to the processing target region.

[0276] Figure 17 This is a conceptual diagram illustrating a bidirectional FRUC method for deriving motion vectors of the processing target area in the encoding device 100 and the decoding device 200. Similar to the template FRUC method, the bidirectional FRUC method does not encode or decode the motion vector information of the processing target area; instead, a common method is used between the encoding device 100 and the decoding device 200 to derive the motion vectors.

[0277] Furthermore, in the bidirectional FRUC method, two reconstructed images of two regions from two reference images are used to derive motion vectors. For example, as... Figure 17 As shown, motion vectors are derived using the reconstructed images of the corresponding regions in the first reference image and the reconstructed images of the symmetrical regions in the second reference image.

[0278] Here, the corresponding region and the symmetrical region are regions specified using candidate motion vectors that are candidates for the motion vectors of the processing object region. Specifically, the corresponding region is the region indicated from the processing object region by the candidate motion vector. The symmetrical region is the region indicated from the processing object region by the symmetrical motion vector. The symmetrical motion vector is a group of motion vectors that constitute the candidate motion vectors for bidirectional prediction. The symmetrical motion vector can also be a motion vector derived by scaling the candidate motion vectors.

[0279] Figure 18 This is a flowchart illustrating the operation of the inter-frame prediction unit 126 of the encoding apparatus 100 in deriving motion vectors according to the template FRUC method or the bidirectional FRUC method. The inter-frame prediction unit 218 of the decoding apparatus 200 operates in the same manner as the inter-frame prediction unit 126 of the encoding apparatus 100.

[0280] First, the inter-frame prediction unit 126 derives candidate motion vectors by referring to the motion vectors of one or more processed regions that are temporally or spatially adjacent to the processing target region.

[0281] In the bidirectional FRUC mode, the inter-frame prediction unit 126 derives candidate motion vectors for bidirectional prediction. That is, the inter-frame prediction unit 126 derives candidate motion vectors as a set of two motion vectors.

[0282] Specifically, in the bidirectional FRUC mode, when the motion vector of the processed region is a bidirectional predicted motion vector, the inter-frame prediction unit 126 directly derives the bidirectional predicted motion vector as a candidate motion vector for bidirectional prediction. When the motion vector of the processed region is a unidirectional predicted motion vector, the inter-frame prediction unit 126 can also derive the bidirectional predicted motion vector from the unidirectional predicted motion vector by scaling, etc., thereby deriving candidate motion vectors for bidirectional prediction.

[0283] More specifically, in bidirectional FRUC mode, the inter-frame prediction unit 126 scales the motion vector of the first reference image according to the display time interval, thereby deriving the motion vector of the second reference image. Thus, the inter-frame prediction unit 126 derives candidate motion vectors that constitute the group of motion vectors for unidirectional prediction and the scaled motion vectors as candidate motion vectors for bidirectional prediction.

[0284] Alternatively, in the bidirectional FRUC mode, the inter-frame prediction unit 126 may derive the motion vector of the processed region as a candidate motion vector if the motion vector of the processed region is a bidirectionally predicted motion vector. Then, if the motion vector of the processed region is a unidirectionally predicted motion vector, the inter-frame prediction unit 126 may not derive the motion vector of the processed region as a candidate motion vector.

[0285] In the template FRUC mode, regardless of whether the motion vector of the processed region is a bidirectional or unidirectional predicted motion vector, the inter-frame prediction unit 126 will derive the motion vector of the processed region as a candidate motion vector.

[0286] Then, the inter-frame prediction unit 126 generates a candidate motion vector list consisting of candidate motion vectors (S201). Here, when the processing target region is a sub-block, that is, when deriving motion vectors at the sub-block unit level, the inter-frame prediction unit 126 may also include block-unit motion vectors as candidate motion vectors in the candidate motion vector list. In this case, the inter-frame prediction unit 126 may also include block-unit motion vectors as the highest priority candidate motion vectors in the candidate motion vector list.

[0287] Furthermore, in the bidirectional FRUC mode, when the motion vector of a block unit is a unidirectional predicted motion vector, the inter-frame prediction unit 126 can also derive bidirectional predicted candidate motion vectors from the unidirectional predicted motion vectors by scaling or the like. For example, the inter-frame prediction unit 126 can also derive bidirectional predicted candidate motion vectors from the unidirectional predicted motion vectors by scaling or the like, just as it can do when the surrounding motion vectors are unidirectional predicted motion vectors.

[0288] Then, the inter-frame prediction unit 126 may also include candidate motion vectors derived from unidirectional predicted motion vectors as candidate motion vectors for bidirectional prediction in the candidate motion vector list.

[0289] Alternatively, in the bidirectional FRUC mode, the inter-frame prediction unit 126 may include the block unit's motion vector as a candidate motion vector in the candidate motion vector list if the block unit's motion vector is a bidirectionally predicted motion vector. Then, the inter-frame prediction unit 126 may not include the block unit's motion vector as a candidate motion vector in the candidate motion vector list if the block unit's motion vector is a unidirectionally predicted motion vector.

[0290] Then, the inter-frame prediction unit 126 selects the best candidate motion vector from one or more candidate motion vectors included in the candidate motion vector list (S202). At this time, the inter-frame prediction unit 126 calculates an evaluation value for each of the one or more candidate motion vectors according to the suitability of the two reconstructed images of the two evaluation object regions.

[0291] Specifically, in the template FRUC approach, the two evaluation object regions are Figure 16 In the bidirectional FRUC approach, such adjacent regions and corresponding adjacent regions represent two evaluation object regions. Figure 17 Such corresponding regions and symmetrical regions. As mentioned above, the corresponding adjacent regions used in the template FRUC method and the corresponding and symmetrical regions used in the bidirectional FRUC method are determined according to the candidate motion vector.

[0292] For example, the higher the fit of the two reconstructed images of the two evaluation target regions, the better the evaluation value calculated by the inter-frame prediction unit 126. Specifically, the inter-frame prediction unit 126 derives the difference value between the two reconstructed images of the two evaluation target regions. Then, the inter-frame prediction unit 126 uses the difference value to calculate the evaluation value. For example, the smaller the difference value, the better the evaluation value calculated by the inter-frame prediction unit 126.

[0293] Furthermore, in calculating the evaluation value, not only the difference value but also other information can be used. That is, the inter-frame prediction unit 126 can use the difference value and other information to calculate the evaluation value. For example, the priority order of more than one candidate motion vector and the amount of coding based on the priority order can also affect the evaluation value.

[0294] Then, the inter-frame prediction unit 126 selects the candidate motion vector with the highest evaluation value from more than one candidate motion vector as the best candidate motion vector.

[0295] Then, the inter-frame prediction unit 126 derives the motion vector of the processing object region by searching the periphery of the best candidate motion vector (S203).

[0296] That is, the inter-frame prediction unit 126 similarly calculates evaluation values ​​for motion vectors indicating the regions surrounding the region indicated by the best candidate motion vector. Then, if there is a motion vector with a better evaluation value than the best candidate motion vector, the inter-frame prediction unit 126 updates the best candidate motion vector using the motion vector with the better evaluation value. Then, the inter-frame prediction unit 126 derives the updated best candidate motion vector as the final motion vector for the processing target region.

[0297] Furthermore, the inter-frame prediction unit 126 may also derive the best candidate motion vector as the final motion vector for the processing target area without performing the surrounding processing of the best candidate motion vector search (S203). Additionally, the best candidate motion vector is not limited to the candidate motion vector with the highest evaluation value. Alternatively, one of more candidate motion vectors with an evaluation value above the benchmark may be selected as the best candidate motion vector according to a predetermined priority order.

[0298] Furthermore, the processing associated with the processing target region and the processed region here is, for example, encoding or decoding. More specifically, the processing associated with the processing target region and the processed region can also be the processing of deriving motion vectors. Alternatively, the processing associated with the processing target region and the processed region can also be the processing of reconstruction.

[0299] [BIO processing] Figure 19 This is a conceptual diagram representing the BIO processing in the encoding device 100 and the decoding device 200. In the BIO processing, a predicted image of the processing object block is generated by referring to the spatial gradient of brightness in the image obtained by performing motion compensation on the processing object block using the motion vector of the processing object block.

[0300] Before BIO processing, two motion vectors, L0 motion vector (MV_L0) and L1 motion vector (MV_L1), are exported as the processing object block. The L0 motion vector (MV_L0) is used to reference the L0 reference image as the processed image, and the L1 motion vector (MV_L1) is used to reference the L1 reference image as the processed image. The L0 and L1 reference images are the two reference images simultaneously referenced in the dual prediction processing of the processing object block.

[0301] As methods for deriving L0 motion vectors (MV_L0) and L1 motion vectors (MV_L1), methods such as the general inter-frame prediction mode, the merge mode, or the FRUC mode can be used. For example, in the general inter-frame prediction mode, the encoding device 100 derives motion vectors by performing motion detection on the image of the processing target block and encodes the motion vector information. Furthermore, in the general inter-frame prediction mode, the decoding device 200 derives motion vectors by decoding the motion vector information.

[0302] Then, in the BIO processing, motion compensation for the processed object block is performed using the L0 motion vector (MV_L0) with reference to the L0 reference image, thereby obtaining the L0 prediction image. For example, the L0 prediction image can also be obtained by applying motion compensation filtering to an image of the L0 reference pixel range, which includes the block indicated by the processed object block in the L0 reference image and its surroundings via the L0 motion vector (MV_L0).

[0303] Additionally, an L0 gradient image representing the spatial gradient of brightness in each pixel of the L0 predicted image is obtained. For example, the L0 gradient image is obtained by referring to the brightness of each pixel in an L0 reference pixel range, which includes the block indicated by the processing object block in the L0 reference image and its surroundings via the L0 motion vector (MV_L0).

[0304] Additionally, referring to the L1 reference image, motion compensation for the processed object block is performed using the L1 motion vector (MV_L1), thereby obtaining the L1 prediction image. For example, the L1 prediction image can be obtained by applying motion compensation filtering to an image of the L1 reference pixel range, which includes the block indicated by the processed object block in the L1 reference image and its surroundings via the L1 motion vector (MV_L1).

[0305] Additionally, an L1 gradient image representing the spatial gradient of brightness in each pixel of the L1 predicted image is obtained. For example, the L1 gradient image is obtained by referring to the brightness of each pixel in an L1 reference pixel range, which includes the block indicated by the processing object block in the L1 reference image and its surroundings via the L1 motion vector (MV_L1).

[0306] Then, local motion estimates are derived for each pixel of the processed object block. Specifically, at this point, the pixel value at the corresponding pixel location in the L0 prediction image, the gradient value at the corresponding pixel location in the L0 gradient image, the pixel value at the corresponding pixel location in the L1 prediction image, and the gradient value at the corresponding pixel location in the L1 gradient image are used. The local motion estimate can also be referred to as the corrected motion vector (corrected MV).

[0307] Then, for each pixel of the processing block, pixel correction values ​​are derived using the gradient values ​​at the corresponding pixel locations in the L0 gradient image, the L1 gradient image, and the local motion estimate. Next, for each pixel of the processing block, predicted pixel values ​​are derived using the pixel values ​​at the corresponding pixel locations in the L0 prediction image, the L1 prediction image, and the pixel correction values. This yields a predicted image with BIO processing applied.

[0308] That is, the predicted pixel value obtained from the pixel value at the corresponding pixel position in the L0 prediction image and the pixel value at the corresponding pixel position in the L1 prediction image is corrected by the pixel correction value. In other words, the predicted image obtained from the L0 prediction image and the L1 prediction image is corrected using the spatial gradient of the brightness in the L0 prediction image and the L1 prediction image.

[0309] Figure 20 This is a flowchart illustrating the operation of the inter-frame prediction unit 126 of the encoding device 100 as a BIO process. The inter-frame prediction unit 218 of the decoding device 200 operates in the same manner as the inter-frame prediction unit 126 of the encoding device 100.

[0310] First, the inter-frame prediction unit 126 obtains the L0 prediction image by referencing the L0 reference image using the L0 motion vector (MV_L0) (S401). Then, the inter-frame prediction unit 126 obtains the L0 gradient image by referencing the L0 reference image using the L0 motion vector (S402).

[0311] Similarly, the inter-frame prediction unit 126 obtains the L1 prediction image by referencing the L1 reference image using the L1 motion vector (MV_L1) (S401). Then, the inter-frame prediction unit 126 obtains the L1 gradient image by referencing the L1 reference image using the L1 motion vector (S402).

[0312] Next, for each pixel of the processing object block, the inter-frame prediction unit 126 derives a local motion estimate (S411). At this time, the pixel value at the corresponding pixel position in the L0 prediction image, the gradient value at the corresponding pixel position in the L0 gradient image, the pixel value at the corresponding pixel position in the L1 prediction image, and the gradient value at the corresponding pixel position in the L1 gradient image are used.

[0313] Then, for each pixel of the processing target block, the inter-frame prediction unit 126 uses the gradient value of the corresponding pixel position in the L0 gradient image, the gradient value of the corresponding pixel position in the L1 gradient image, and the local motion estimation value to derive a pixel correction value. Then, for each pixel of the processing target block, the inter-frame prediction unit 126 uses the pixel value of the corresponding pixel position in the L0 prediction image, the pixel value of the corresponding pixel position in the L1 prediction image, and the pixel correction value to derive a predicted pixel value (S412).

[0314] Through the above actions, the inter-frame prediction unit 126 generates a predicted image that has applied BIO processing.

[0315] In addition, in the derivation of local motion estimates and pixel correction values, specifically, the following equation (3) can also be used.

[0316]

Formula 3

[0317] Furthermore, in equation (3), I0[x, y] is the pixel value at pixel position [x, y] in the L0 predicted image. I1[x, y] is the pixel value at pixel position [x, y] in the L1 predicted image. ΔI[x, y] is the difference between the pixel value at pixel position [x, y] in the L0 predicted image and the pixel value at pixel position [x, y] in the L1 predicted image.

[0318] Additionally, in equation (3), Ω is, for example, the set of pixel positions contained in the region with pixel position [x, y] at the center. w[i, j] is a weighting coefficient relative to pixel position [i, j]. w[i, j] can use the same value. Gx[x, y], Gy[x, y], GxGy[x, y], sGxGy[x, y], sGx2[x, y], sGy2[x, y], sGxdI[x, y], and sGydI[x, y] are auxiliary calculation values.

[0319] Additionally, in equation (3), u[x, y] is the horizontal value constituting the local motion estimate at pixel position [x, y]. v[x, y] is the vertical value constituting the local motion estimate at pixel position [x, y]. b[x, y] is the pixel correction value at pixel position [x, y]. p[x, y] is the predicted pixel value at pixel position [x, y].

[0320] In addition, in the above description, the inter-frame prediction unit 126 derives local motion estimates per pixel, but it can also derive local motion estimates per sub-block, which is a unit of image data that is coarser than a pixel and finer than the block being processed.

[0321] For example, in equation (3) above, Ω can also be the set of pixel positions contained in the sub-block. Furthermore, sGxGy[x,y], sGx2[x,y], sGy2[x,y], sGxdI[x,y], sGydI[x,y], u[x,y] and v[x,y] can be calculated not per pixel but per sub-block.

[0322] Furthermore, the encoding device 100 and the decoding device 200 can apply shared BIO processing. That is, the encoding device 100 and the decoding device 200 can apply BIO processing in the same way.

[0323] [Installation example of the encoding device] Figure 21 This is a block diagram showing an installation example of the encoding device 100 according to Embodiment 1. The encoding device 100 includes circuitry 160 and a memory 162. For example, Figure 1 and Figure 11 The multiple components of the encoding device 100 shown are composed of Figure 21 The circuit 160 and memory 162 shown are implemented.

[0324] Circuit 160 is an information processing circuit that accesses memory 162. For example, circuit 160 is a dedicated or general-purpose electronic circuit for encoding moving images. Circuit 160 can also be a processor like a CPU. Alternatively, circuit 160 can be an assembly of multiple electronic circuits. Furthermore, for example, circuit 160 can also function as… Figure 1 The functions of multiple components of the encoding device 100 shown, excluding the component for storing information.

[0325] Memory 162 is a dedicated or general-purpose memory used by storage circuit 160 to encode moving images. Memory 162 can be an electronic circuit or connected to circuit 160. Alternatively, memory 162 can be included within circuit 160. Alternatively, memory 162 can be an assembly of multiple electronic circuits. Furthermore, memory 162 can be a disk or optical disk, or it can be represented as storage or a recording medium. Furthermore, memory 162 can be either non-volatile or volatile memory.

[0326] For example, memory 162 can store the encoded motion image, or it can store the bit string corresponding to the encoded motion image. Additionally, memory 162 can also store the program used by circuit 160 to encode the motion image.

[0327] Additionally, for example, memory 162 can also serve as Figure 1 The encoding device 100 shown above has multiple components that serve as components for storing information. Specifically, the memory 162 can function as... Figure 1 The block memory 118 and frame memory 122 shown have the following functions. More specifically, memory 162 can store reconstructed blocks and reconstructed images, etc.

[0328] Alternatively, the encoding device 100 may not require installation of [something]. Figure 1 All of the multiple constituent elements shown above can also be processed without the aforementioned multiple processing steps. Figure 1 A portion of the multiple constituent elements shown may be included in other devices, or a portion of the aforementioned multiple processes may be performed by other devices. Furthermore, in the encoding device 100, an assembly is installed... Figure 1 One of the multiple components shown above can be used to efficiently perform motion compensation by performing one of the multiple processes described above.

[0329] Specifically, the encoding device 100 derives the first motion vector by using a first-frame prediction method that employs the fit of two reconstructed images from two different regions within two distinct images, in the form of prediction block units that segment the images contained in the moving image. Figure 13 (S104 or S106). Here, the first-frame prediction method refers to, for example, the FRUC method described above. Specifically, the first-frame prediction method includes at least one of the template FRCU method and the bidirectional FRUC method. That is, the two regions mentioned above in the first-frame prediction method are (1) the region within the object image adjacent to the object prediction block and the region within the reference image, or (2) two regions within two different reference images.

[0330] In other words, the first-frame prediction method refers to a method in which motion vectors are derived using the same method on both the encoding and decoding sides. Furthermore, in the first-frame prediction method, information representing motion vectors is not signaled to the encoding stream and is not transmitted from the encoding side to the decoding side. Additionally, in the first-frame prediction method, the encoding device 100 uses the pixel values ​​of the already encoded prediction blocks and does not use the pixel values ​​of the object prediction blocks to derive the motion vectors.

[0331] Next, the encoding device 100 performs a first motion compensation process, which generates a prediction image on a block-by-block basis by referring to the spatial gradient of brightness in the image generated using motion compensation derived from the first motion vector. Figure 13 (S103A). Here, the first motion compensation process refers to, for example, the BIO process described above, including correction using a brightness gradient. Furthermore, in the first motion compensation process, the predicted image is corrected in units smaller than the predicted block (e.g., pixel units or block units). Additionally, in the first motion compensation process, the predicted image is generated using a region within a reference image represented by a motion vector and the pixels surrounding that region.

[0332] Therefore, by performing motion vector derivation processing based on the first inter-frame prediction method and first motion compensation processing on a prediction block basis, the encoding apparatus 100 can reduce the processing load compared to, for example, performing these processes on a sub-block basis. Furthermore, since the first motion compensation processing, which includes correction using luminance gradients, can be implemented in units smaller than the prediction block unit, the reduction in coding efficiency that would occur without processing on a sub-block basis can be suppressed. Thus, the encoding apparatus 100 can reduce the processing load while suppressing a reduction in coding efficiency.

[0333] Additionally, the encoding device 100 derives the second motion vector in units of prediction blocks by using a second-frame prediction method that assesses the fit of the reconstructed image based on the region contained in the object prediction block and the reference image. Figure 13 (S102). Then, the encoding device 100 generates an encoded bitstream containing information for determining the second motion vector.

[0334] Here, the second inter-frame prediction method refers to, for example, the typical inter-frame prediction method described above. In other words, the second inter-frame prediction method refers to a method of deriving motion vectors using different methods on the encoding and decoding sides. Specifically, the encoding device 100 derives motion vectors using the pixel values ​​of the encoded prediction blocks and the pixel values ​​of the object prediction blocks. Then, the encoding device 100 signals information representing the derived motion vectors to the encoding stream. Thus, information representing the motion vectors derived by the encoding device 100 is transmitted from the encoding device 100 to the decoding device 200. The decoding device 200 derives motion vectors using this information contained in the encoding stream.

[0335] Next, the encoding device 100 performs a second motion compensation process, which generates a prediction image on a block-by-block basis, referring to the spatial gradient of brightness in the image generated by motion compensation using the derived second motion vector. Figure 13 (S103A). Here, the second motion compensation process refers to, for example, the BIO process described above, including correction using a brightness gradient. Furthermore, in the second motion compensation process, the predicted image is corrected in units smaller than the predicted block (e.g., pixel units or block units). Additionally, in the second motion compensation process, the predicted image is generated using a region within a reference image represented by a motion vector and the pixels surrounding that region.

[0336] In addition, the second motion compensation process can be the same as the first motion compensation process, or it can be a part of a different process.

[0337] Therefore, whether using the first-frame prediction method or the second-frame prediction method, the processing unit for motion compensation processing can be the same. This simplifies the installation of motion compensation processing.

[0338] Furthermore, in the first operation mode (the first operation mode in S111), the encoding device 100 derives a first motion vector by using a first inter-frame prediction method, in prediction block units that segment the images contained in the moving image (S112), and performs a first motion compensation process using the derived first motion vector in prediction block units (S113). Here, the first motion compensation process is, for example, BIO processing, which is a motion compensation process for generating the predicted image by referring to the spatial gradient of brightness in the image generated by motion compensation using the derived first motion vector.

[0339] Furthermore, in the second operation mode (the second operation mode in S111), the encoding device 100 derives a second motion vector in sub-block units that divide the prediction block using a second inter-frame prediction method (S114), and performs a second motion compensation process using the second motion vector in sub-block units (S115). Here, the second motion compensation process refers to, for example, motion compensation processing that does not apply BIO processing, and motion compensation processing that generates the prediction image without referring to the spatial gradient of brightness in the image generated by motion compensation using the second motion vector.

[0340] Therefore, in the first operating mode, the encoding device 100 reduces the processing workload by performing motion vector derivation processing and first motion compensation processing on a prediction block basis, compared to performing these processes on a sub-block basis. Furthermore, since the first motion compensation processing, which generates the prediction image based on the spatial gradient of brightness, can be implemented in units smaller than the prediction block unit, the reduction in coding efficiency when processing is not performed on a sub-block basis can be suppressed. In the second operating mode, the encoding device 100 performs motion vector derivation processing and second motion compensation processing on a sub-block basis. Here, the second motion compensation processing, since it does not refer to the spatial gradient of brightness, requires less processing compared to the first motion compensation processing. Thus, by having these two operating modes, the encoding device 100 can improve coding efficiency. In this way, the encoding device 100 can reduce the processing workload while suppressing the reduction in coding efficiency.

[0341] For example, the prediction method between the first frame and the prediction method between the second frame are different. Specifically, the prediction method between the second frame is an inter-frame prediction method that uses the fit of two reconstructed images of two different regions within two different images, such as the FRUC method.

[0342] Therefore, this inter-frame prediction method, which enables the calculation of motion vectors based on sub-block units, significantly improves coding efficiency. Thus, it enhances coding efficiency.

[0343] For example, the first inter-frame prediction method is one of (1) the third inter-frame prediction method (e.g., template FRUC method) which uses the fit of the reconstructed image of the region in the object picture adjacent to the object prediction block and the reconstructed image of the region in the reference picture, and (2) the fourth inter-frame prediction method (e.g., bidirectional FRUC method) which uses the fit of the two reconstructed images of two regions in two different reference pictures, and the second inter-frame prediction method is the other of the third and fourth inter-frame prediction methods.

[0344] For example, the prediction method between the first frame is the prediction method between the third frame (e.g., template FRUC method), and the prediction method between the second frame is the prediction method between the fourth frame (e.g., bidirectional FRUC method).

[0345] Therefore, this inter-frame prediction method, which enables the calculation of motion vectors based on sub-block units, significantly improves coding efficiency. Thus, it enhances coding efficiency.

[0346] For example, the first inter-frame prediction method is an inter-frame prediction method that uses the fit of the reconstructed image of the region contained in the object prediction block and the reference picture (typical inter-frame prediction method), and the encoding device 100 generates an encoded bitstream containing information for determining the derived first motion vector.

[0347] [Installation example of the decoding device] Figure 22 This is a block diagram illustrating an installation example of the decoding device 200 according to Embodiment 1. The decoding device 200 includes circuitry 260 and a memory 262. For example, Figure 10 and Figure 12 The multiple components of the decoding device 200 shown are composed of Figure 22 The circuit 260 and memory 262 shown are implemented.

[0348] Circuit 260 is a circuit that performs information processing and can access memory 262. For example, circuit 260 may be a dedicated or general-purpose electronic circuit for decoding moving images. Circuit 260 can also be a processor like a CPU. Alternatively, circuit 260 can be an assembly of multiple electronic circuits. Furthermore, circuit 260 can also, for example, function as... Figure 10 The functions of multiple components of the decoding device 200 shown, excluding the component for storing information.

[0349] Memory 262 is a dedicated or general-purpose memory that stores information used by circuit 260 to decode moving images. Memory 262 can be an electronic circuit or connected to circuit 260. Alternatively, memory 262 can be included within circuit 260. Alternatively, memory 262 can be an assembly of multiple electronic circuits. Alternatively, memory 262 can be a disk or optical disk, or it can take the form of a memory or recording medium. Furthermore, memory 262 can be either non-volatile or volatile memory.

[0350] For example, memory 262 can store bit strings corresponding to the encoded motion image, or it can store motion images corresponding to the decoded bit strings. Additionally, memory 262 can also store a program for circuit 260 to decode the motion image.

[0351] Additionally, for example, memory 262 can serve as Figure 10 The decoding device 200 shown above has multiple components, including the component for storing information. Specifically, the memory 262 can serve as... Figure 10 The block memory 210 and frame memory 214 shown herein serve a specific purpose. More specifically, memory 262 can store reconstructed blocks and reconstructed images, etc.

[0352] In addition, in the decoding device 200, it is not necessary to install Figure 10 All of the multiple constituent elements shown above can also be processed without the aforementioned multiple processing steps. Figure 10 A portion of the multiple components shown may be included in other devices, or a portion of the aforementioned multiple processes may be performed by other devices. Then, in the decoding device 200, an installation is performed... Figure 10 One of the multiple components shown above can be used to efficiently perform motion compensation by performing one of the multiple processes described above.

[0353] Specifically, the decoding device 200 derives the first motion vector by dividing the motion image into prediction block units and using a first-frame prediction method based on the fit of two reconstructed images of two different regions within two images. Figure 13 (S104 or S106). Here, the first-frame prediction method is, for example, the FRUC method described above. Specifically, the first-frame prediction method includes at least one of the template FRCU method and the bidirectional FRUC method. That is, the two regions mentioned above in the first-frame prediction method are (1) the region within the object image adjacent to the object prediction block and the region within the reference image, or (2) two regions within two different reference images.

[0354] In other words, the first-frame prediction method refers to a method in which motion vectors are derived using the same method on both the encoding and decoding sides. Furthermore, in the first-frame prediction method, information representing motion vectors is not signaled to the encoded stream and is not transmitted from the encoding side to the decoding side. Additionally, in the first-frame prediction method, the decoding device 200 uses the pixel values ​​of the decoded prediction blocks and does not use the pixel values ​​of the object prediction blocks to derive the motion vectors.

[0355] Next, the decoding device 200 performs a first motion compensation process, which generates a prediction image on a block-by-block basis, referring to the spatial gradient of brightness in the image generated by motion compensation using the derived first motion vector. Figure 13(S103A). Here, the first motion compensation process refers to, for example, the BIO process described above, including correction using a brightness gradient. Furthermore, in the first motion compensation process, the predicted image is corrected in units smaller than the predicted block (e.g., pixel units or block units). Additionally, in the first motion compensation process, the predicted image is generated using a region within a reference image represented by a motion vector and the pixels surrounding that region.

[0356] Therefore, by performing motion vector derivation processing based on the first inter-frame prediction method and first motion compensation processing on a prediction block basis, the decoding apparatus 200 can reduce the processing load compared to, for example, performing these processes on a sub-block basis. Furthermore, since the first motion compensation processing, which includes correction using luminance gradients, can be implemented in units smaller than the prediction block unit, the reduction in coding efficiency that would occur without processing on a sub-block basis can be suppressed. Thus, the decoding apparatus 200 can reduce the processing load while suppressing a reduction in coding efficiency.

[0357] Furthermore, the decoding device 200 obtains information from the encoded bitstream for determining the second motion vector in prediction block units. The decoding device 200 derives the second motion vector in prediction block units using a second-frame prediction method that utilizes the aforementioned information. Figure 13 (S102).

[0358] Here, the second inter-frame prediction method refers to, for example, the typical inter-frame prediction method described above. In other words, the second inter-frame prediction method refers to a method of deriving motion vectors using different methods on the encoding and decoding sides. Specifically, the encoding device 100 derives motion vectors using the pixel values ​​of the encoded prediction blocks and the pixel values ​​of the object prediction blocks. Then, the encoding device 100 signals information representing the derived motion vectors to the encoding stream. Thus, information representing the motion vectors derived by the encoding device 100 is transmitted from the encoding device 100 to the decoding device 200. The decoding device 200 derives motion vectors using this information contained in the encoding stream.

[0359] Next, the decoding device 200 performs a second motion compensation process, which generates a prediction image on a block-by-block basis, referring to the spatial gradient of brightness in the image generated by motion compensation using the derived second motion vector. Figure 13 (S103A). Here, the second motion compensation process refers to, for example, the BIO process described above, including correction using a brightness gradient. Furthermore, in the second motion compensation process, the predicted image is corrected in units smaller than the predicted block (e.g., pixel units or block units). Additionally, in the second motion compensation process, the predicted image is generated using a region within a reference image represented by a motion vector and the pixels surrounding that region.

[0360] In addition, the second motion compensation process can be the same as the first motion compensation process, or it can be a part of a different process.

[0361] Therefore, whether using the first-frame prediction method or the second-frame prediction method, the processing unit for motion compensation processing can be the same. This simplifies the installation of motion compensation processing.

[0362] Furthermore, in the first operation mode (the first operation mode in S111), the decoding device 200 derives a first motion vector by using a first inter-frame prediction method, dividing the image contained in the moving image into prediction block units (S112), and performs a first motion compensation process using the derived first motion vector in prediction block units (S113). Here, the first motion compensation process is, for example, BIO processing, which is a motion compensation process for generating the predicted image by referring to the spatial gradient of brightness in the image generated by motion compensation using the derived first motion vector.

[0363] Furthermore, in the second operation mode (the second operation mode in S111), the decoding device 200 derives a second motion vector in sub-block units that divide the prediction block using a second inter-frame prediction method (S114), and performs a second motion compensation process using the second motion vector in sub-block units (S115). Here, the second motion compensation process refers to, for example, motion compensation processing that does not apply BIO processing, and motion compensation processing that generates the prediction image without referring to the spatial gradient of brightness in the image generated by motion compensation using the second motion vector.

[0364] Therefore, in the first operation mode, the decoding device 200 reduces the processing workload by performing motion vector derivation processing and the first motion compensation processing on a prediction block basis, compared to performing these processes on a sub-block basis. Furthermore, since the first motion compensation processing, which generates the prediction image based on the spatial gradient of brightness, can achieve correction in units smaller than the prediction block unit, the reduction in coding efficiency when processing is not performed on a sub-block basis can be suppressed. In the second operation mode, the decoding device 200 performs motion vector derivation processing and the second motion compensation processing on a sub-block basis. Here, the second motion compensation processing, since it does not refer to the spatial gradient of brightness, requires less processing compared to the first motion compensation processing. Thus, by having these two operation modes, the decoding device 200 can improve coding efficiency. In this way, the decoding device 200 can reduce the processing workload while suppressing the reduction in coding efficiency.

[0365] For example, the prediction method between the first frame and the prediction method between the second frame are different. Specifically, the prediction method between the second frame is an inter-frame prediction method that uses the fit of two reconstructed images of two different regions within two different images, such as the FRUC method.

[0366] Therefore, this inter-frame prediction method, which enables the calculation of motion vectors based on sub-block units, significantly improves coding efficiency. Thus, it enhances coding efficiency.

[0367] For example, the first inter-frame prediction method is one of (1) the third inter-frame prediction method (e.g., template FRUC method) which uses the fit of the reconstructed image of the region in the object picture adjacent to the object prediction block and the reconstructed image of the region in the reference picture, and (2) the fourth inter-frame prediction method (e.g., bidirectional FRUC method) which uses the fit of the two reconstructed images of two regions in two different reference pictures, and the second inter-frame prediction method is the other of the third and fourth inter-frame prediction methods.

[0368] For example, the prediction method between the first frame is the prediction method between the third frame (e.g., template FRUC method), and the prediction method between the second frame is the prediction method between the fourth frame (e.g., bidirectional FRUC method).

[0369] Therefore, this inter-frame prediction method, which enables the calculation of motion vectors based on sub-block units, significantly improves coding efficiency. Thus, it enhances coding efficiency.

[0370] For example, in the first-frame prediction mode, the decoding device 200 obtains information from the encoded bitstream for determining the second motion vector in sub-block units, and uses the information to derive the second motion vector.

[0371] [Additional Explanation] Furthermore, the encoding device 100 and decoding device 200 of this embodiment can be used as an image encoding device and an image decoding device, respectively, or as a moving image encoding device and a moving image decoding device. Alternatively, the encoding device 100 and decoding device 200 can be used as inter-frame prediction devices (inter-picture prediction devices).

[0372] That is, the encoding device 100 and the decoding device 200 may correspond only to the inter-frame prediction unit (inter-picture prediction unit) 126 and the inter-frame prediction unit (inter-picture prediction unit) 218, respectively. Furthermore, other components such as the transformation unit 106 and the inverse transformation unit 206 may also be included in other devices.

[0373] Furthermore, in this embodiment, each component can be constructed using dedicated hardware, or it can be implemented by executing software programs suitable for each component. Each component can also be implemented by a program execution unit such as a CPU or processor reading and executing software programs recorded on a recording medium such as a hard disk or semiconductor memory.

[0374] Specifically, the encoding device 100 and the decoding device 200 may each include a processing circuit and a storage device electrically connected to and accessible from the processing circuit. For example, the processing circuit corresponds to circuit 160 or 260, and the storage device corresponds to memory 162 or 262.

[0375] The processing circuit includes at least one of dedicated hardware and a program execution unit, and uses a storage device to perform the processing. Additionally, if the processing circuit includes a program execution unit, the storage device stores a software program executed by that program execution unit.

[0376] Here, the software for implementing the encoding device 100 or decoding device 200 of this embodiment is the following program.

[0377] Furthermore, as mentioned above, each component can also be a circuit. These circuits can either form a single circuit as a whole or be separate circuits. Additionally, each component can be implemented using either a general-purpose processor or a dedicated processor.

[0378] Alternatively, other components may perform the processing performed by a specific component. Furthermore, the order of processing can be changed, and multiple processes can be performed simultaneously. Alternatively, the encoding / decoding apparatus may include an encoding device 100 and a decoding device 200.

[0379] The above description illustrates the configurations of the encoding device 100 and the decoding device 200 based on the embodiments, but the configurations of the encoding device 100 and the decoding device 200 are not limited to this embodiment. As long as they do not depart from the spirit of the invention, various modifications that can be conceived by those skilled in the art to this embodiment, and configurations constructed by combining the constituent elements of different embodiments, can also be included within the scope of the configurations of the encoding device 100 and the decoding device 200.

[0380] This method can also be implemented in combination with at least a portion of other methods in this invention. Furthermore, a portion of the processing described in the flowchart of this method, a portion of the device structure, a portion of the syntax, etc., can be combined with other methods to implement it.

[0381] (Implementation Method 2) In the above embodiments, each functional block is typically implemented using an MPU and memory. Furthermore, the processing of each functional block is usually achieved by a program execution unit such as a processor reading and executing software (programs) recorded in a recording medium such as ROM. This software can be distributed via download or by recording it in a recording medium such as semiconductor memory. Alternatively, each functional block can also be implemented in hardware (dedicated circuitry).

[0382] Furthermore, the processing described in each embodiment can be implemented either centrally using a single device (system) or distributedly using multiple devices. Additionally, the processor executing the above-described program can be either single or multiple. That is, it can be either centrally processed or distributed.

[0383] The present invention is not limited to the above embodiments, and various modifications can be made, which are also included within the scope of the present invention.

[0384] Furthermore, examples of applications of the motion picture encoding method (image encoding method) or motion picture decoding method (image decoding method) shown in the above embodiments and systems using them will be described here. The system is characterized by having an image encoding apparatus using the image encoding method, an image decoding apparatus using the image decoding method, and an image encoding / decoding apparatus possessing both. Other structures within the system can be appropriately modified as needed.

[0385] [Usage Examples] Figure 23 This is a diagram showing the overall structure of the content delivery system ex100 that implements content distribution services. The provision of communication services is divided into desired sizes, and each unit is equipped with base stations ex106, ex107, ex108, ex109, and ex110, which serve as fixed wireless stations.

[0386] In this content delivery system ex100, various devices such as computer ex111, game console ex112, camera ex113, home appliance ex114, and smartphone ex115 are connected to the Internet ex101 via an Internet service provider ex102 or a communication network ex104, and base stations ex106 to ex110. This content delivery system ex100 can also combine some of the above elements for connection. Alternatively, the devices can be directly or indirectly interconnected via telephone networks or short-range wireless connections without using base stations ex106 to ex110, which are fixed wireless stations. Furthermore, a streaming media server ex103 is connected to the computer ex111, game console ex112, camera ex113, home appliance ex114, and smartphone ex115 via the Internet ex101, etc. Additionally, the streaming media server ex103 is connected to terminals within a hotspot inside an aircraft ex117 via satellite ex116.

[0387] Alternatively, it can replace base stations ex106 to ex110 by using wireless access points or hotspots. Furthermore, the streaming media server ex103 can connect directly to the communication network ex104 without going through the Internet ex101 or the Internet service provider ex102, or it can connect directly to the aircraft ex117 without going through the satellite ex116.

[0388] The camera ex113 is a digital camera or similar device capable of capturing still and moving images. Additionally, the smartphone ex115 refers to a smartphone, mobile phone, or PHS (Personal Handyphone System) corresponding to mobile communication systems commonly referred to as 2G, 3G, 3.9G, 4G, and the future 5G.

[0389] Home appliance EX118 refers to refrigerators or other equipment included in a home fuel cell combined heat and power system.

[0390] In the content supply system ex100, terminals with photography capabilities are connected to the streaming media server ex103 via a base station ex106, enabling on-site distribution. During on-site distribution, terminals (such as computers ex111, game consoles ex112, cameras ex113, home appliances ex114, smartphones ex115, and terminals within airplanes ex117) perform encoding processing on still or moving images captured by the user using the terminal, as described in the above embodiments. The encoded image data and the encoded audio data are multiplexed, and the resulting data is sent to the streaming media server ex103. In other words, each terminal functions as an image encoding device according to one aspect of the present invention.

[0391] On the other hand, the streaming media server ex103 distributes the content data sent by requesting clients. Clients are terminals such as computers ex111, game consoles ex112, cameras ex113, home appliances ex114, smartphones ex115, or airplanes ex117 capable of decoding the encoded data. Each device receiving the distributed data decodes and reproduces the received data. That is, each device functions as an image decoding device according to one aspect of the present invention.

[0392] [Decentralized processing] Furthermore, the streaming media server ex103 can also be multiple servers or multiple computers, distributing data through decentralized processing or recording. For example, the streaming media server ex103 can also be implemented using a CDN (Contents Delivery Network), which distributes content by connecting many edge servers scattered around the world. In a CDN, physically nearby edge servers are dynamically allocated based on the client. Furthermore, by caching and distributing content to these edge servers, latency can be reduced. Moreover, in the event of an error or a change in communication status due to increased traffic, processing can be distributed across multiple edge servers, the distribution entity can be switched to other edge servers, or the faulty part of the network can be bypassed to continue distribution, thus achieving high-speed and stable distribution.

[0393] Furthermore, beyond the decentralized processing of distribution itself, the encoding processing of the captured data can be performed by each terminal, on the server side, or shared among them. For example, the encoding process typically involves two processing loops. In the first loop, the complexity or code size of the image per frame or scene unit is detected. In the second loop, processing is performed to maintain image quality while improving encoding efficiency. For instance, by having the terminal perform the first encoding processing and the server receiving the content perform the second, the processing load on each terminal can be reduced while improving both content quality and efficiency. In this case, if there is a request for near real-time reception and decoding, the data encoded by the terminal in the first round can be received and reproduced by other terminals, enabling more flexible real-time distribution.

[0394] As another example, cameras like the ex113 extract features from images, compress the data about these features as metadata, and send it to a server. The server then uses these features to determine the importance of the target and adjusts the quantization precision accordingly, performing compression that corresponds to the meaning of the image. The feature data is particularly effective in improving the accuracy and efficiency of motion vector prediction during further compression on the server. Alternatively, simple encoding such as VLC (Variable Length Coding) can be performed by the terminal, while more demanding encoding methods like CABAC (Context Adaptive Binary Arithmetic Coding) can be used by the server.

[0395] As another example, in stadiums, shopping malls, or factories, there may be multiple images of roughly the same scene captured by multiple terminals. In such cases, the data is distributed and encoded separately using the terminals that captured the images, as well as other terminals and servers that did not capture images, as needed. This can be done by assigning encoding and processing data to different units, such as GOP (Group of Pictures), image units, or tile units obtained by segmenting images. This reduces latency and improves real-time performance.

[0396] Furthermore, since multiple image datasets depict roughly the same scene, the server can manage and / or instruct the data to be cross-referenced between images captured by different terminals. Alternatively, the server can receive encoded data from each terminal and change the reference relationships between the multiple datasets, or modify or replace the images themselves and re-encode them. This allows for the generation of streams with improved quality and efficiency for each data item.

[0397] In addition, the server can also transcode the image data by changing its encoding method before distributing it. For example, the server can convert MPEG encoding to VP encoding, or H.264 to H.265.

[0398] In this way, encoding processing can be performed by a terminal or one or more servers. Therefore, the terms "server" or "terminal" will be used below to refer to the main body performing the processing, but it is also possible to perform part or all of the processing performed by the server by the terminal, or vice versa. Furthermore, the same applies to decoding processing.

[0399] [3D, Multi-angle] In recent years, there has been an increase in the use of images or videos captured by multiple cameras (ex113 and / or smartphones (ex115)) at roughly the same time, capturing different scenes or capturing the same scene from different angles. These images are then merged based on the relative positions of the cameras or regions containing consistent feature points within the images.

[0400] The server not only encodes 2D moving images, but can also automatically or at user-specified times encode still images based on scene analysis of moving images and send them to the receiving terminal. Furthermore, when the server can obtain the relative positions between shooting terminals, it can generate 3D shapes of scenes not only from 2D moving images, but also from images of the same scene captured from different angles. Additionally, the server can separately encode 3D data generated from point clouds, and can also select or reconstruct images from multiple terminals based on the results of identifying or tracking people or targets using 3D data to generate images to be sent to the receiving terminal.

[0401] In this way, users can freely select images corresponding to each shooting terminal to appreciate the scene, and can also appreciate the content of images extracted from any viewpoint from 3D data reconstructed using multiple images or videos. Furthermore, similar to the images, sound can also be collected from multiple different angles, and the server can match it with the images to multiplex and send sound from a specific angle or space with the images.

[0402] Furthermore, in recent years, content that establishes a correspondence between the real world and the virtual world, such as Virtual Reality (VR) and Augmented Reality (AR), has become increasingly popular. In the case of VR images, the server creates separate viewpoint images for the right and left eyes. This can be achieved through Multi-View Coding (MVC) to allow reference between the viewpoint images, or by encoding them as separate streams without reference. During decoding of these different streams, they can be synchronously reproduced according to the user's viewpoint to recreate a virtual three-dimensional space.

[0403] In the case of AR images, the server can overlay virtual object information in virtual space onto camera information in real space based on the 3D position or the user's viewpoint movement. The decoding device acquires or holds the virtual object information and 3D data, generates a 2D image based on the user's viewpoint movement, and creates overlay data by smoothly connecting them. Alternatively, the decoding device can send the user's viewpoint movement to the server in addition to the virtual object information. The server creates overlay data by matching the received viewpoint movement with the 3D data held on the server, encodes the overlay data, and distributes it to the decoding device. Furthermore, the overlay data has an α value representing transmittance in addition to RGB. The server sets the α value of the portion outside the target created from the 3D data to 0, etc., and encodes the portion in a state of transmittance. Alternatively, the server can set a predetermined RGB value as the background, similar to a chroma key, to generate data where the portion outside the target is set as the background color.

[0404] Similarly, the decoding of distributed data can be performed by the individual terminals acting as clients, on the server side, or distributed among them. For example, one terminal could first send a receive request to the server, other terminals could receive the content corresponding to that request, decode it, and then send the decoded signal to a device with a display. By distributing the processing independently of the capabilities of the communicating terminals and selecting appropriate content, data with better image quality can be reproduced. Furthermore, as another example, large-format image data can be received by a TV, and the viewer's personal terminal can decode and display a segmented area of ​​the image, such as tiles. This allows for the sharing of the overall image while simultaneously allowing the viewer to identify their own area of ​​responsibility or areas they wish to examine in more detail.

[0405] Furthermore, it is envisioned that in the future, with the availability of multiple short-range, medium-range, or long-range wireless communications both indoors and outdoors, content can be seamlessly received while switching appropriate data between connected communications using distribution system standards such as MPEG-DASH. This would allow users to switch in real-time not only using their own terminals but also freely choosing decoding or display devices such as monitors installed indoors or outdoors. Additionally, decoding can be performed by switching between decoding and display terminals based on the user's location information. This would also allow movement towards a destination while displaying map information on a portion of the wall or ground of a building adjacent to a display device. Furthermore, the bit rate of the received data can be switched based on the ease of accessing encoded data to the network, such as when the encoded data is cached on a server accessible only briefly from the receiving terminal or copied to an edge server of the content distribution service.

[0406] [Hyper-level coding] Regarding content switching, use Figure 24 The scalable stream, which is compressed using the motion picture coding method described in the above embodiments, will be explained. For the server, multiple streams with the same content but different qualities can exist as a single stream, or the content structure can be switched using the characteristics of a temporally / spatially scalable stream achieved through layered coding, as shown in the illustration. That is, by having the decoding side determine which layer to decode to based on intrinsic factors such as performance and extrinsic factors such as the state of the communication band, the decoding side can freely switch between decoding low-resolution and high-resolution content. For example, if one wants to watch a follow-up video viewed on a smartphone (ex115) while on the go, and then watch it at home on an internet TV or similar device, the device only needs to decode the same stream to a different layer, thus reducing the burden on the server side.

[0407] Furthermore, in addition to the hierarchical structure described above, where images are encoded layer by layer and enhancement layers exist above the base layers, the enhancement layer can also contain metadata such as image statistics. The decoding side then uses this metadata to perform super-resolution on the base layer images to generate high-quality content. Super-resolution can be either an increase in the signal-to-noise ratio (SN ratio) at the same resolution or an increase in resolution. The metadata includes information used to determine the linear or nonlinear filtering coefficients used in the super-resolution process, or information determining the parameter values ​​for filtering, machine learning, or least-squares operations used in the super-resolution process.

[0408] Alternatively, the image can be segmented into tiles based on the meaning of targets within it, and the decoding side can select tiles to decode, thus decoding only a portion of the region. Furthermore, by storing the target's attributes (people, cars, balls, etc.) and its position within the image (coordinates within the same image, etc.) as metadata, the decoding side can determine the desired target's location based on this metadata and decide which tiles to include that target. For example, as... Figure 25 As shown, metadata is stored using data storage structures different from pixel data, such as SEI messages in HEVC. This metadata may represent, for example, the position, size, or color of the main target.

[0409] Furthermore, metadata can be stored in units consisting of multiple images, such as streams, sequences, or random access units. This allows the decoder to obtain information such as the time a specific person appears in the image, and by matching this information with the image unit information, it can determine the image containing the target and the target's location within that image.

[0410] [Web page optimization] Figure 26 This is an example of a web page display screen in a computer such as ex111. Figure 27 This is an example image showing the display screen of a web page in a smartphone such as the ex115. Figure 26 and Figure 27 As shown, in cases where a web page contains multiple linked images that serve as links to image content, their visibility varies depending on the viewing device. When multiple linked images are visible on the screen, before the user explicitly selects a linked image, or before the linked image is near the center of the screen, or before the entire linked image enters the screen, the display device (decoding device) displays still images or I-images of each content as linked images, or displays images like GIF animations using multiple still images or I-images, or only receives the basic layer and decodes and displays the image.

[0411] When a user selects a linked image, the display device prioritizes decoding the base layer. Additionally, if the HTML constituting the webpage contains information indicating tiered content, the display device can also decode up to the enhancement layer. Furthermore, in situations where real-time performance is ensured before selection or when communication bandwidth is extremely limited, the display device can reduce the delay between decoding and displaying the first image (the delay from the start of content decoding to the start of display) by decoding and displaying only the preceding reference images (I images, P images, and B images only used for preceding reference). Alternatively, the display device can forcibly ignore image reference relationships and coarsely decode all B and P images as preceding references, performing normal decoding as more images are received over time.

[0412] [Autonomous Driving] Furthermore, when receiving and transmitting still images or video data such as two-dimensional or three-dimensional map information for the purpose of autonomous driving or driving assistance, the receiving terminal can also receive information such as weather or construction as metadata, in addition to image data belonging to more than one layer, and decode them by establishing correspondences. Moreover, metadata can belong to a layer or be multiplexed only with image data.

[0413] In this scenario, since the vehicle, drone, or aircraft containing the receiving terminal is in motion, the receiving terminal can seamlessly receive and decode data by switching between base stations ex106 to ex110 by sending its location information when a request is received. Furthermore, the receiving terminal can dynamically adjust the level of metadata reception or map information updates based on user selection, user status, or the status of the communication frequency band.

[0414] As described above, in the content delivery system ex100, the client can receive, decode, and reproduce the encoded information sent by the user in real time.

[0415] [Distribution of Personal Content] Furthermore, the content delivery system ex100 not only handles high-quality, long-duration content provided by video distribution providers, but also enables unicast or multicast distribution of low-quality, short-duration content provided by individuals. Moreover, it's conceivable that such personal content will increase in the future. To improve the quality of personal content, the server can also perform encoding processing after editing. This can be achieved, for example, through a structure like the following.

[0416] During or after capturing images, the server performs image processing such as shooting errors, scene search, meaning analysis, and target detection based on the original images or encoded data. Furthermore, based on the recognition results, the server manually or automatically corrects focus deviations or camera shake, deletes scenes of lower importance (e.g., scenes with lower brightness or out-of-focus areas), emphasizes target edges, or adjusts tones. The server then encodes the edited data. Additionally, recognizing that longer shooting times reduce audiovisual quality, the server can automatically limit not only low-importance scenes as described above but also scenes with minimal movement to specific timeframes based on image processing results. Alternatively, the server can generate and encode summaries based on the meaning analysis results of the scenes.

[0417] Furthermore, personal content may contain elements that infringe on copyrights, the author's moral rights, or portrait rights in its original state, or the sharing scope may exceed the intended scope, causing inconvenience to the individual. Therefore, for example, the server could forcibly encode images such as faces of people in the periphery of a scene or a home, replacing them with out-of-focus images. Additionally, the server could identify whether a face different from a pre-registered person is captured within the image to be encoded, and if so, apply processing such as mosaic to the face. Alternatively, as pre- or post-processing for encoding, from a copyright perspective, the user can specify the person or background area to be processed, and the server can replace the specified area with another image or blur the focus. If it is a person, the face image can be replaced while tracking the person in a moving image.

[0418] Furthermore, personal content with small data volumes has strong real-time requirements for audiovisual presentation. Therefore, although bandwidth also plays a role, the decoding device prioritizes receiving, decoding, and reproducing the base layer. The decoding device can also receive enhancement layers during this process, including them in high-quality image reproduction if the playback is looped or repeated more than twice. In this way, if the stream is scalably encoded, it can provide an experience where the motion images are initially coarse but gradually become smoother and the image quality improves. Besides scalable encoding, the same experience can be provided when the first, coarser stream and a second stream encoded based on the first motion image are combined into a single stream.

[0419] [Other Use Cases] Furthermore, these encoding or decoding processes are typically handled within the LSIex500 chip present in each terminal. The LSIex500 can be a single-chip or a multi-chip structure. Alternatively, software for motion image encoding or decoding can be installed on a recording medium (CD-ROM, floppy disk, hard disk, etc.) that can be read by a computer such as the ex111, and the encoding and decoding processes can be performed using this software. Furthermore, when the ex115 smartphone has a camera, motion image data acquired by that camera can also be transmitted. This motion image data is encoded using the LSIex500 chip present in the ex115 smartphone.

[0420] Alternatively, the LSIex500 can also be a structure that downloads and activates application software. In this case, the terminal first determines whether it corresponds to the content's encoding method or whether it has the capability to execute a specific service. If the terminal does not correspond to the content's encoding method or does not have the capability to execute a specific service, the terminal downloads the codec or application software, and then retrieves and reproduces the content.

[0421] Furthermore, not limited to the content delivery system ex100 via the Internet ex101, at least one of the moving image encoding device (image encoding device) or moving image decoding device (image decoding device) of the above embodiments can also be assembled in a digital broadcasting system. Since multiplexed data that multiplexes images and sound is carried and transmitted and received using radio waves for broadcasting via satellites or the like, it is more suitable for multicasting than the unicast-friendly structure of the content delivery system ex100, but the same applications can be performed for encoding and decoding processing.

[0422] [Hardware Structure] Figure 28 This is a diagram representing the EX115 smartphone. Additionally, Figure 29This diagram illustrates a structural example of a smartphone ex115. The smartphone ex115 includes an antenna ex450 for transmitting and receiving radio waves with a base station ex110, a camera unit ex465 for capturing images and still images, and a display unit ex458 for displaying decoded data such as images captured by the camera unit ex465 and images received by the antenna ex450. The smartphone ex115 also includes an operation unit ex466, such as a touch panel; a sound output unit ex457, such as a speaker, for outputting sound or audio; a sound input unit ex456, such as a microphone, for inputting sound; a memory unit ex467 capable of storing encoded or decoded data such as captured images or still images, recorded audio, received images or still images, and emails; and a slot unit ex464 serving as an interface with a SIM ex468 used to identify the user and authenticate access to various data sources, such as the network. Alternatively, an external memory may be used instead of the memory unit ex467.

[0423] Furthermore, the main control unit ex460, which performs integrated control of the display unit ex458 and the operation unit ex466, is interconnected with the power supply circuit unit ex461, the operation input control unit ex462, the image signal processing unit ex455, the camera interface unit ex463, the display control unit ex459, the modulation / demodulation unit ex452, the multiplexing / demultiplexing unit ex453, the audio signal processing unit ex454, the slot unit ex464, and the memory unit ex467 via the bus ex470.

[0424] If the power button is turned on by the user, the power circuit section ex461 will supply power to each part from the battery pack, thus activating the smartphone ex115 into an operational state.

[0425] The smart phone ex115 is controlled by a main control unit ex460, which includes a CPU, ROM, and RAM, for call and data communication processing. During a call, the audio signal collected by the audio input unit ex456 is converted into a digital audio signal by the audio signal processing unit ex454. This digital signal is then subjected to spectral diffusion processing by the modulation / demodulation unit ex452, and finally, digital-to-analog conversion and frequency conversion processing by the transmitting / receiving unit ex451 before being transmitted via the antenna ex450. Similarly, received data is amplified and subjected to frequency conversion and analog-to-digital conversion processing. The data is then subjected to inverse spectral diffusion processing by the modulation / demodulation unit ex452, converted into an analog audio signal by the audio signal processing unit ex454, and output from the audio output unit ex457. During data communication, text, still images, or video data are sent to the main control unit ex460 via the operation input control unit ex462 through the operation unit ex466 of the main unit, and are also processed for transmission and reception. In data communication mode, when transmitting images, still images, or images and sound, the image signal processing unit ex455 compresses and encodes the image signal stored in the memory unit ex467 or the image signal input from the camera unit ex465 using the moving image encoding method described in the above embodiments, and sends the encoded image data to the multiplexing / demultiplexing unit ex453. Furthermore, the sound signal processing unit ex454 encodes the sound signal collected by the sound input unit ex456 during the capture of images, still images, etc., by the camera unit ex465, and sends the encoded sound data to the multiplexing / demultiplexing unit ex453. The multiplexing / demultiplexing unit ex453 multiplexes the encoded image data and encoded sound data in a prescribed manner, performs modulation and conversion processing by the modulation / demodulation unit (modulation / demodulation circuit unit) ex452 and the transmitting / receiving unit ex451, and transmits the data via the antenna ex450.

[0426] When receiving images attached to emails or chat tools, or images linked to web pages, the multiplexing / demultiplexing unit ex453, in order to decode the multiplexed data received via antenna ex450, separates the multiplexed data into a bitstream of image data and a bitstream of audio data. The encoded image data is supplied to the image signal processing unit ex455 via the synchronization bus ex470, and the encoded audio data is supplied to the audio signal processing unit ex454. The image signal processing unit ex455 decodes the image signal using a motion picture decoding method corresponding to the motion picture encoding method described in the above embodiments, and displays the image or still image contained in the linked motion picture file from the display unit ex458 via the display control unit ex459. Furthermore, the audio signal processing unit ex454 decodes the audio signal and outputs audio from the audio output unit ex457. Additionally, since real-time streaming is becoming increasingly common, depending on the user's situation, the reproduction of audio may be unsuitable for certain situations. Therefore, as an initial consideration, a structure that reproduces only the image data and not the audio signal is preferred. It can also reproduce sound synchronously only when the user performs actions such as clicking on the image data.

[0427] Furthermore, while the example given here is the ex115 smartphone, as a terminal, three installation methods can be considered: a transmitting terminal with both an encoder and a decoder, a transmitting terminal with only an encoder, and a receiving terminal with only a decoder. Moreover, in a digital broadcasting system, the example described involves receiving and transmitting multiplexed data, such as audio data, multiplexed within video data. However, in addition to audio data, multiplexed data can also multiplex character data associated with the video, or the video data itself can be received or transmitted without using multiplexed data.

[0428] Furthermore, while the description assumes the CPU's main control unit (ex460) controls the encoding or decoding process, many terminals also possess GPUs. Therefore, a structure can be implemented that utilizes GPU performance to process larger regions simultaneously, using shared memory between the CPU and GPU, or memory that manages addresses in a shared manner. This reduces encoding time, ensures real-time performance, and achieves low latency. In particular, it is even more efficient to perform motion estimation, deblocking filtering, SAO (Sample Adaptive Offset), and transform / quantization processing on the GPU, image-by-image, rather than using the CPU.

[0429] This method can also be implemented in combination with at least a portion of other methods in this invention. Furthermore, a portion of the processing described in the flowchart of this method, a portion of the device structure, a portion of the syntax, etc., can be combined with other methods to implement it.

[0430] Industrial availability This invention can be used in, for example, television receivers, digital video recorders, car navigation systems, mobile phones, digital cameras, digital video cameras, video conferencing systems, or electronic mirrors.

[0431] Label Explanation 100 encoding device 102 Division 104 Subtraction Section 106 Transformer Quantitative Department 108 110 Entropy Coding Department 112, 204 Inverse Quantization Section Inverse Transformation Units 114 and 206 116, 208 Addition Department 118 and 210 memory blocks 120, 212 Circular Filter Section 122, 214 frame memory Intra-frame prediction units 124 and 216 Inter-frame prediction units 126 and 218 Predictive Control Department 128, 220 160, 260 circuits 162, 262 memory 200 Decoding Device 202 Entropy Decoding Department

Claims

1. A decoding device, characterized in that, have: Circuits; and Memory; The circuit described above uses the memory described above to perform the following processing. In the first action mode, Using a candidate list containing motion vector candidates, the first motion vector is derived in units of predicted patches obtained from image segmentation. By referring to the spatial gradient of brightness generated based on the first motion vector mentioned above, a prediction image corresponding to the prediction block is generated. In the second action mode, The second motion vector is derived by dividing the predicted block into sub-blocks. The predicted image corresponding to the above sub-blocks is generated without referring to the spatial gradient of brightness.

2. An encoding device, characterized in that, have: Circuits; and Memory; The circuit described above uses the memory described above to perform the following processing. In the first action mode, Using a candidate list containing motion vector candidates, the first motion vector is derived in units of predicted patches obtained from image segmentation. By referring to the spatial gradient of brightness generated based on the first motion vector mentioned above, a prediction image corresponding to the prediction block is generated. In the second action mode, The second motion vector is derived by dividing the predicted block into sub-blocks. The predicted image corresponding to the above sub-blocks is generated without referring to the spatial gradient of brightness.

3. A decoding method, characterized in that, In the first action mode, Using a candidate list containing motion vector candidates, the first motion vector is derived in units of predicted patches obtained from image segmentation. By referring to the spatial gradient of brightness generated based on the first motion vector mentioned above, a prediction image corresponding to the prediction block is generated. In the second action mode, The second motion vector is derived by dividing the predicted block into sub-blocks. The predicted image corresponding to the above sub-blocks is generated without referring to the spatial gradient of brightness.

4. A bit stream transmission device, characterized in that, have: Circuit; as well as Memory; The circuit described above uses the memory described above to generate a bit stream by executing an encoding method and then transmits the bit stream. The above encoding methods include: In the first action mode, Using a candidate list containing motion vector candidates, the first motion vector is derived in units of predicted patches obtained from image segmentation. By referring to the spatial gradient of brightness generated based on the first motion vector mentioned above, a prediction image corresponding to the prediction block is generated. In the second action mode, The second motion vector is derived by dividing the predicted block into sub-blocks. The predicted image corresponding to the above sub-blocks is generated without referring to the spatial gradient of brightness.

Citation Information

Patent Citations

  • Dynamic image coding device, dynamic image coding method and dynamic image coding program

    JP2017034408A

  • Video encoding device, video encoding method, video encoding program, transmitting device, transmitting method and transmitting program; and video decoding device, video decoding method, video decoding program, receiving device, receiving method and receiving program

    WO2013046707A1