Encoding methods, decoding methods, and non-transitory computer-readable media

Abstract

In inter-prediction mode, the encoding device generates a first predicted image of the target block based on the derived motion vector, and applies an update process to the first predicted image to generate the final predicted image of the target block. Here, the update process candidates include a first process and a second process; the first process is a BDOF process, and the second process is a process that mixes the second predicted image generated by the intra-prediction of the target block with the first predicted image. Then, in the application of the update process, the first process and the second process are applied exclusively.

Description

Technical Field

[0001] This disclosure relates to a video coding system, such as systems, components, and methods for encoding and decoding moving images. Prior Technology

[0002] Video coding technologies have progressed from H.261 and MPEG-1 to H.264 / AVC (Advanced Video Coding), MPEG-LA, H.265 / HEVC (High Efficiency Video Coding), and H.266 / VVC (Versatile Video Codec). Along with this progress, improvements and optimizations of video coding technologies are frequently needed to handle the ever-increasing volume of digital video data across a wide range of applications.

[0003] Additionally, Non-Patent Document 1 is an example of a known specification related to the aforementioned video coding technology.

[0004] [Preliminary Technology Documents] [Non-patent literature] [Non-patent literature 1] H.265 (ISO / IEC 23008-2 HEVC) / HEVC (High Efficiency Video Coding) Summary of the Invention

[0005] [Invention Summary] [The problem the invention aims to solve] Regarding the encoding method described above, in order to improve encoding efficiency, image quality, reduce processing load, reduce circuit size, or make appropriate selections of elements or actions such as filters, blocks, sizes, motion vectors, reference images, or reference blocks, we hope to have new proposals.

[0006] This disclosure provides a configuration or method that contributes to one or more of the following: improved encoding efficiency; improved image quality; reduced processing load; reduced circuit size; improved processing speed; and appropriate selection of elements or actions. Furthermore, this disclosure may include configurations or methods that contribute to benefits other than those described above. [Methods used to solve problems]

[0007] For example, the encoding device of the present invention includes: a circuit; and a memory connected to the circuit. In operation, in an inter-prediction mode, the circuit generates a first predicted image of the processing target block based on a derived movement vector, and applies an update process to the first predicted image to generate a final predicted image of the processing target block. The candidates for the update process include a first process and a second process. The first process is a BDOF (bi-directional optical flow) process, and the second process is a process that mixes the second predicted image generated by the intra-prediction of the processing target block with the first predicted image. In the application of the update process, the first process and the second process are applied exclusively.

[0008] The various installation methods described in this disclosure can improve encoding efficiency, simplify encoding / decoding processes, accelerate encoding / decoding speed, and efficiently select appropriate components / actions used in encoding and decoding, such as appropriate filtering, block size, motion vector, reference image, and reference block.

[0009] Further advantages and effects of the disclosed embodiment are readily apparent from the specification and drawings. These advantages and / or effects can be obtained separately by means of the various embodiments and the features described in the specification and drawings, but it is not necessary to provide all of them in order to obtain one or more advantages and / or effects.

[0010] Furthermore, such comprehensive or specific forms can also be realized by systems, methods, integrated circuits, computer programs, recording media, or any combination thereof.

[0011] [Invention Effects] The configuration or method of the state disclosed herein may contribute to one or more of the following: improved encoding efficiency; improved image quality; reduced processing load; reduced circuit size; improved processing speed; and appropriate selection of elements or actions. Furthermore, the configuration or method of the state disclosed herein may also contribute to benefits other than those mentioned above. Simple Explanation of the Diagram

[0012] Figure 1 is a block diagram showing the functional configuration of the encoding device in the implementation form.

[0013] Figure 2 is a flowchart showing an example of the overall encoding process performed by the encoding device.

[0014] Figure 3 is a conceptual diagram showing one example of block segmentation.

[0015] Figure 4A is a conceptual diagram showing an example of the composition of a slice.

[0016] Figure 4B is a conceptual diagram showing one example of the composition of a tile.

[0017] Figure 5A is a table showing the transformation basis functions corresponding to various transformation types.

[0018] Figure 5B is a conceptual diagram showing one example of SVT (Spatially Varying Transform).

[0019] Figure 6A is a conceptual diagram showing one example of the filter shape used in ALF (adaptive loop filter).

[0020] Figure 6B is a conceptual diagram showing another example of the filter shape used in ALF.

[0021] Figure 6C is a conceptual diagram showing another example of the filter shape used in ALF.

[0022] Figure 7 is a block diagram showing an example of the detailed configuration of the loop filter section that functions as a DBF (deblocking filter).

[0023] Figure 8 is a conceptual diagram showing an example of a block filter with symmetrical filtering characteristics at block boundaries.

[0024] Figure 9 is a conceptual diagram illustrating the block boundaries for deblocking filtering.

[0025] Figure 10 is a conceptual diagram showing one example of the Bs value.

[0026] Figure 11 is a flowchart showing an example of the processing performed in the prediction processing unit of the encoding device.

[0027] Figure 12 is a flowchart showing another example of the processing performed in the prediction processing unit of the encoding device.

[0028] Figure 13 is a flowchart showing another example of the processing performed in the prediction processing unit of the encoding device.

[0029] Figure 14 is a conceptual diagram showing one example of 67 internal prediction patterns in the implementation configuration.

[0030] Figure 15 is a flowchart of one example of the basic processing flow for display prediction.

[0031] Figure 16 is a flowchart showing an example of deriving a movement vector.

[0032] Figure 17 is a flowchart showing another example of deriving a move vector.

[0033] Figure 18 is a flowchart showing another example of moving vector derivation.

[0034] Figure 19 is a flowchart showing an example of prediction between general patterns.

[0035] Figure 20 is a flowchart showing an example of prediction between merged patterns.

[0036] Figure 21 is a conceptual diagram illustrating one example of the move vector derivation process performed in the merge mode.

[0037] Figure 22 is a flowchart showing one example of FRUC (frame rate up conversion) processing.

[0038] Figure 23 is a conceptual diagram illustrating one example of pattern matching (bidirectional matching) between two blocks along a movement trajectory.

[0039] Figure 24 is a conceptual diagram illustrating one example of pattern matching (template matching) between a template in the current image and a block in the reference image.

[0040] Figure 25A is a conceptual diagram illustrating one example of the derivation of a movement vector for a sub-block unit based on the movement vectors of a plurality of adjacent blocks.

[0041] Figure 25B is a conceptual diagram illustrating one example of the derivation of the movement vector of a sub-block unit in an affine pattern with 3 control points.

[0042] Figure 26A is a conceptual diagram used to illustrate the affine merging pattern.

[0043] Figure 26B is a conceptual diagram illustrating an affine merging pattern with two control points.

[0044] Figure 26C is a conceptual diagram illustrating an affine merging pattern with three control points.

[0045] Figure 27 is a flowchart showing one example of the affine merging process.

[0046] Figure 28A is a conceptual diagram illustrating an affine pattern with two control points.

[0047] Figure 28B is a conceptual diagram illustrating an affine pattern with three control points.

[0048] Figure 29 is a flowchart showing one example of the processing of affine inter-pattern.

[0049] Figure 30A is a conceptual diagram illustrating an affine pattern where the current block has 3 control points and adjacent blocks have 2 control points.

[0050] Figure 30B is a conceptual diagram illustrating an affine pattern where the current block has 2 control points and adjacent blocks have 3 control points.

[0051] Figure 31A is a flowchart showing the merging mode including DMVR (decoder motion vector refinement).

[0052] Figure 31B is a conceptual diagram illustrating one example of DMVR processing.

[0053] Figure 32 is a flowchart showing one example of how a predicted image is generated.

[0054] Figure 33 is a flowchart showing another example of the generation of a predicted image.

[0055] Figure 34 is a flowchart showing another example of the generation of a predicted image.

[0056] Figure 35 is a flowchart illustrating one example of the predictive image correction process performed by OBMC (overlapped block motion compensation) processing.

[0057] Figure 36 is a conceptual diagram illustrating one example of the predictive image correction process performed by OBMC.

[0058] Figure 37 is a conceptual diagram illustrating the generation of the predicted image of two triangles.

[0059] Figure 38 is a conceptual diagram used to illustrate the model that assumes uniform linear motion.

[0060] Figure 39 is a conceptual diagram illustrating an example of a predictive image generation method that uses luminance correction processing with LIC (local illumination compensation).

[0061] Figure 40 is a block diagram showing an installation example of the encoding device.

[0062] Figure 41 is a block diagram showing the functional configuration of the decoding device in the implementation form.

[0063] Figure 42 is a flowchart showing an example of the overall decoding process performed by the decoding device.

[0064] Figure 43 is a flowchart showing an example of the processing performed in the prediction processing unit of the decoding device.

[0065] Figure 44 is a flowchart showing another example of the processing performed in the prediction processing unit of the decoding device.

[0066] Figure 45 is a flowchart showing an example of inter-mode prediction performed in a decoding device.

[0067] Figure 46 is a block diagram showing an installation example of the decoding device.

[0068] Figure 47 is a diagram illustrating the block size during predictive processing.

[0069] Figure 48 is a schematic diagram showing the first example of prediction processing performed by the decoding device in Embodiment 2 as a pipeline processing example.

[0070] Figure 49 is a flowchart showing a second example of prediction processing performed by the decoding device in Embodiment 2.

[0071] Figure 50 is a schematic diagram showing a second example of prediction processing performed by the decoding device in Embodiment 2 as a pipeline processing example.

[0072] Figure 51 is a flowchart showing a third example of prediction processing performed by the decoding device in Embodiment 2.

[0073] Figure 52 is a schematic diagram showing a third example of prediction processing performed by the decoding device in Embodiment 2 as a pipeline processing example.

[0074] Figure 53 is a flowchart showing the fourth example of prediction processing performed by the decoding device in Embodiment 2.

[0075] Figure 54 is a schematic diagram showing the fourth example of prediction processing performed by the decoding device in Embodiment 2 as a pipeline processing example.

[0076] Figure 55 is a flowchart showing the fifth example of prediction processing performed by the decoding device in Embodiment 2.

[0077] Figure 56 is a schematic diagram showing the fifth example of prediction processing performed by the decoding device in Embodiment 2 as a pipeline processing example.

[0078] Figure 57 is a flowchart showing the sixth example of prediction processing performed by the decoding device in Embodiment 2.

[0079] Figure 58 is a schematic diagram showing the sixth example of prediction processing performed by the decoding device in Embodiment 2 as a pipeline processing example.

[0080] Figure 59 is a flowchart showing the seventh example of prediction processing performed by the decoding device in Embodiment 2.

[0081] Figure 60 is a schematic diagram showing the seventh example of prediction processing performed by the decoding device in Embodiment 2 as a pipeline processing example.

[0082] Figure 61 is a flowchart showing an example of the encoding and decoding processes in Embodiment 2.

[0083] Figure 62 is a flowchart showing another example of the encoding and decoding processes in Embodiment 2.

[0084] Figure 63 is a block diagram showing the overall structure of the content supply system that implements content publishing services.

[0085] Figure 64 is a conceptual diagram showing an example of encoding construction in scalable encoding.

[0086] Figure 65 is a conceptual diagram showing an example of the encoding construction in adjustable encoding.

[0087] Figure 66 is a conceptual diagram of an example of a webpage display.

[0088] Figure 67 is a conceptual diagram of an example of a webpage display.

[0089] Figure 68 is a block diagram showing an example of a smartphone.

[0090] Figure 69 is a block diagram showing an example of the structure of a smartphone. Implementation

[0091] [The form in which the invention is implemented] The disclosed encoding device includes: a circuit; and a memory connected to the circuit. In operation, in an inter-prediction mode, the circuit generates a first predicted image of a processing target block based on a derived motion vector, and applies an update process to the first predicted image to generate a final predicted image of the processing target block. The candidates for the update process include a first process and a second process. The first process is a BDOF (bi-directional optical flow) process, and the second process is a process that mixes the second predicted image generated by the intra-prediction of the processing target block with the first predicted image. In the application of the update process, the first process and the second process are applied exclusively.

[0092] Therefore, regardless of whether the update process is the first process or the second process, no processing other than the update process will be applied to the first predicted image. In other words, the application of processing other than the update process is prohibited. Thus, for example, when a pipeline process used to encode an image includes both the first and second processes in one stage, performing the first and second processes together increases the likelihood of suppressing the time spent on that stage. That is, it increases the likelihood of suppressing an increase in processing time.

[0093] Furthermore, the aforementioned circuit can also determine whether the aforementioned second processing is applicable to the aforementioned first predicted image during the application of the aforementioned update processing. When it is determined that the aforementioned second processing is applicable, the aforementioned second processing is used as the aforementioned update processing. When it is determined that the aforementioned second processing is not applicable, it determines whether the aforementioned first processing is applicable to the aforementioned first predicted image. When it is determined that the aforementioned first processing is applicable, the aforementioned first processing is used as the aforementioned update processing.

[0094] This increases the likelihood of having the option to perform appropriate update processing.

[0095] Furthermore, the aforementioned circuit can also exclusively perform the aforementioned first process and the aforementioned second process in one stage of the pipeline processing used to encode the image, and in the same stage as the reconstruction process, wherein the aforementioned reconstruction process is the process of adding the generated final predicted image and the residual image to generate the reconstructed image.

[0096] This allows, for example, a reduction in the number of stages involved in pipeline processing compared to performing the first and second processes at a different stage than the reconstruction process. Consequently, it increases the likelihood of suppressing an increase in the circuit size of the encoding device.

[0097] Furthermore, the decoding apparatus disclosed herein includes: a circuit; and a memory connected to the aforementioned circuit. During operation, in an inter-prediction mode, the aforementioned circuit generates a first predicted image of the processing target block based on a derived motion vector, and applies an update process to the aforementioned first predicted image, thereby generating a final predicted image of the aforementioned processing target block. The candidates for the aforementioned update process include a first process and a second process. The aforementioned first process is a BDOF (bi-directional optical flow) process, and the aforementioned second process is a process that mixes the second predicted image generated by the intra-prediction of the aforementioned processing target block with the aforementioned first predicted image. In the application of the aforementioned update process, the aforementioned first process and the aforementioned second process are applied exclusively.

[0098] Therefore, regardless of whether the update process is the first process or the second process, no processing other than the update process will be applied to the first predicted image. In other words, the application of processing other than the update process is prohibited. Thus, for example, when a pipeline process for decoding an encoded image includes both the first and second processes in one stage, performing the first and second processes together can increase the likelihood of the processing time in that stage becoming longer. That is, it can increase the likelihood of increased processing time.

[0099] Furthermore, the aforementioned circuit can also determine whether the aforementioned second processing is applicable to the aforementioned first predicted image during the application of the aforementioned update processing. When it is determined that the aforementioned second processing is applicable, the aforementioned second processing is used as the update processing. When it is determined that the aforementioned second processing is not applicable, it determines whether the aforementioned first processing is applicable to the aforementioned first predicted image. When it is determined that the aforementioned first processing is applicable, the aforementioned first processing is used as the update processing.

[0100] This increases the likelihood of having the option to perform appropriate update processing.

[0101] Furthermore, the aforementioned circuit can also exclusively execute the aforementioned first process and the aforementioned second process in one stage of the pipeline processing used to decode the encoded image, and in the same stage as the reconstruction process, wherein the aforementioned reconstruction process is the process of adding the generated final predicted image and the residual image to generate the reconstructed image.

[0102] This allows, for example, a reduction in the number of stages involved in pipeline processing compared to performing the first and second processes at a different stage than the reconstruction process. Consequently, it increases the likelihood of suppressing an increase in the circuit size of the decoding device.

[0103] For example, the encoding device for a single-state sample disclosed herein includes: a segmentation unit, an internal prediction unit, an inter-prediction unit, a prediction control unit, a conversion unit, a quantization unit, and an entropy encoding unit.

[0104] The aforementioned segmentation unit segments the encoded object image constituting the dynamic image into a plurality of blocks. The aforementioned intra-prediction unit performs intra-prediction using a reference image from the aforementioned encoded object image to generate an intra-prediction image of the processing object block in the aforementioned encoded object image. The aforementioned inter-prediction unit performs inter-prediction using a reference image from a reference image different from the aforementioned encoded object image to generate an inter-prediction image of the processing object block.

[0105] The aforementioned prediction control unit controls the intra-prediction performed by the aforementioned intra-prediction unit and the inter-prediction performed by the aforementioned inter-prediction unit. The aforementioned conversion unit converts the prediction image, which is composed of at least one of the aforementioned intra-prediction image or the aforementioned inter-prediction image, and the residual image between the image of the aforementioned processing target block to generate the conversion coefficient signal of the aforementioned processing target block. The aforementioned quantization unit quantizes the aforementioned conversion coefficient signal. The aforementioned entropy encoding unit encodes the quantized aforementioned conversion coefficient signal.

[0106] Here, in the inter-prediction mode, the aforementioned inter-prediction unit generates a first predicted image of the processing target block based on the derived motion vector, and applies an update process to the first predicted image to generate the final predicted image of the processing target block. The candidates for the update process include a first process and a second process. The first process is a BDOF (bi-directional optical flow) process, and the second process is a process that mixes the second predicted image generated by the intra-prediction of the processing target block with the first predicted image. Then, in the application of the update process, the first process and the second process are applied exclusively.

[0107] For example, the decoding device for a single state disclosed herein is a decoding device that uses a predicted image to decode a dynamic image, and includes: an entropy decoding unit, an inverse quantization unit, an inverse conversion unit, an internal prediction unit, an inter-prediction unit, a prediction control unit, and an addition unit (reconstruction unit).

[0108] The aforementioned entropy decoding unit decodes the quantized conversion coefficient signals of the processing object block in the decoding object image constituting the aforementioned dynamic image. The aforementioned inverse quantization unit inverse quantizes the quantized conversion coefficient signals. The aforementioned inverse conversion unit inverse converts the conversion coefficient signals to obtain the residual image of the aforementioned processing object block.

[0109] The aforementioned intra-prediction unit performs intra-prediction using a reference image from the aforementioned decoded object image to generate an intra-prediction image for the aforementioned processed object block. The aforementioned inter-prediction unit performs inter-prediction using a reference image from a different reference image from the aforementioned decoded object image to generate an inter-prediction image for the aforementioned processed object block. The aforementioned prediction control unit controls the intra-prediction performed by the aforementioned intra-prediction unit and the inter-prediction performed by the aforementioned inter-prediction unit.

[0110] The aforementioned addition unit will add the predicted image, which is composed of at least one of the aforementioned intra-predicted images or the aforementioned inter-predicted images, to the aforementioned residual image to reconstruct the image of the aforementioned processing target block.

[0111] Here, in the inter-prediction mode, the aforementioned inter-prediction unit generates a first predicted image of the processing target block based on the derived motion vector, and applies an update process to the first predicted image to generate the final predicted image of the processing target block. The candidates for the update process include a first process and a second process. The first process is a BDOF (bi-directional optical flow) process, and the second process is a process that mixes the second predicted image generated by the intra-prediction of the processing target block with the first predicted image. Then, in the application of the update process, the first process and the second process are applied exclusively.

[0112] Furthermore, these general or specific states can also be implemented by systems, devices, methods, integrated circuits, computer programs, or non-transitory recording media such as computer-readable CD-ROMs, or by any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0113] The following descriptions, with reference to the accompanying drawings, detail the specific embodiments. Furthermore, the embodiments described below are general or specific examples. The numerical values, shapes, materials, constituent elements, the arrangement and connection of constituent elements, steps, relationships, and sequences of steps shown in the following embodiments are merely examples and are not intended to limit the scope of the patent application.

[0114] The following describes embodiments of the encoding and decoding apparatus. These embodiments are examples of encoding and decoding apparatuses to which the processing and / or configuration described in the various embodiments of this disclosure are applicable. The processing and / or configuration may also be implemented in encoding and decoding apparatuses different from those of the embodiments. For example, regarding the processing and / or configuration applicable to the embodiments, any of the following may also be implemented.

[0115] (1) Any one of the constituent elements of the encoding or decoding device of the embodiment described in the various embodiments of this disclosure may be replaced or combined with other constituent elements described in any embodiment of this disclosure.

[0116] (2) In the encoding or decoding apparatus of the embodiment, the functions or processes performed by a portion of the constituent elements of the plurality of constituent elements of the encoding or decoding apparatus may be arbitrarily changed, such as by adding, replacing, or deleting functions or processes. For example, any function or process may be replaced or combined with other functions or processes described in any of the embodiments of this disclosure.

[0117] (3) In the method implemented by the encoding or decoding device of the embodiment, any changes such as addition, substitution, and deletion can be made to a portion of the processing included in the method. For example, any processing in the method can be substituted or combined with other processing described in any of the various forms disclosed herein.

[0118] (4) A component that constitutes a portion of the plurality of components of the encoding or decoding device of the embodiment may also be combined with the components described in any of the various states of the present disclosure, or with a portion of the components that have the functions described in any of the various states of the present disclosure, or with a portion of the components that perform the processing performed by implementing the components described in the various states of the present disclosure.

[0119] (5) A component element that has part of the function of the encoding or decoding device of the embodiment, or a component element that has part of the processing of the encoding or decoding device of the embodiment, may be combined or replaced with a component element described in any of the embodiments of the present disclosure, a component element that has part of the function of any of the embodiments of the present disclosure, or a component element that has part of the processing of any of the embodiments of the present disclosure.

[0120] (6) In the method implemented by the encoding or decoding device of the embodiment, any of the plurality of processes included in the method may be replaced or combined with any of the processes described in any of the various states of the present disclosure, or any of the same processes.

[0121] (7) A portion of the processing included in the method implemented by the encoding or decoding device of the embodiment may also be combined with the processing described in any of the embodiments of this disclosure.

[0122] (8) The implementation methods of processing and / or configuration described in the various embodiments of this disclosure are not limited to the encoding or decoding apparatus of the embodiments. For example, the processing and / or configuration may also be implemented in apparatuses used for different purposes than the motion image encoding or motion image decoding disclosed in the embodiments.

[0123] (Implementation Form 1) [Encoding device] First, the encoding device of the embodiment will be described. Figure 1 is a block diagram showing the functional configuration of the encoding device 100 of the embodiment. The encoding device 100 is a motion image encoding device that encodes motion images in block units.

[0124] As shown in Figure 1, the encoding device 100 is an image encoding device that encodes images in block units, and includes: a segmentation unit 102, a subtraction unit 104, a conversion unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse conversion unit 114, an addition unit 116, a block memory 118, a loop filter unit 120, a frame memory 122, an intra prediction unit 124, an inter-prediction unit 126, and a prediction control unit 128.

[0125] The encoding device 100 can be implemented using, for example, a general-purpose processor and memory. In this case, when the processor executes the software program stored in memory, the processor functions as a segmentation unit 102, a subtraction unit 104, a conversion unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse conversion unit 114, an addition unit 116, a loop filter unit 120, an internal prediction unit 124, an inter-prediction unit 126, and a prediction control unit 128. Alternatively, the encoding device 100 can be implemented using one or more dedicated electronic circuits, and these dedicated electronic circuits correspond to the segmentation unit 102, the subtraction unit 104, the conversion unit 106, the quantization unit 108, the entropy encoding unit 110, the inverse quantization unit 112, the inverse conversion unit 114, the addition unit 116, the loop filter unit 120, the internal prediction unit 124, the inter-prediction unit 126, and the prediction control unit 128.

[0126] The following describes the overall processing flow of the encoding device 100, followed by an explanation of the constituent elements of the encoding device 100.

[0127] [Overall Encoding Process] Figure 2 is a flowchart illustrating an example of the overall encoding process performed by the encoding device 100.

[0128] First, the segmentation unit 102 of the encoding device 100 segments each image contained in the dynamic image, i.e., the input image, into a plurality of fixed-size blocks (e.g., 128×128 pixels) (step Sa_1). Next, the segmentation unit 102 selects a segmentation pattern (also called a block shape) for the fixed-size blocks (step Sa_2). That is, the segmentation unit 102 further segments the fixed-size blocks into a plurality of blocks constituting the selected segmentation pattern. Then, the encoding device 100 performs steps Sa_3 to Sa_9 on each of the plurality of blocks (i.e., the block to be encoded).

[0129] In other words, the prediction processing unit, which consists of all or part of the internal prediction unit 124, the inter-prediction unit 126 and the prediction control unit 128, generates a prediction signal (also known as a prediction block) for the encoded target block (also known as the current block) (step Sa_3).

[0130] Next, the subtraction unit 104 generates the difference between the encoded object block and the prediction block, which is used as the prediction residual (also known as the difference block) (step Sa_4).

[0131] Next, the conversion unit 106 and the quantization unit 108 convert and quantize the difference block, thereby generating a plurality of quantization coefficients (step Sa_5). Furthermore, the block composed of the plurality of quantization coefficients is also called a coefficient block.

[0132] Next, the entropy coding unit 110 encodes the coefficient block and the prediction parameters generated by the related prediction signal (specifically, entropy coding), thereby generating a coded signal (step Sa_6). The coded signal is also called a coded bitstream, a compressed bitstream, or a stream.

[0133] Next, the dequantization unit 112 and the inverse transformation unit 114 dequantize and inverse transform the coefficient block to restore multiple prediction residuals (i.e., difference blocks) (step Sa_7).

[0134] Next, the addition unit 116 adds the prediction block to the restored difference block, thereby reconstructing the current block into a reconstructed image (also known as a reconstructed block or a decoded image block) (step Sa_8). This generates the reconstructed image.

[0135] Once the reconstructed image is generated, the loop filter unit 120 filters the reconstructed image as needed (step Sa_9).

[0136] Next, the encoding device 100 determines whether the encoding of the entire image has ended (step Sa_10). If it is determined that the encoding has not ended (no in step Sa_10), the processing starting from step Sa_2 is repeated.

[0137] Furthermore, in the example described above, although the encoding device 100 selects one segmentation pattern for blocks of a fixed size and encodes each block according to that segmentation pattern, it can also encode each block according to multiple segmentation patterns. In this case, the encoding device 100 can also evaluate the cost of each of the multiple segmentation patterns and select the encoded signal obtained by encoding according to, for example, the segmentation pattern with the lowest cost as the encoded signal to be output.

[0138] As shown in the figure, the processing steps Sa_1 to Sa_10 are performed sequentially by the encoding device 100. Alternatively, multiple processing steps of a portion thereof can be performed in parallel, or the order of the processing steps can be reversed.

[0139] [Divider] The segmentation unit 102 divides the images contained in the input dynamic image into a plurality of blocks and outputs each block to the subtraction unit 104. For example, the segmentation unit 102 first segments the image into blocks of a fixed size (e.g., 128x128). Other fixed block sizes may also be used. This fixed-size block is sometimes called a coding tree unit (CTU). Next, the segmentation unit 102 divides each fixed-size block into blocks of variable size (e.g., 64×64 or less) according to, for example, a recursive quadtree and / or binary tree block segmentation. That is, the segmentation unit 102 selects a segmentation pattern. This variable-size block is sometimes called a coding unit (CU), a prediction unit (PU), or a transformation unit (TU). Alternatively, in various processing examples, it may not be necessary to distinguish between CU, PU, ​​and TU, and some or all of the blocks in the image may become processing units of CU, PU, ​​and TU.

[0140] Figure 3 is a conceptual diagram showing an example of block partitioning in an implementation. In Figure 3, solid lines represent block boundaries in quaternion tree block partitioning, and dashed lines represent block boundaries in binary tree block partitioning.

[0141] Here, block 10 is a square block of 128×128 pixels (128×128 block). This 128×128 block 10 is first divided into 4 square blocks of 64×64 (quadtree block partitioning).

[0142] The top-left 64×64 block is further vertically divided into two rectangular 32×64 blocks, and the left 32×64 block is further vertically divided into two rectangular 16×64 blocks (binary tree block partitioning). As a result, the top-left 64×64 block is divided into two 16×64 blocks 11 and 12, and a 32×64 block 13.

[0143] The 64×64 block in the upper right corner is horizontally divided into two rectangular 64×32 blocks, 14 and 15 (binary tree block division).

[0144] The bottom left 64×64 block is divided into four 32×32 square blocks (quadtree block partitioning). Of these four 32×32 blocks, the top left and bottom right blocks are further partitioned. The top left 32×32 block is vertically divided into two 16×32 rectangular blocks, and the right 16×32 block is further horizontally divided into two 16×16 blocks (binary tree block partitioning). The bottom right 32×32 block is horizontally divided into two 32×16 blocks (binary tree block partitioning). As a result, the bottom left 64×64 block is divided into 16×32 block 16, two 16×16 blocks 17 and 18, two 32×32 blocks 19 and 20, and two 32×16 blocks 21 and 22.

[0145] The 64x64 block 23 in the lower right corner is not divided.

[0146] As shown above, in Figure 3, block 10 is divided into 13 variable-size blocks 11 to 23 based on recursive quad-tree and binary tree block partitioning. This partitioning is sometimes referred to as QTBT (quad-tree plus binary tree) partitioning.

[0147] Additionally, in Figure 3, a block is divided into four or two blocks (quadtree or binary tree block partitioning), but the partitioning is not limited to these. For example, a block can also be divided into three blocks (ternary tree block partitioning). Such partitioning involving ternary tree block partitioning is sometimes referred to as MBT (multi-type tree) partitioning.

[0148] [Image composition slices / tiles] To decode images side-by-side, images are sometimes constructed from slice units or tile units. Images composed of slice units or tile units can also be constructed using segmentation units 102.

[0149] A slice is the basic coding unit that makes up an image. An image is composed of one or more slices. Furthermore, a slice is composed of one or more consecutive CTUs (Coding Tree Units).

[0150] Figure 4A is a conceptual diagram illustrating an example of the structure of a slice. For example, an image contains 11×8 CTUs and is divided into four slices (slices 1 to 4). Slice 1 consists of 16 CTUs, slice 2 consists of 21 CTUs, slice 3 consists of 29 CTUs, and slice 4 consists of 22 CTUs. Here, each CTU within the image belongs to any one slice. The slices are shaped to divide the image horizontally. The boundaries of the slices do not need to be at the edge of the screen; they can be anywhere within the boundaries of the CTUs in the screen. The processing order (encoding order or decoding order) of the CTUs within a slice is, for example, a raster scan order. Furthermore, each slice contains header information and encoded data. The header information may also describe the characteristics of the slice, such as the CTU address at the beginning of the slice and the slice type.

[0151] A tile is a unit that constitutes a rectangular area of ​​an image. Each tile can also be assigned a number called TileId according to the line-by-line scanning sequence.

[0152] Figure 4B is a conceptual diagram showing an example of tile composition. For example, an image contains 11×8 CTUs and is divided into four rectangular regions (tiles 1 to 4). The processing order of the CTUs changes when tiles are used compared to when tiles are not used. When tiles are not used, the multiple CTUs within the image are processed in a line-by-line scan order. When tiles are used, at least one CTU in each of the multiple tiles is processed in a line-by-line scan order. For example, as shown in Figure 4B, the processing order of the multiple CTUs in tile 1 is: from the left end of the first column of tile 1 towards the right end of the first column of tile 1, and then from the left end of the second column of tile 1 towards the right end of the second column of tile 1.

[0153] Additionally, a tile can contain more than one slice, and a slice can contain more than one tile.

[0154] [Subtraction Section] The subtraction unit 104 subtracts the prediction signal (the prediction sample input from the prediction control unit 128 shown later) from the original signal (original sample) using block units input from the segmentation unit 102 and segmented by the segmentation unit 102. In other words, the subtraction unit 104 calculates the prediction error (also called the residual) of the encoded target block (hereinafter referred to as the current block). Then, the subtraction unit 104 outputs the calculated prediction error (residual) to the conversion unit 106.

[0155] The original signal is the input signal of the encoding device 100, which represents the image of each picture that constitutes the moving image (e.g., luminance signal and two chroma signals). Hereinafter, the signal representing the image will sometimes be referred to as a sample.

[0156] [Transition Section] The conversion unit 106 converts the prediction error in the spatial domain into conversion coefficients in the frequency domain and outputs the conversion coefficients to the quantization unit 108. Specifically, the conversion unit 106 performs a predetermined discrete cosine transform (DCT) or discrete sine transform (DST) on the prediction error in the spatial domain, for example. The predetermined DCT or DST can also be determined in advance.

[0157] Alternatively, the transformation unit 106 can adaptively select a transformation type from a plurality of transformation types and use a transformation basis function corresponding to the selected transformation type to convert the prediction error into transformation coefficients. Such a transformation is sometimes called EMT (explicit multiple core transform) or AMT (adaptive multiple transform).

[0158] The multiple transformation types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII. Figure 5A is a table showing the transformation basis functions corresponding to the transformation type examples. In Figure 5A, N represents the number of input pixels. The selection of the transformation type from these multiple transformation types can be based, for example, on the type of prediction (intra-prediction and inter-prediction) or on the intra-prediction mode.

[0159] Information indicating whether EMT or AMT is applicable (e.g., EMT flag or AMT flag) and information indicating the selected conversion type are typically signaled at the CU level. However, the signaling of this information is not limited to the CU level; it can also be at other levels (e.g., bit sequence level, picture level, slice level, tile level, or CTU level).

[0160] Furthermore, the transformation unit 106 can also perform a further transformation on the transformation coefficients (transformation results). Such a further transformation is sometimes called AST (adaptive secondary transform) or NSST (non-separable secondary transform). For example, the transformation unit 106 performs a further transformation based on the individual sub-blocks (e.g., 4×4 sub-blocks) contained in the block corresponding to the transformation coefficients of the internal prediction error. Information indicating whether NSST is applicable and information about the transformation matrix used in NSST are usually signaled at the CU level. However, the signaling of such information is not limited to the CU level and can also be at other levels (e.g., sequence level, image level, slice level, tile level, or CTU level).

[0161] In the transformation unit 106, both separable and non-separable transformations can be applied. A separable transformation refers to a method of performing multiple transformations by separating the dimensions of the input in a direction, while a non-separable transformation refers to a method of combining two or more dimensions into one dimension and then performing the transformation together when the input is multi-dimensional.

[0162] For example, the following example can be used as an example of an inseparable transformation: when the input is a 4×4 block, the aforementioned block is regarded as an array with 16 elements, and the aforementioned array is transformed using a 16×16 transformation matrix.

[0163] Furthermore, in a further example of an inseparable transformation, a transformation such as performing multiple Givens rotations on the array can be performed after treating the 4×4 input block as an array with 16 elements.

[0164] During the conversion in the conversion unit 106, the type of substrate for conversion to the frequency domain can be switched according to the region within the CU. For example, there is SVT (Spatially Varying Transform). In SVT, as shown in Figure 5B, the CU is divided into two equal parts along a horizontal or vertical direction, and only one side of the region is converted to the frequency domain. The type of conversion substrate can be set for each region; for example, DST7 and DCT8 can be used. In this example, only one side of the two regions within the CU is converted, while the other side is not converted, but both regions can also be converted. Furthermore, the division method is not limited to two equal parts; it can also be four equal parts, or a more flexible method can be used, such as encoding the information representing the division separately and transmitting it in the same way as the CU division. Additionally, SVT is sometimes also called SBT (Sub-block Transform).

[0165] [Quantitative Department] The quantization unit 108 quantizes the conversion coefficients output from the conversion unit 106. Specifically, the quantization unit 108 scans the conversion coefficients of the current block in a predetermined scan order, and quantizes the conversion coefficients according to the quantization parameters (QP) corresponding to the scanned conversion coefficients. Then, the quantization unit 108 outputs the quantized conversion coefficients (hereinafter referred to as quantization coefficients) of the current block to the entropy encoding unit 110 and the inverse quantization unit 112. The predetermined scan order can also be determined in advance.

[0166] The predetermined scan order is the order in which the conversion coefficients are quantized / dequantized. For example, the predetermined scan order can also be defined by ascending frequency (from low frequency to high frequency) or descending frequency (from high frequency to low frequency).

[0167] The quantization parameter (QP) is a parameter that defines the quantization step size (quantization width). For example, increasing the value of the quantization parameter will increase the quantization step size. In other words, increasing the value of the quantization parameter will increase the quantization error.

[0168] Furthermore, quantization sometimes uses quantization matrices. For example, various types of quantization matrices are sometimes used to correspond to frequency conversion sizes such as 4×4 and 8×8, prediction modes such as intra-prediction and inter-prediction, and pixel components such as luminance and chromatic aberration. Additionally, quantization refers to digitizing values ​​sampled at predetermined intervals and corresponding to predetermined levels. In this field, other representation methods such as rounding, scaling, etc., can be used for reference, or rounding, scaling, etc., can also be employed. The predetermined intervals and levels can also be determined in advance.

[0169] There are two methods for using quantization matrices: one is to use a quantization matrix that is directly set on the encoding device side, and the other is to use a default quantization matrix. By directly setting the quantization matrix on the encoding device side, a quantization matrix corresponding to the image features can be set. However, in this case, there is a drawback: the encoding of the quantization matrix increases the amount of code.

[0170] On the other hand, there are methods that quantize both high-frequency and low-frequency coefficients without using a quantization matrix. This method is equivalent to using a quantization matrix (a planar matrix) where all coefficients have the same value.

[0171] The quantization matrix can also be specified using, for example, SPS (Sequence Parameter Set) or PPS (Picture Parameter Set). SPS contains parameters that can be used for sequences, and PPS contains parameters that can be used for pictures. SPS and PPS are sometimes simply referred to as parameter sets.

[0172] [Entropy Coding Department] The entropy coding unit 110 generates a coded signal (coded bit stream) based on the quantization coefficients input from the quantization unit 108. Specifically, the entropy coding unit 110, for example, binarizes the quantization coefficients, performs arithmetic coding on the binary signal, and outputs a compressed bit stream or sequence.

[0173] [Anti-Quantitative Department] The dequantization unit 112 dequantizes the quantization coefficients input from the quantization unit 108. Specifically, the dequantization unit 112 dequantizes the quantization coefficients of the current block in a predetermined scan order. Then, the dequantization unit 112 outputs the dequantized conversion coefficients of the current block to the inversion conversion unit 114. The predetermined scan order can also be determined in advance.

[0174] [Reverse Conversion Section] The inverse conversion unit 114 inversely converts the conversion coefficients input from the inverse quantization unit 112, thereby restoring the prediction error (residual). Specifically, the inverse conversion unit 114 restores the prediction error of the current block by performing an inverse conversion on the conversion coefficients corresponding to the conversion of the conversion unit 106. Then, the inverse conversion unit 114 outputs the restored prediction error to the addition unit 116.

[0175] Furthermore, since the restored prediction error usually loses information due to quantization, it will not be consistent with the prediction error calculated by the subtraction unit 104. That is, the restored prediction error usually includes quantization error.

[0176] [Addition Department] The adder 116 adds the prediction error input from the inverse conversion unit 114 to the prediction sample input from the prediction control unit 128, thereby reconstructing the current block. Then, the adder 116 outputs the reconstructed block to the block memory 118 and the loop filter unit 120. The reconstructed block is sometimes also called a local decoding block.

[0177] [Block Memory] Block memory 118 is a memory unit used to store blocks such as those referenced in the internal prediction and those within the encoded object image (referred to as the current image). Specifically, block memory 118 stores the reconstructed blocks output from the addition unit 116.

[0178] [Frame Memory] Frame memory 122 is, for example, a memory unit used to store reference images used in inter-prediction, and is sometimes also called a frame buffer. Specifically, frame memory 122 stores reconstructed blocks that have been filtered by loop filter unit 120.

[0179] [Loop Filtering Section] The loop filter 120 applies loop filtering to the block reconstructed by the adder 116 and outputs the filtered reconstructed block to the frame memory 122. The loop filter is a filter used within the coding loop (in-loop filter), including, for example, deblocking filters (DF or DBF), sample adaptive offset (SAO), and adaptive loop filter (ALF).

[0180] In ALF, the least squares error filter is used to remove coding distortion. For example, for each 2×2 sub-block within the current block, a filter is selected from a plurality of filters based on the direction and activity of the local gradient.

[0181] Specifically, sub-blocks (e.g., 2×2 sub-blocks) are first classified into multiple groups (e.g., 15 or 25 groups). The classification of sub-blocks is based on the direction and activity of the gradient. For example, the classification value C (e.g., C = 5D + A) is calculated using the gradient direction value D (e.g., 0 to 2 or 0 to 4) and the gradient activity value A (e.g., 0 to 4). Then, the sub-blocks are classified into multiple groups based on the classification value C.

[0182] The gradient direction value D is derived, for example, by comparing the gradients in multiple directions (e.g., horizontal, vertical, and two diagonal directions). Similarly, the gradient activity value A is derived, for example, by adding the gradients in multiple directions and then quantizing the result.

[0183] Based on this classification result, the filter to be used for the sub-block is determined from a plurality of filters.

[0184] As the filter shape used in ALF, a circularly symmetrical shape can be used, for example. Figures 6A to 6C are diagrams showing several examples of filter shapes used in ALF. Figure 6A shows a 5×5 diamond-shaped filter, Figure 6B shows a 7×7 diamond-shaped filter, and Figure 6C shows a 9×9 diamond-shaped filter. The information displaying the filter shape is usually signaled at the image level. However, the signaling of the filter shape information is not limited to the image level and can also be at other levels (e.g., sequence level, slice level, tile level, CTU level, or CU level).

[0185] The on / off state of ALF can also be determined at, for example, picture level or CU level. For instance, luminance can be assessed at the CU level, and chromatic aberration at the picture level. Information indicating ALF on / off is typically signaled at picture level or CU level. However, the signaling of ALF on / off status is not limited to picture level or CU level; it can also be at other levels (e.g., sequence level, slice level, tile level, or CTU level).

[0186] The coefficient set of selectable multiple filters (e.g., up to 15 or 25 filters) is typically signaled at the picture level. Alternatively, the signaling of the coefficient set is not limited to the picture level, but can also be at other levels (e.g., sequence level, slice level, tile level, CTU level, CU level, or sub-block level).

[0187] [Loop Filtering Section > Deblocking Filter] In the deblocking filter, the loop filter unit 120 filters the block boundaries of the reconstructed image, thereby reducing the distortion caused by the block boundaries.

[0188] Figure 7 is a block diagram showing an example of the detailed configuration of the loop filter unit 120, which functions as a deblocking filter.

[0189] The loop filter unit 120 includes: a boundary determination unit 1201, a filter determination unit 1203, a filter processing unit 1205, a processing determination unit 1208, a filter characteristic determination unit 1207, and switches 1202, 1204, and 1206.

[0190] The boundary determination unit 1201 determines whether the pixel (i.e., the target pixel) to be subjected to block filtering processing exists near the block boundary. Then, the boundary determination unit 1201 outputs its determination result to the switch 1202 and the processing determination unit 1208.

[0191] When the boundary determination unit 1201 determines that the object pixel exists near the block boundary, the switch 1202 outputs the image before filtering to the switch 1204. Conversely, when the boundary determination unit 1201 determines that the object pixel does not exist near the block boundary, the switch 1202 outputs the image before filtering to the switch 1206.

[0192] The filtering determination unit 1203 determines whether to perform deblocking filtering on the target pixel based on the pixel values ​​of at least one surrounding pixel located around the target pixel. Then, the filtering determination unit 1203 outputs its determination result to the switch 1204 and the processing determination unit 1208.

[0193] When the filtering determination unit 1203 determines that deblocking filtering processing should be performed on the target pixels, the switch 1204 outputs the image before filtering processing, which has been obtained through the switch 1202, to the filtering processing unit 1205. Conversely, when the filtering determination unit 1203 determines that deblocking filtering processing should not be performed on the target pixels, the switch 1204 outputs the image before filtering processing, which has been obtained through the switch 1202, to the switch 1206.

[0194] When the image before filtering has been obtained through switches 1202 and 1204, the filtering unit 1205 performs deblocking filtering on the target pixels, wherein the deblocking filtering has filtering characteristics determined by the filtering characteristic determination unit 1207. Then, the filtering unit 1205 outputs the filtered pixels to switch 1206.

[0195] In response to the control of the processing determination unit 1208, the switch 1206 selectively outputs pixels that have not been deblocked and pixels that have been deblocked by the filtering processing unit 1205.

[0196] The processing determination unit 1208 controls the switch 1206 based on the determination results of the boundary determination unit 1201 and the filtering determination unit 1203. That is, when the boundary determination unit 1201 determines that an object pixel exists near a block boundary, and the filtering determination unit 1203 determines that deblocking filtering processing should be performed on the object pixel, the processing determination unit 1208 outputs the deblocked pixel from the switch 1206. In other cases, the processing determination unit 1208 outputs the un-deblocked pixel from the switch 1206. By repeatedly outputting such pixels, a filtered image is output from the switch 1206.

[0197] Figure 8 is a conceptual diagram showing an example of a block filter with symmetrical filtering characteristics at block boundaries.

[0198] In deblocking filtering, for example, pixel values ​​and quantization parameters are used to select two deblocking filters with different characteristics, i.e., either a strong filter or a weak filter. In the strong filter, as shown in Figure 8, when pixels p0 to p2 and pixels q0 to q2 exist across the block boundary, the pixel values ​​of pixels q0 to q2 are changed to pixel values ​​q'0 to q'2 by performing operations such as those shown in the following formulas. q'0=(p1+2×p0+2×q0+2×q1+q2+4) / 8 q'1=(p0+q0+q1+q2+2) / 4 q'2=(p0+q0+q1+3×q2+2×q3+4) / 8

[0199] Furthermore, in the above formulas, p0 to p2 and q0 to q2 are the pixel values ​​of pixels p0 to p2 and q0 to q2, respectively. Also, q3 is the pixel value of pixel q3, which is adjacent to pixel q2 on the opposite side of the block boundary. Furthermore, the coefficients multiplied by the pixel values ​​of each pixel used in the deblocking filtering process on the right-hand side of the above formulas are the filtering coefficients.

[0200] Furthermore, in the deblocking filtering process, to prevent the calculated pixel values ​​from exceeding a threshold, a clipping process can be performed. In this clipping process, the pixel values ​​calculated according to the above formula are clipped to "the pixel value of the operation ± 2 × the threshold" using a threshold determined by the quantization parameter. This prevents excessive smoothing.

[0201] Figure 9 is a conceptual diagram illustrating the block boundaries during deblocking filtering. Figure 10 is a conceptual diagram showing an example of the Bs value.

[0202] The block boundaries for deblocking filtering are, for example, the boundaries of the PU (Prediction Unit) or TU (Transform Unit) of an 8×8 pixel block as shown in Figure 9. Deblocking filtering can be performed in units of 4 columns or 4 rows. First, for blocks P and Q shown in Figure 9, the Bs (Boundary Strength) value is determined as shown in Figure 10.

[0203] Even for blocks belonging to the same image, the Bs value in Figure 10 determines whether to perform deblocking filtering of different intensities. Deblocking filtering of chrominance signals is performed when the Bs value is 2. Deblocking filtering of luminance signals is performed when the Bs value is 1 or higher and predetermined conditions are met. These predetermined conditions can also be set in advance. Furthermore, the criteria for determining the Bs value are not limited to those shown in Figure 10 and can be determined based on other parameters.

[0204] [Forecasting Processing Department (Internal Forecasting Department, Intermediate Forecasting Department, Forecasting Control Department)] Figure 11 is a flowchart showing an example of the processing performed by the prediction processing unit of the encoding device 100. The prediction processing unit is composed of all or part of the components of the inner prediction unit 124, the intermediate prediction unit 126, and the prediction control unit 128.

[0205] The prediction processing unit generates a prediction image for the current block (step Sb_1). This prediction image is also called a prediction signal or a prediction block. The prediction signal may include, for example, an intra-prediction signal or an inter-prediction signal. Specifically, the prediction processing unit uses the reconstructed image obtained by generating prediction blocks, difference blocks, coefficient blocks, restoring difference blocks, and generating decoded image blocks to generate a prediction image for the current block.

[0206] The reconstructed image can be, for example, an image of a reference image, or an image containing the current block, i.e., an image of the encoded blocks within the current image. The encoded blocks within the current image can be, for example, adjacent blocks of the current block.

[0207] Figure 12 is a flowchart showing another example of the processing performed in the prediction processing unit of the encoding device 100.

[0208] The prediction processing unit generates a prediction image in three ways: a first method (step Sc_1a), a second method (step Sc_1b), and a third method (step Sc_1c). The first, second, and third methods are different methods for generating the prediction image, and may be, for example, inter-prediction, intra-prediction, or other prediction methods. The reconstructed image described above can also be used in these prediction methods.

[0209] Next, the prediction processing unit selects any one of the plurality of prediction images generated in steps Sc_1a, Sc_1b, and Sc_1c (step Sc_2). This selection of the prediction image, i.e., the selection of the method or mode used to obtain the final prediction image, can also be based on the calculated cost of each generated prediction image. Alternatively, the selection of the prediction image can be based on parameters used in the encoding process. The encoding device 100 can also signal the information used to specify the selected prediction image, method, or mode, making it an encoded signal (also called an encoded bit stream). This information can also be, for example, a flag. In this way, the decoding device can generate a prediction image based on this information, according to the method or mode selected in the encoding device 100. Furthermore, in the example shown in FIG12, the prediction processing unit selects any one prediction image after generating prediction images in each method. However, the prediction processing unit can also select a method or mode based on parameters used in the above-described encoding process before generating such prediction images, and generate prediction images according to that method or mode.

[0210] For example, the first method and the second method can be internal prediction and inter-prediction, respectively, and the prediction processing unit selects the final prediction image for the current block from the prediction images generated according to these prediction methods.

[0211] Figure 13 is a flowchart showing another example of the processing performed in the prediction processing unit of the encoding device 100.

[0212] First, the prediction processing unit generates a prediction image by intra-prediction (step Sd_1a) and by inter-prediction (step Sd_1b). The prediction image generated by intra-prediction is also called the intra-prediction image, and the prediction image generated by inter-prediction is also called the inter-prediction image.

[0213] Next, the prediction processing unit evaluates the intra-predicted image and the inter-predicted image separately (step Sd_2). Cost can also be used in this evaluation. That is, the prediction processing unit calculates the cost C for each of the intra-predicted image and the inter-predicted image. This cost C can be calculated using the formula of the RD optimization model, for example, C=D+λ×R. In this formula, D is the coding distortion of the predicted image, and is represented by, for example, the sum of the absolute differences between the pixel values ​​of the current block and the pixel values ​​of the predicted image. R is the amount of coding required to generate the predicted image, specifically the amount of coding needed to generate motion information, etc., for the predicted image. λ is, for example, a Lagrange multiplier.

[0214] Next, the prediction processing unit selects the prediction image with the lowest calculated cost C from the internal and inter-prediction images as the final prediction image for the current block (step Sd_3). In other words, it selects the prediction method or mode used to generate the prediction image for the current block.

[0215] [Internal Forecasting Department] The intraprediction unit 124 references blocks within the current image stored in the block memory 118 and performs intraprediction (also known as intra-image prediction) of the current block, thereby generating a prediction signal (intraprediction signal). Specifically, the intraprediction unit 124 references samples (e.g., brightness values, color difference values) of blocks adjacent to the current block to perform intraprediction, thereby generating an intraprediction signal, and outputs the intraprediction signal to the prediction control unit 128.

[0216] For example, the internal prediction unit 124 uses one of a predetermined plurality of internal prediction patterns to perform internal prediction. The plurality of internal prediction patterns typically includes one or more non-directional prediction patterns and a plurality of directional prediction patterns. The predetermined plurality of patterns can also be predetermined.

[0217] One or more non-directional prediction modes include, for example, planar prediction modes and direct current (DC) prediction modes as specified in the H.265 / HEVC standard.

[0218] Multiple directional prediction modes may include, for example, the 33 directional prediction modes specified in the H.265 / HEVC standard. Additionally, multiple directional prediction modes may further include 32 additional directional prediction modes (a total of 65 directional prediction modes). Figure 14 is a conceptual diagram showing the 67 intraprediction modes (2 non-directional prediction modes and 65 directional prediction modes) available in the intraprediction. Solid arrows represent the 33 directions specified in the H.265 / HEVC standard, and dashed arrows represent the additional 32 directions (the 2 non-directional prediction modes are not shown in Figure 14).

[0219] In various processing examples, the luminance block can also be referenced in the intra-prediction of the chromatic difference block. That is, the chromatic difference component of the current block can be predicted based on the luminance component of the current block. This intra-prediction is sometimes called CCLM (cross-component linear model) prediction. Such intra-prediction modes that reference the luminance block (e.g., called CCLM modes) can also be added as one type of intra-prediction mode for chromatic difference blocks.

[0220] The intraprediction unit 124 can also correct the pixel values ​​after intraprediction based on the gradient of reference pixels in the horizontal / vertical directions. This intraprediction with correction is sometimes called PDPC (position dependent intraprediction combination). Information indicating whether PDPC is applicable (e.g., a PDPC flag) is usually signaled at the CU level. However, the signaling of this information is not limited to the CU level and can also be at other levels (e.g., sequence level, image level, tile level, or CTU level).

[0221] [Inter-departmental Forecasting] Inter-prediction unit 126 refers to a reference image stored in frame memory 122, which is different from the current image, to perform inter-prediction (also called inter-frame prediction) of the current block, thereby generating a prediction signal (inter-prediction signal). Inter-prediction is performed on a unit basis, either the current block or the current sub-block within the current block (e.g., a 4×4 block). For example, inter-prediction unit 126 performs motion estimation within the reference image for the current block or current sub-block to find the reference block or sub-block that best matches the current block or current sub-block. Next, inter-prediction unit 126 obtains motion information (e.g., a motion vector) from the reference block or sub-block to compensate for the movement or change of the current block or sub-block. Inter-prediction unit 126 performs motion compensation (or motion prediction) based on this motion information to generate the inter-prediction signal for the current block or sub-block. Inter-prediction unit 126 outputs the generated inter-prediction signal to prediction control unit 128.

[0222] Motion information used for motion compensation can also be signaled in various forms to serve as indirect prediction signals. For example, motion vectors can be signaled. As another example, the difference between the motion vector and the predicted motion vector can also be signaled.

[0223] [Basic process of inter-satellite forecasting] Figure 15 is a flowchart of one example of the basic process of display prediction.

[0224] First, the prediction unit 126 generates a prediction image (steps Se_1 to Se_3). Second, the subtraction unit 104 generates the difference between the current block and the prediction image as the prediction residual (step Se_4).

[0225] Here, the intermediate prediction unit 126 generates the predicted image by determining the motion vector (MV) of the current block (steps Se_1 and Se_2) and performing motion compensation (step Se_3). Furthermore, in determining the MV, the intermediate prediction unit 126 determines the MV by selecting candidate motion vectors (candidate MVs) (step Se_1) and deriving the MV (step Se_2). The selection of candidate MVs is, for example, performed by selecting at least one candidate MV from a candidate MV list. In MV derivation, the intermediate prediction unit 126 may further select at least one candidate MV from the at least one candidate MV, thereby determining the selected at least one candidate MV as the MV of the current block. Alternatively, the intermediate prediction unit 126 may search for the region of the reference image indicated by each of the selected at least one candidate MV, thereby determining the MV of the current block. This search for the region of the reference image may also be referred to as motion estimation.

[0226] Furthermore, in the above example, although steps Se_1 to Se_3 are performed by the inter-prediction unit 126, the processing of steps such as Se_1 or Se_2 can also be performed by other components included in the encoding device 100.

[0227] [Process for exporting movement vectors] Figure 16 is a flowchart showing an example of deriving a movement vector.

[0228] The prediction unit 126 derives the MV of the current block using a mode that encodes motion information (e.g., MV). In this case, for example, the motion information is encoded as a prediction parameter and signaled. That is, the encoded motion information is contained in the encoded signal (also called the coded bit stream).

[0229] Alternatively, the prediction unit 126 may derive the MV in a mode that does not encode mobile information. In this case, the encoded signal does not contain mobile information.

[0230] Here, the modes for deriving the MV can include general inter-mode, merge mode, FRUC mode, and affine mode, as described later. Among these modes, the modes that encode motion information include general inter-mode, merge mode, and affine mode (specifically, affine inter-mode and affine merge mode). Furthermore, the motion information may include not only the MV but also the predicted motion vector selection information, as described later. Also, modes that do not encode motion information include FRUC mode, etc. The inter-prediction unit 126 selects a mode from these multiple modes for deriving the MV of the current block and uses the selected mode to derive the MV of the current block.

[0231] Figure 17 is a flowchart showing another example of deriving a move vector.

[0232] The prediction unit 126 derives the MV of the current block using a mode that encodes the differential MV. In this case, for example, the differential MV is encoded as a prediction parameter and signaled. That is, the encoded differential MV is contained in the encoded signal. This differential MV is the difference between the MV of the current block and its predicted MV.

[0233] Alternatively, the prediction unit 126 may derive the MV in a mode that does not encode the differential MV. In this case, the encoded signal does not contain the encoded differential MV.

[0234] Here, as mentioned above, the modes for deriving MV include general inter-mode, merge mode, FRUC mode, and affine mode, which will be described later. Among these modes, the modes that encode differential MV include general inter-mode and affine mode (specifically, affine inter-mode). Furthermore, the modes that do not encode differential MV include FRUC mode, merge mode, and affine mode (specifically, affine merge mode). The inter-prediction unit 126 selects a mode from these multiple modes for deriving the MV of the current block, and uses the selected mode to derive the MV of the current block.

[0235] [Process for exporting movement vectors] Figure 18 is a flowchart showing another example of move vector derivation. There are several modes for deriving the MV, i.e., the in-between prediction modes, which can be broadly categorized into: modes that encode the differential MV and modes that do not encode the differential move vector. Modes that do not encode the differential MV include the merge mode, the FRUC mode, and the affine mode (specifically, the affine merge mode). Details of these modes will be described later. In short, the merge mode derives the MV of the current block by selecting move vectors from surrounding encoded blocks, the FRUC mode derives the MV of the current block by searching among encoded regions, and the affine mode derives the MV of the current block by envisioning an affine transformation to derive the move vectors of the multiple sub-blocks constituting the current block.

[0236] Specifically, as shown in the diagram, when the inter-prediction mode information displays 0 (0 in Sf_1), the inter-prediction unit 126 derives the movement vector (Sf_2) using the merge mode. When the inter-prediction mode information displays 1 (1 in Sf_1), the inter-prediction unit 126 derives the movement vector (Sf_3) using the FRUC mode. When the inter-prediction mode information displays 2 (2 in Sf_1), the inter-prediction unit 126 derives the movement vector (Sf_4) using the affine mode (specifically, the affine merge mode). When the inter-prediction mode information displays 3 (3 in Sf_1), the inter-prediction unit 126 derives the movement vector (Sf_5) using the differential MV encoding mode (e.g., the general inter-prediction mode).

[0237] [MV Export > Normal Mode] The general inter-prediction pattern is as follows: from the region of the reference image shown in the candidate MV, the MV of the current block is derived based on the image blocks similar to the current block. Furthermore, in this general inter-prediction pattern, the differential MV is encoded.

[0238] Figure 19 is a flowchart showing an example of prediction between general patterns.

[0239] The intermediate prediction unit 126 first obtains a plurality of candidate MVs for the current block based on information such as the MVs of a plurality of encoded blocks located in the vicinity of the current block in time or space (step Sg_1). In other words, the intermediate prediction unit 126 creates a list of candidate MVs.

[0240] Next, the prediction unit 126 selects N (N is an integer greater than 2) candidate MVs from the plurality of candidate MVs obtained in step Sg_1 according to a predetermined priority order, and treats each candidate MV as a prediction movement vector candidate (also called a prediction MV candidate) (step Sg_2). Alternatively, this priority order can be a pre-determined order of the N candidate MVs.

[0241] Next, the inter-prediction unit 126 selects one prediction move vector candidate from the N prediction move vector candidates as the prediction move vector (also called prediction MV) for the current block (step Sg_3). At this time, the inter-prediction unit 126 encodes the prediction move vector selection information used to identify the selected prediction move vector into a stream. The stream is either the encoded signal or the encoded bit stream described above.

[0242] Next, the inter-prediction unit 126 derives the MV of the current block from the encoded reference image (step Sg_4). At this time, the inter-prediction unit 126 further encodes the difference between the derived MV and the predicted movement vector as a difference MV into a stream. Furthermore, the encoded reference image is an image composed of a plurality of reconstructed blocks after encoding.

[0243] Finally, the inter-prediction unit 126 uses the exported MV and the encoded reference image to perform motion compensation on the current block, thereby generating a prediction image of the current block (step Sg_5). The prediction image is the inter-prediction signal described above.

[0244] Furthermore, the information contained in the encoded signal regarding the inter-prediction patterns (general inter-prediction patterns in the example above) used in the generation of the predicted image is encoded as, for example, prediction parameters.

[0245] Furthermore, the candidate MV list can be used interchangeably with lists used in other modes. Also, processing related to the candidate MV list can be applied to processing related to lists used in other modes. This processing related to the candidate MV list includes, for example, retrieving or selecting candidate MVs from the candidate MV list, rearranging candidate MVs, or deleting candidate MVs.

[0246] [MV Export > Merge Mode] The merge mode is the following inter-prediction mode: select a candidate MV from the candidate MV list as the MV of the current block, thereby deriving the MV.

[0247] Figure 20 is a flowchart showing an example of prediction between merged patterns.

[0248] The intermediate prediction unit 126 first obtains a plurality of candidate MVs for the current block based on information such as the MVs of a plurality of encoded blocks located in the vicinity of the current block in time or space (step Sh_1). In other words, the intermediate prediction unit 126 creates a list of candidate MVs.

[0249] Next, the inter-prediction unit 126 selects one candidate MV from the plurality of candidate MVs obtained in step Sh_1, thereby deriving the MV of the current block (step Sh_2). At this time, the inter-prediction unit 126 encodes the MV selection information used to identify the selected candidate MV into a stream.

[0250] Finally, the intermediate prediction unit 126 uses the exported MV and the encoded reference image to perform motion compensation on the current block, thereby generating a prediction image of the current block (step Sh_3).

[0251] Furthermore, the information contained in the encoded signal regarding the inter-prediction mode (in the example above, the merging mode) used in the generation of the predicted image is encoded as, for example, prediction parameters.

[0252] Figure 21 is a conceptual diagram illustrating one example of the motion vector export process of the current image in the merge mode.

[0253] First, a list of candidate predicted MVs that have been registered is generated. Candidate predicted MVs include the following: spatial adjacency predicted MVs, which are the MVs of multiple encoded blocks spatially surrounding the object block; temporal adjacency predicted MVs, which are the MVs of blocks near the location of the object block projected in the encoded reference image; combined predicted MVs, which are the MVs generated by combining the MV values ​​of spatial adjacency predicted MVs and temporal adjacency predicted MVs; and zero predicted MVs, which are MVs with a value of zero.

[0254] Next, select one predicted MV from the multiple predicted MVs already logged in the predicted MV list, thereby determining it as the MV of the target block.

[0255] Furthermore, in the variable-length encoding section, the signal indicating which prediction MV has been selected, namely merge_idx, is recorded in the stream and encoded.

[0256] Additionally, the predicted MV listed in the predicted MV list illustrated in Figure 21 is only one example. It may be a different number than the number shown in the figure, or it may be a type that does not include the predicted MVs in the figure, or it may be a predicted MV that adds to the types of predicted MVs in the figure.

[0257] Alternatively, the MV of the object block derived by the merge mode can be used for the DMVR (decoder motion vector refinement) process described later, thereby determining the final MV.

[0258] Furthermore, the candidate MVs for prediction are the aforementioned candidate MVs, and the list of predicted MVs is the aforementioned list of candidate MVs. Also, the list of candidate MVs can also be called a candidate list. Furthermore, merge_idx contains MV selection information.

[0259] [MV Export > FRUC Mode] Motion information can also be signaled on the decoding side without being encoded on the encoding side. Alternatively, as mentioned above, the merging mode specified in the H.265 / HEVC standard can also be used. Furthermore, for example, motion search can be performed on the decoding side to export motion information. In an embodiment, motion search is performed on the decoding side without using the pixel values ​​of the current block.

[0260] Here, we will explain the motion search mode performed on the decoding device side. This motion search mode on the decoding device side is sometimes called PMMVD (pattern matched motion vector derivation) mode or FRUC (frame rate up-conversion) mode.

[0261] Figure 22 illustrates an example of FRUC processing using a flowchart. First, referencing the movement vectors of spatially or temporally adjacent encoded blocks, a plurality of candidate lists (i.e., candidate MV lists, which can also be shared with the merge list) are generated, each with a predicted movement vector (MV) (step Si_1). Second, the best candidate MV is selected from the plurality of candidate MVs registered in the candidate MV list (step Si_2). For example, the evaluation value of each candidate MV in the candidate MV list is calculated, and one candidate MV is selected based on the evaluation value. Next, the movement vector for the current block is derived based on the selected candidate movement vector (step Si_4). Specifically, for example, the selected candidate movement vector (best candidate MV) can be directly derived as the movement vector for the current block. Alternatively, pattern matching can be performed on the surrounding area of ​​the position in the reference image corresponding to the selected candidate movement vector to derive the movement vector for the current block. That is, the surrounding area of ​​the best candidate MV can also be searched using pattern matching and evaluation values ​​from the reference image. If an MV with a better evaluation value is found, the best candidate MV is updated to the aforementioned MV and used as the final MV for the current block. Alternatively, it can be configured to not perform the update process and not update to the MV with the better evaluation value.

[0262] Finally, the intermediate prediction unit 126 uses the exported MV and the encoded reference image to perform motion compensation on the current block, thereby generating a prediction image of the current block (step Si_5).

[0263] When processing at the sub-block level, the exact same processing can be performed.

[0264] Evaluation values ​​can also be calculated using various methods. For example, comparing the reconstructed image of a region within a reference image corresponding to the motion vector with the reconstructed image of a predetermined region (such as shown below, or a region of another reference image, or a region of an adjacent block in the current image). The predetermined region can also be determined in advance.

[0265] Furthermore, the difference between the pixel values ​​of the two reconstructed images can be calculated and used as the evaluation value of the motion vector. In addition to the difference value, other information can also be used to calculate the evaluation value.

[0266] Secondly, a detailed explanation of an example of pattern matching is provided. First, one candidate MV contained in the candidate MV list (e.g., a merged list) is selected as the starting point for the pattern matching search. For example, as a pattern match, either a first pattern match or a second pattern match can be used. The first and second pattern matches are sometimes referred to as bilateral matching and template matching, respectively.

[0267] [MV Export > FRUC > Two-way Matching] The first pattern matching is performed between two blocks: two blocks within two different reference images, and two blocks along the motion trajectory of the current block. Therefore, in the first pattern matching, the predetermined region used to calculate the aforementioned candidate evaluation values ​​is a region within another reference image along the motion trajectory of the current block. This predetermined region can also be determined in advance.

[0268] Figure 23 is a conceptual diagram illustrating an example of first pattern matching (bidirectional matching) between two blocks in two reference images along a movement trajectory. As shown in Figure 23, in first pattern matching, two movement vectors (MV0, MV1) are derived by searching for the most matching pair among two blocks in two different reference images (Ref0, Ref1) along the movement trajectory of the current block. Specifically, the difference between the reconstructed image of a specified position in the first encoded reference image (Ref0) and the reconstructed image of a specified position in the second encoded reference image (Ref1) is derived for the current block, and the obtained difference value is used to calculate the evaluation value. The specified position in the first encoded reference image is the position specified by the candidate MV, and the specified position in the second encoded reference image is the position specified by the symmetrical MV after scaling the candidate MV using the display time interval. Being able to select the candidate MV with the best evaluation value from multiple candidate MVs as the final MV can bring good results.

[0269] Under the assumption of continuous movement trajectories, the movement vectors (MV0, MV1) indicating two reference blocks are proportional to the temporal distances (TD0, TD1) between the current image (Cur Pic) and the two reference images (Ref0, Ref1). For example, if the current image is located between two reference images in time, and the temporal distances from the current image to the two reference images are equal, then in the first pattern matching, a mirror-symmetric bidirectional movement vector will be derived.

[0270] [MV Export > FRUC > Template Matching] In the second pattern matching (template matching), pattern matching is performed between the template in the current image (the block adjacent to the current block in the current image (e.g., the top and / or left adjacent block)) and the block in the reference image. Therefore, in the second pattern matching, the predetermined area used to calculate the aforementioned candidate evaluation value is the block adjacent to the current block in the current image.

[0271] Figure 24 is a conceptual diagram illustrating one example of pattern matching (template matching) between a template in the current image and a block in the reference image. As shown in Figure 24, in the second pattern matching, the movement vector of the current block is derived by searching in the reference image (Ref0) for the block that best matches the block adjacent to the current block in the current image (Cur block). Specifically, for the current block, the difference between the reconstructed image of the encoded regions of its left and top neighbors (or one of them) and the reconstructed image at the same position in the encoded reference image (Ref0) specified by the candidate MV is derived. The obtained difference value is used to calculate the evaluation value, and the candidate MV with the best evaluation value is selected from among the multiple candidate MVs as the best candidate MV.

[0272] Information indicating whether this FRUC mode applies (e.g., a FRUC flag) can also be signaled at the CU level. Furthermore, when the FRUC mode applies (e.g., when the FRUC flag is true), information indicating the applicable pattern matching method (first pattern matching or second pattern matching) (e.g., a FRUC mode flag) can also be signaled at the CU level. Additionally, the signaling of this information is not limited to the CU level; it can also be at other levels (e.g., sequence level, image level, slice level, tile level, CTU level, or sub-block level).

[0273] [MV Export > Affine Mode] Secondly, the affine model for deriving movement vectors in sub-block units based on the movement vectors of a plurality of adjacent blocks is explained. This model is sometimes called the affine motion compensation prediction model.

[0274] Figure 25A is a conceptual diagram illustrating an example of deriving a movement vector for a sub-block unit based on the movement vectors of a plurality of adjacent blocks. In Figure 25A, the current block contains 16 4×4 sub-blocks. Here, the movement vector v0 of the top-left control point of the current block is derived from the movement vectors of the adjacent blocks, and similarly, the movement vector v1 of the top-right control point of the current block is derived from the movement vectors of the adjacent sub-blocks. Then, the movement vectors (vx, vy) of each sub-block within the current block can be derived by projecting the two movement vectors v0 and v1 using the following equation (1A). [Number 1]

[0275] Here, x and y represent the horizontal and vertical positions of the sub-block, respectively, and w represents the predetermined weighting coefficient. The predetermined weighting coefficient can also be determined in advance.

[0276] Information indicating this affine pattern (e.g., an affine flag) can also be signaled at the CU level. Furthermore, the signaling of this affine pattern information is not limited to the CU level; it can also be at other levels (e.g., sequence level, image level, slice level, tile level, CTU level, or sub-block level).

[0277] Furthermore, this affine mode can also include several modes with different methods for deriving the movement vectors of the upper left and upper right control points. For example, there are two modes in the affine mode: the inter-affine (also known as the general inter-affine) mode and the affine merging mode.

[0278] [MV Export > Affine Mode] Figure 25B is a conceptual diagram illustrating an example of deriving the movement vector of a sub-block unit in an affine pattern with three control points. In Figure 25B, the current block contains 16 4×4 sub-blocks. Here, based on the movement vectors of adjacent blocks, the movement vector v0 of the top-left control point of the current block is derived; similarly, based on the movement vectors of adjacent blocks, the movement vector v1 of the top-right control point of the current block is derived; and based on the movement vectors of adjacent blocks, the movement vector v2 of the bottom-left control point of the current block is derived. Then, the movement vectors (vx, vy) of each sub-block within the current block can be derived by projecting the three movement vectors v0, v1, and v2 using the following equation (1B). [Number 2]

[0279] Here, x and y represent the horizontal and vertical positions of the sub-block center, respectively, w represents the width of the current block, and h represents the height of the current block.

[0280] Affine modes with different numbers of control points (e.g., 2 and 3) can also be switched and signaled at the CU level. Furthermore, information indicating the number of control points for the affine mode used at the CU level can also be signaled at other levels (e.g., sequence level, image level, slice level, tile level, CTU level, or sub-block level).

[0281] Furthermore, this affine pattern with three control points can also include several patterns that differ from the methods used to derive the movement vectors of the upper left, upper right, and lower left control points. For example, there are two types of affine patterns: the inter-affine (also known as the general inter-affine) pattern and the affine merging pattern.

[0282] [MV Export > Affine Merge Mode] Figures 26A, 26B, and 26C are conceptual diagrams used to illustrate affine merging patterns.

[0283] In affine merging mode, as shown in Figure 26A, for example, among the encoded blocks A (left), B (top), C (top right), D (bottom left), and E (top left) adjacent to the current block, the predicted movement vectors for the control points of the current block are calculated based on a plurality of movement vectors corresponding to the blocks encoded in affine mode. Specifically, the blocks are examined in the order of encoded blocks A (left), B (top), C (top right), D (bottom left), and E (top left) to identify the first valid block encoded in affine mode. The predicted movement vectors for the control points of the current block are calculated based on a plurality of movement vectors corresponding to this identified block.

[0284] For example, as shown in Figure 26B, when encoding in an affine pattern with two control points in block A adjacent to the left of the current block, movement vectors v3 and v4 are derived, projected onto the upper left and upper right corners of the encoded block containing block A. Then, from the derived movement vectors v3 and v4, the predicted movement vector v0 for the upper left corner control point and the predicted movement vector v1 for the upper right corner control point of the current block are calculated.

[0285] For example, as shown in Figure 26C, when encoding in an affine pattern with three control points in block A adjacent to the left of the current block, movement vectors v3, v4, and v5 are derived, projected onto the top-left, top-right, and bottom-left corners of the encoded block containing block A. Then, from the derived movement vectors v3, v4, and v5, the predicted movement vector v0 for the top-left control point, the predicted movement vector v1 for the top-right control point, and the predicted movement vector v2 for the bottom-left control point of the current block are calculated.

[0286] Furthermore, the method of deriving the predicted movement vectors of the control points of the current block in step Sj_1 of Figure 29, which will be described later, can also be used to derive the predicted movement vectors.

[0287] Figure 27 is a flowchart showing one example of an affine merging pattern.

[0288] In the affine merging mode, as shown in the figure, firstly, the inter-prediction unit 126 derives the prediction MV of each control point of the current block (step Sk_1). The control points are shown in Figure 25A, which are the points at the top left and top right corners of the current block, or as shown in Figure 25B, which are the points at the top left, top right, and bottom left corners of the current block.

[0289] In other words, as shown in Figure 26A, the inter-prediction unit 126 checks the blocks in the order of the encoded blocks A (left), B (top), C (top right), D (bottom left), and E (top left) to identify the first valid block encoded in affine mode.

[0290] Next, when block A has been specified and block A has two control points, as shown in FIG26B, the inter-prediction unit 126 calculates the movement vector v0 of the control point at the top left corner and the movement vector v1 of the control point at the top right corner of the current block from the movement vectors v3 and v4 of the top left and top right corners of the encoded block containing block A. For example, the inter-prediction unit 126 projects the movement vectors v3 and v4 of the top left and top right corners of the encoded block onto the current block, thereby calculating the predicted movement vector v0 of the control point at the top left corner and the predicted movement vector v1 of the control point at the top right corner of the current block.

[0291] Alternatively, when block A has been specified and block A has 3 control points, as shown in Figure 26C, the inter-prediction unit 126 calculates the movement vector v0 of the control point at the top left corner, the movement vector v1 of the control point at the top right corner, and the movement vector v2 of the control point at the bottom left corner of the current block from the movement vectors v3, v4, and v5 of the top left, top right, and bottom left corners of the encoded block containing block A. For example, the inter-prediction unit 126 projects the movement vectors v3, v4, and v5 of the top left, top right, and bottom left corners of the encoded block onto the current block, thereby calculating the predicted movement vector v0 of the control point at the top left corner, the predicted movement vector v1 of the control point at the top right corner, and the movement vector v2 of the control point at the bottom left corner of the current block.

[0292] Next, the inter-prediction unit 126 performs motion compensation for each of the plurality of sub-blocks contained in the current block. That is, for each of the plurality of sub-blocks, the inter-prediction unit 126 uses two predicted motion vectors v0 and v1 with the above equation (1A), or three predicted motion vectors v0, v1 and v2 with the above equation (1B), to calculate the motion vector of each sub-block as an affine MV (step Sk_2). Then, the inter-prediction unit 126 uses the affine MV and the encoded reference image to perform motion compensation for the sub-blocks (step Sk_3). As a result, motion compensation is performed on the current block, and a predicted image of the current block is generated.

[0293] [MV Export > Affine Mode] Figure 28A is a conceptual diagram illustrating an affine pattern with two control points.

[0294] In this affine pattern, as shown in Figure 28A, the movement vector selected from the movement vectors of the coded blocks A, B, and C adjacent to the current block is used as the predicted movement vector v0 for the top-left control point of the current block. Similarly, the movement vector selected from the movement vectors of the coded blocks D and E adjacent to the current block is used as the predicted movement vector v1 for the top-right control point of the current block.

[0295] Figure 28B is a conceptual diagram illustrating an affine pattern with three control points.

[0296] In this affine pattern, as shown in Figure 28B, the movement vector selected from the movement vectors of the adjacent encoded blocks A, B, and C is used as the predicted movement vector v0 for the top-left control point of the current block. Similarly, the movement vector selected from the movement vectors of the adjacent encoded blocks D and E is used as the predicted movement vector v1 for the top-right control point of the current block. Furthermore, the movement vector selected from the movement vectors of the adjacent encoded blocks F and G is used as the predicted movement vector v2 for the bottom-left control point of the current block.

[0297] Figure 29 is a flowchart showing one example of an affine pattern.

[0298] As shown in the figure, in the affine inter-block mode, firstly, the inter-block prediction unit 126 derives the prediction MV(v0, v1) or (v0, v1, v2) for each of the two or three control points of the current block (step Sj_1). As shown in Figure 25A or Figure 25B, the control points are the points at the top left, top right, or bottom left corners of the current block.

[0299] In other words, the inter-prediction unit 126 selects the movement vector of any one of the encoded blocks near each control point of the current block shown in Figure 28A or Figure 28B, thereby deriving the predicted movement vector (v0, v1) or (v0, v1, v2) of the control point of the current block. At this time, the inter-prediction unit 126 encodes the predicted movement vector selection information used to identify the two selected movement vectors into a stream.

[0300] For example, the prediction unit 126 may also use cost evaluation or the like to determine which block's movement vector should be selected from the coded blocks adjacent to the current block as the prediction movement vector of the control point, and describe the flag indicating which prediction movement vector was selected in the bit stream.

[0301] Next, the inter-prediction unit 126 updates the predicted movement vectors selected or derived in step Sj_1 (step Sj_2) while performing movement search (steps Sj_3 and Sj_4). That is, the inter-prediction unit 126 uses the above-mentioned equation (1A) or equation (1B) to calculate the movement vector of each sub-block corresponding to the updated predicted movement vector, and uses it as an affine MV (step Sj_3). Then, the inter-prediction unit 126 uses these affine MVs and the encoded reference image to perform movement compensation on each sub-block (step Sj_4). As a result, in the movement search loop, the inter-prediction unit 126 determines the predicted movement vector that can obtain the minimum cost as the movement vector of the control point (step Sj_5). At this time, the inter-prediction unit 126 further encodes the difference between the determined MV and the predicted movement vector as a differential MV into a stream.

[0302] Finally, the prediction unit 126 uses the determined MV and the encoded reference image to perform motion compensation on the current block, thereby generating a prediction image of the current block (step Sj_6).

[0303] [MV Export > Affine Mode] When switching between affine modes with different numbers of control points (e.g., 2 and 3) at the CU level and signaling, there are cases where the number of control points in the encoded block differs from that in the current block. Figures 30A and 30B are conceptual diagrams illustrating the method for deriving the prediction vector of control points when the number of control points in the encoded block differs from that in the current block.

[0304] For example, as shown in Figure 30A, when encoding in an affine pattern where the current block has three control points (top left, top right, and bottom left) and the adjacent block A to the left of the current block has two control points, movement vectors v3 and v4 are derived, projected onto the top left and top right corners of the encoded block containing block A. Then, from the derived movement vectors v3 and v4, the predicted movement vector v0 for the top left control point and the predicted movement vector v1 for the top right control point of the current block are calculated. Furthermore, from the derived movement vectors v0 and v1, the predicted movement vector v2 for the bottom left control point is calculated.

[0305] For example, as shown in Figure 30B, when encoding in an affine pattern where the current block has two control points at its top left and top right corners, and the adjacent block A to the left of the current block has three control points, movement vectors v3, v4, and v5 are derived, projected onto the top left, top right, and bottom left corners of the encoded block containing block A. Then, from the derived movement vectors v3, v4, and v5, the predicted movement vector v0 for the top left control point and the predicted movement vector v1 for the top right control point of the current block are calculated.

[0306] The method for deriving the predicted movement vectors of the control points of the current block in step Sj_1 of Figure 29 can also be used.

[0307] [MV Export > DMVR] Figure 31A is a flowchart showing the relationship between the merge mode and DMVR.

[0308] The inter-prediction unit 126 derives the movement vector of the current block using a merge mode (step S1_1). Next, the inter-prediction unit 126 determines whether to perform a movement vector search, i.e., whether to perform a movement search (step S1_2). Here, if the inter-prediction unit 126 determines that a movement search should not be performed (no in step S1_2), it determines the movement vector derived in step S1_1 as the final movement vector for the current block (step S1_4). That is, in this case, the movement vector of the current block is determined using a merge mode.

[0309] On the other hand, when it is determined in step Sl_1 that a move search is to be performed (as in step Sl_2), the intermediate prediction unit 126 searches the surrounding area of ​​the reference image shown by the move vector derived in step Sl_1, and derives the final move vector for the current block (step Sl_3). That is, in this case, the move vector of the current block is determined by DMVR.

[0310] Figure 31B is a conceptual diagram illustrating one example of the DMVR process used to determine the MV.

[0311] First, (for example in merge mode) the best MVP for the current block is set as the candidate MV. Next, based on the candidate MV (L0), reference pixels are identified from the encoded image in the L0 direction, i.e., the first reference image (L0). Similarly, based on the candidate MV (L1), reference pixels are identified from the encoded image in the L1 direction, i.e., the second reference image (L1). A template is generated by averaging these reference pixels.

[0312] Next, using the aforementioned template, the surrounding areas of the candidate MVs in the first reference image (L0) and the second reference image (L1) are searched respectively, and the MV with the lowest cost is determined as the final MV. Alternatively, the cost value can be calculated using, for example, the difference between each pixel value of the template and each pixel value of the search area, as well as the candidate MV value.

[0313] Furthermore, typically, the structure and operation of the processing described herein are essentially the same in both the encoding device and the decoding device described later.

[0314] Even if it's not the processing example described here, any processing method that can search for the surrounding elements of candidate MVs to derive the final MV is acceptable.

[0315] [Movement Compensation > BIO / OBMC] In motion compensation, there are modes that generate a predicted image and correct that predicted image. Examples of such modes include BIO and OBMC, which will be discussed later.

[0316] Figure 32 is a flowchart showing one example of how a predicted image is generated.

[0317] The prediction unit 126 generates a prediction image (step Sm_1) and corrects the prediction image using, for example, any of the above-described modes (step Sm_2).

[0318] Figure 33 is a flowchart showing another example of the generation of a predicted image.

[0319] The inter-prediction unit 126 determines the movement vector of the current block (step Sn_1). Next, the inter-prediction unit 126 generates a prediction image (step Sn_2) and determines whether to perform correction processing (step Sn_3). Here, if the inter-prediction unit 126 determines that correction processing is required (yes in step Sn_3), it corrects the prediction image, thereby generating a final prediction image (step Sn_4). On the other hand, if the inter-prediction unit 126 determines that correction processing is not required (no in step Sn_3), it does not perform correction and outputs the prediction image as the final prediction image (step Sn_5).

[0320] Furthermore, in motion compensation, there is a mode that corrects brightness when generating the predicted image. This mode is, for example, LIC, which will be described later.

[0321] Figure 34 is a flowchart showing another example of the generation of a predicted image.

[0322] The inter-prediction unit 126 derives the motion vector of the current block (step So_1). Next, the inter-prediction unit 126 determines whether to perform brightness correction processing (step So_2). Here, when the inter-prediction unit 126 determines that brightness correction processing should be performed (yes in step So_2), it generates a prediction image while performing brightness correction (step So_3). That is, the prediction image is generated using LIC. On the other hand, when the inter-prediction unit 126 determines that brightness correction processing should not be performed (no in step So_2), it generates a prediction image by means of normal motion compensation without performing brightness correction (step So_4).

[0323] [Movement Compensation > OBMC] Alternatively, motion information from the current block obtained through motion search can be used, along with motion information from adjacent blocks, to generate inter-block prediction signals. Specifically, a weighted sum can be made of the prediction signal based on motion information obtained through motion search (within the reference image) and the prediction signal based on motion information from adjacent blocks (within the current image), thereby generating inter-block prediction signals using sub-block units within the current block. This type of inter-block prediction (motion compensation) is sometimes called OBMC (overlapped block motion compensation).

[0324] In OBMC mode, information displaying the size of sub-blocks used by OBMC (e.g., OBMC block size) can also be signaled at the sequence level. Furthermore, information displaying whether OBMC mode is applicable (e.g., OBMC flag) can also be signaled at the CU level. Additionally, the signaling level for this information is not limited to the sequence level and CU level; it can also be other levels (e.g., image level, slice level, tile level, CTU level, or sub-block level).

[0325] A more specific example of the OBMC mode will be provided. Figures 35 and 36 are flowcharts and conceptual diagrams illustrating the general outline of the predictive image correction processing performed in OBMC processing.

[0326] First, as shown in Figure 36, the motion vector (MV) assigned to the processing object (current) block is used to obtain the predicted image (Pred) with normal motion compensation. In Figure 36, the arrow "MV" points to the reference image, indicating which image the current block of the current image is referencing in order to obtain the predicted image.

[0327] Next, the movement vector (MV_L) derived from the encoded left adjacent block is applied (reused) to the encoded target block to obtain the predicted image (Pred_L). The movement vector (MV_L) is represented by the arrow "MV_L" pointing from the current block to the reference image. Then, the two predicted images Pred and Pred_L are overlapped to perform the first correction of the predicted image. This has the effect of blending the boundaries between adjacent blocks.

[0328] Similarly, the movement vector (MV_U) derived from the encoded upper adjacent block is applied (reused) to the encoded target block to obtain the prediction image (Pred_U). The movement vector (MV_U) is represented by the arrow "MV_U" pointing from the current block to the reference image. Next, the prediction image Pred_U is overlaid on the prediction image that has undergone the first correction (e.g., Pred and Pred_L) to perform the second correction of the prediction image. This has the effect of blending the boundaries between adjacent blocks. The prediction image obtained by the second correction is the final prediction image of the current block, whose boundaries with adjacent blocks have been blended (smoothed).

[0329] Furthermore, although the above example uses a two-fold correction method involving the left and top adjacent blocks, this correction method can also be a three-fold or more correction method that uses the right and / or bottom adjacent blocks.

[0330] Alternatively, the overlapping area may not be the entire pixel area of ​​the block, but only a portion of the area near the block boundary.

[0331] Furthermore, the OBMC prediction image correction process used here to overlay additional prediction images Pred_L and Pred_U from a single reference image to obtain a single prediction image Pred is described. However, when correcting prediction images based on multiple reference images, the same process can be applied to each of the multiple reference images. In this case, by performing OBMC image correction based on multiple reference images, corrected prediction images are obtained from each reference image, and then these obtained multiple corrected prediction images are further overlaid to obtain the final prediction image.

[0332] Additionally, in OBMC, the unit of an object block can be a predicted block unit or a sub-block unit after further dividing the predicted block.

[0333] One method for determining whether OBMC processing is applicable is using the `obmc_flag`, which is a signal indicating whether OBMC processing is applicable. For example, the encoding device can also determine whether the target block belongs to a region with complex motion. If it belongs to a region with complex motion, the encoding device sets a value of 1 as `obmc_flag` and applies OBMC processing for encoding; if it does not belong to a region with complex motion, it sets a value of 0 as `obmc_flag` and does not apply OBMC processing for encoding the block. On the other hand, in the decoding device, the `obmc_flag` recorded in the stream (e.g., a compressed sequence) is decoded, thereby switching whether OBMC processing is applicable and performing decoding based on this value.

[0334] In the example described above, the intermediate prediction unit 126 generates a rectangular prediction image for the current block of the rectangle. However, the intermediate prediction unit 126 may also generate multiple prediction images of different shapes from the rectangle for the current block of the rectangle, and combine these multiple prediction images to generate the final rectangular prediction image. For example, a shape different from a rectangle could be a triangle.

[0335] Figure 37 is a conceptual diagram illustrating the generation of the predicted image of two triangles.

[0336] The intermediate prediction unit 126 performs motion compensation on the first partition of the triangle within the current block using the first MV of the first partition, thereby generating a predicted image of the triangle. Similarly, the intermediate prediction unit 126 performs motion compensation on the second partition of the triangle within the current block using the second MV of the second partition, thereby generating a predicted image of the triangle. Next, the intermediate prediction unit 126 combines these predicted images to generate a predicted image that is the same rectangle as the current block.

[0337] Furthermore, in the example shown in Figure 37, although the first and second partitions are triangular, they could also be trapezoidal, or they could be different shapes. Moreover, in the example shown in Figure 37, although the current block is composed of two partitions, it could also be composed of three or more partitions.

[0338] Furthermore, partitions 1 and 2 can be duplicated. That is, partitions 1 and 2 can contain the same pixel area. In this case, the predicted image of partition 1 and the predicted image of partition 2 can also be used to generate the predicted image of the current block.

[0339] Furthermore, this example shows an instance where two partitions are generated using inter-partition prediction to produce the predicted image, but it is also possible to generate the predicted image using intra-partition prediction for at least one partition.

[0340] [Movement Compensation > BIO] Secondly, the method for deriving the translation vector will be explained. First, the mode of deriving the translation vector based on a model will be explained, where the aforementioned model assumes uniform linear motion. This mode is sometimes referred to as the BIO (bi-directional optical flow) mode.

[0341] Figure 38 is a conceptual diagram illustrating the model assuming uniform linear motion. In Figure 38, (vx, vy) represents the velocity vector, and τ0 and τ1 represent the time distances between the current image (Cur Pic) and two reference images (Ref 0, Ref 1). (MVx 0, MVy 0) represents the movement vector corresponding to reference image Ref 0, and (MVx 1, MVy 1) represents the movement vector corresponding to reference image Ref 1.

[0342] At this point, under the assumption that the velocity vector (vx,vy) is a constant linear motion, (MVx 0,MVy 0) and (MVx 1,MVy 1) are expressed as (v xτ 0,vy τ 0) and (-v xτ 1,-vy τ 1) respectively, and the following optical flow equation (2) can also be used. [Number 3]

[0343] Here, I(k) represents the brightness value of the reference image k (k=0,1) after motion compensation. This optical flow equation states that (i) the time derivative of the brightness value, (ii) the product of the horizontal velocity and the horizontal component of the spatial gradient of the reference image, and (iii) the product of the vertical velocity and the vertical component of the spatial gradient of the reference image equals zero. Alternatively, the motion vector of the block unit obtained from the merge list, etc., can be corrected in pixels by combining this optical flow equation with Hermite interpolation.

[0344] Alternatively, a different method can be used to derive the movement vector on the decoding device side, instead of deriving it based on a model assuming constant linear motion. For example, the movement vector can be derived on a sub-block basis from the movement vectors of a plurality of adjacent blocks.

[0345] [Movement Compensation>LIC] Secondly, an example of a mode for generating a predicted image (prediction) using LIC (local illumination compensation) processing will be explained.

[0346] Figure 39 is a conceptual diagram illustrating one example of a predictive image generation method that uses luminance correction processing with LIC processing.

[0347] First, export the MV from the encoded reference image to obtain the reference image corresponding to the current block.

[0348] Next, information indicating how the brightness value changes in the current block is extracted from the reference image and the current image. This extraction is performed based on the brightness pixel values ​​of the encoded left adjacent reference region (surrounding reference region) and the encoded upper adjacent reference region (surrounding reference region) in the current image, and the brightness pixel values ​​at the same positions in the reference image specified by the exported MV. Then, using the information indicating how the brightness value changes, the brightness correction parameters are calculated.

[0349] For the reference image within the reference image specified by MV, a brightness correction process is performed using the aforementioned brightness correction parameters, thereby generating a predicted image for the current block.

[0350] Furthermore, the shape of the aforementioned surrounding reference area in Figure 39 is only one example; other shapes may also be used.

[0351] Furthermore, although the process of generating a prediction image from a single reference image is explained here, the same applies to generating a prediction image from multiple reference images. The reference images obtained from each reference image can be brightness-corrected using the same method as described above before generating the prediction image.

[0352] One method for determining whether LIC processing is applicable is using a `lic_flag`, which is a signal indicating whether LIC processing is applicable. For example, in an encoding device, it is determined whether the current block belongs to an area where brightness changes occur. If it does, a value of 1 is set as `lic_flag` and LIC processing is applied for encoding. If it does not belong to an area where brightness changes occur, a value of 0 is set as `lic_flag` and LIC processing is not applied for encoding. On the other hand, in a decoding device, the `lic_flag` recorded in the stream can be decoded, and the application of LIC processing can be switched according to its value.

[0353] As another method for determining whether LIC processing is applicable, there is a method based on whether LIC processing has been applied to surrounding blocks. For example, when the current block is in merge mode, it is determined whether the surrounding blocks selected during MV export in merge mode processing have undergone LIC processing and have been encoded. Based on the result, the application of LIC processing is switched and encoding is performed accordingly. Furthermore, in this example, the same processing can be applied to the decoding device side.

[0354] The state of LIC processing (luminance correction processing) has been illustrated using Figure 39, and its details are explained below.

[0355] First, the inter-prediction unit 126 derives a motion vector from the encoded image, i.e., the reference image, to obtain a reference image corresponding to the encoded object block.

[0356] Next, the inter-prediction unit 126 uses the luminance pixel values ​​of the coded surrounding reference regions (left and top adjacent) and the luminance pixel values ​​at the same position in the reference image specified by the movement vector to extract information indicating how the luminance value changes between the reference image and the coded object image, and calculates luminance correction parameters. For example, let the luminance pixel value of a pixel in the surrounding reference region of the coded object image be p0, and let the luminance pixel value of a pixel in the surrounding reference region of the reference image at the same position be p1. The inter-prediction unit 126 calculates coefficients A and B for a plurality of pixels in the surrounding reference region to optimize A×p1+B=p0, and uses them as luminance correction parameters.

[0357] Next, the inter-prediction unit 126 performs brightness correction processing on the reference image within the reference image specified by the movement vector using brightness correction parameters, thereby generating a prediction image for the coded object block. For example, let the brightness pixel value in the reference image be p2, and let the brightness pixel value in the brightness-corrected prediction image be p3. The inter-prediction unit 126 calculates A×p2+B=p3 for each pixel in the reference image, thereby generating the brightness-corrected prediction image.

[0358] Furthermore, the shape of the peripheral reference region in Figure 39 is only one example; other shapes may also be used. Also, a portion of the peripheral reference region shown in Figure 39 may be used. For example, a region containing pixels spaced a predetermined number from its upper adjacent pixel and left adjacent pixel may be used as the peripheral reference region. Furthermore, the peripheral reference region is not limited to the region adjacent to the coded object block; it may also be a region not adjacent to the coded object block. The predetermined number of pixels can also be determined in advance.

[0359] Furthermore, in the example shown in Figure 39, the surrounding reference region within the reference image is the region specified by the movement vector of the coded object image from the surrounding reference region within the coded object image, but it can also be the region specified by other movement vectors. For example, the other movement vector can also be the movement vector of the surrounding reference region within the coded object image.

[0360] Furthermore, although the operation in the encoding device 100 is described here, the operation in the decoding device 200 is typically the same.

[0361] Furthermore, LIC processing can be applied not only to luminance but also to chromatic aberration. In this case, correction parameters can be exported separately for Y, Cb, and Cr, or a common correction parameter can be used for any of them.

[0362] Furthermore, LIC processing can also be applied at the sub-block level. For example, the surrounding reference area of ​​the current sub-block and the surrounding reference area of ​​the reference sub-block within the reference image specified by the MV of the current sub-block can be used to export correction parameters.

[0363] [Forecasting and Control Department] The prediction control unit 128 selects either the internal prediction signal (the signal output from the internal prediction unit 124) or the inter-prediction signal (the signal output from the inter-prediction unit 126), and outputs the selected signal as the prediction signal to the subtraction unit 104 and the addition unit 116.

[0364] As shown in Figure 1, in various encoding device examples, the prediction control unit 128 can also output the prediction parameters input to the entropy encoding unit 110. The entropy encoding unit 110 can also generate an encoded bitstream (or sequence) based on the prediction parameters input from the prediction control unit 128 and the quantization coefficients input from the quantization unit 108. The prediction parameters can also be used in the decoding device. The decoding device can also receive the encoded bitstream and decode it, and perform the same processing as the prediction processing performed in the inner prediction unit 124, the inter-prediction unit 126, and the prediction control unit 128. The prediction parameters may also include: selecting a prediction signal (e.g., a movement vector, prediction type, or prediction mode used in the inner prediction unit 124 or the inter-prediction unit 126); or, based on the prediction processing performed in the inner prediction unit 124, the inter-prediction unit 126, and the prediction control unit 128, or any index, flag, or value representing that prediction processing.

[0365] [Installation example of the encoding device] Figure 40 is a block diagram showing an example of the installation of the encoding device 100. The encoding device 100 includes a processor a1 and a memory a2. For example, a plurality of components of the encoding device 100 shown in Figure 1 are installed using the processor a1 and memory a2 shown in Figure 40.

[0366] Processor a1 is a circuit that performs information processing and can access memory a2. For example, processor a1 is a dedicated or general-purpose electronic circuit for encoding dynamic images. Processor a1 can also be a processor such as a CPU. Furthermore, processor a1 can also be an assembly of multiple electronic circuits. Also, for example, processor a1 can perform the functions of multiple components of the encoding device 100 shown in FIG1, etc.

[0367] Memory a2 is a dedicated or general-purpose memory that stores information used by processor a1 to encode dynamic images. Memory a2 can be an electronic circuit or connected to processor a1. Alternatively, memory a2 can be contained within processor a1. Furthermore, memory a2 can be a collection of multiple electronic circuits. Also, memory a2 can be a magnetic disk or optical disk, or it can be presented as a storage device or recording medium. Furthermore, memory a2 can be non-volatile memory or volatile memory.

[0368] For example, memory a2 can store the motion image to be encoded, or it can store the bit array corresponding to the encoded motion image. Also, memory a2 can store the program used by processor a1 to encode the motion image.

[0369] Furthermore, for example, memory a2 can also function as one of the components of the encoding device 100 shown in FIG1 that stores information. For example, memory a2 can also function as block memory 118 and frame memory 122 shown in FIG1. ​​More specifically, memory a2 can also store reconstructed blocks and reconstructed images, etc.

[0370] Alternatively, in the encoding device 100, it is not necessary to install all of the plurality of constituent elements shown in FIG1, etc., nor is it necessary to perform all of the plurality of processes described above. A portion of the plurality of constituent elements shown in FIG1, etc., may also be included in other devices, and a portion of the plurality of processes described above may also be executed by other devices.

[0371] [Decoding device] Next, a decoding device that can decode, for example, the encoded signal (encoded bit stream) output from the encoding device 100 described above will be described. Figure 41 is a block diagram showing the functional configuration of the decoding device 200 in the embodiment. The decoding device 200 is a motion picture decoding device that decodes motion pictures in block units.

[0372] As shown in Figure 41, the decoding device 200 includes: an entropy decoding unit 202, an inverse quantization unit 204, an inverse conversion unit 206, an addition unit 208, a block memory 210, a loop filter unit 212, a frame memory 214, an intra-prediction unit 216, an inter-prediction unit 218, and a prediction control unit 220.

[0373] The decoding device 200 can be implemented using, for example, a general-purpose processor and memory. In this case, when the software program stored in memory is executed by the processor, the processor functions as an entropy decoding unit 202, an inverse quantization unit 204, an inverse conversion unit 206, an adder 208, a loop filter unit 212, an internal prediction unit 216, an inter-prediction unit 218, and a prediction control unit 220. Alternatively, the decoding device 200 can be implemented as one or more dedicated electronic circuits corresponding to the entropy decoding unit 202, the inverse quantization unit 204, the inverse conversion unit 206, the adder 208, the loop filter unit 212, the internal prediction unit 216, the inter-prediction unit 218, and the prediction control unit 220.

[0374] The following section will explain the overall processing flow of the decoding device 200, followed by a description of each component of the decoding device 200.

[0375] [Overall Decoding Process] Figure 42 is a flowchart showing an example of the overall decoding process performed by the decoding device 200.

[0376] First, the entropy decoding unit 202 of the decoding device 200 identifies a segmentation pattern of a fixed-size block (e.g., 128×128 pixels) (step Sp_1). This segmentation pattern is the segmentation pattern selected by the encoding device 100. Next, the decoding device 200 performs steps Sp_2 to Sp_6 on each of the plurality of blocks constituting the segmentation pattern.

[0377] In other words, the entropy decoding unit 202 decodes the encoded quantization coefficients and prediction parameters of the decoding target block (also known as the current block) (specifically, entropy decoding) (step Sp_2).

[0378] Next, the dequantization unit 204 and the inversion conversion unit 206 dequantize and invert the plurality of quantization coefficients, thereby restoring the plurality of prediction residuals (i.e. difference blocks) (step Sp_3).

[0379] Next, the prediction processing unit, which consists of all or part of the internal prediction unit 216, the intermediate prediction unit 218 and the prediction control unit 220, generates the prediction signal of the current block (also known as the prediction block) (step Sp_4).

[0380] Next, the addition unit 208 adds the prediction block to the difference block, thereby reconstructing the current block into a reconstructed image (also known as a decoded image block) (step Sp_5).

[0381] Next, after the reconstructed image is generated, the loop filter unit 212 filters the reconstructed image (step Sp_6).

[0382] Next, the decoding device 200 determines whether the decoding of the entire image has been completed (step Sp_7). If it is determined that the decoding has not been completed (no in step Sp_7), the processing starting from step Sp_1 is repeated.

[0383] As shown in the figure, steps Sp_1 to Sp_7 are performed sequentially by the decoding device 200. Alternatively, multiple steps of one part of these processes can be performed in parallel or their order can be reversed.

[0384] [Entropy Decoding Department] The entropy decoding unit 202 entropy decodes the encoded bitstream. Specifically, the entropy decoding unit 202 arithmetically decodes the encoded bitstream into a binary signal, for example. Then, the entropy decoding unit 202 debinarizes the binary signal. The entropy decoding unit 202 outputs the quantization coefficients to the dequantization unit 204 in block units. The entropy decoding unit 202 can also output the prediction parameters contained in the encoded bitstream (refer to FIG. 1) to the inner prediction unit 216, the interpretation unit 218, and the prediction control unit 220 in the embodiment. The inner prediction unit 216, the interpretation unit 218, and the prediction control unit 220 can perform the same prediction processing as that performed by the inner prediction unit 124, the interpretation unit 126, and the prediction control unit 128 on the encoding device side.

[0385] [Anti-Quantitative Department] The dequantization unit 204 dequantizes the quantization coefficients of the decoded target block (hereinafter referred to as the current block) input from the entropy decoding unit 202. Specifically, the dequantization unit 204 dequantizes each quantization coefficient of the current block according to the quantization parameter corresponding to that quantization coefficient. Then, the dequantization unit 204 outputs the dequantized quantization coefficients (i.e., conversion coefficients) of the current block to the inverse conversion unit 206.

[0386] [Reverse Conversion Section] The inverse conversion unit 206 inversely converts the conversion coefficients input from the inverse quantization unit 204, thereby restoring the prediction error.

[0387] For example, when the information interpreted from the encoded bit stream indicates that EMT or AMT is applicable (e.g., the AMT flag is true), the inverse conversion unit 206 inversely converts the conversion coefficients of the current block based on the information of the already interpreted conversion type.

[0388] For example, when the information representation interpreted from the encoded bit stream is subject to NSST, the inverse conversion unit 206 applies inverse reconversion to the conversion coefficients.

[0389] [Addition Department] The adder 208 reconstructs the current block by adding the prediction error input from the inverse conversion unit 206 to the prediction sample input from the prediction control unit 220. Then, the adder 208 outputs the reconstructed block to the block memory 210 and the loop filter unit 212.

[0390] [Block Memory] Block memory 210 is a memory unit used to store blocks that are referenced in the internal prediction and blocks within the decoded object image (hereinafter referred to as the current image). Specifically, block memory 210 stores the reconstructed blocks output from the addition unit 208.

[0391] [Loop Filtering Section] The loop filter 212 applies loop filtering to the block reconstructed by the adder 208 and outputs the filtered reconstructed block to the frame memory 214 and the display device, etc.

[0392] When the information indicating the on / off state of ALF is interpreted from the encoded bitstream, indicating that ALF is on, one filter is selected from a plurality of filters based on the direction and activity of the local gradient, and the selected filter is applied to the reconstructed block.

[0393] [Frame Memory] Frame memory 214 is a memory unit used to store reference images used in prediction, and is sometimes called a frame buffer. Specifically, frame memory 214 stores reconstructed blocks filtered by loop filter 212.

[0394] [Forecasting Processing Department (Internal Forecasting Department, Intermediate Forecasting Department, Forecasting Control Department)] Figure 43 is a flowchart showing an example of the processing performed in the prediction processing unit of the decoding device 200. The prediction processing unit is composed of all or part of the constituent elements of the inner prediction unit 216, the intermediate prediction unit 218, and the prediction control unit 220.

[0395] The prediction processing unit generates a prediction image for the current block (step Sq_1). This prediction image is also called a prediction signal or a prediction block. The prediction signal may include, for example, an intra-prediction signal or an inter-prediction signal. Specifically, the prediction processing unit uses the reconstructed image obtained by generating prediction blocks, difference blocks, coefficient blocks, restoring difference blocks, and generating decoded image blocks to generate a prediction image for the current block.

[0396] The reconstructed image can be, for example, an image of a reference image, or an image containing the current block, i.e., an image of a decoded block within the current image. A decoded block within the current image can be, for example, an adjacent block of the current block.

[0397] Figure 44 is a flowchart showing another example of the processing performed in the prediction processing unit of the decoding device 200.

[0398] The prediction processing unit determines the method or mode used to generate the prediction image (step Sr_1). For example, this method or mode can also be determined based on, for example, prediction parameters.

[0399] When the prediction processing unit determines that the mode for generating the prediction image is the first mode, it generates the prediction image according to the first mode (step Sr_2a). Furthermore, when the prediction processing unit determines that the mode for generating the prediction image is the second mode, it generates the prediction image according to the second mode (step Sr_2b). Furthermore, when the prediction processing unit determines that the mode for generating the prediction image is the third mode, it generates the prediction image according to the third mode (step Sr_2c).

[0400] Methods 1, 2, and 3 are different methods for generating the predicted image, and may also be, for example, inter-prediction methods, intra-prediction methods, and other prediction methods. The reconstructed image described above can also be used in these prediction methods.

[0401] [Internal Forecasting Department] The internal prediction unit 216 performs internal prediction based on the internal prediction pattern interpreted from the encoded bit stream, referring to blocks within the current image stored in the block memory 210, thereby generating a prediction signal (internal prediction signal). Specifically, the internal prediction unit 216 performs internal prediction by referring to samples (e.g., luminance values, chromaticity values) of blocks adjacent to the current block, thereby generating an internal prediction signal, and outputs the internal prediction signal to the prediction control unit 220.

[0402] Additionally, when the reference luminance block's internal prediction mode is selected in the internal prediction of the chromatic difference block, the internal prediction unit 216 can also predict the chromatic difference component of the current block based on the luminance component of the current block.

[0403] Furthermore, when the information representation interpreted from the encoded bit stream is applicable to PDPC, the internal prediction unit 216 corrects the pixel value after internal prediction based on the gradient of the reference pixel in the horizontal / vertical direction.

[0404] [Inter-departmental Forecasting] Inter-prediction unit 218 refers to a reference image stored in frame memory 214 to predict the current block. Prediction is performed on a unit of the current block or sub-blocks within the current block (e.g., 4×4 blocks). For example, inter-prediction unit 218 uses motion information (e.g., motion vectors) interpreted from the encoded bit stream (e.g., prediction parameters output from entropy decoding unit 202) to perform motion compensation, thereby generating an inter-prediction signal for the current block or sub-block, and outputs the inter-prediction signal to prediction control unit 220.

[0405] When the information interpreted from the encoded bit stream indicates that the OBMC mode is applicable, the interpretation unit 218 can not only use the movement information of the current block obtained by the movement search, but also use the movement information of adjacent blocks to generate interpretation signals.

[0406] Furthermore, when the information representation interpreted from the coded bitstream is in FRUC mode, the intermediate prediction unit 218 performs a motion search according to the pattern matching method (bidirectional matching or template matching) interpreted from the coded bitstream, thereby deriving motion information. Then, the intermediate prediction unit 218 uses the derived motion information to perform motion compensation (prediction).

[0407] Furthermore, when the BIO mode is applied, the inter-prediction unit 218 derives the movement vector based on a model assuming constant velocity linear motion. Also, when the information representation interpreted from the encoded bit stream is subject to affine motion compensation prediction mode, the inter-prediction unit 218 derives the movement vector in sub-block units based on the movement vectors of a plurality of adjacent blocks.

[0408] [MV Export > Normal Mode] When the information interpreted from the encoded bit stream is in a general inter-interpretation mode, the inter-interpretation unit 218 derives the MV based on the information interpreted from the encoded stream and uses the MV to perform motion compensation (prediction).

[0409] Figure 45 is a flowchart showing an example of inter-interpretation performed in a general inter-interpretation mode in the decoding device 200.

[0410] The inter-prediction unit 218 of the decoding device 200 performs motion compensation for each block. Based on information such as the MVs of multiple decoded blocks located in the vicinity of the current block in time or space, the inter-prediction unit 218 obtains multiple candidate MVs for the current block (step Ss_1). In other words, the inter-prediction unit 218 creates a list of candidate MVs.

[0411] Next, the intermediate prediction unit 218 selects N (N is an integer greater than 2) candidate MVs from the plurality of candidate MVs obtained in step Ss_1 according to a predetermined priority order, and uses them as prediction movement vector candidates (also called prediction MV candidates) (step Ss_2). Alternatively, this priority order can also be predetermined for each of the N prediction MV candidates.

[0412] Next, the inter-prediction unit 218 decodes the prediction movement vector selection information from the input stream (i.e., the encoded bit stream), and uses the decoded prediction movement vector selection information to select one prediction movement vector candidate from the N prediction movement vector candidates as the prediction movement vector (also called prediction movement vector) of the current block (step Ss_3).

[0413] Next, the inter-prediction unit 218 decodes the differential MV from the input stream, adds the decoded differential MV, i.e. the differential value, to the selected prediction move vector, thereby deriving the MV of the current block (step Ss_4).

[0414] Finally, the inter-prediction unit 218 uses the exported MV and the decoded reference image to perform motion compensation on the current block, thereby generating a prediction image of the current block (step Ss_5).

[0415] [Forecasting and Control Department] The prediction control unit 220 selects either the internal prediction signal or the inter-prediction signal, and outputs the selected signal as the prediction signal to the adder 208. Overall, the configuration, function, and processing of the prediction control unit 220, internal prediction unit 216, and inter-prediction unit 218 on the decoding device side can also correspond to the configuration, function, and processing of the prediction control unit 128, internal prediction unit 124, and inter-prediction unit 126 on the encoding device side.

[0416] [Installation example of the decoding device] Figure 46 is a block diagram showing an example of the installation of the decoding device 200. The decoding device 200 includes a processor b1 and a memory b2. For example, the plurality of components of the decoding device 200 shown in Figure 41 are installed using the processor b1 and memory b2 shown in Figure 46.

[0417] Processor b1 is a circuit that performs information processing and can access memory b2. For example, processor b1 is a dedicated or general-purpose electronic circuit that decodes encoded moving images (i.e., encoded bitstreams). Processor b1 can also be a processor such as a CPU. Furthermore, processor b1 can also be an assembly of multiple electronic circuits. Also, for example, processor b1 can perform the functions of multiple components of the multiple components of the decoding device 200 shown in FIG41, etc.

[0418] Memory b2 is a dedicated or general-purpose memory that stores information used by processor b1 to decode the encoded bit stream. Memory b2 can be an electronic circuit or connected to processor b1. Alternatively, memory b2 can be contained within processor b1. Furthermore, memory b2 can be an assembly of multiple electronic circuits. Also, memory b2 can be a magnetic disk or optical disk, or it can be presented as a storage device or recording medium. Furthermore, memory b2 can be non-volatile memory or volatile memory.

[0419] For example, memory b2 can store moving images or encoded bitstreams. Furthermore, memory b2 can also store the program used by processor b1 to decode the encoded bitstream.

[0420] Furthermore, for example, memory b2 can also function as a component for storing information among the plurality of components of the decoding device 200 shown in FIG41. Specifically, memory b2 can also function as block memory 210 and frame memory 214 shown in FIG41. More specifically, memory b2 can also store reconstructed blocks and reconstructed images, etc.

[0421] Alternatively, the decoding device 200 may not require the installation of all of the plurality of constituent elements shown in FIG41, etc., nor may it require the performance of all of the aforementioned plurality of processes. A portion of the plurality of constituent elements shown in FIG41, etc., may also be included in other devices, and a portion of the aforementioned plurality of processes may also be performed by other devices.

[0422] [Definitions for each term] For example, the terms can also be defined as follows.

[0423] The image is an array of multiple luminance samples in monochrome format, or an array of multiple luminance samples and two corresponding arrays of multiple chrominance samples in color format (4:2:0, 4:2:2, and 4:4:4). The image can also be a frame or a field.

[0424] A frame is composed of the top field generated by a plurality of sample columns 0, 2, 4... and the bottom field generated by a plurality of sample columns 1, 3, 5...

[0425] A slice is an integer number of code tree units contained in a single slice segment and, if any, all subsequent sub-slices preceding the next single slice segment within the same access unit.

[0426] A tile is a rectangular area of ​​multiple coding tree blocks within a specific tile row and a specific tile column in an image. Tiles can still be used for loop filtering across tile edges, but can also be rectangular areas of frames that are intentionally designed to be decoded and encoded independently.

[0427] A block is an M×N (N columns, M rows) array of a plurality of samples, or an M×N array of a plurality of conversion coefficients. A block can also be a square or rectangular region of a plurality of pixels composed of a plurality of matrices of 1 luminance and 2 chrominance.

[0428] A CTU (Coding Tree Unit) can be a coding tree block of a plurality of luminance samples from an image with three sample arrays, or it can be two corresponding coding tree blocks of a plurality of chrominance samples. Alternatively, a CTU can also be a coding tree block of a plurality of samples from either a monochrome image or an image encoded using a syntax construct, wherein the aforementioned syntax construct is used in the encoding of three separate color planes and a plurality of samples.

[0429] A superblock can consist of one or two pattern information blocks, or it can be a 64×64 pixel square block that is recursively divided into four 32×32 blocks and can be further divided.

[0430] (Implementation Form 2) The encoding device 100 in this embodiment has the same configuration as that in Embodiment 1. Furthermore, the encoding device 100 in this embodiment has additional or substitute functions compared to Embodiment 1. Similarly, the decoding device 200 in this embodiment has the same configuration as that in Embodiment 1. Furthermore, the decoding device 200 in this embodiment has additional or substitute functions compared to Embodiment 1.

[0431] The decoding apparatus 200 in this embodiment processes images using units called VPDUs and performs prediction processing as described in Examples 1 to 7 below. Furthermore, pipeline processing is used in Examples 1 to 7 of the prediction processing described below. This pipeline processing includes a plurality of stages (e.g., Stages 1 to 6), each stage being executed by hardware such as circuitry or a processor corresponding to that stage.

[0432] [VPDU] Figure 47 is a diagram illustrating the block size during predictive processing.

[0433] The largest unit constituting an image or picture is the CTU. Predictive processing is basically performed in CU units, which can be smaller than the CTU. For example, when the CTU size is 128×128 pixels, the CU size can be 4×4 pixels, 4×8 pixels, 8×4 pixels, 8×8 pixels… or 128×128 pixels.

[0434] Furthermore, another unit is called a VPDU (Virtual Pipeline Decoding Unit). When performing pipeline processing in hardware, the VPDU is a fixed unit that can be processed in one stage. Specifically, the VPDU size is usually the largest size that can be taken from the block sizes applicable to the conversion processing, for example, setting the VPDU to a size of 64×64 pixels.

[0435] Generally, when the CU being processed is smaller than the VPDU, it is envisioned that the multiple CUs contained in the VPDU be aggregated and processed in one stage of pipeline processing. On the other hand, when the CU being processed is larger than the VPDU, it is envisioned that the CU be divided into multiple VPDUs and processed in one stage of pipeline processing for each CU.

[0436] [Example of predictive processing] Figure 48 is a schematic diagram showing a first example of prediction processing performed by the decoding device 200 of this embodiment as a configuration example of pipeline processing.

[0437] The pipeline treatment PL1 shown in Figure 48 includes stages 1 to 6.

[0438] Entropy decoding processing (St1) is performed in Stage 1. That is, in Stage 1, the decoding device 200 performs entropy decoding on the input stream of the object to be decoded, thereby obtaining the information required for decoding the encoded image. Furthermore, the input stream is the aforementioned encoded bit stream, and entropy decoding processing (St1) is performed by the entropy decoding unit 202 of the decoding device 200 shown in FIG. 41. Generally, the processing in Stage 1 is not performed in VPDU units as shown in FIG. 47, but in CU units, CTU units, or larger units.

[0439] In stage 2, MV export processing St21 and memory transfer processing St22 are performed. That is, in MV export processing St21, the decoding device 200 exports the movement vector, i.e., the MV. Then, in memory transfer processing St22, the decoding device 200 uses its exported MV, for example, to transfer at least a portion of the reference image from memory such as frame memory 214. Generally, the processing in each stage after stage 2 is performed in VPDU units as shown in FIG47. For example, in each stage after stage 2, if the VPDU is greater than the CU, processing is performed on each CU until processing of all CUs within the VPDU is completed. Furthermore, if the VPDU is less than the CU, for example, one VPDU within the CU is processed, and the processing result of that one VPDU (e.g., the exported MV) is applied to the other VPDUs within the CU.

[0440] In Stage 3, DMVR processing (St3) is performed. That is, in Stage 3, the decoding device 200 searches for the periphery of the MV as described above through DMVR processing and performs correction on the MV.

[0441] In stage 4, motion compensation processing (St4) is performed. That is, in stage 4, the decoding device 200 uses the corrected motion compensation (MV) to perform motion compensation processing.

[0442] Furthermore, the processing up to stages 2 to 4 described above is performed by the inter-prediction unit 218 shown in Figure 41.

[0443] In stage 5, LIC processing (St51a), intra-prediction processing (St52), intra / inter-mixing processing (St53), mode switching processing (St57), inverse quantization and inverse conversion processing (St54), and reconstruction processing (St56) are performed.

[0444] In LIC processing St51a, the decoding device 200 applies LIC processing to the inter-prediction image obtained through motion compensation processing to correct the inter-prediction image. Furthermore, LIC processing St51a is performed by the inter-prediction unit 218 shown in FIG41.

[0445] Furthermore, in the intra-prediction process St52, which is performed simultaneously with the LIC process St51a, the decoding device 200 performs intra-prediction to generate an intra-predicted image. The intra-prediction process St52 is performed by the intra-prediction unit 216 shown in FIG41.

[0446] In the intra / inter-prediction blending process St53, the decoding device 200 uses an adder to blend or overlap the corrected inter-prediction image with the intra-prediction image, thereby generating an intra / inter-prediction blended image. For example, the average pixel values ​​of the intra-prediction image and the inter-prediction image can also be used as the pixel values ​​of the intra / inter-prediction blended image. Furthermore, the intra / inter-prediction blending process St53 is also known as CIIP (Combined inter-merge / intra-prediction). In the mode switching process St57, the decoding device 200 selectively switches between the intra-prediction image, the intra / inter-prediction blended image, and the corrected inter-prediction image, and uses the final prediction image used for decoding the encoded image.

[0447] The inter-prediction fusion process St53 performs MC processing St4 using inter-prediction, and after generating the inter-prediction image, it then blends the intra-prediction image. Therefore, it can also be considered as a type of MV-derived mode in inter-prediction. Furthermore, MV-derived modes in inter-prediction modes can be classified, for example, as shown in Figure 16 or Figure 17, but can also be included in any of the classifications. For example, when classifying MV-derived modes as either differential MV-encoded modes or non-differential MV-encoded modes as shown in Figure 17, they can be included in any classification as long as they are included in either one. The same applies to other classifications such as Figure 16.

[0448] Furthermore, the inter-internal / internal hybrid processing St53 and the mode switching processing St57 are performed by at least one of the inter-internal prediction unit 218, the inter-internal prediction unit 216, and the prediction control unit 220 shown in FIG41.

[0449] In the dequantization and inverse transformation process St54, the decoding device 200 performs dequantization and inverse transformation on the input stream to generate a residual image. The residual image is the prediction residual or difference block mentioned above. In the reconstruction process St56, the decoding device 200 adds the residual image and the prediction image switched above to generate a reconstructed image. The generated reconstructed image is used as an adjacent reference pixel in the processing of the next block or CU, and is therefore fed back to the LIC process St51a and the intraprediction process St52.

[0450] Furthermore, the dequantization and inversion conversion process St54 is performed by the dequantization unit 204 and the inversion conversion unit 206 shown in FIG41, and the reconstruction process St56 is performed by the addition unit 208 shown in FIG41.

[0451] In Stage 6, a loop filtering process (St6) is performed. That is, in Stage 6, the decoding device 200 applies loop filtering, such as deblocking filtering, to the reconstructed image to generate a decoded image. Furthermore, this loop filtering process (St6) is performed by the loop filtering unit 212 shown in FIG41.

[0452] Additionally, the pipeline processing PL1 shown in Figure 48 is just an example. At least one processing contained in the pipeline processing PL1 can be removed, or other processing can be added to the pipeline processing PL1 to change the segmentation method of the stage.

[0453] As illustrated in Figure 47, when the processing object CU is smaller than the VPDU, it is envisioned that the multiple CUs contained in the VPDU are aggregated and processed in one stage of the pipeline. For example, when the VPDU is 64×64 pixels in size and each of the CUs contained in the VPDU is 8×8 pixels in size, 64 CUs are processed in one stage of pipeline processing PL1 shown in Figure 48.

[0454] Here, in the mode switching process St57 of stage 5, in the worst case, there is a situation where the inner / outer prediction image is used in the decoding of all 64 CUs contained in the VPDU. In the inner / outer prediction image generation process St53, the decoding device 200 mixes the inner prediction image with the corrected inter-prediction image. Therefore, the decoding device 200 needs to confirm that both the inner prediction process St52 and the inter-prediction process for each CU are completed before mixing the inner prediction image with the corrected inter-prediction image. In addition, the above-mentioned inter-prediction process includes MV export process St21, memory transfer process St22, DMVR process St3, MC process St4, and LIC process St51a.

[0455] In other words, in the pipeline processing PL1 shown in Figure 48, the decoding device 200 pairs per CU must wait for a relatively long time during the completion of the inner prediction processing St52 and the inter-prediction processing. As a result, in the worst case, there is a possibility of up to 64 wait times in one stage of processing.

[0456] This waiting time significantly increases the processing time of stage 5, resulting in situations where decoding of a single image cannot be completed within the allotted processing time. In other words, there is a possibility that the decoded image cannot be displayed within the predetermined time.

[0457] Therefore, the decoding device 200 of this embodiment can also perform the second to seventh examples of prediction processing as shown below.

[0458] [Example 2 of predictive processing] Figure 49 is a flowchart showing a second example of the prediction processing performed by the decoding apparatus 200 of this embodiment. Furthermore, this flowchart shows the processing flow for the CU in the second example of prediction processing.

[0459] The decoding device 200 performs steps S11 to S17a on a unit-by-unit basis. Specifically, first, the decoding device 200 derives the motion prediction image (MV) from the CU being processed (step S11), and then performs DMVR processing (step S12). In this DMVR processing, the decoding device 200 searches for the periphery of the MV derived in step S11, thereby correcting the MV. Then, the decoding device 200 uses the corrected MV to perform MC processing, i.e., motion compensation processing (step S13). This yields the inter-predictive image.

[0460] Next, the decoding device 200 determines whether LIC processing is applicable to the processing object CU (step S14a). Here, when the decoding device 200 determines that LIC processing is applicable (yes in step S14a), it performs LIC processing correction on the inter-prediction image obtained by MC processing, thereby generating the final prediction image (step S16a). Furthermore, the final prediction image refers to the prediction image used to generate the reconstructed image, and is the prediction image added to the residual image.

[0461] On the other hand, when the decoding device 200 determines that LIC processing is not applicable (No in step S14a), it further determines whether inter-prediction / inter-prediction mixing processing is applicable (step S15a). Here, when the decoding device 200 determines that inter-prediction / inter-prediction mixing processing is applicable (Yes in step S15a), it mixes the inter-prediction image obtained by MC processing with the intra-prediction image obtained by intra-prediction processing (step S17a). By this mixing, an inter-prediction / inter-prediction image is generated as the final prediction image. On the other hand, when the decoding device 200 determines that inter-prediction / inter-prediction mixing processing is not applicable (No in step S15a), it uses the inter-prediction image obtained by MC processing unchanged as the final prediction image in the generation of the reconstructed image.

[0462] Additionally, in the flowchart of Figure 49, the decoding device 200 first determines whether LIC processing is applicable, and then determines whether internal / internal mixing processing is applicable, but these determinations can also be performed in the reverse order.

[0463] Additionally, the flowchart in Figure 49 is just an example. At least one step in the flowchart can be removed, or other processing or condition determination steps can be added to the flowchart.

[0464] Alternatively, the processing shown in the flowchart of Figure 49 can also be performed by the encoding device 100. That is, the encoding device 100 and the decoding device 200 perform the same prediction processing, the difference being that the prediction processing is used to encode the image into a stream, or to decode the stream into an image. Therefore, the prediction processing flow shown in the flowchart of Figure 49 is basically the same in the encoding device 100.

[0465] Furthermore, the determination of whether LIC processing is applicable (step S14a) and whether inter-process / inter-process mixing is applicable (step S15a) in the flowchart of Figure 49 can also be performed by parsing flags. That is, the decoding device 200 can also parse the flags recorded in the stream to perform such determinations. Also, the encoding device 100 can also calculate the cost of using the RD optimization model to perform such determinations. That is, in each of the above determinations, the encoding device 100 calculates the cost of the predicted image obtained by processing the object to be determined and the cost of the predicted image obtained by not applying the processing. Then, the encoding device 100 performs the above determinations to use the predicted image corresponding to the smaller cost among these costs as the final predicted image.

[0466] Figure 50 is a schematic diagram showing a second example of prediction processing performed by the decoding device 200 in this embodiment as a configuration example of pipeline processing.

[0467] The pipeline treatment PL2 shown in Figure 50 includes stages 1 to 6. Furthermore, the treatment of each stage from stage 2 down is performed, for example, in VPDU units.

[0468] Phase 5 of phases 1 to 6 of pipeline processing PL2 differs from phase 5 of pipeline processing PL1 shown in FIG48. That is, in phase 5 of pipeline processing PL2, inter-phase / inter-phase mixing processing St53 is performed when LIC processing St51a is not applied. In other words, the decoding device 200 does not perform LIC processing St51a on the inter-prediction image generated by MC processing St4, but instead generates an inter-phase / inter-phase mixed prediction image by mixing the inter-prediction image and the inter-prediction image.

[0469] In this pipeline processing PL2, at the time point of performing the inter-prediction image generation for ...

[0470] Additionally, the pipeline processing PL2 shown in Figure 50 is just an example. At least one processing contained in the pipeline processing PL2 can be removed, or other processing can be added to the pipeline processing PL2 to change the segmentation method of the stage.

[0471] Thus, in the second example of predictive processing, by using the processing shown in Figures 49 and 50, even when encoding a VPDU containing multiple CUs, all stages of processing can still be completed within a pre-specified processing time. As a result, the possibility of achieving high-speed processing while suppressing the degradation of coding performance can be improved.

[0472] [Changes in the second case of predictive processing] Furthermore, in the second example of prediction processing, when the condition of selecting the applicability of LIC processing is met, inter-process mixing is always prohibited. However, the prohibition of inter-process mixing is not limited to this condition. For example, inter-process mixing can also be prohibited when the condition of selecting the applicability of LIC processing and the size of the processing object CU is less than a certain threshold is met. In this way, although the processing time of stage 5 in generating the reconstructed image increases, this increase in time can still be limited to a certain extent, which can improve the possibility of suppressing the degradation of coding performance while suppressing the increase in processing time.

[0473] Additionally, LIC processing can be disabled when the condition of selecting the appropriate internal / internal mixed processing and the size of the processing object CU is less than a specific threshold is met. Furthermore, the condition "the size of the processing object CU is less than a specific threshold" can be replaced with "the number of CUs contained in one VPDU is greater than a specific threshold".

[0474] Furthermore, in the second example of prediction processing, although in the combination of LIC processing and inter-prediction / inter-prediction hybrid processing, one processing is applied and the other is prohibited, this is not limited to this combination. The same application and prohibition can be performed in other combinations as described above. For example, in stage 5 of generating the reconstructed image, any other combination can be any combination of the prediction-side processing that requires feedback to reconstruct the image and the processing that requires waiting time. The processing that requires waiting time is the processing of the inter-prediction side that requires the later completion time between the inter-prediction side processing and the intra-prediction side processing. Additionally, the inter-prediction side processing that requires feedback to reconstruct the image can also be, for example, filtering processing performed on the predicted image obtained through inter-prediction.

[0475] [Example 3 of predictive processing] Figure 51 is a flowchart showing a third example of prediction processing performed by the decoding apparatus 200 in this embodiment. Furthermore, the flowchart shows the processing flow of the CU in the third example of prediction processing.

[0476] The decoding device 200 performs steps S11 to S13, S14b, S15a, S16b, and S17a on a CU-by-CU basis. Specifically, firstly, the decoding device 200 derives the MV (Motion Video Image) of the CU being processed (step S11), and then performs DMVR (Motion Detection and Reduction) processing (step S12). In this DMVR processing, the decoding device 200 searches for the periphery of the MV derived in step S11, thereby correcting the MV. Then, the decoding device 200 uses the corrected MV to perform MC (Motion Compensation) processing, i.e., motion compensation processing (step S13). This yields the inter-prediction image of the CU being processed.

[0477] Next, the decoding device 200 determines whether to apply filtering processing (also known as prediction map filtering processing) to the inter-prediction image of the processing object CU (step S14b). Here, filtering processing means multiplying each of a plurality of pixels by a weight to correct or update the pixel value. In addition, prediction map filtering processing specifically means correcting or updating the pixel values ​​contained in the prediction image.

[0478] When the decoding device 200 determines that filtering is applicable (in step S14b), it performs filtering on the inter-prediction image obtained by MC processing to generate a filtered inter-prediction image as the final prediction image (in step S16b). For example, a matrix of 5×5 filtering coefficients can be used in the filtering process. Furthermore, pixels contained in the reconstructed images of CUs adjacent to the processing target CU (hereinafter referred to as adjacent CUs) can also be used in the filtering process. For example, pixels in the inter-prediction image of the processing target CU that are close to the boundary between the processing target CU and the adjacent CU are filtered using pixels contained in the reconstructed image of the adjacent CU.

[0479] On the other hand, when the decoding device 200 determines that filtering processing is not applicable (No in step S14b), it further determines whether inter-prediction / inter-prediction blending processing is applicable (step S15a). Here, when the decoding device 200 determines that inter-prediction / inter-prediction blending processing is applicable (Yes in step S15a), it blends the inter-prediction image obtained by MC processing with the intra-prediction image obtained by intra-prediction processing (step S17a). By this blending, an inter-prediction / inter-prediction image is generated as the final prediction image. On the other hand, when the decoding device 200 determines that inter-prediction / inter-prediction blending processing is not applicable (No in step S15a), it uses the inter-prediction image obtained by MC processing unchanged as the final prediction image in the generation of the reconstructed image.

[0480] Additionally, in the flowchart of Figure 51, the decoding device 200 first determines whether filtering is applicable to the inter-prediction image, and then determines whether intra / inter-mixing is applicable, but these determinations can also be performed in the reverse order.

[0481] Additionally, the flowchart in Figure 51 is just an example. At least one step in the flowchart can be removed, or other processing or condition determination steps can be added to the flowchart.

[0482] Alternatively, the processing shown in the flowchart of Figure 51 can also be performed by the encoding device 100. That is, the encoding device 100 and the decoding device 200 perform the same prediction processing, the difference being that the prediction processing is used to encode the image into a stream, or to decode the stream into an image. Therefore, the prediction processing flow shown in the flowchart of Figure 51 is basically the same in the encoding device 100.

[0483] Furthermore, the determination of whether filtering processing is applicable (step S14b) and whether internal / inter-interval mixing processing is applicable (step S15a) in the flowchart of Figure 51 can also be performed by parsing flags. That is, the decoding device 200 can, for example, parse the flags recorded in the stream to perform such determinations. Also, the encoding device 100 can, for example, calculate the cost of using the RD optimization model to perform such determinations. That is, in each of the above determinations, the encoding device 100 calculates the cost of the predicted image obtained by processing the object to be determined, and the cost of the predicted image obtained by not applying the processing. Then, the encoding device 100 performs the above determinations to use the predicted image corresponding to the smaller cost among these costs as the final predicted image.

[0484] Figure 52 is a schematic diagram showing a third example of prediction processing performed by the decoding device 200 in this embodiment as a configuration example of pipeline processing.

[0485] The pipeline treatment PL3 ​​shown in Figure 52 includes stages 1 to 6. Furthermore, the treatment of each stage from stage 2 down is performed, for example, in VPDU units.

[0486] Phase 5 of phases 1 to 6 of the pipeline processing PL3 differs from phase 5 of the pipeline processing PL2 shown in Figure 50. That is, in phase 5 of the pipeline processing PL3, a prediction graph filtering process St51b is performed instead of the LIC process St51a. This prediction graph filtering process St51b is the filtering process described above.

[0487] In other words, in stage 5 of pipeline processing PL2, when prediction map filtering St51b is not applied, inter-prediction mixing St53 is performed. That is, the decoding device 200 does not apply prediction map filtering St51b to the inter-prediction image generated by MC processing St4, but instead generates an inter-prediction image by mixing the inter-prediction image and the inter-prediction image. On the other hand, when the decoding device 200 applies prediction map filtering St51b to the inter-prediction image generated by MC processing St4, it uses the inter-prediction image that has already undergone prediction map filtering St51b as the final prediction image for the generation of the reconstructed image.

[0488] In this pipeline processing PL3, at the time point of performing the inter-prediction image generation for ...

[0489] Additionally, the pipeline processing PL3 shown in Figure 52 is just an example. At least one of the processes included in the pipeline processing PL3 can be removed, or other processes can be added to the pipeline processing PL3 to change the segmentation method of the stage.

[0490] Additionally, the prediction image filtering process St51b is configured to feed back the reconstructed image generated by adding the residual image of the processed block to the prediction image, and also add the aforementioned reconstructed image into the input for processing. However, it is not limited to this; it can also be configured to not feed back or input the aforementioned reconstructed image, but only perform filtering processing on the prediction image within the processing target block.

[0491] In other words, in stage 5 of pipeline processing PL3 in Figure 52, the reconstructed image generated by adding the residual image of the processed CU to the predicted image is fed back to the prediction image filtering process St51b. Next, in the prediction image filtering process St51b, processing is performed using the reconstructed image and the inter-predicted image generated by the MC process St4. That is, the pixels of the inter-predicted image are filtered using the pixels of the reconstructed image. However, this third example of prediction processing is not limited to such feedback. That is, it is also possible not to feed back the reconstructed image to the prediction image filtering process St51b. In this case, in the prediction image filtering process St51b, the reconstructed image is not used, and instead, the inter-predicted image generated by the MC process St4 is processed.

[0492] Without such feedback, stage 5 can be divided into a stage performing the prediction graph filtering process St51b, and a stage containing several processes such as the reconstruction process St56, which includes the prediction graph filtering process St51b. However, dividing stage 5 into two stages may increase the required memory and circuitry due to the addition of stages. However, in pipeline processing PL3, since the prediction graph filtering process St51b is performed in the same stage as the reconstruction process St56 and other processes, the possibility of avoiding the increase in memory and circuitry is improved.

[0493] In the third example of prediction processing, when encoding a VPDU containing multiple CUs using the processing shown in Figures 51 and 52, all stages of processing can still be completed within the pre-specified processing time. As a result, the possibility of achieving high-speed processing while suppressing the degradation of encoding performance can be increased. Furthermore, as mentioned above, since the prediction graph filtering process St51b is installed at the same stage as the reconstruction process St56, the possibility of suppressing the increase in the circuit size of the encoding device 100 and the decoding device 200 can be increased.

[0494] [Case 4 of Predictive Processing] Figure 53 is a flowchart showing a fourth example of prediction processing performed by the decoding apparatus 200 in this embodiment. Furthermore, this flowchart shows the processing flow of the CU in the fourth example of prediction processing.

[0495] The decoding device 200 performs steps S11 to S13, S14c, S15a, S16c, and S17a on a CU-by-CU basis. Specifically, first, the decoding device 200 derives the MV (Motion Video Image) of the CU being processed (step S11), and then performs DMVR (Motion-Driven Video Recognition) processing (step S12). In this DMVR processing, the decoding device 200 searches for the periphery of the MV derived in step S11, thereby correcting the MV. Then, the decoding device 200 uses the corrected MV to perform MC (Motion Compensation) processing, i.e., motion compensation processing (step S13). This yields the inter-prediction image of the CU being processed.

[0496] Next, the decoding device 200 determines whether BIO processing is applicable to the inter-prediction image of the processing object CU (step S14c). Here, if it is determined that BIO processing is applicable (yes in step S14c), the decoding device 200 performs BIO processing on the inter-prediction image obtained by MC processing, thereby generating the final prediction image (step S16c). Furthermore, the BIO processing is the aforementioned BIO mode, also known as BDOF (bi-directional optical flow) processing. This BIO processing corrects the inter-prediction image based on a model assuming constant linear motion, derived or corrected by the motion vector. The inter-prediction image corrected by BIO processing is then used as the final prediction image in the generation of the reconstructed image.

[0497] On the other hand, when the decoding device 200 determines that BIO processing is not applicable (No in step S14c), it further determines whether intra / interval blending processing is applicable (step S15a). Here, when the decoding device 200 determines that intra / interval blending processing is applicable (Yes in step S15a), it blends the inter-prediction image obtained by MC processing with the intra-prediction image obtained by intra-prediction processing (step S17a). By this blending, an intra / interval blended prediction image is generated as the final prediction image. On the other hand, when the decoding device 200 determines that intra / interval blending processing is not applicable (No in step S15a), it uses the inter-prediction image obtained by MC processing unchanged as the final prediction image in the generation of the reconstructed image.

[0498] Additionally, in the flowchart of Figure 53, the decoding device 200 first determines whether BIO processing is applicable to the inter-prediction image, and then determines whether intra / inter-mixing processing is applicable, but these determinations can also be performed in the reverse order.

[0499] Additionally, the flowchart in Figure 53 is just an example. At least one step in the flowchart can be removed, or other processing or condition determination steps can be added to the flowchart.

[0500] Alternatively, the processing shown in the flowchart of Figure 53 can also be performed by the encoding device 100. That is, the encoding device 100 and the decoding device 200 perform the same prediction processing, the difference being that the prediction processing is used to encode the image into a stream, or to decode the stream into an image. Therefore, the prediction processing flow shown in the flowchart of Figure 53 is basically the same in the encoding device 100.

[0501] Furthermore, the determination of whether BIO processing is applicable (step S14c) and whether internal / internal hybrid processing is applicable (step S15a) in the flowchart of Figure 53 can also be performed by parsing flags. That is, the decoding device 200 can, for example, parse the flags recorded in the stream to perform such determinations. Also, the encoding device 100 can, for example, calculate the cost of using the RD optimization model to perform such determinations. That is, in each of the above determinations, the encoding device 100 calculates the cost of the predicted image obtained by processing the object to be determined, and the cost of the predicted image obtained by not applying the processing. Then, the encoding device 100 performs the above determinations to use the predicted image corresponding to the smaller cost among these costs as the final predicted image.

[0502] Figure 54 is a schematic diagram showing a fourth example of prediction processing performed by the decoding device 200 in this embodiment as a configuration example of pipeline processing.

[0503] The pipeline treatment PL4 shown in Figure 54 includes stages 1 to 6. Furthermore, the treatment of each stage from stage 2 down is performed, for example, in VPDU units.

[0504] Phase 5 of phases 1 to 6 of pipeline processing PL4 differs from phase 5 of pipeline processing PL2 shown in Figure 50. Specifically, in phase 5 of pipeline processing PL4, BIO processing St51c replaces LIC processing St51a. Furthermore, in this BIO processing St51c, the reconstructed image generated by adding the residual image to the predicted image is not fed back.

[0505] In stage 5 of such pipeline processing PL4, when BIO processing St51c is not applied, inter-prediction mixing processing St53 is performed. In other words, the decoding device 200 does not perform BIO processing St51c on the inter-prediction image generated by MC processing St4, but instead generates an inter-prediction image by mixing the inter-prediction image and the inter-prediction image. On the other hand, when the decoding device 200 applies BIO processing St51c to the inter-prediction image generated by MC processing St4, it does not perform inter-prediction mixing processing St53. Therefore, the decoding device 200 uses the inter-prediction image that has already undergone BIO processing St51c as the final prediction image in the generation of the reconstructed image.

[0506] In other words, in the fourth example of prediction processing, one of the BIO processing St51c and the inter-process / inter-process hybrid processing St53 is exclusively selected, and only the selected processing is applied to the inter-prediction image generated by the MC processing St4. By applying the selected processing, the final prediction image of the processing object CU is generated and used in the generation of the reconstructed image.

[0507] In this pipeline processing PL4, at the time point of performing the inter-prediction image required for the inter-prediction image processing St53, the generation of the inter-prediction image is already completed in stage 4. Therefore, the waiting time required for the completion of the inter-prediction image processing in pipeline processing PL1 shown in FIG48 is not required in pipeline processing PL4. Thus, the inter-prediction image processing St53 can be performed immediately at the time point of completion of the inter-prediction image processing St52. As a result, the generation of unnecessary waiting time can be suppressed. That is, the processing time of stage 5 when performing the inter-prediction image processing St53 can be made almost equal to the processing time of stage 5 when the inter-prediction image processing St53 is not performed.

[0508] Additionally, the pipeline processing PL4 shown in Figure 54 is just an example. At least one processing contained in the pipeline processing PL4 can be removed, or other processing can be added to the pipeline processing PL4 to change the segmentation method of the stage.

[0509] Furthermore, in stage 5 of pipeline processing PL4 in Figure 54, the reconstructed image generated by adding the residual image of the processed CU to the predicted image is not fed back to BIO processing St51c. Therefore, stage 5 can be divided into a stage that performs BIO processing St51c and a stage that includes several processes such as reconstruction processing St56, in addition to BIO processing St51c. However, if stage 5 is divided into two stages, the added stages may increase the required memory and circuitry. However, in pipeline processing PL4, since BIO processing St51c and several other processes such as reconstruction processing St56 are performed in the same stage, the possibility of avoiding the increase in memory and circuitry is improved.

[0510] In the fourth example of predictive processing, when encoding a VPDU containing multiple CUs is performed using the processing shown in Figures 53 and 54, all stages of processing can still be completed within the pre-specified processing time. As a result, the possibility of achieving high-speed processing while suppressing the degradation of encoding performance can be increased. Furthermore, as mentioned above, since the BIO processing St51c is installed at the same stage as the reconstruction processing St56, the possibility of suppressing the increase in the circuit size of the encoding device 100 and the decoding device 200 can be increased.

[0511] As described above, in the inter-prediction mode, the decoding apparatus 200 in this embodiment generates a first predicted image of the target block based on the derived motion vector. Then, the decoding apparatus 200 applies an update process to the first predicted image to generate a final predicted image of the target block. Here, the update process candidates include a first process and a second process. The first process is a BDOF (bi-directional optical flow) process, and the second process is a process that mixes the second predicted image generated by the intra-prediction of the target block with the aforementioned first predicted image. Furthermore, in the application of the update process, the first and second processes are applied exclusively. For example, in the generation of the final predicted image, when the update process is the first process, the decoding apparatus 200 does not apply the second process to the first predicted image, but instead applies the first process to generate the final predicted image. Similarly, when the update process is the second process, the decoding apparatus 200 does not apply the first process to the first predicted image, but instead applies the second process to generate the final predicted image. Additionally, the first predicted image is an inter-predicted image, the second predicted image is an intra-predicted image, and the processing object block is, for example, the processing object CU.

[0512] Therefore, regardless of whether the update process is the first process or the second process, no processing other than the update process in the first and second processes will be applied to the first predicted image. In other words, the application of processing other than the update process is prohibited. Thus, for example, when a pipeline process for decoding an encoded image includes both the first and second processes in one stage, performing the first and second processes together increases the likelihood of suppressing the time spent on that stage. That is, it increases the likelihood of suppressing an increase in processing time.

[0513] Furthermore, during the application of the aforementioned update process, the decoding device 200 determines whether the second process should be applied to the first predicted image. If it determines that the second process should be applied, it performs the second process as the update process. On the other hand, if it determines that the second process should not be applied, the decoding device 200 determines whether the first process should be applied to the first predicted image. If it determines that the first process should be applied, it performs the first process as the update process.

[0514] This increases the likelihood of having the option to perform appropriate update processing.

[0515] Furthermore, in a pipeline processing stage for decoding the encoded image, and in the same stage as the reconstruction process of adding the generated final predicted image to the residual image to generate the reconstructed image, the decoding device 200 exclusively performs the first and second processes.

[0516] In this way, for example, compared to performing the first and second processes at a different stage than the reconstruction process, the number of stages included in the pipeline processing can be reduced. As a result, the possibility of suppressing the increase in the circuit size of the decoding device 200 can be improved.

[0517] Similar to the decoding apparatus 200, the encoding apparatus 100 in this embodiment generates a first predicted image of the target block in inter-prediction mode based on the derived movement vector. Then, the encoding apparatus 100 applies an update process to the first predicted image to generate a final predicted image of the target block. Here, the update process candidates include a first process and a second process. The first process is a BDOF (bi-directional optical flow) process, and the second process is a process that mixes the second predicted image generated by the intra-prediction of the target block with the aforementioned first predicted image. Then, in the application of the update process, the first and second processes are applied exclusively. For example, in the generation of the final predicted image, when the update process is the first process, the encoding apparatus 100 does not apply the second process to the first predicted image, but instead applies the first process to generate the final predicted image. Similarly, when the update process is the second process, the encoding apparatus 100 does not apply the first process to the first predicted image, but instead applies the second process to generate the final predicted image. Additionally, the first predicted image is an inter-predicted image, the second predicted image is an intra-predicted image, and the processing object block is, for example, the processing object CU.

[0518] Therefore, regardless of whether the update process is the first process or the second process, no processing other than the update process will be applied to the first predicted image. In other words, the application of processing other than the update process is prohibited. Thus, for example, when a pipeline process used to encode an image includes both the first and second processes in one stage, performing the first and second processes together increases the likelihood of suppressing the time spent on that stage. That is, it increases the likelihood of suppressing an increase in processing time.

[0519] Furthermore, during the application of the aforementioned update process, the encoding device 100 determines whether the second process should be applied to the first predicted image. If it determines that the second process should be applied, the second process is used as the update process. On the other hand, if it determines that the second process should not be applied, the encoding device 100 determines whether the first process should be applied to the first predicted image. If it determines that the first process should be applied, the first process is used as the update process.

[0520] This increases the likelihood of having the option to perform appropriate update processing.

[0521] Furthermore, in a stage of the pipeline processing for encoding the image, which is the same stage as the reconstruction processing of adding the generated final predicted image to the residual image to generate the reconstructed image, the encoding device 100 exclusively performs the first processing and the second processing.

[0522] In this way, for example, compared to performing the first and second processes at a different stage than the reconstruction process, the number of stages included in the pipeline processing can be reduced. As a result, the possibility of suppressing the increase in the circuit size of the encoding device 100 can be improved.

[0523] [Case 5 of Predictive Processing] Figure 55 is a flowchart showing a fifth example of prediction processing performed by the decoding apparatus 200 in this embodiment. Furthermore, this flowchart shows the processing flow of the CU in the fifth example of prediction processing.

[0524] The decoding device 200 performs steps S11 to S13, S14c, S15b, S16c, and S17b on a CU-by-CU basis. Specifically, first, the decoding device 200 derives the MV (Motion Video Image) of the CU being processed (step S11), and then performs DMVR (Motion Detection and Reduction) processing (step S12). In this DMVR processing, the decoding device 200 searches for the periphery of the MV derived in step S11, thereby correcting the MV. Then, the decoding device 200 uses the corrected MV to perform MC (Motion Compensation) processing, i.e., motion compensation processing (step S13). This yields the inter-prediction image of the CU being processed.

[0525] Next, the decoding device 200 determines whether BIO processing is applicable to the inter-prediction image of the processing object CU (step S14c). Here, when it is determined that BIO processing is applicable (yes in step S14c), the decoding device 200 performs BIO processing on the inter-prediction image obtained by MC processing, thereby generating the final prediction image (step S16c). The inter-prediction image corrected by BIO processing is used as the final prediction image in the generation of the reconstructed image.

[0526] On the other hand, when the decoding device 200 determines that BIO processing is not applicable (No in step S14c), it further determines whether filtering processing (also known as prediction image filtering processing) is applicable (step S15b). Here, when the decoding device 200 determines that filtering processing is applicable (Yes in step S15b), it performs filtering processing on the inter-prediction image obtained by MC processing (step S17b). Through this filtering processing, a final prediction image used in the generation of the reconstructed image is generated. On the other hand, when the decoding device 200 determines that filtering processing is not applicable (No in step S15b), it uses the inter-prediction image obtained by MC processing unchanged as the final prediction image in the generation of the reconstructed image.

[0527] Here, the filtering process can be the same as that in the third example of the prediction process described above. As mentioned above, it can also be a process of multiplying each of the plurality of pixels by a weight to correct or update the pixel value. Furthermore, prediction map filtering specifically refers to the process of correcting or updating the pixel values ​​contained in the prediction image. Or / and, this prediction map filtering process can also, for example, be a process of correcting or updating the pixels of the inter-prediction image obtained by MC processing.

[0528] Additionally, in the flowchart of Figure 55, the decoding device 200 first determines whether BIO processing is applicable to the inter-prediction image, and then determines whether filtering processing is applicable, but these determinations can also be performed in the reverse order.

[0529] Additionally, the flowchart in Figure 55 is just an example. At least one step in the flowchart can be removed, or other processing or condition determination steps can be added to the flowchart.

[0530] Alternatively, the processing shown in the flowchart of Figure 55 can also be performed by the encoding device 100. That is, the encoding device 100 and the decoding device 200 perform the same prediction processing, the difference being that the prediction processing is used to encode the image into a stream, or to decode the stream into an image. Therefore, the prediction processing flow shown in the flowchart of Figure 55 is basically the same in the encoding device 100.

[0531] Furthermore, the determination of whether BIO processing is applicable (step S14c) and whether filtering processing is applicable (step S15b) in the flowchart of Figure 55 can also be performed by parsing flags. That is, the decoding device 200 can, for example, parse the flags recorded in the stream to perform such determinations. Also, the encoding device 100 can, for example, calculate the cost of using the RD optimization model to perform such determinations. That is, in each of the above determinations, the encoding device 100 calculates the cost of the predicted image obtained by processing the object to be determined, and the cost of the predicted image obtained by not applying the processing. Then, the encoding device 100 performs the above determinations to use the predicted image corresponding to the smaller cost among these costs as the final predicted image.

[0532] Figure 56 is a schematic diagram showing the fifth example of prediction processing performed by the decoding device 200 in this embodiment as a configuration example of pipeline processing.

[0533] The pipeline treatment PL5 shown in Figure 56 includes stages 1 to 6. Furthermore, the treatment of each stage from stage 2 down is performed, for example, in VPDU units.

[0534] Phase 5 of phases 1 to 6 of pipeline processing PL5 differs from phase 5 of pipeline processing PL2 shown in Figure 50. That is, in phase 5 of pipeline processing PL5, only one of the BIO processing St51c or the prediction map filtering processing St51b is applied to the inter-prediction image generated by MC processing St4 to produce the final prediction image. Furthermore, the prediction map filtering processing St51b is the filtering process described above.

[0535] In other words, in stage 5 of pipeline processing PL5, BIO processing St51c is performed when prediction graph filtering processing St51b is not applied. That is, the decoding device 200 does not perform prediction graph filtering processing St51b on the inter-prediction image generated by MC processing St4, but instead generates the final prediction image by applying BIO processing St51c. Alternatively, the decoding device 200 does not perform BIO processing St51c on the inter-prediction image generated by MC processing St4, but instead generates the final prediction image by applying prediction graph filtering processing St51b.

[0536] In other words, in the fifth example of prediction processing, one of the BIO processing St51c and the prediction image filtering processing St51b is exclusively selected, and only the selected processing is applied to the inter-prediction image generated by the MC processing St4. By applying the selected processing, the final prediction image of the processing object CU is generated and used in the generation of the reconstructed image.

[0537] When both BIO processing St51c and prediction graph filtering processing St51b are applied simultaneously, the processing time or number of processing cycles in stage 5 increases due to the continuous execution of these two processes. Therefore, in such cases, there is a possibility that stage 5 may not be completed within the specified processing time. However, in the pipeline processing PL5 shown in Figure 56, only either BIO processing St51c or prediction graph filtering processing St51b is performed. Therefore, the increase in the processing time or number of processing cycles in stage 5 can be suppressed, and stage 5 can be completed within the specified processing time.

[0538] Additionally, the pipeline processing PL5 shown in Figure 56 is just an example. At least one processing included in the pipeline processing PL5 can be removed, or other processing can be added to the pipeline processing PL5 to change the segmentation method of the stage.

[0539] Additionally, the prediction image filtering process St51b is configured to feed back the reconstructed image generated by adding the residual image of the processed block to the prediction image, and also add the aforementioned reconstructed image to the input for processing. However, it is not limited to this; it can also be configured to not feed back or input the aforementioned reconstructed image, but only perform filtering processing on the prediction image within the processing target block.

[0540] In other words, in stage 5 of pipeline processing PL5 in Figure 56, the reconstructed image generated by adding the residual image of the processed CU and the predicted image is fed back to the prediction image filtering process St51b. Next, in the prediction image filtering process St51b, processing is performed using the reconstructed image and the inter-prediction image generated by the MC process St4. That is, the pixels of the inter-prediction image are filtered using the pixels of the reconstructed image. However, this fifth example of prediction processing is not limited to such feedback. That is, the feedback of the reconstructed image to the prediction image filtering process St51b may not be performed. In this case, in the prediction image filtering process St51b, the reconstructed image is not used, and instead, the inter-prediction image generated by the MC process St4 is processed.

[0541] Similarly, no image reconstruction feedback is performed in BIO processing St51c. Therefore, stage 5 can be divided into a stage containing BIO processing St51c, and a stage containing several processes such as prediction image filtering processing St51b and reconstruction processing St56, in addition to BIO processing St51c. However, dividing stage 5 into two stages may increase the required memory and circuitry due to the addition of stages. However, in pipeline processing PL5, since prediction image filtering processing St51b and BIO processing St51c are performed in the same stage as reconstruction processing St56 and other processes, the possibility of avoiding increased memory and circuitry is improved.

[0542] In the fifth example of prediction processing, when encoding a VPDU containing multiple CUs using the processing shown in Figures 55 and 56, all stages of processing can still be completed within the pre-specified processing time. As a result, the possibility of achieving high-speed processing while suppressing the degradation of encoding performance can be increased. Furthermore, as mentioned above, since the prediction graph filtering process St51b and the BIO process St51c are installed in the same stages as the reconstruction process St56, the possibility of suppressing the increase in the circuit size of the encoding device 100 and the decoding device 200 can be increased.

[0543] As described above, in the inter-prediction mode, the decoding apparatus 200 in this embodiment generates a predicted image of the processing target block based on the derived motion vector, and applies an update process to the predicted image to generate a final predicted image of the processing target block. Here, the candidates for the update process include a first process and a second process. The first process is a BDOF (bi-directional optical flow) process, and the second process is a filtering process that updates the pixel values ​​of the predicted image. Then, in the application of the update process, the first process and the second process are applied exclusively. For example, in the generation of the final predicted image, when the update process is the first process, the decoding apparatus 200 does not apply the second process to the predicted image, but instead applies the first process to generate the final predicted image. Similarly, when the update process is the second process, the decoding apparatus 200 does not apply the first process to the predicted image, but instead applies the second process to generate the final predicted image. Furthermore, the processing target block is, for example, the processing target CU. Also, the predicted image generated in the inter-prediction mode is an inter-prediction image.

[0544] Therefore, regardless of whether the update process is the first or the second process, no processing other than the update process will be applied to the predicted image. In other words, the application of processing other than the update process is prohibited. Thus, for example, when a pipeline process for decoding an encoded image includes both the first and second processes in one stage, performing the first and second processes together increases the likelihood of suppressing the time spent on that stage. That is, it increases the likelihood of suppressing an increase in processing time.

[0545] Furthermore, during the application of the update process, the decoding device 200 determines whether the second process should be applied to the predicted image. If it is determined that the second process should be applied, the second process is used as the update process. On the other hand, if it is determined that the second process should not be applied, the decoding device 200 determines whether the first process should be applied to the predicted image. If it is determined that the first process should be applied, the first process is used as the update process.

[0546] This increases the likelihood of being able to perform appropriate updates.

[0547] Furthermore, the second process updates the pixel values ​​of the predicted image based on the predicted image. For example, in the second process, the reconstructed image is not used for processing. Alternatively, the second process updates the pixel values ​​of the predicted image based on (1) the predicted image and (2) the reconstructed image of a block that is different from the processing target block.

[0548] This increases the likelihood of appropriate pixel value updates performed through the second process.

[0549] Furthermore, in a pipeline processing stage used to decode the encoded image, and in the same stage as the reconstruction process of adding the generated final predicted image to the residual image to generate the reconstructed image, the decoding device 200 exclusively performs the first and second processes.

[0550] This allows, for example, a reduction in the number of stages involved in pipeline processing compared to performing the first and second processes at a different stage than the reconstruction process. As a result, it increases the likelihood of suppressing an increase in the circuit size of the decoding device 200.

[0551] Here, the update processing candidates may further include a third processing, different from the first and second processing. The third processing is a process of mixing the second predicted image generated by the intra-prediction of the processing object block and the first predicted image generated by the inter-prediction mode. In this case, the first, second, and third processings are applied exclusively in the application of the update processing. For example, the third processing is the intra / inter-mixing processing described above.

[0552] This increases the variety of applicable update processes, such as generating predicted images with high accuracy.

[0553] Furthermore, in the application of the update process, (1) when the inter-prediction mode is an inter-prediction mode in which the third process can be selected, the decoding device 200 can also determine whether the third process should be applied to the predicted image. Then, when the decoding device 200 determines that the third process should be applied, it applies the third process as the update process; when it determines that the third process should not be applied, it determines whether the first process should be applied to the predicted image. Then, when the decoding device 200 determines that the first process should be applied, it applies the first process as the update process. On the other hand, (2) when the inter-prediction mode is an inter-prediction mode in which the third process cannot be selected, the decoding device 200 can also determine whether the second process should be applied to the predicted image. Then, when the decoding device 200 determines that the second process should be applied, it applies the second process as the update process; when it determines that the second process should not be applied, it determines whether the first process should be applied to the predicted image. Then, when the decoding device 200 determines that the first process should be applied, it applies the first process as the update process.

[0554] This increases the likelihood of being able to perform appropriate updates.

[0555] Furthermore, in a pipeline processing stage used to decode the encoded image, and in the same stage as the reconstruction process of adding the generated final predicted image to the residual image to generate the reconstructed image, the decoding device 200 exclusively performs the first, second, and third processes.

[0556] This allows, for example, a reduction in the number of stages involved in pipeline processing compared to performing the first, second, and third processes at stages different from the reconstruction process. Consequently, it increases the likelihood of suppressing an increase in the circuit size of the decoding device 200.

[0557] Furthermore, the decoding device 200 performs filtering processing to update the pixel values ​​contained in the reconstructed image, which is the sum of the final predicted image and the residual signal. For example, this filtering processing is loop filtering such as deblocking filtering.

[0558] This allows us to remove distortions from the reconstructed image.

[0559] Similar to the decoding apparatus 200, the encoding apparatus 100 in this embodiment generates a predicted image of the processing target block based on the derived motion vector in inter-prediction mode, and applies an update process to the predicted image to generate a final predicted image of the processing target block. Here, the candidates for the update process include a first process and a second process; the first process is a BDOF (bi-directional optical flow) process, and the second process is a filtering process that updates the pixel values ​​of the predicted image. Then, in the application of the update process, the first process and the second process are applied exclusively. For example, in the generation of the final predicted image, when the update process is the first process, the encoding apparatus 100 does not apply the second process to the predicted image, but instead applies the first process to generate the final predicted image. Also, when the update process is the second process, the encoding apparatus 100 does not apply the first process to the predicted image, but instead applies the second process to generate the final predicted image. Furthermore, the processing target block is, for example, the processing target CU. Also, the predicted image generated in the inter-prediction mode is an inter-prediction image.

[0560] Therefore, regardless of whether the update process is the first process or the second process, no processing other than the update process in the first and second processes will be applied to the predicted image. In other words, the application of processing other than the update process is prohibited. Thus, for example, when a pipeline process used to encode an image includes both the first and second processes in one stage, performing the first and second processes together increases the likelihood of suppressing the time spent on that stage. That is, it increases the likelihood of suppressing an increase in processing time.

[0561] Furthermore, during the application of the update process, the encoding device 100 determines whether the second process should be applied to the predicted image. If it determines that the second process should be applied, the second process is used as the update process. On the other hand, if it determines that the second process should not be applied, the encoding device 100 determines whether the first process should be applied to the predicted image. If it determines that the first process should be applied, the first process is used as the update process.

[0562] This increases the likelihood of being able to perform appropriate updates.

[0563] Furthermore, the second process updates the pixel values ​​of the predicted image based on the predicted image. For example, in the second process, the reconstructed image is not used for processing. Alternatively, the second process updates the pixel values ​​of the predicted image based on (1) the predicted image and (2) the reconstructed image of a block that is different from the processing target block.

[0564] This increases the likelihood of appropriate pixel value updates performed through the second process.

[0565] Furthermore, in a stage of the pipeline processing used to encode the image, and in the same stage as the reconstruction processing that adds the final predicted image and the residual image to generate the reconstructed image, the encoding device 100 exclusively performs the first processing and the second processing.

[0566] In this way, for example, compared to performing the first and second processes at a different stage than the reconstruction process, the number of stages included in the pipeline processing can be reduced. As a result, the possibility of suppressing the increase in the circuit size of the encoding device 100 can be improved.

[0567] Here, the update processing candidates may further include a third processing, different from the first and second processing. The third processing is a process of mixing the second predicted image generated by the intra-prediction of the processing object block and the first predicted image generated by the inter-prediction mode. In this case, the first, second, and third processings are applied exclusively in the application of the update processing. For example, the third processing may also be the intra / inter-mixing processing described above.

[0568] This increases the variety of applicable update processes, such as generating predicted images with high accuracy.

[0569] Furthermore, in the application of the update process, (1) when the inter-prediction mode is an inter-prediction mode in which the third process can be selected, the encoding device 100 can also determine whether the third process should be applied to the predicted image. Then, when the encoding device 100 determines that the third process should be applied, it uses the third process as the update process; when it determines that the third process should not be applied, it determines whether the first process should be applied to the predicted image. Then, when the encoding device 100 determines that the first process should be applied, it uses the first process as the update process. On the other hand, (2) when the inter-prediction mode is an inter-prediction mode in which the third process cannot be selected, the encoding device 100 can also determine whether the second process should be applied to the predicted image. Then, when the encoding device 100 determines that the second process should be applied, it uses the second process as the update process; when it determines that the second process should not be applied, it determines whether the first process should be applied to the predicted image. Then, when the encoding device 100 determines that the first process should be applied, it uses the first process as the update process.

[0570] This increases the likelihood of being able to perform appropriate updates.

[0571] Furthermore, in a stage of pipeline processing for encoding the image, and in the same stage as the reconstruction processing that adds the final predicted image to the residual image to generate the reconstructed image, the encoding device 100 exclusively performs the first, second, and third processes.

[0572] In this way, for example, compared to performing the first, second, and third processes at a different stage than the reconstruction process, the number of stages included in the pipeline processing can be reduced. As a result, the possibility of suppressing the increase in the circuit size of the encoding device 100 can be improved.

[0573] Furthermore, the encoding device 100 performs filtering processing to update the pixel values ​​contained in the reconstructed image, which is the sum of the final predicted image and the residual signal. For example, this filtering processing is loop filtering such as deblocking filtering.

[0574] This allows us to remove distortions from the reconstructed image.

[0575] [Case 6 of Predictive Processing] Figure 57 is a flowchart showing a sixth example of prediction processing performed by the decoding apparatus 200 in this embodiment. Furthermore, this flowchart shows the processing flow of the CU in the sixth example of prediction processing.

[0576] The decoding device 200 performs steps S11 to S13, S14d, S15a, S16d, and S17a on a CU-by-CU basis. Specifically, firstly, the decoding device 200 derives the MV (Motion Video Image) of the CU being processed (step S11), and then performs DMVR (Motion-Driven Video Image) processing (step S12). In this DMVR processing, the decoding device 200 searches for the periphery of the MV derived in step S11, thereby correcting the MV. Then, the decoding device 200 uses the corrected MV to perform MC (Motion Compensation) processing, i.e., motion compensation processing (step S13). This yields the inter-prediction image of the CU being processed.

[0577] Next, the decoding device 200 determines whether OBMC processing is applicable to the inter-prediction image of the processing object CU (step S14d). Here, when it is determined that OBMC processing is applicable (yes in step S14d), the decoding device 200 performs OBMC processing on the inter-prediction image obtained by MC processing, thereby generating the final prediction image (step S16d).

[0578] On the other hand, when the decoding device 200 determines that OBMC processing is not applicable (No in step S14d), it further determines whether intra / interval blending processing is applicable (step S15a). Here, when the decoding device 200 determines that intra / interval blending processing is applicable (Yes in step S15a), it blends the inter-prediction image obtained by MC processing with the intra-prediction image obtained by intra-prediction processing (step S17a). The final prediction image is generated by this blending. On the other hand, when the decoding device 200 determines that intra / interval blending processing is not applicable (No in step S15a), it uses the inter-prediction image obtained by MC processing unchanged as the final prediction image in the generation of the reconstructed image.

[0579] Additionally, in the flowchart of Figure 57, the decoding device 200 first determines whether OBMC processing is applicable to the inter-prediction image, and then determines whether intra / inter-mixing processing is applicable, but these determinations can also be performed in the reverse order.

[0580] Additionally, the flowchart in Figure 57 is just an example. At least one step in the flowchart can be removed, or other processing or condition determination steps can be added to the flowchart.

[0581] Alternatively, the processing shown in the flowchart of Figure 57 can also be performed by the encoding device 100. That is, the encoding device 100 and the decoding device 200 perform the same prediction processing, the difference being that the prediction processing is used to encode the image into a stream, or to decode the stream into an image. Therefore, the prediction processing flow shown in the flowchart of Figure 57 is basically the same in the encoding device 100.

[0582] Furthermore, the determination of whether OBMC processing is applicable (step S14d) and whether internal / internal mixing processing is applicable (step S15a) in the flowchart of Figure 57 can also be performed by parsing flags. That is, the decoding device 200 can, for example, parse the flags recorded in the stream to perform such determinations. Also, the encoding device 100 can, for example, calculate the cost of using the RD optimization model to perform such determinations. That is, in each of the above determinations, the encoding device 100 calculates the cost of the predicted image obtained by processing the object to be determined, and the cost of the predicted image obtained by not applying the processing. Then, the encoding device 100 performs the above determinations to use the predicted image corresponding to the smaller cost among these costs as the final predicted image.

[0583] Figure 58 is a schematic diagram showing a sixth example of prediction processing performed by the decoding device 200 in this embodiment as a configuration example of pipeline processing.

[0584] The pipeline treatment PL6 shown in Figure 58 comprises stages 1 to 6. Furthermore, the treatment of each stage from stage 2 down is performed, for example, in VPDU units.

[0585] Phase 5 of phases 1 to 6 of pipeline processing PL6 differs from phase 5 of pipeline processing PL2 shown in Figure 50. Specifically, in phase 5 of pipeline processing PL6, OBMC processing St51d is performed instead of LIC processing St51a. Furthermore, in this OBMC processing St51d, the reconstructed image generated by adding the residual image to the predicted image is not fed back.

[0586] In stage 5 of such pipeline processing PL6, when OBMC processing St51d is not applied, inter-prediction / inter-prediction mixing processing St53 is performed. In other words, the decoding device 200 does not perform OBMC processing St51d on the inter-prediction image generated by MC processing St4, but instead generates an inter-prediction / inter-prediction image by mixing the inter-prediction image and the inter-prediction image. On the other hand, when the decoding device 200 applies OBMC processing St51d to the inter-prediction image generated by MC processing St4, it uses the inter-prediction image with applied OBMC processing St51d as the final prediction image in the generation of the reconstructed image.

[0587] In this pipeline processing PL6, at the time point of performing the inner / inner blending process St53, the generation of the inter-prediction image required for St53 has already been completed in stage 4. Therefore, the waiting time required for the completion of the inter-prediction process in pipeline processing PL1 shown in FIG48 is not required in pipeline processing PL6. Thus, the inner / inner blending process St53 can be performed immediately at the time point of completion of the inner prediction process St52. As a result, the generation of unnecessary waiting time can be suppressed. That is, the processing time of stage 5 when performing the inner / inner blending process St53 can be made almost equal to the processing time of stage 5 when the inner / inner blending process St53 is not performed.

[0588] Additionally, the pipeline processing PL6 shown in Figure 58 is just an example. At least one processing contained in the pipeline processing PL6 can be removed, or other processing can be added to the pipeline processing PL6 to change the segmentation method of the stage.

[0589] Furthermore, in stage 5 of pipeline processing PL6 in Figure 58, the reconstructed image generated by adding the residual image of the processed CU to the predicted image is not fed back to OBMC processing St51d. Therefore, stage 5 can be divided into a stage that performs OBMC processing St51d and a stage that includes several processes such as reconstruction processing St56, in addition to OBMC processing St51d. However, dividing stage 5 into two stages may increase the required memory and circuitry due to the addition of stages. However, in pipeline processing PL6, since OBMC processing St51d is performed in the same stage as reconstruction processing St56 and other processes, the possibility of avoiding the increase in memory and circuitry can be improved.

[0590] In the sixth example of predictive processing, when encoding a VPDU containing multiple CUs is performed using the processing shown in Figures 57 and 58, all stages of processing can still be completed within the pre-specified processing time. As a result, the possibility of achieving high-speed processing while suppressing the degradation of encoding performance can be increased. Furthermore, as mentioned above, since the OBMC processing St51d is installed at the same stage as the reconstruction processing St56, the possibility of suppressing the increase in the circuit size of the encoding device 100 and the decoding device 200 can be increased.

[0591] [Case 7 of Predictive Processing] Figure 59 is a flowchart showing a seventh example of prediction processing performed by the decoding apparatus 200 in this embodiment. Furthermore, this flowchart shows the processing flow of the CU in the seventh example of prediction processing.

[0592] The decoding device 200 performs steps S11 to S13, S14d, S15b, S16d, and S17b on a CU-by-CU basis. Specifically, the decoding device 200 first derives the MV (Motion Video Image) from the CU being processed (step S11), and then performs DMVR (Motion Detection and Reduction) processing (step S12). In this DMVR processing, the decoding device 200 searches for the periphery of the MV derived in step S11, thereby correcting the MV. Then, the decoding device 200 uses the corrected MV to perform MC (Motion Compensation) processing, i.e., motion compensation processing (step S13). This yields the inter-prediction image of the CU being processed.

[0593] Next, the decoding device 200 determines whether OBMC processing is applicable to the inter-prediction image of the processing object CU (step S14d). Here, when it is determined that OBMC processing is applicable (yes in step S14d), the decoding device 200 performs OBMC processing on the inter-prediction image obtained by MC processing, thereby generating the final prediction image (step S16d).

[0594] On the other hand, when the decoding device 200 determines that OBMC processing is not applicable (No in step S14d), it further determines whether filtering processing (also known as prediction image filtering processing) is applicable (step S15b). Here, when the decoding device 200 determines that filtering processing is applicable (Yes in step S15b), it performs filtering processing on the inter-prediction image obtained by MC processing (step S17b). The final prediction image is generated by this filtering processing. On the other hand, when the decoding device 200 determines that filtering processing is not applicable (No in step S15b), it uses the inter-prediction image obtained by MC processing unchanged as the final prediction image in the generation of the reconstructed image.

[0595] Additionally, in the flowchart of Figure 59, the decoding device 200 first determines whether OBMC processing is applicable to the inter-prediction image, and then determines whether filtering processing is applicable, but these determinations can also be performed in the reverse order.

[0596] Additionally, the flowchart in Figure 59 is just an example. At least one step in the flowchart can be removed, or other processing or condition determination steps can be added to the flowchart.

[0597] Alternatively, the processing shown in the flowchart of Figure 59 can also be performed by the encoding device 100. That is, the encoding device 100 and the decoding device 200 perform the same prediction processing, the difference being that the prediction processing is used to encode the image into a stream, or to decode the stream into an image. Therefore, the prediction processing flow shown in the flowchart of Figure 59 is basically the same in the encoding device 100.

[0598] Furthermore, the determination of whether OBMC processing is applicable (step S14d) and the determination of whether filtering processing is applicable (step S15b) in the flowchart of Figure 59 can also be performed by parsing flags. That is, the decoding device 200 can, for example, parse the flags recorded in the stream to perform such determinations. Also, the encoding device 100 can, for example, calculate the cost of using the RD optimization model to perform such determinations. That is, in each of the above determinations, the encoding device 100 calculates the cost of the predicted image obtained by processing the object to be determined, and the cost of the predicted image obtained by not applying the processing. Then, the encoding device 100 performs the above determinations to use the predicted image corresponding to the smaller cost among these costs as the final predicted image.

[0599] Figure 60 is a schematic diagram showing the seventh example of prediction processing performed by the decoding device 200 in this embodiment as a configuration example of pipeline processing.

[0600] The pipeline treatment PL7 shown in Figure 60 includes stages 1 to 6. Furthermore, the treatment of each stage from stage 2 down is performed, for example, in VPDU units.

[0601] Phase 5 of phases 1 to 6 of pipeline processing PL7 differs from phase 5 of pipeline processing PL2 shown in Figure 50. That is, in phase 5 of pipeline processing PL7, only one of the OBMC processing St51d or the prediction map filtering processing St51b is applied to the inter-prediction image generated by MC processing St4 to produce the final prediction image. Furthermore, the prediction map filtering processing St51b is the filtering process described above.

[0602] In other words, in stage 5 of pipeline processing PL7, OBMC processing St51d is performed when prediction graph filtering St51b is not applied, and prediction graph filtering St51b is performed when OBMC processing St51d is not applied. In other words, the decoding device 200 does not perform prediction graph filtering St51b on the inter-prediction image generated by MC processing St4, but instead generates the final prediction image by applying OBMC processing St51d. Alternatively, the decoding device 200 does not perform OBMC processing St51d on the inter-prediction image generated by MC processing St4, but instead generates the final prediction image by applying prediction graph filtering St51b.

[0603] When both OBMC processing (St51d) and predictive graph filtering (St51b) are applied simultaneously, the processing time or number of processing cycles in stage 5 increases due to the continuous execution of these two processes. Therefore, in such cases, there is a possibility that stage 5 may not be completed within the specified processing time. However, in the pipeline processing PL7 shown in Figure 60, only either OBMC processing (St51d) or predictive graph filtering (St51b) is performed. Therefore, the increase in the processing time or number of processing cycles in stage 5 can be suppressed, and stage 5 can be completed within the specified processing time.

[0604] Additionally, the pipeline processing PL7 shown in Figure 60 is just an example. At least one processing included in the pipeline processing PL7 can be removed, or other processing can be added to the pipeline processing PL7 to change the segmentation method of the stage.

[0605] Furthermore, in stage 5 of pipeline processing PL7 in Figure 60, the reconstructed image generated by adding the residual image of the processed CU and the predicted image is fed back to the prediction image filtering process St51b. Next, in the prediction image filtering process St51b, processing is performed using the reconstructed image and the inter-prediction image generated by MC processing St4. That is, pixels around the boundaries of the inter-prediction image are filtered using pixels from the reconstructed image. However, the seventh example of this prediction processing is not limited to this feedback. That is, the feedback of the reconstructed image to the prediction image filtering process St51b may not be performed. In this case, in the prediction image filtering process St51b, the reconstructed image is not used, and instead, the inter-prediction image generated by MC processing St4 is processed.

[0606] Similarly, no image reconstruction feedback is performed in the OBMC processing St51d. Therefore, stage 5 can be divided into a stage containing the OBMC processing St51d, and a stage containing several other processes such as the prediction image filtering process St51b and the reconstruction process St56. However, dividing stage 5 into two stages may increase the required memory and circuitry due to the addition of a stage. However, in pipeline processing PL5, since the prediction image filtering process St51b and the OBMC processing St51d are performed in the same stage as the reconstruction process St56 and other processes, the possibility of avoiding increased memory and circuitry is improved.

[0607] In the seventh example of prediction processing, when encoding a VPDU containing multiple CUs using the processing shown in Figures 59 and 60, all stages of processing can still be completed within the pre-specified processing time. As a result, the possibility of achieving high-speed processing while suppressing the degradation of encoding performance can be increased. Furthermore, as mentioned above, since the prediction graph filtering process St51b and the OBMC process St51d are installed in the same stages as the reconstruction process St56, the possibility of suppressing the increase in the circuit size of the encoding device 100 and the decoding device 200 can be increased.

[0608] [Variation Example] Alternatively, multiple examples from the first, second, third, fourth, fifth, sixth, and seventh examples of the prediction processing in this embodiment can be combined. For example, five processes including LIC processing, BIO processing, OBMC processing, prediction map filtering processing, and inter-process / inter-process mixing processing can be installed in the same stage as the reconstruction processing. Then, only one of these five processes can be selected for application to the inter-process prediction image. Alternatively, at least two of these five processes can be installed in the same stage as the reconstruction processing. Furthermore, only one of these at least two processes can be selected for application to the inter-process prediction image. Also, in the above examples, only one process is selected from five processes, but only one process can be selected from six or more processes for application to the inter-process prediction image. In this case, other processes different from LIC processing, BIO processing, OBMC processing, prediction map filtering processing, and inter-process / inter-process mixing processing can also be included in the above six or more processes. Furthermore, the selected candidate processing is not limited to LIC processing, BIO processing, OBMC processing, prediction graph filtering processing, and inner / outer mixing processing, but can be any processing.

[0609] [Representative examples of composition and processing] The following shows a representative example of the configuration and processing of the encoding device 100 and the decoding device 200 described above.

[0610] Figure 61 is a flowchart illustrating an example of encoding and decoding processing. For example, encoding device 100 and decoding device 200 each have circuitry and memory connected to the circuitry. The circuitry and memory of encoding device 100 can also correspond to processor a1 and memory a2 shown in Figure 40, and the circuitry and memory of decoding device 200 can also correspond to processor b1 and memory b2 shown in Figure 46. The circuitry of each of encoding device 100 and decoding device 200 performs the following processing during operation.

[0611] Specifically, as in the fourth example of predictive processing, the circuit performs steps S1 and S2a during operation. More specifically, the circuit first generates a first predicted image of the target block in the inter-prediction mode based on the derived movement vector (step S1).

[0612] Next, the circuit applies an update process to the first predicted image, thereby generating a final predicted image of the target block (step S2a). Here, the candidates for the update process include the first process and the second process. The first process is a BDOF (bi-directional optical flow) process, i.e., the aforementioned BIO process. The second process is the aforementioned inter-internal / inter-internal mixing process. That is, the second process is a process that mixes the second predicted image generated by the inter-prediction of the target block with the aforementioned first predicted image. Then, in the application of the update process, the first and second processes are applied exclusively. That is, when the update process is the first process, the circuit does not apply the second process to the first predicted image, but instead generates the final predicted image by applying the first process. On the other hand, when the update process is the second process, the circuit does not apply the first process to the first predicted image, but instead generates the final predicted image by applying the second process.

[0613] Therefore, regardless of whether the update process is the first process or the second process, no processing other than the update process in the first and second processes will be applied to the inter-predicted image. In other words, the application of processing other than the update process is prohibited. Thus, for example, when a pipeline process used to encode an image includes both the first and second processes in one stage, performing the first and second processes together increases the likelihood of suppressing the time spent on that stage. That is, it increases the likelihood of suppressing an increase in processing time.

[0614] Figure 62 is a flowchart showing another example of encoding and decoding processing. For example, encoding device 100 and decoding device 200 each have circuitry and memory connected to the circuitry. The circuitry and memory of encoding device 100 can also correspond to processor a1 and memory a2 shown in Figure 40, and the circuitry and memory of decoding device 200 can also correspond to processor b1 and memory b2 shown in Figure 46. The circuitry of each of encoding device 100 and decoding device 200 performs the following processing during operation.

[0615] Specifically, as in the fifth example of predictive processing, the circuit performs steps S1 and S2b during operation. More specifically, the circuit first generates a predicted image of the target block in the inter-prediction mode based on the derived movement vector (step S1).

[0616] Next, the circuit applies an update process to the predicted image, thereby generating the final predicted image of the processing target block (step S2b). Here, the candidates for the update process include a first process and a second process. The first process is a BDOF (bi-directional optical flow) process, i.e., the aforementioned BIO process. The second process is a filtering process that updates the pixel values ​​of the predicted image. Then, in the application of the update process, the first and second processes are applied exclusively. That is, when the update process is the first process, the circuit does not apply the second process to the predicted image, but instead generates the final predicted image by applying the first process. On the other hand, when the update process is the second process, the circuit does not apply the first process to the predicted image, but instead generates the final predicted image by applying the second process.

[0617] Therefore, regardless of whether the update process is the first process or the second process, no processing other than the update process in the first and second processes will be applied to the inter-predicted image. In other words, the application of processing other than the update process is prohibited. Thus, for example, when a pipeline process used to encode an image includes both the first and second processes in one stage, performing the first and second processes together increases the likelihood of suppressing the time spent on that stage. That is, it increases the likelihood of suppressing an increase in processing time.

[0618] [Other examples] The encoding device 100 and decoding device 200 in the above examples can each be used as an image encoding device and an image decoding device, or as a motion image encoding device and a motion image decoding device.

[0619] Furthermore, the encoding device 100 and the decoding device 200 may process (more specifically, encode or decode) only a portion of the aforementioned plurality of processing object elements, while other devices may process (more specifically, encode or decode) the remaining processing object elements. Also, the encoding device 100 and the decoding device 200 may possess only a portion of the aforementioned plurality of constituent elements, while other devices may possess the remaining constituent elements.

[0620] Furthermore, at least some of the above examples can also be used as encoding methods, decoding methods, prediction methods, or other methods.

[0621] Furthermore, each component can be implemented using dedicated hardware or by executing software programs suitable for each component. Alternatively, the program execution unit of the CPU or processor can read and execute software programs recorded on recording media such as hard drives or semiconductor memory to implement each component.

[0622] Specifically, the encoding device 100 and the decoding device 200 may each have a processing circuitry and a storage device, the aforementioned storage device being electrically connected to the processing circuitry and accessible by the processing circuitry. For example, the processing circuitry corresponds to processor a1 or b1, and the storage device corresponds to memory a2 or b2.

[0623] The processing circuit includes at least one of dedicated hardware and a program execution unit, and uses a storage device to perform processing. Furthermore, when the processing circuit includes a program execution unit, the storage device stores the software program executed by the program execution unit.

[0624] Here, the software that implements the above-mentioned encoding device 100 or decoding device 200 is a program that allows the computer to execute the processing shown in Figures 47 to 62.

[0625] Furthermore, as mentioned above, each component can also be a circuit. These circuits can be configured as a single circuit or as individual circuits. Moreover, each component can be implemented using a general-purpose processor or a dedicated processor.

[0626] Furthermore, other components can be allowed to perform the processing required by a specific component. Also, the order of processing can be changed, and multiple processes can be executed in parallel. Furthermore, the encoding / decoding device may include an encoding device 100 and a decoding device 200.

[0627] Furthermore, the ordinal numbers 1 and 2 used in the description may be appropriately replaced. Also, ordinal numbers may be reassigned to constituent elements, or they may be removed.

[0628] The forms of the encoding device 100 and the decoding device 200 have been described above using a number of examples, but the forms of the encoding device 100 and the decoding device 200 are not limited to these examples. As long as they do not depart from the spirit of this disclosure, various modifications that can be conceived by those skilled in the art can be applied to each example, or forms constructed by combining the constituent elements of different examples, can also be included within the scope of the forms of the encoding device 100 and the decoding device 200.

[0629] It is also possible to implement one or more states disclosed herein in combination with at least a portion of other states disclosed herein. Furthermore, it is also possible to implement a portion of the processing, a portion of the apparatus configuration, a portion of the syntax, etc., described in the flowchart of one or more states disclosed herein in combination with other states.

[0630] [Implementation and Application] In the above embodiments, each functional or operational block can typically be implemented using an MPU (microprocessor unit) and memory. Furthermore, the processing performed by each functional block can also be implemented by a program execution unit such as a processor that reads and executes software (programs) recorded on a recording medium such as ROM. This software can also be distributed. The software can also be recorded on various recording media such as semiconductor memory. Alternatively, each functional block can be implemented using hardware (dedicated circuitry). Various combinations of hardware and software can be employed.

[0631] Furthermore, the processing described in each embodiment can be implemented by centralized processing using a single device (system), or by distributed processing using multiple devices. Also, the number of processors executing the above program can be singular or multiple. That is, centralized processing or distributed processing can be performed.

[0632] The embodiments disclosed herein are not limited to the above-described embodiments, and various modifications can be made, which are also included within the scope of the embodiments disclosed herein.

[0633] Furthermore, examples of application of the motion image encoding method (image encoding method) or motion image decoding method (image decoding method) shown in the above embodiments and various systems implementing such examples will be described. Such a system may also include an image encoding device using the image encoding method, an image decoding device using the image decoding method, or an image encoding / decoding device possessing both. Other configurations of such a system can be appropriately modified depending on the circumstances.

[0634] [Usage Example] Figure 63 is a diagram showing the overall structure of an appropriate content delivery system ex100 for implementing content publishing services. The area for providing communication services is divided into desired sizes, and fixed wireless stations, i.e., base stations ex106, ex107, ex108, ex109, and ex110, are set up in each cell, as shown in the example.

[0635] In this content delivery system ex100, various devices such as computers ex111, game consoles ex112, cameras ex113, home appliances ex114, and smartphones ex115 are connected to the Internet ex101 via Internet service provider ex102 or communication network ex104, and base stations ex106 to ex110. The content delivery system ex100 can also combine and connect any of the aforementioned devices. In various implementations, the devices may also connect directly or indirectly to each other without using base stations ex106 to ex110, but via telephone networks or short-range wireless connections. Furthermore, the streaming server ex103 can also connect to the computers ex111, game consoles ex112, cameras ex113, home appliances ex114, and smartphones ex115 via the Internet ex101. Additionally, the streaming server ex103 can also connect to terminals within a hotspot inside an aircraft ex117 via satellite ex116.

[0636] Alternatively, wireless access points or hotspots can be used to replace base stations ex106 to ex110. Furthermore, the streaming server ex103 can connect directly to the communication network ex104 without going through the Internet ex101 or the Internet service provider ex102, and can also connect directly to the aircraft ex117 without going through the satellite ex116.

[0637] The EX113 camcorder is a digital camera or similar device capable of capturing still images and moving images. The EX115 smartphone is a smartphone, mobile phone, or PHS (Personal Handy-phone System) that supports 2G, 3G, 3.9G, 4G, and future 5G mobile communication systems.

[0638] Home appliances EX114 refers to refrigerators or other appliances included in household fuel cell cogeneration systems.

[0639] In the content delivery system ex100, terminals with photography capabilities can be connected to the streaming server ex103 via base station ex106, thereby enabling live streaming. During live streaming, terminals (such as computers ex111, game consoles ex112, cameras ex113, home appliances ex114, smartphones ex115, and terminals within airplanes ex117) can perform the encoding processing described in the aforementioned embodiments on still images or moving images captured by the user using the terminal. They can also multiplex the encoded image data and the corresponding encoded audio data, and send the obtained data to the streaming server ex103. In other words, each terminal functions as an image encoding device as described in this disclosure.

[0640] On the other hand, the streaming server ex103 sends content data to requesting clients for streaming publication. Clients can be terminals such as computers ex111, game consoles ex112, cameras ex113, home appliances ex114, smartphones ex115, or aircraft ex117 that can decode the encoded data. Each machine receiving the published data can also decode and play the received data. That is, each machine can also function as an image decoding device as described in this disclosure.

[0641] [Distributed processing] Furthermore, the streaming server ex103 can be multiple servers or multiple computers, distributing, recording, and publishing data. For example, the streaming server ex103 can also be implemented using a CDN (Contents Delivery Network), which connects numerous edge servers distributed around the world via the Internet to achieve content publishing. In a CDN, physically close edge servers can be dynamically allocated according to the client. Moreover, by caching and publishing content on these edge servers, latency can be reduced. Furthermore, in the event of various types of errors or changes in communication status due to increased traffic, high-speed and stable publishing can be achieved because multiple edge servers can be used for distributed processing, the publishing entity can be switched to other edge servers, or the publishing can be continued by bypassing the network portion where obstacles occur.

[0642] Furthermore, the distributed processing of the data transmission itself is not limited to the transmission itself; the encoding of the captured data can also be performed on each terminal, on the server side, or distributed among different terminals. For example, encoding typically involves two processing loops. In the first loop, the complexity or amount of data encoded per frame or scene is determined. In the second loop, processing is performed to maintain image quality and improve encoding efficiency. For instance, the terminal performs the first encoding, while the server receiving the content performs the second encoding. This reduces the processing load on each terminal and improves content quality and efficiency. In this case, if there is a requirement for near-instantaneous reception and decoding, other terminals can receive and play the data encoded in the first iteration, making more flexible real-time transmission possible.

[0643] For another example, a camera like the ex113 extracts features (quantities of features or characteristics) from an image, compresses this feature data into metadata, and sends it to a server. The server uses these features to determine the importance of the object and adjusts the quantization precision accordingly, compressing the image based on its meaning (or the importance of its content). Feature data is particularly effective in improving the accuracy and efficiency of motion vector prediction during further compression on the server. Alternatively, simple encoding methods like VLC (Variable Length Coding) can be performed on the terminal, while more demanding encoding methods like CABAC (Adaptive Binary Arithmetic Coding) can be used on the server.

[0644] To give another example, in stadiums, shopping malls, or factories, there are sometimes multiple images of almost the same scene captured by multiple terminals. In this case, the encoding process is distributed and processed separately using multiple terminals that took the photos, other terminals that did not take photos as needed, and servers, in units such as GOP (Group of Pictures), image units, or tile units after image segmentation. This reduces latency and achieves real-time processing.

[0645] Since multiple image data sets depict nearly identical scenes, the server can manage and / or instruct the references between image data captured by each terminal. Furthermore, the server can receive encoded data from each terminal, modify the reference relationships or correct / replace the images themselves among the multiple data sets, and then re-encode them. This allows for streaming with improved quality and efficiency for each data set.

[0646] Furthermore, the server can also transcode the image data after changing its encoding method before publishing the image data. For example, the server can convert the encoding method of the MPEG system to the VP system (such as VP9), or convert H.264 to H.265, etc.

[0647] In this way, encoding processing can be performed via a terminal or one or more servers. Therefore, although "server" or "terminal" is used below to refer to the subject of processing, the terminal can perform part or all of the processing performed by the server, and the server can perform part or all of the processing performed by the terminal. Furthermore, the same applies to decoding processing.

[0648] [3D, Multi-angle] There is a growing trend of integrating and utilizing images or videos of different scenes captured by multiple cameras (ex113) and / or smartphones (ex115) that are almost synchronized with each other, or images or videos of the same scene captured from different angles. The images captured by each terminal can be integrated based on the relative positional relationship between the terminals or the fact that the feature points contained in the images are consistent regions.

[0649] The server can not only encode 2D moving images, but also automatically or at user-specified times encode still images based on scene analysis of the moving images, and send them to the receiving terminal. Furthermore, when the server can obtain the relative positional relationship between the camera terminals, it can generate the 3D shape of the scene not only based on 2D moving images, but also based on images of the same scene captured from different angles. The server can also encode 3D data generated by point clouds, etc., and can use 3D data to identify or track people or targets. Based on the identification or tracking results, it can select or reconstruct images from images captured by multiple terminals and generate images to be sent to the receiving terminal.

[0650] In this way, users can freely select images corresponding to each camera terminal to appreciate the scene, or appreciate the content of the image cut out from the selected viewpoint from 3D data reconstructed using multiple images or videos. Furthermore, sound is also picked up from multiple different angles along with the images, and the server can also multiplex sound from a specific angle or space with the corresponding images and send the multiplexed images and sound.

[0651] Furthermore, in recent years, content that aligns the real world with the virtual world, such as Virtual Reality (VR) and Augmented Reality (AR), has become increasingly popular. In the case of VR images, the server can create separate viewpoint images for the right and left eyes. Multi-View Coding (MVC) is used to encode these images, allowing for reference between different viewpoints, or they can be encoded into different streams without cross-referencing. When decoding different streams, the virtual 3D space is recreated according to the user's viewpoint, allowing the streams to play synchronously.

[0652] In the case of AR images, the server can also overlay virtual object information in virtual space onto camera information in real space based on the 3D position or the user's viewpoint movement. The decoding device can also acquire or retain virtual object information and 3D data, and generate 2D images in response to the user's viewpoint movement, creating overlay data by smoothly combining these. Alternatively, the decoding device can also send the user's viewpoint movement to the server in addition to requests for virtual object information. The server can also create overlay data in conjunction with viewpoint movement received from 3D data held on the server, encode the overlay data, and publish it to the decoding device. Furthermore, the overlay data typically includes an α value representing penetration, in addition to RGB values. The server can also set the α value of parts other than the target created from 3D data to 0, and encode these parts as transparent. Alternatively, the server can set the background to a predetermined RGB value, similar to a chroma key, and generate data where parts other than the target are set to the background color. The predetermined RGB values ​​can also be determined in advance.

[0653] Similarly, the decoding of published data can be performed on the client side (e.g., a terminal), on the server side, or distributed among them. For example, one terminal can first send a receiving request to the server, and then other terminals can receive the content corresponding to that request, decode it, and send a signal indicating completion to a device with a display. By distributing the processing and selecting appropriate content without relying on the performance of the communicating terminals themselves, high-quality data can be played. As another example, large-format image data can be received by a television or similar device, and then decoded and displayed as segmented areas, such as tiles, on the viewer's personal terminal. This allows for sharing of the entire image and enabling users to easily identify their assigned area or areas requiring more detailed examination.

[0654] In situations where multiple wireless communications are available for short, medium, or long distances both indoors and outdoors, seamless content reception using publishing system specifications such as MPEG-DASH may be possible. Users can also freely select and switch between their own terminals, decoding devices such as displays located indoors and outdoors, and so on in real time. Furthermore, they can use their location information to switch between the terminals to be decoded and the terminals to be displayed while decoding is being performed. This allows information to be mapped and displayed on a portion of the wall or floor of a building adjacent to a display device while the user is moving towards their destination. Additionally, the bit rate of the received data can be switched based on the ease of access to the encoded data on the network. This ease of access refers to the encoded data being cached to a server that can be accessed from the receiving terminal in a short time, or copied to an edge server in a Contents Delivery Service.

[0655] [Adjustable Encoding] Regarding content switching, the adjustable stream, which uses the dynamic image encoding method described in the above embodiments for compression encoding, as shown in Figure 64, will be used for illustration. While it is acceptable for a server to have multiple streams with the same content but different qualities as individual streams, they can also be configured as layered encodings as shown in the figure. This achieves temporal / spatial adjustable streaming, and the characteristics of this adjustable streaming are utilized to switch content. In other words, the decoding side determines which layer to decode based on intrinsic factors such as performance and extrinsic factors such as communication band status. This allows the decoding side to freely switch between low-resolution and high-resolution content during decoding. For example, after a user watches a video on a smartphone (ex115) while on the go, and then wants to watch it on an internet TV device at home, the device only needs to decode the same stream to a different layer, thus reducing the burden on the server side.

[0656] Furthermore, as mentioned above, in addition to encoding the image layer by layer and achieving scalability through enhancement layers above the base layer, the enhancement layers may also contain metadata based on statistical information of the image. The decoder can also perform super-resolution on the base layer image based on the metadata, thereby producing high-quality content. Super-resolution can also maintain and / or increase resolution and improve the S / N ratio. The metadata includes information useful for specifying linear or nonlinear filtering coefficients used in similar super-resolution processing, or information specifying parameter values ​​in filtering, machine learning, or least squares operations used in super-resolution processing.

[0657] Alternatively, the image can be segmented into tiles based on the meaning of targets within the image. The decoder selects tiles to decode to decode only a portion of the region. Furthermore, by storing the attributes of the target (people, vehicles, balls, etc.) and its position within the image (coordinates within the same image, etc.) as metadata, the decoder can determine the desired target's location based on the metadata and identify the tile containing that target. For example, as shown in Figure 65, metadata can also be stored using a data storage structure different from pixel data, such as SEI (supplementary enhancement information) messages in HEVC. This metadata is used to represent, for example, the position, size, or color of the main target.

[0658] Meta-information can also be stored in units consisting of multiple images, such as streaming, sequences, or random access units. The decoding side can obtain information such as the time when a specific person appears in the image. By comparing the information of the image unit with the time information, the image in which the target exists can be identified, and the position of the target within the image can be determined.

[0659] [Webpage Optimization] Figure 66 shows an example of a webpage display on a computer (e.g., ex111). Figure 67 shows an example of a webpage display on a smartphone (e.g., ex115). As shown in Figures 66 and 67, webpages sometimes contain multiple links to image content. The viewing method varies depending on the browsing device. When multiple links are visible on the screen, the display device (decoding device) can display still images or I-images of each content as link images until the user explicitly selects a link image, or until the link image is near the center of the screen or the entire link image enters the screen. Alternatively, it can display images like GIF animations using multiple still images or I-images, or it can receive only the basic layer, decode, and display the image.

[0660] When a user selects a linked image, the display device, for example, prioritizes the base layer and decodes it simultaneously. Additionally, when the HTML of a webpage contains information indicating that it is editable content, the display device can also decode up to the enhancement layer. Furthermore, to ensure real-time performance, before selection or when communication bandwidth is very limited, the display device only decodes and displays images that reference the preceding image (I-images, P-images, and only B-images that reference the preceding image), thereby reducing the delay between the decoding and display of the initial image (the delay from the start of content decoding to the start of display). Moreover, the display device can also deliberately ignore image reference relationships, rendering all B-images and P-images referenced to the preceding image and decoding them coarsely, then performing normal decoding as time passes and more images are received.

[0661] [Autonomous Driving] Furthermore, when sending and receiving still images or data such as 2D or 3D maps for the purpose of autonomous driving or supporting driving, the receiving terminal can receive image data belonging to more than one layer, as well as weather or construction information as metadata, and decode them accordingly. Additionally, metadata can belong to a layer or simply be multiplexed with image data.

[0662] In this scenario, since the vehicle, drone, or aircraft containing the receiving terminal is mobile, the receiving terminal can seamlessly receive and decode data while switching base stations from ex106 to ex110 by transmitting its location information. Furthermore, the receiving terminal can dynamically adjust the level of metadata reception and map updates based on user selection, user status, and / or the status of the communication band.

[0663] In the content delivery system ex100, the client can receive encoded information sent by the user in real time, decode it, and play it.

[0664] [Publishing personal content] Furthermore, in the content delivery system ex100, not only high-quality, long-duration content from video publishers, but also low-quality, short-duration content from individuals can be unicast or multicast. It's conceivable that such personal content will gradually increase in the future. To further enhance the quality of personal content, the server can also perform editing before encoding. This can be achieved using, for example, the following configuration.

[0665] During or after shooting, the server performs image processing on the original or encoded data, including error detection, scene searching, meaning analysis, and target detection. Based on the identification results, the server then performs manual or automatic editing, such as correcting blur or camera shake, deleting scenes of lower importance (e.g., scenes with lower brightness or out of focus), emphasizing target edges, and adjusting color tones. The server then encodes the edited data based on these changes. Furthermore, given that longer shooting times can decrease viewership, the server can automatically crop scenes with less dynamic content, in addition to those of lower importance, based on image processing results to create content within a specific timeframe, adapting to the shooting duration. Alternatively, the server can generate and encode a digest based on the meaning analysis of the scene.

[0666] There are cases where direct broadcasting of personal content could infringe on copyrights, moral rights, or portrait rights, and there are also situations where sharing exceeds the intended scope, causing inconvenience for the individual. Therefore, servers can, for example, intentionally alter the focus of images such as faces or homes at the periphery of the screen before re-encoding. Furthermore, servers can identify whether the image being encoded contains faces different from those previously registered, and if so, add mosaic or other processing to the faces. Alternatively, users can specify the people or background areas in the image they want to process, based on copyright concerns, for pre- or post-processing. Servers can also replace specified areas with other images or blur the focus. If it involves people, the server can track the person in the video and replace the face portion of the image.

[0667] For viewing personal content with limited data volume, real-time processing is crucial. Therefore, regardless of video bandwidth, the decoding device prioritizes receiving the base layer for decoding and playback. The decoding device can also receive enhancement layers during this process, and in cases of loop playback or playback more than twice, high-definition video including the enhancement layers will be played. Streams with this adjustable encoding provide an experience where, initially, the stream is a coarse animation, but gradually becomes smarter and the image improves. In addition to adjustable encoding, combining the initial coarse stream with a second stream encoded based on the first animation into a single stream also provides the same experience.

[0668] [Other Implementation Examples] Furthermore, such encoding or decoding processing is generally performed within the LSIex500 integrated circuitry (LSI) of each terminal. The LSIex500 (refer to Figure 63) can be a single chip or composed of multiple chips. Alternatively, software for encoding or decoding motion images can be installed on a recording medium (CD-ROM, floppy disk, or hard disk, etc.) that can be read by a computer such as the ex111, and the encoding or decoding processing can be performed using this software. Furthermore, when the smartphone ex115 has a built-in camera, motion image data captured by that camera can also be sent. This motion image data can also be data that has been encoded within the LSIex500 integrated circuitry of the smartphone ex115.

[0669] Alternatively, the LSIex500 can also be configured to download and activate application software. In this case, the terminal first determines whether it corresponds to the content's encoding method or whether it has the capability to execute a specific service. If the terminal does not correspond to the content's encoding method or does not have the capability to execute a specific service, the terminal can also download a codec or application software before obtaining and playing the content.

[0670] Furthermore, not limited to the content delivery system ex100 via the Internet ex101, at least one of the aforementioned motion picture encoding devices (image encoding devices) or motion picture decoding devices (image decoding devices) can also be installed in digital broadcasting systems. Since it uses satellites or other means to transmit and receive multiplexed data that has already been multiplexed for video and audio via radio waves, it is easier to unicast than the content delivery system ex100. The difference is that it is suitable for multicast, but the same applications can be used for encoding and decoding processing.

[0671] [Hardware Configuration] Figure 68 is a diagram showing the smartphone ex115 shown in Figure 63 in more detail. Figure 69 is a diagram showing an example of the configuration of the smartphone ex115. The smartphone ex115 includes: an antenna ex450 for transmitting and receiving radio waves between itself and the base station ex110; a camera unit ex465 for capturing images and still images; and a display unit ex458 for displaying decoded data such as images captured by the camera unit ex465 and images received by the antenna ex450. The smartphone ex115 further includes: an operation unit ex466, which is a touch panel, etc.; a sound output unit ex457, which is a speaker for outputting sound or audio; a sound input unit ex456, which is a microphone for inputting sound, etc.; a memory unit ex467, which can store encoded or decoded data such as captured images or still images, recorded audio, received images or still images, and emails; and a slot unit ex464, which is the interface with SIMex468. The aforementioned SIMex468 can be used to authenticate specific users for access to various data, including network data. Alternatively, external memory can be used to replace the memory unit ex467.

[0672] The main control unit ex460, power circuit unit ex461, operation input control unit ex462, image signal processing unit ex455, camera interface unit ex463, display control unit ex459, modulation / demodulation unit ex452, multiplexing / splitting unit ex453, audio signal processing unit ex454, slot unit ex464, and memory unit ex467, which can coordinately control the display unit ex458 and operation unit ex466, are connected through a synchronous bus ex470.

[0673] When the power button is turned on by the user, the power circuit section ex461 starts the smartphone ex115 into an operational state and supplies power to various parts from the battery pack.

[0674] The smartphone ex115 performs call and data communication processing under the control of the main control unit ex460, which includes a CPU, ROM, and RAM. During a call, the audio signal received by the audio input unit ex456 is converted into a digital audio signal by the audio signal processing unit ex454, then subjected to spread spectrum processing by the modulation / demodulation unit ex452, followed by digital-to-analog conversion and frequency conversion processing by the transmit / receive unit ex451. The resulting signal is then transmitted through the antenna ex450. Similarly, the received data is amplified, subjected to frequency conversion and analog-to-digital conversion processing, despread spectrum processing by the modulation / demodulation unit ex452, converted into an analog audio signal by the audio signal processing unit ex454, and then output from the audio output unit ex457. In data communication mode, text, still images, or video data can be sent out under the control of the main control unit ex460 via the operation input control unit ex462, according to the operation of the main unit ex466 and other components. The same sending and receiving processes are performed. In data communication mode, when sending images, still images, or images and sound, the image signal processing unit ex455 uses the motion image encoding method shown in the above embodiments to compress and encode the image signal stored in the memory unit ex467 or the image signal input from the camera unit ex465, and sends the encoded image data to the multiplexing / splitting unit ex453. The sound signal processing unit ex454 encodes the sound signal received by the sound input unit ex456 when the camera unit ex465 captures images or still images, and sends the encoded sound data to the multiplexing / splitting unit ex453. The multiplexing / splitting unit ex453 multiplexes the encoded image data and encoded sound data in a predetermined manner, then performs modulation and conversion processing on the modulation / demodulation unit (modulation / demodulation circuit unit) ex452 and the transmitting / receiving unit ex451, and transmits it through the antenna ex450. The predetermined method can also be determined in advance.

[0675] When receiving images attached to emails or chat rooms, or images linked to web pages, in order to decode the multiplexed data received through the antenna ex450, the multiplexing / demultiplexing unit ex453 separates the multiplexed data into bit streams of image data and bit streams of audio data by demultiplexing the multiplexed data. The encoded image data is then supplied to the image signal processing unit ex455 via the synchronization bus ex470, and the encoded audio data is supplied to the audio signal processing unit ex454. The image signal processing unit ex455 decodes the image signal using a motion picture decoding method corresponding to the motion picture encoding method shown in the above embodiments, and displays the image or still image contained in the linked motion picture file on the display unit ex458 via the display control unit ex459. The audio signal processing unit ex454 decodes the audio signal and outputs sound from the audio output unit ex457. Due to the increasing popularity of real-time streaming, the playback of sound may be socially inappropriate depending on the user's situation. Therefore, as an initial value, it is ideal to play only video data without playing sound signals. Alternatively, sound can be played synchronously only when the user clicks on the video data.

[0676] Furthermore, although the EX115 smartphone is used as an example here, as a terminal, besides a transmitting and receiving terminal with both an encoder and a decoder, it can also be conceived as a transmitting terminal with only an encoder, or a receiving terminal with only a decoder, among other installation forms. The case of multiplexing video data with audio data in a digital broadcasting system has already been explained. However, in addition to audio data, text data associated with the video can also be multiplexed into the multiplexed data. Furthermore, it is also possible to receive or transmit the video data itself, rather than multiplexed data.

[0677] Furthermore, while the case of the CPU's main control unit (ex460) controlling encoding or decoding processing has been explained, many terminals also possess GPUs. Therefore, a configuration can also be implemented as follows: by utilizing shared memory between the CPU and GPU, or memory with managed addresses for shared use, the GPU's performance can be leveraged to process large areas at once. This shortens encoding time, ensures real-time performance, and achieves low latency. It is particularly efficient when using the GPU instead of the CPU to perform motion search, deblocking filtering, SAO (Sample Adaptive Offset), and conversion / quantization processing on an image-by-image basis.

[0678] [Industrial Applicability] This disclosure can be applied to, for example, televisions, digital video recorders, car navigation systems, mobile phones, digital cameras, digital video cameras, video conferencing systems, or electronic mirrors.

[0679] Blocks 10 to 23 100: Encoding device 102: Segmentation 104: Subtraction Section 106: Conversion Section 108: Quantitative Department 110: Entropy Coding Department 112,204: Anti-quantitative department 114,206: Reverse conversion section 116,208: Addition Department 118,210: Block memory 120, 212: Loop Filtering Section 122,214: Frame memory 124,216: Internal Forecasting Department 126,218: Interim Prediction Department 128,220: Predictive Control Department 200: Decoding device 202: Entropy Decoding Department 1201: Boundary Determination Unit 1202, 1204, 1206: Switches 1203: Filtering Determination Unit 1205: Filtering Processing Department 1207: Filter Characteristics Determination Unit 1208: Processing and Judgment Department a1, b1: Processor a2, b2: Memory ex100: Content Supply System ex101: Internet ex102: Internet Service Provider ex103: Streaming Server ex104: Communication Network EX106 to EX110: Base Station ex111: Computer ex112: Game console ex113: Camera ex114: Home Appliances ex115: Smartphone ex116: Satellite ex117: Airplane ex450: Antenna ex451: Transmitter / Receiver Unit ex452: Modulation / Demodulation Unit ex453: Multiplexing / Separation Section ex454: Audio Signal Processing Unit EX455: Image Signal Processing Unit ex456: Audio Input Section ex457: Audio Output Section ex458: Display Section ex459: Display Control Unit (LCD Control Unit) EX460: Main Control Unit ex461: Power Supply Circuit Section ex462: Operation Input Control Unit ex463: Camera Interface ex464: Slot section ex465: Camera Department ex466: Operations Department ex467: Memory Section ex468:SIM ex500:LSI S1, S2a, S2b, Sa_1 to Sa_10, Sb_1, Sc_1a to Sc_2, Sd_1a, Sd_1b, Sd_2, Sd_3, Se_1 to Se_4, Sf_1 to Sf_5, Sg_1 to Sg_5, Sh_1 to Sh_3, Si_1 to Si_5, Sj_1 to Sj_6, Sl_1 to Sl_4, Sm_1, Sm_2, Sn_1 to Sn_5, So_1 to So_4, Sp_1 to Sp_7, Sq_1, Sr_1, Sr_2a to Sr_2c, Ss_1 to Ss_5, S11 to S13, S14a to S14d, S15a, S15b, S16a to S16d, S17a, S17b: Steps PL1 to PL7: Piping treatment St1: Entropy Decoding Processing St21: MV Export Processing St22: Memory Transfer Processing St3: DMVR Processing St4:MC processing St51a:LIC processing St51b: Predictive graph filtering St51c: BIO Processing St51d:OBMC Processing St52: Intrapredictive processing St53: Internal / Internal Mixed Treatment St54: Inverse quantization and inverse conversion processing St55, St57: Mode switching processing St56: Refactoring Process St6: Loop Filtering Process TD0, TD1: Distance

Claims

1. An encoding method comprising: in a prediction mode, generating a first prediction image of a current block to be processed based on a derived motion vector, and applying an update process to the first prediction image to generate a final prediction image of the current block, wherein a plurality of candidates for the update process include a first process and a second process, wherein the first process is a bidirectional optical flow (BDOF) process, and the second process is a process of mixing the first prediction image with a second prediction image generated in the intraprediction of the current block, wherein the first process and the second process are applied exclusively in the application of the update process.

2. A decoding method comprising: in a prediction mode, generating a first prediction image of a current block to be processed based on a derived motion vector, and applying an update process to the first prediction image to generate a final prediction image of the current block, wherein a plurality of candidates for the update process include a first process and a second process, wherein the first process is a bidirectional optical flow (BDOF) process, and the second process is a process of mixing the first prediction image with a second prediction image generated in the intraprediction of the current block, wherein the first process and the second process are applied exclusively in the application of the update process.

3. A non-transitory computer-readable medium storing a bitstream, the bitstream including decoding information that causes a decoding device to perform a decoding process, the decoding process comprising: in a prediction mode, generating a first prediction image of a current block to be processed based on a derived motion vector, and applying an update process to the first prediction image to generate a final prediction image of the current block, the plurality of candidates for the update process including a first process and a second process, the first process being a bidirectional optical flow (BDOF) process, the second process being a process of mixing the first prediction image with a second prediction image generated in the intraprediction of the current block, and the first process and the second process being applied exclusively in the application of the update process.

Images