Encoder and decoder
Patent Information
- Application Number
- IN202248048704
- Authority / Receiving Office
- IN · IN
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-02-13
- Filing Date
- 2022-08-26
- Publication Date
- 2026-10-13
- Estimated Expiration
- 2041-02-01
Smart Images

Figure 00000183_0000 
Figure 00000184_0000 
Figure 00000185_0000
Abstract
Description
[Technical Field]
[0001] The present disclosure relates to an encoder, a decoder, anencoding method, and a decoding method.[Background Art]
[0002] With advancement in video coding technology, from H.261 andMPEG-1 to H.264 / AVC (Advanced Video Coding), MPEG-LA, H.265 / HEVC(High Efficiency Video Coding) and H.266 / VVC (Versatile Video Codec),there remains a constant need to provide improvements andoptimizations to the video coding technology to process an ever-increasingamount of digital video data in various applications. The presentdisclosure relates to further developments, improvements andoptimizations in video coding.
[0003] Note that Non patent Literature (NPL) 1 relates to one example ofa conventional standard regarding the above-described video codingtechnology.[Citation List][Non Patent Literature]
[0004] [NPL 1] H.265 (ISO / IEC 23008-2 HEVC) / HEVC (High Efficiency VideoCoding)[Summary of Invention][Technical Problem]
[0005] Regarding the coding scheme as described above, proposals of newschemes have been desired in order to (i) improve coding efficiency,enhance image quality, reduce processing amounts, reduce circuit scales,or (ii) appropriately select an element or an option. The element is, forexample, a filter, a block, a size, a motion vector, a reference picture, or areference block.
[0006] The present disclosure provides, for example, a configuration or amethod which can contribute to at least one of increase in codingefficiency, increase in image quality, reduction in processing amount,reduction in circuit scale, appropriate selection of an element or anoperation, etc. It is to be noted that the present disclosure mayencompass possible configurations or methods which can contribute toadvantages other than the above advantages.[Solution to Problem]
[0007] For example, an encoder according to an aspect of the presentdisclosure includes circuitry and memory coupled to the circuitry. Inoperation, the circuitry: switches whether to encode a first flag, accordingto whether a multilayer coding scheme is used to encode one or morepictures, the first flag indicating whether use of one or more interlayerreference pictures for interlayer prediction is allowed.
[0008] In video coding technology, new methods are desired to be proposedin order to improve coding efficiency, enhance image quality, and reducecircuit scales.
[0009] Each of embodiments, or each of part of constituent elements andmethods in the present disclosure enables, for example, at least one of thefollowing: improvement in coding efficiency, enhancement in imagequality, reduction in processing amount of encoding / decoding, reductionin circuit scale, improvement in processing speed of encoding / decoding,etc. Alternatively, each of embodiments, or each of part of constituentelements and methods in the present disclosure enables, in encoding anddecoding, appropriate selection of an element or an operation. Theelement is, for example, a filter, a block, a size, a motion vector, areference picture, or a reference block. It is to be noted that the presentdisclosure includes disclosure regarding configurations or methods whichmay provide advantages other than the above-described ones. Examplesof such configurations or methods include a configuration or a method forimproving coding efficiency while reducing increase in processingamount.
[0010] Additional benefits and advantages according to an aspect of thepresent disclosure will become appa rent from the specification anddrawings. The benefits and / or advantages may be individually obtainedby the various embodiments and features of the specification anddrawings, and not all of which need to be provided in order to obtain oneor more of such benefits and / or advantages.
[0011] It is to be noted that these general or specific aspects may beimplemented using a system, an integrated circuit, a computer program,or a computer readable medium (recording medium) such as a CD-ROM,or any combination of systems, methods, integrated circuits, computerprograms, and media.[Advantageous Effects of Invention]
[0012] A configuration or method according to an aspect of the presentdisclosure enables, for example, at least one of the following:improvement in coding efficiency, enhancement in image quality,reduction in processing amount, reduction in circuit scale, improvementin processing speed, appropriate selection of an element or an operation,etc. It is to be noted that the configuration or method according to anaspect of the present disclosure may provide advantages other than theabove-described ones.[Brief Description of Drawings]
[0013] [FIG. 1]FIG. 1 is a schematic diagram illustrating one example of aconfiguration of a transmission system according to an embodiment.[FIG. 2]FIG. 2 is a diagram illustrating one example of a hierarchicalstructure of data in a stream.[FIG. 3]FIG. 3 is a diagram illustrating one example of a sliceconfiguration.[FIG. 4]FIG. 4 is a diagram illustrating one example of a tileconfiguration.[FIG. 5]FIG. 5 is a diagram illustrating one example of an encodingstructure in scalable encoding.[FIG. 6]FIG. 6 is a diagram illustrating one example of an encodingstructure in scalable encoding.[FIG. 7]FIG. 7 is a block diagram illustrating one example of a functionalconfiguration of an encoder according to an embodiment.[FIG. 8]FIG. 8 is a block diagram illustrating a mounting example of theencoder.[FIG. 9]FIG. 9 is a flow chart illustrating one example of an overallencoding process performed by the encoder.[FIG. 10]FIG. 10 is a diagram illustrating one example of block splitting.[FIG. 11]FIG. 11 is a diagram illustrating one example of a functionalconfiguration of a splitter.[FIG. 12]FIG. 12 is a diagram illustrating examples of splitting patterns.[FIG. 13A]FIG. 13A is a diagram illustrating one example of a syntax tree ofa splitting pattern.[FIG. 13B]FIG. 13B is a diagram illustrating another example of a syntaxtree of a splitting pattern.[FIG. 14]FIG. 14 is a chart illustrating transform basis functions for eachtransform type.[FIG. 15]FIG. 15 is a diagram illustrating examples of SVT.[FIG. 16]FIG. 16 is a flow chart illustrating one example of a processperformed by a transformer.[FIG. 17]FIG. 17 is a flow chart illustrating another example of a processperformed by the transformer.[FIG. 18]FIG. 18 is a block diagram illustrating one example of a functionalconfiguration of a quantizer.[FIG. 19]FIG. 19 is a flow chart illustrating one example of quantizationperformed by the quantizer.[FIG. 20]FIG. 20 is a block diagram illustrating one example of a functionalconfiguration of an entropy encoder.[FIG. 21]FIG. 21 is a diagram illustrating a flow of CABAC in the entropyencoder.[FIG. 22]FIG. 22 is a block diagram illustrating one example of a functionalconfiguration of a loop filter.[FIG. 23A]FIG. 23A is a diagram illustrating one example of a filter shapeused in an adaptive loop filter (ALF).[FIG. 23B]FIG. 23B is a diagram illustrating another example of a filtershape used in an ALF.[FIG. 23C]FIG. 23C is a diagram illustrating another example of a filtershape used in an ALF.[FIG. 23D]FIG. 23D is a diagram illustrating an example where Y samples(first component) are used for a cross component ALF (CCALF) for Cb anda CCALF for Cr (components different from the first component).[FIG. 23E]FIG. 23E is a diagram illustrating a diamond shaped filter.[FIG. 23F]FIG. 23F is a diagram illustrating an example for a joint chromaCCALF (JC-CCALF).[FIG. 23G]FIG. 23G is a diagram illustrating an example for JC-CCALFweight index candidates.[FIG. 24]FIG. 24 is a block diagram illustrating one example of a specificconfiguration of a loop filter which functions as a DBF.[FIG. 25]FIG. 25 is a diagram illustrating an example of a deblocking filterhaving a symmetrical filtering characteristic with respect to a blockboundary.[FIG. 26]FIG. 26 is a diagram for illustrating a block boundary on which adeblocking filter process is performed.[FIG. 27]FIG. 27 is a diagram illustrating examples of Bs values.[FIG. 28]FIG. 28 is a flow chart illustrating one example of a processperformed by a predictor of the encoder.[FIG. 29]FIG. 29 is a flow chart illustrating another example of a processperformed by the predictor of the encoder.[FIG. 30]FIG. 30 is a flow chart illustrating another example of a processperformed by the predictor of the encoder.[FIG. 31]FIG. 31 is a diagram illustrating one example of sixty-seven intraprediction modes used in intra prediction.[FIG. 32]FIG. 32 is a flow chart illustrating one example of a processperformed by an intra predictor.[FIG. 33]FIG. 33 is a diagram illustrating examples of reference pictures.[FIG. 34]FIG. 34 is a diagram illustrating examples of reference picturelists.[FIG. 35]FIG. 35 is a flow chart illustrating a basic processing flow of interprediction.[FIG. 36]FIG. 36 is a flow chart illustrating one example of MV derivation.[FIG. 37]FIG. 37 is a flow chart illustrating another example of MVderivation.[FIG. 38A]FIG. 38A is a diagram illustrating one example of categorization ofmodes for MV derivation.[FIG. 38B]FIG. 38B is a diagram illustrating one example of categorization ofmodes for MV derivation.[FIG. 39]FIG. 39 is a flow chart illustrating an example of inter predictionby normal inter mode.[FIG. 40]FIG. 40 is a flow chart illustrating an example of inter predictionby normal merge mode.[FIG. 41]FIG. 41 is a diagram for illustrating one example of an MVderivation process by normal merge mode.[FIG. 42]FIG. 42 is a diagram for illustrating one example of an MVderivation process by HMVP mode.[FIG. 43]FIG. 43 is a flow chart illustrating one example of frame rate upconversion (FRUC).[FIG. 44]FIG. 44 is a diagram for illustrating one example of patternmatching (bilateral matching) between two blocks located along a motiontrajectory.[FIG. 45]FIG. 45 is a diagram for illustrating one example of patternmatching (template matching) between a template in a current pictureand a block in a reference picture.[FIG. 46A]FIG. 46A is a diagram for illustrating one example of MVderivation in units of a sub-block in affine mode in which two controlpoints are used.[FIG. 46B]FIG. 46B is a diagram for illustrating one example of MVderivation in units of a sub-block in affine mode in which three controlpoints are used.[FIG. 47A]FIG. 47A is a conceptual diagram for illustrating one example ofMV derivation at control points in an affine mode.[FIG. 47B]FIG. 47B is a conceptual diagram for illustrating one example ofMV derivation at control points in an affine mode.[FIG. 47C]FIG. 47C is a conceptual diagram for illustrating one example ofMV derivation at control points in an affine mode.[FIG. 48A]FIG. 48A is a diagram for illustrating an affine mode in which twocontrol points are used.[FIG. 48B]FIG. 48B is a diagram for illustrating an affine mode in whichthree control points are used.[FIG. 49A]FIG. 49A is a conceptual diagram for illustrating one example of amethod for MV derivation at control points when the number of controlpoints for an encoded block and the number of control points for a currentblock are different from each other.[FIG. 49B]FIG. 49B is a conceptual diagram for illustrating another exampleof a method for MV derivation at control points when the number ofcontrol points for an encoded block and the number of control points for acurrent block are different from each other.[FIG. 50]FIG. 50 is a flow chart illustrating one example of a process inaffine merge mode.[FIG. 51]FIG. 51 is a flow chart illustrating one example of a process inaffine inter mode.[FIG. 52A]FIG. 52A is a diagram for illustrating generation of two triangularprediction images.[FIG. 52B]FIG. 52B is a conceptual diagram illustrating examples of a firstportion of a first partition and first and second sets of samples.[FIG. 52C]FIG. 52C is a conceptual diagram illustrating a first portion of afirst partition.[FIG. 53]FIG. 53 is a flow chart illustrating one example of a triangle mode.[FIG. 54]FIG. 54 is a diagram illustrating one example of an ATMVP modein which an MV is derived in units of a sub-block.[FIG. 55]FIG. 55 is a diagram illustrating a relationship between a mergemode and dynamic motion vector refreshing (DMVR).[FIG. 56]FIG. 56 is a conceptual diagram for illustrating one example ofDMVR.[FIG. 57]FIG. 57 is a conceptual diagram for illustrating another example ofDMVR for determining an MV.[FIG. 58A]FIG. 58A is a diagram illustrating one example of motionestimation in DMVR.[FIG. 58B]FIG. 58B is a flow chart illustrating one example of motionestimation in DMVR.[FIG. 59]FIG. 59 is a flow chart illustrating one example of generation of aprediction image.[FIG. 60]FIG. 60 is a flow chart illustrating another example of generationof a prediction image.[FIG. 61]FIG. 61 is a flow chart illustrating one example of a correctionprocess of a prediction image by overlapped block motion compensation(OBMC).[FIG. 62]FIG. 62 is a conceptual diagram for illustrating one example of aprediction image correction process by OBMC.[FIG. 63]FIG. 63 is a diagram for illustrating a model assuming uniformlinear motion.[FIG. 64]FIG. 64 is a flow chart illustrating one example of inter predictionaccording to BIO.[FIG. 65]FIG. 65 is a diagram illustrating one example of a functionalconfiguration of an inter predictor which performs inter predictionaccording to BIO.[FIG. 66A]FIG. 66A is a diagram for illustrating one example of a predictionimage generation method using a luminance correction process by localillumination compensation (LIC).[FIG. 66B]FIG. 66B is a flow chart illustrating one example of a predictionimage generation method using a luminance correction process by LIC.[FIG. 67]FIG. 67 is a block diagram illustrating a functional configurationof a decoder according to an embodiment.[FIG. 68]FIG. 68 is a block diagram illustrating a mounting example of adecoder.[FIG. 69]FIG. 69 is a flow chart illustrating one example of an overalldecoding process performed by the decoder.[FIG. 70]FIG. 70 is a diagram illustrating a relationship between a splittingdeterminer and other constituent elements.[FIG. 71]FIG. 71 is a block diagram illustrating one example of a functionalconfiguration of an entropy decoder.[FIG. 72]FIG. 72 is a diagram illustrating a flow of CABAC in the entropydecoder.[FIG. 73]FIG. 73 is a block diagram illustrating one example of a functionalconfiguration of an inverse quantizer.[FIG. 74]FIG. 74 is a flow chart illustrating one example of inversequantization performed by the inverse quantizer.[FIG. 75]FIG. 75 is a flow chart illustrating one example of a processperformed by an inverse transformer.[FIG. 76]FIG. 76 is a flow chart illustrating another example of a processperformed by the inverse transformer.[FIG. 77]FIG. 77 is a block diagram illustrating one example of a functionalconfiguration of a loop filter.[FIG. 78]FIG. 78 is a flow chart illustrating one example of a processperformed by a predictor of the decoder.[FIG. 79]FIG. 79 is a flow chart illustrating another example of a processperformed by the predictor of the decoder.[FIG. 80A]FIG. 80A is a flow chart illustrating a portion of other example of aprocess performed by the predictor of the decoder.[FIG. 80B]FIG. 80B is a flow chart illustrating the remaining portion of theother example of the process performed by the predictor of the decoder.[FIG. 81]FIG. 81 is a diagram illustrating one example of a processperformed by an intra predictor of the decoder.[FIG. 82]FIG. 82 is a flow chart illustrating one example of MV derivationin the decoder.[FIG. 83]FIG. 83 is a flow chart illustrating another example of MVderivation in the decoder.[FIG. 84]FIG. 84 is a flow chart illustrating an example of inter predictionby normal inter mode in the decoder.[FIG. 85]FIG. 85 is a flow chart illustrating an example of inter predictionby normal merge mode in the decoder.[FIG. 86]FIG. 86 is a flow chart illustrating an example of inter predictionby FRUC mode in the decoder.[FIG. 87]FIG. 87 is a flow chart illustrating an example of inter predictionby affine merge mode in the decoder.[FIG. 88]FIG. 88 is a flow chart illustrating an example of inter predictionby affine inter mode in the decoder.[FIG. 89]FIG. 89 is a flow chart illustrating an example of inter predictionby triangle mode in the decoder.[FIG. 90]FIG. 90 is a flow chart illustrating an example of motionestimation by DMVR in the decoder.[FIG. 91]FIG. 91 is a flow chart illustrating one specific example of motionestimation by DMVR in the decoder.[FIG. 92]FIG. 92 is a flow chart illustrating one example of generation of aprediction image in the decoder.[FIG. 93]FIG. 93 is a flow chart illustrating another example of generationof a prediction image in the decoder.[FIG. 94]FIG. 94 is a flow chart illustrating another example of correction ofa prediction image by OBMC in the decoder.[FIG. 95]FIG. 95 is a flow chart illustrating another example of correction ofa prediction image by BIO in the decoder.[FIG. 96]FIG. 96 is a flow chart illustrating another example of correction ofa prediction image by LIC in the decoder.[FIG. 97]FIG. 97 is a diagram indicating one example of a syntax structurefor notifying of information used to configure a reference picture list.[FIG. 98]FIG. 98 is a diagram indicating one example of an SPS syntaxstructure for notifying of an interlayer reference picture present flag inAspect 1.[FIG. 99]FIG. 99 is a diagram indicating one example of an operation fornotifying of the interlayer reference picture present flag in Aspect 1.[FIG. 100]FIG. 100 is a diagram indicating one example of an SPS syntax fornotifying of an interlayer reference picture present flag in Aspect 2.[FIG. 101]FIG. 101 is a flow chart indicating an operation that is performedby an encoder according to an embodiment.[FIG. 102]FIG. 102 is a flow chart indicating a more detailed operation thatis performed by the encoder according to the embodiment.[FIG. 103]FIG. 103 is a flow chart indicating an operation that is performedby a decoder according to the embodiment.[FIG. 104]FIG. 104 is a flow chart indicating a more detailed operation thatis performed by the decoder according to the embodiment.[FIG. 105]FIG. 105 is a diagram illustrating an overall configuration of acontent providing system for implementing a content distribution service.[FIG. 106]FIG. 106 is a diagram illustrating an example of a display screenof a web page.[FIG. 107]FIG. 107 is a diagram illustrating an example of a display screenof a web page.[FIG. 108]FIG. 108 is a diagram illustrating one example of a smartphone.[FIG. 109]FIG. 109 is a block diagram illustrating an example of aconfiguration of a smartphone.[Description of Embodiments]
[0014] [Introduction]In encoding of a video, there is a case in which a multilayer codingscheme that is for example a spatially scalable coding scheme, amultiview coding scheme, or the like is used. In the multilayer codingscheme, in order to increase the coding efficiency, interlayer predictionmay be performed using one or more interlayer reference pictures whichare reference pictures in a reference layer different from a current layer.The one or more interlayer reference pictures used for the interlayerprediction may be included in a reference picture list that can include areference picture used for normal inter prediction.
[0015] For example, whether the one or more interlayer referencepictures are included in the reference picture list can be controlled basedon a flag indicating whether use of the one or more interlayer referencepictures is allowed, a flag indicating whether the one or more interlayerreference pictures are used, etc. These flags may be encoded or decodedas coding parameters. In this way, information for controlling interlayerprediction is shared between an encoder and a decoder.
[0016] However, there is a possibility that encoding and decoding theinformation for controlling the interlayer prediction as the codingparameters increase the amount of codes.
[0017] In view of this, for example, an encoder according to an aspect ofthe present disclosure includes circuitry and memory coupled to thecircuitry. In operation, the circuitry: switches whether to encode a firstflag, according to whether a multilayer coding scheme is used to encodeone or more pictures, the first flag indicating whether use of one or moreinterlayer reference pictures for interlayer prediction is allowed.
[0018] In this way, there is a possibility that the encoder is capable ofskipping encoding of the first flag indicating whether the use of the one ormore interlayer reference pictures is allowed, according to whether themultilayer coding scheme is used. Accordingly, there is a possibility thatthe encoder is capable of contributing reduction in the amount of codesregarding information for controlling the interlayer prediction.
[0019] In addition, for example, the circuitry: encodes the first flag whenthe multilayer coding scheme is used to encode the one or more pictures;and does not encode the first flag when the multilayer coding scheme isnot used to encode the one or more pictures.
[0020] In this way, there is a possibility that the encoder is capable ofadaptively controlling whether the use of the one or more interlayerreference pictures is allowed, when the multilayer coding scheme is used.In other words, in this case, there is a possibility that the encoder iscapable of adaptively controlling whether the use of the interlayerprediction is allowed.
[0021] Furthermore, there is a possibility that the encoder is capable ofskipping encoding of the first flag because the one or more interlayerreference pictures are not used when the multilayer coding scheme is notused. Accordingly, there is a possibility that the encoder is capable ofcontributing reduction in the amount of codes. It is to be noted thatunuse of the multilayer coding scheme corresponds to use of a single layercoding scheme.
[0022] In addition, for example, the use of the one or more interlayerreference pictures is allowed when (i) the first flag is encoded and (ii) thefirst flag indicates that the use of the one or more interlayer referencepictures is allowed, the use of the one or more interlayer referencepictures is not allowed when (i) the first flag is encoded and (ii) the firstflag indicates that the use of the one or more interlayer reference picturesis not allowed, and the use of the one or more interlayer referencepictures is not allowed when the first flag is not encoded.
[0023] In this way, there is a possibility that the encoder is capable ofcontrolling whether the one or more interlayer reference pictures areused so as to conform to the first flag when the first flag is encoded. Inaddition, there is a possibility that the encoder is capable of simplifyingthe process because the use of the one or more interlayer referencepictures is not allowed when the first flag is not encoded.
[0024] In addition, for example, the circuitry encodes a second flag whichindicates whether any of the one or more interlayer reference pictures isused as a reference picture that is included in a reference picture list,when the use of the one or more interlayer reference pictures is allowed.
[0025] In this way, there is a possibility that the encoder is capable ofadaptively controlling whether the use of the one or more interlayerreference pictures is allowed as the reference pictures in the referencepicture list, when the use of the one or more interlayer reference picturesis allowed. Accordingly, in this case, there is a possibility that theencoder is capable of appropriately encoding a current picture eitherusing the one or more interlayer reference pictures or without using theone or more interlayer reference pictures.
[0026] In addition, for example, a reference picture different from any ofthe one or more interlayer reference pictures is used as the referencepicture that is included in the reference picture list, when (I-i) the secondflag is encoded and (I-ii) the second flag indicates that any of the one ormore interlayer reference pictures is not used as the reference picturethat is included in the reference picture list, or (II) the second flag is notto be encoded.
[0027] In this way, there is a possibility that the encoder is capable ofusing the reference picture other than the one or more interlayerreference pictures either when the use of the one or more interlayerreference pictures is not allowed or when the one or more interlayerreference pictures are not used. Accordingly, there is a possibility thatthe encoder is capable of appropriately encoding the current picture usingthe reference picture other than the one or more interlayer referencepictures.
[0028] In addition, for example, the circuitry switches whether to encodethe first flag, according to a video parameter set ID corresponding towhether the multilayer coding scheme is used to encode the one or morepictures, the video parameter set ID is 0 or greater, when the videoparameter set ID is 0, the multilayer coding scheme is not used to encodethe one or more pictures, and when the video parameter set ID is greaterthan 0, the multilayer coding scheme is used to encode the one or morepictures, and the video parameter set ID indicates an identifier of a videoparameter set for encoding the one or more pictures using the multilayercoding scheme.
[0029] In this way, there is a possibility that the encoder is capable ofskipping encoding of the first flag according to the video parameter set ID.Accordingly, there is a possibility that the encoder is capable ofcontributing reduction in the amount of codes regarding the informationfor controlling the interlayer prediction. Furthermore, there is apossibility that the encoder is capable of appropriately encoding the oneor more pictures using the video parameter set identified according to thevideo parameter set ID and the multilayer coding scheme, when the videoparameter set ID is greater than 0.
[0030] Furthermore, for example, an encoder according to an aspect of thepresent disclosure includes circuitry and memory coupled to the circuitry.In operation, the circuitry: switches whether to decode a first flag,according to whether a multilayer coding scheme is used to decode one ormore pictures, the first flag indicating whether use of one or moreinterlayer reference pictures for interlayer prediction is allowed.
[0031] In this way, there is a possibility that the decoder is capable ofskipping decoding of the first flag indicating whether the use of the one ormore interlayer reference pictures is allowed, according to whether themultilayer coding scheme is used. Accordingly, there is a possibility thatthe decoder is capable of contributing reduction in the amount of codesregarding the information for controlling the interlayer prediction.
[0032] In addition, for example, the circuitry: decodes the first flag whenthe multilayer coding scheme is used to decode the one or more pictures;and does not decode the first flag when the multilayer coding scheme isnot used to decode the one or more pictures.
[0033] In this way, there is a possibility that the decoder is capable ofadaptively controlling whether the use of the one or more interlayerreference pictures is allowed, when the multilayer coding scheme is used.In other words, in this case, there is a possibility that the decoder iscapable of adaptively controlling whether the use of the interlayerprediction is allowed.
[0034] Furthermore, there is a possibility that the decoder is capable ofskipping decoding of the first flag because the one or more interlayerreference pictures are not used when the multilayer coding scheme is notused. Accordingly, there is a possibility that the decoder is capable ofcontributing reduction in the amount of codes. It is to be noted thatunuse of the multilayer coding scheme corresponds to use of a single layercoding scheme.
[0035] In addition, for example, the use of the one or more interlayerreference pictures is allowed when (i) the first flag is decoded and (ii) thefirst flag indicates that the use of the one or more interlayer referencepictures is allowed, the use of the one or more interlayer referencepictures is not allowed when (i) the first flag is decoded and (ii) the firstflag indicates that the use of the one or more interlayer reference picturesis not allowed, and the use of the one or more interlayer referencepictures is not allowed when the first flag is not decoded.
[0036] In this way, there is a possibility that the decoder is capable ofcontrolling whether the one or more interlayer reference pictures areused so as to conform to the first flag when the first flag is decoded. Inaddition, there is a possibility that the decoder is capable of simplifyingthe process because the use of the one or more interlayer referencepictures is not allowed when the first flag is not decoded.
[0037] In addition, for example, the circuitry decodes a second flag whichindicates whether any of the one or more interlayer reference pictures isused as a reference picture that is included in a reference picture list,when the use of the one or more interlayer reference pictures is allowed.
[0038] In this way, there is a possibility that the decoder is capable ofadaptively controlling whether the use of the one or more interlayerreference pictures is allowed as the reference pictures in the referencepicture list, when the use of the one or more interlayer reference picturesis allowed. Accordingly, in this case, there is a possibility that thedecoder is capable of appropriately decoding a current picture eitherusing the interlayer reference pictures or without using the interlayerreference pictures.
[0039] In addition, for example, a reference picture different from any ofthe one or more interlayer reference pictures is used as the referencepicture that is included in the reference picture list, when (I-i) the secondflag is decoded and (I-ii) the second flag indicates that any of the one ormore interlayer reference pictures is not used as the reference picturethat is included in the reference picture list, or (II) the second flag is notto be decoded.
[0040] In this way, there is a possibility that the decoder is capable ofusing the reference picture other than the one or more interlayerreference pictures either when the use of the one or more interlayerreference pictures is not allowed or when the one or more interlayerreference pictures are not used. Accordingly, there is a possibility thatthe decoder is capable of appropriately decoding the current picture usingthe reference picture other than the one or more interlayer referencepictures.
[0041] In addition, for example, the circuitry switches whether to decodethe first flag, according to a video parameter set ID corresponding towhether the multilayer coding scheme is used to decode the one or morepictures, the video parameter set ID is 0 or greater, when the videoparameter set ID is 0, the multilayer coding scheme is not used to decodethe one or more pictures, and when the video parameter set ID is greaterthan 0, the multilayer coding scheme is used to decode the one or morepictures, and the video parameter set ID indicates an identifier of a videoparameter set for decoding the one or more pictures using the multilayercoding scheme.
[0042] In this way, there is a possibility that the decoder is capable ofskipping decoding of the first flag according to the video parameter set ID.Accordingly, there is a possibility that the decoder is capable ofcontributing reduction in the amount of codes regarding information forcontrolling the interlayer prediction. Furthermore, there is a possibilitythat the decoder is capable of appropriately decoding the one or morepictures using the video parameter set identified according to the videoparameter set ID and the multilayer coding scheme, when the videoparameter set ID is greater than 0.
[0043] In addition, for example, an encoding method according to anaspect of the present disclosure is an encoding method which is performedby an encoder and includes: determining whether a multilayer codingscheme is used to encode one or more pictures; and switching whether toencode a first flag, according to whether the multilayer coding scheme isused to encode the one or more pictures, the first flag indicating whetheruse of one or more interlayer reference pictures for interlayer predictionis allowed.
[0044] In this way, there is a case in which encoding of the first flagindicating whether the use of the one or more interlayer referencepictures is allowed can be skipped, according to whether the multilayercoding scheme is used. Accordingly, there is a case in which contributionto reduction in the amount of codes can be made regarding information forcontrolling the interlayer prediction.
[0045] In addition, for example, a decoding method according to an aspectof the present disclosure is a decoding method which is performed by adecoder and includes: determining whether a multilayer coding scheme isused to decode one or more pictures; and switching whether to decode afirst flag, according to whether the multilayer coding scheme is used todecode the one or more pictures, the first flag indicating whether use ofone or more interlayer reference pictures for interlayer prediction isallowed.
[0046] In this way, there is a case in which decoding of the first flagindicating whether the use of the one or more interlayer referencepictures is allowed can be skipped, according to whether the multilayercoding scheme is used. Accordingly, there is a case in which contributionto reduction in the amount of codes can be made regarding information forcontrolling the interlayer prediction.
[0047] In addition, for example, an encoder according to an aspect of thepresent disclosure includes an inputter, a splitter, an intra predictor, aninter predictor, a loop filter unit, a transformer, a quantizer, an entropyencoder, and outputter.
[0048] A current picture is input to the inputter. The splitter splits thecurrent picture into a plurality of blocks.
[0049] The intra predictor generates prediction signals of a current blockincluded in the current picture, using a reference image included in thecurrent picture. The inter predictor generates prediction signals of acurrent block included in the current picture, using a reference imageincluded in a reference picture different from the current picture. Theloop filter unit applies a filter to a reconstructed block in a current blockincluded in the current picture.
[0050] The transformer generates transformed coefficients bytransforming prediction errors between original signals of the currentblock included in the current picture and prediction signals generated byeither the intra predictor or the inter predictor. The quantizer quantizesthe transform coefficients to generate quantized coefficients. Theentropy encoder applies variable length encoding on the quantizedcoefficients to generate an encoded bitstream. The quantizedcoefficients to which the variable length encoding has been applied andthe encoded bitstream including control information are then output fromthe outputter.
[0051] In addition, for example, in operation, the entropy encoderswitches whether to encode a first flag, according to whether a multilayercoding scheme is used to encode one or more pictures, the first flagindicating whether use of one or more interlayer reference pictures forinterlayer prediction is allowed.
[0052] In addition, for example, a decoder according to an aspect of thepresent disclosure includes an inputter, an entropy decoder, an inversequantizer, an intra predictor, an inter predictor, a loop filter unit, andoutputter.
[0053] An encoded bitstream is input to the inputter. The entropydecoder applies variable length decoding on the encoded bitstream toderive quantized coefficients. The inverse quantizer inverse quantizesthe quantized coefficients to derive transform coefficients. The inversetransformer inverse transforms the transformed coefficients to deriveprediction errors.
[0054] The intra predictor generates prediction signals of a current blockincluded in the current picture, using a reference image included in thecurrent picture. The inter predictor generates prediction signals of acurrent block included in the current picture, using a reference imageincluded in a reference picture different from the current picture.
[0055] The loop filter unit applies a filter to a reconstructed block in acurrent block included in the current picture. The current picture isthen output from the outputter.
[0056] For example, the entropy decoder switches whether to decode afirst flag, according to whether a multilayer coding scheme is used todecode one or more pictures, the first flag indicating whether use of one ormore interlayer reference pictures for interlayer prediction is allowed.
[0057] Furthermore, these general and specific aspects may beimplemented using, a system, a device, a method, an integrated circuit, acomputer program, or a non-transitory computer readable medium suchas a CD-ROM, or any combination of, systems, devices, methods,integrated circuits, computer programs, or computer readable media.
[0058] [Definitions of Terms]The respective terms may be defined as indicated below asexamples.
[0059] (1) imageAn image is a data unit configured with a set of pixels, is a pictureor includes blocks smaller than a picture. Images include a still imagein addition to a video.
[0060] (2) pictureA picture is an image processing unit configured with a set ofpixels, and is also referred to as a frame or a field.
[0061] (3) blockA block is a processing unit which is a set of a particular number ofpixels. The block is also referred to as indicated in the followingexamples. The shapes of blocks are not limited. Examples include arectangle shape of MxN pixels and a square shape of MxM pixels for thefirst place, and also include a triangular shape, a circular shape, andother shapes.
[0062] (examples of blocks)- slice / tile / brick- CTU / super block / basic splitting unit- VPDU / processing splitting unit for hardware- CU / processing block unit / prediction block unit (PU) / orthogonal transform block unit (TU) / unit- sub-block
[0063] (4) pixel / sampleA pixel or sample is a smallest point of an image. Pixels orsamples include not only a pixel at an integer position but also a pixel at asub-pixel position generated based on a pixel at an integer position.
[0064] (5) pixel value / sample valueA pixel value or sample value is an eigen value of a pixel. Pixel orsample values naturally include a luma value, a chroma value, an RGBgradation level and also covers a depth value, or a binary value of 0 or 1.
[0065] (6) flagA flag indicates one or more bits, and may be, for example, aparameter or index represented by two or more bits. Alternatively, theflag may indicate not only a binary value represented by a binary numberbut also a multiple value represented by a number other than the binarynumber.
[0066] (7) signalA signal is the one symbolized or encoded to convey information.Signals include a discrete digital signal and an analog signal which takesa continuous value.
[0067] (8) stream / bitstreamA stream or bitstream is a digital data string or a digital data flow.A stream or bitstream may be one stream or may be configured with aplurality of streams having a plurality of hierarchical layers. A streamor bitstream may be transmitted in serial communication using a singletransmission path, or may be transmitted in packet communication usinga plurality of transmission paths.
[0068] (9) differenceIn the case of scalar quantity, it is only necessary that a simpledifference (x - y) and a difference calculation be included. Differencesinclude an absolute value of a difference (|x - y|), a squared difference(x^2 - y^2), a square root of a difference (√(x - y)), a weighted difference(ax - by: a and b are constants), an offset difference (x - y + a: a is anoffset).
[0069] (10) sumIn the case of scalar quantity, it is only necessary that a simplesum (x + y) and a sum calculation be included. Sums include an absolutevalue of a sum (|x + y|), a squared sum (x^2 + y^2), a square root of asum (√(x + y)), a weighted difference (ax + by: a and b are constants), anoffset sum (x + y + a: a is an offset).
[0070] (11) based onA phrase "based on something" means that a thing other than thesomething may be considered. In addition, "based on" may be used in acase in which a direct result is obtained or a case in which a result isobtained through an intermediate result.
[0071] (12) used, usingA phrase "something used" or "using something" means that athing other than the something may be considered. In addition, "used"or "using" may be used in a case in which a direct result is obtained or acase in which a result is obtained through an intermediate result.
[0072] (13) prohibit, forbidThe term "prohibit" or "forbid" can be rephrased as "does notpermit" or "does not allow". In addition, "being not prohibited / forbidden"or "being permitted / allowed" does not always mean "obligation".
[0073] (14) limit, restriction / restrict / restrictedThe term "limit" or "restriction / restrict / restricted" can berephrased as "does not permit / allow" or "being not permitted / allowed".In addition, "being not prohibited / forbidden" or "being permitted / allowed"does not always mean "obligation". Furthermore, it is only necessarythat part of something be prohibited / forbidden quantitatively orqualitatively, and something may be fully prohibited / forbidden.
[0074] (15) chromaAn adjective, represented by the symbols Cb and Cr, specifyingthat a sample array or single sample is representing one of the two colordifference signals related to the primary colors. The term chroma maybe used instead of the term chrominance.
[0075] (16) lumaAn adjective, represented by the symbol or subscript Y or L,specifying that a sample array or single sample is representing themonochrome signal related to the primary colors. The term luma may beused instead of the term luminance.
[0076] [Notes Related to the Descriptions]In the drawings, same reference numbers indicate same or similarcomponents. The sizes and relative locations of components are notnecessarily drawn by the same scale.
[0077] Hereinafter, embodiments will be described with reference to thedrawings. Note that the embodiments described below each show ageneral or specific example. The numerical values, shapes, materials,components, the arrangement and connection of the components, steps,the relation and order of the steps, etc., indicated in the followingembodiments are mere examples, and are not intended to limit the scopeof the claims.
[0078] Embodiments of an encoder and a decoder will be described below.The embodiments are examples of an encoder and a decoder to which theprocesses and / or configurations presented in the description of aspects ofthe present disclosure are applicable. The processes and / orconfigurations can also be implemented in an encoder and a decoderdifferent from those according to the embodiments. For example,regarding the processes and / or configurations as applied to theembodiments, any of the following may be implemented:
[0079] (1) Any of the components of the encoder or the decoder accordingto the embodiments presented in the description of aspects of the presentdisclosure may be substituted or combined with another componentpresented anywhere in the description of aspects of the presentdisclosure.
[0080] (2) In the encoder or the decoder according to the embodiments,discretionary changes may be made to functions or processes performedby one or more components of the encoder or the decoder, such as addition,substitution, removal, etc., of the functions or processes. For example,any function or process may be substituted or combined with anotherfunction or process presented anywhere in the description of aspects ofthe present disclosure.
[0081] (3) In methods implemented by the encoder or the decoderaccording to the embodiments, discretionary changes may be made suchas addition, substitution, and removal of one or more of the processesincluded in the method. For example, any process in the method may besubstituted or combined with another process presented anywhere in thedescription of aspects of the present disclosure.
[0082] (4) One or more components included in the encoder or the decoderaccording to embodiments may be combined with a component presentedanywhere in the description of aspects of the present disclosure, may becombined with a component including one or more functions presentedanywhere in the description of aspects of the present disclosure, and maybe combined with a component that implements one or more processesimplemented by a component presented in the description of aspects ofthe present disclosure.
[0083] (5) A component including one or more functions of the encoder orthe decoder according to the embodiments, or a component thatimplements one or more processes of the encoder or the decoder accordingto the embodiments, may be combined or substituted with a componentpresented anywhere in the description of aspects of the present disclosure,with a component including one or more functions presented anywhere inthe description of aspects of the present disclosure, or with a componentthat implements one or more processes presented anywhere in thedescription of aspects of the present disclosure.
[0084] (6) In methods implemented by the encoder or the decoderaccording to the embodiments, any of the processes included in themethod may be substituted or combined with a process presentedanywhere in the description of aspects of the present disclosure or withany corresponding or equivalent process.
[0085] (7) One or more processes included in methods implemented by theencoder or the decoder according to the embodiments may be combinedwith a process presented anywhere in the description of aspects of thepresent disclosure.
[0086] (8) The implementation of the processes and / or configurationspresented in the description of aspects of the present disclosure is notlimited to the encoder or the decoder according to the embodiments. Forexample, the processes and / or configurations may be implemented in adevice used for a purpose different from the moving picture encoder or themoving picture decoder disclosed in the embodiments.
[0087] [System Configuration]FIG. 1 is a schematic diagram illustrating one example of aconfiguration of a transmission system according to an embodiment.
[0088] Transmission system Trs is a system which transmits a streamgenerated by encoding an image and decodes the transmitted stream.Transmission system Trs like this includes, for example, encoder 100,network Nw, and decoder 200 as illustrated in FIG. 1.
[0089] An image is input to encoder 100. Encoder 100 generates astream by encoding the input image, and outputs the stream to networkNw. The stream includes, for example, the encoded image and controlinformation for decoding the encoded image. The image is compressedby the encoding.
[0090] It is to be noted that a previous image before being encoded andbeing input to encoder 100 is also referred to as the original image, theoriginal signal, or the original sample. The image may be a video or astill image. The image is a generic concept of a sequence, a picture, anda block, and thus is not limited to a spatial region having a particular sizeand to a temporal region having a particular size unless otherwisespecified. The image is an array of pixels or pixel values, and the signalrepresenting the image or pixel values are also referred to as samples.The stream may be referred to as a bitstream, an encoded bitstream, acompressed bitstream, or an encoded signal. Furthermore, the encodermay be referred to as an image encoder or a video encoder. The encodingmethod performed by encoder 100 may be referred to as an encodingmethod, an image encoding method, or a video encoding method.
[0091] Network Nw transmits the stream generated by encoder 100 todecoder 200. Network Nw may be the Internet, the Wide Area Network(WAN), the Local Area Network (LAN), or any combination of thesenetworks. Network Nw is not always limited to a bi-directionalcommunication network, and may be a uni-directional communicationnetwork which transmits broadcast waves of digital terrestrialbroadcasting, satellite broadcasting, or the like. Alternatively, networkNw may be replaced by a medium such as a Digital Versatile Disc (DVD)and a Blu-Ray Disc (BD) (R), etc. on which a stream is recorded.
[0092] Decoder 200 generates, for example, a decoded image which is anuncompressed image by decoding a stream transmitted by network Nw.For example, the decoder decodes a stream according to a decodingmethod corresponding to an encoding method by encoder 100.
[0093] It is to be noted that the decoder may also be referred to as animage decoder or a video decoder, and that the decoding methodperformed by decoder 200 may also be referred to as a decoding method,an image decoding method, or a video decoding method.
[0094] [Data Structure]FIG. 2 is a diagram illustrating one example of a hierarchicalstructure of data in a stream. A stream includes, for example, a videosequence. As illustrated in (a) of FIG. 2, the video sequence includes avideo parameter set (VPS), a sequence parameter set (SPS), a pictureparameter set (PPS), supplemental enhancement information (SEI), anda plurality of pictures.
[0095] In a video having a plurality of layers, a VPS includes: a codingparameter which is common between some of the plurality of layers; and acoding parameter related to some of the plurality of layers included in thevideo or an individual layer.
[0096] An SPS includes a parameter which is used for a sequence, that is,a coding parameter which decoder 200 refers to in order to decode thesequence. For example, the coding parameter may indicate the width orheight of a picture. It is to be noted that a plurality of SPSs may bepresent.
[0097] A PPS includes a parameter which is used for a picture, that is, acoding parameter which decoder 200 refers to in order to decode each ofthe pictures in the sequence. For example, the coding parameter mayinclude a reference value for the quantization width which is used todecode a picture and a flag indicating application of weighted prediction.It is to be noted that a plurality of PPSs may be present. Each of theSPS and the PPS may be simply referred to as a parameter set.
[0098] As illustrated in (b) of FIG. 2, a picture may include a pictureheader and at least one slice. A picture header includes a codingparameter which decoder 200 refers to in order to decode the at least oneslice.
[0099] As illustrated in (c) of FIG. 2, a slice includes a slice header and atleast one brick. A slice header includes a coding parameter whichdecoder 200 refers to in order to decode the at least one brick.
[0100] As illustrated in (d) of FIG. 2, a brick includes at least one codingtree unit (CTU).
[0101] It is to be noted that a picture may not include any slice and mayinclude a tile group instead of a slice. In this case, the tile groupincludes at least one tile. In addition, a brick may include a slice.
[0102] A CTU is also referred to as a super block or a basis splitting unit.As illustrated in (e) of FIG. 2, a CTU like this includes a CTU header andat least one coding unit (CU). A CTU header includes a codingparameter which decoder 200 refers to in order to decode the at least oneCU.
[0103] A CU may be split into a plurality of smaller CUs. As illustratedin (f) of FIG. 2, a CU includes a CU header, prediction information, andresidual coefficient information. Prediction information is informationfor predicting the CU, and the residual coefficient information isinformation indicating a prediction residual to be described later.Although a CU is basically the same as a prediction unit (PU) and atransform unit (TU), it is to be noted that, for example, an SBT to bedescribed later may include a plurality of TUs smaller than the CU. Inaddition, the CU may be processed for each virtual pipeline decoding unit(VPDU) included in the CU. The VPDU is, for example, a fixed unitwhich can be processed at one stage when pipeline processing isperformed in hardware.
[0104] It is to be noted that a stream may not include part of thehierarchical layers illustrated in FIG. 2. The order of the hierarchicallayers may be exchanged, or any of the hierarchical layers may bereplaced by another hierarchical layer. Here, a picture which is a targetfor a process which is about to be performed by a device such as encoder100 or decoder 200 is referred to as a current picture. A current picturemeans a current picture to be encoded when the process is an encodingprocess, and a current picture means a current picture to be decodedwhen the process is a decoding process. Likewise, for example, a CU or ablock of CUs which is a target for a process which is about to be performedby a device such as encoder 100 or decoder 200 is referred to as a currentblock. A current block means a current block to be encoded when theprocess is an encoding process, and a current block means a current blockto be decoded when the process is a decoding process.
[0105] [Picture Structure: Slice / Tile]A picture may be configured with one or more slice units or tileunits in order to decode the picture in parallel.
[0106] Slices are basic encoding units included in a picture. A picturemay include, for example, one or more slices. In addition, a sliceincludes one or more successive coding tree units (CTUs).
[0107] FIG. 3 is a diagram illustrating one example of a sliceconfiguration. For example, a picture includes 11x8 CTUs, and is splitinto four slices (slices 1 to 4). Slice 1 includes sixteen CTUs, slice 2includes twenty-one CTUs, slice 3 includes twenty-nine CTUs, and slice 4includes twenty-two CTUs. Here, each CTU in the picture belongs to oneof the slices. The shape of each slice is a shape obtained by splitting thepicture horizontally. A boundary of each slice does not need to coincidewith an image end, and may coincide with any of the boundaries betweenCTUs in the image. The processing order of the CTUs in a slice (anencoding order or a decoding order) is, for example, a raster-scan order.A slice includes a slice header and encoded data. Features of the slicemay be written in the slice header. The features include a CTU addressof a top CTU in the slice, a slice type, etc.
[0108] A tile is a unit of a rectangular region included in a picture. Eachof tiles may be assigned with a number referred to as TileId in raster-scanorder.
[0109] FIG. 4 is a diagram illustrating one example of a tile configuration.For example, a picture includes 11x8 CTUs, and is split into four tiles ofrectangular regions (tiles 1 to 4). When tiles are used, the processingorder of CTUs is changed from the processing order in the case where notile is used. When no tile is used, a plurality of CTUs in a picture areprocessed in raster-scan order. When a plurality of tiles are used, atleast one CTU in each of the plurality of tiles is processed in raster-scanorder. For example, as illustrated in FIG. 4, the processing order of theCTUs included in tile 1 is the order which starts from the left-end of thefirst column of tile 1 toward the right-end of the first column of tile 1 andthen starts from the left-end of the second column of tile 1 toward theright-end of the second column of tile 1.
[0110] It is to be noted that one tile may include one or more slices, andone slice may include one or more tiles.
[0111] It is to be noted that a picture may be configured with one or moretile sets. A tile set may include one or more tile groups, or one or moretiles. A picture may be configured with only one of a tile set, a tile group,and a tile. For example, an order for scanning a plurality of tiles foreach tile set in raster scan order is assumed to be a basic encoding orderof tiles. A set of one or more tiles which are continuous in the basicencoding order in each tile set is assumed to be a tile group. Such apicture may be configured by splitter 102 (see FIG. 7) to be describedlater.
[0112] [Scalable Encoding]FIGs. 5 and 6 are diagrams illustrating examples of scalablestream structures.
[0113] As illustrated in FIG. 5, encoder 100 may generate atemporally / spatially scalable stream by dividing each of a plurality ofpictures into any of a plurality of layers and encoding the picture in thelayer. For example, encoder 100 encodes the picture for each layer,thereby achieving scalability where an enhancement layer is presentabove a base layer. Such encoding of each picture is also referred to asscalable encoding. In this way, decoder 200 is capable of switchingimage quality of an image which is displayed by decoding the stream. Inother words, decoder 200 determines up to which layer to decode based oninternal factors such as the processing ability of decoder 200 and externalfactors such as a state of a communication bandwidth. As a result,decoder 200 is capable of decoding a content while freely switchingbetween low resolution and high resolution. For example, the user of thestream watches a video of the stream halfway using a smartphone on theway to home, and continues watching the video at home on a device suchas a TV connected to the Internet. It is to be noted that each of thesmartphone and the device described above includes decoder 200 havingthe same or different performances. In this case, when the devicedecodes layers up to the higher layer in the stream, the user can watchthe video at high quality at home. In this way, encoder 100 does not needto generate a plurality of streams having different image qualities of thesame content, and thus the processing load can be reduced.
[0114] Furthermore, the enhancement layer may include metainformation based on statistical information on the image. Decoder 200may generate a video whose image quality has been enhanced byperforming super-resolution imaging on a picture in the base layer basedon the metadata. Super-resolution imaging may be any of improvementin the SN ratio in the same resolution and increase in resolution.Metadata may include information for identifying a linear or a non-linearfilter coefficient, as used in a super-resolution process, or informationidentifying a parameter value in a filter process, machine learning, or aleast squares method used in super-resolution processing.
[0115] Alternatively, a configuration may be provided in which a pictureis divided into, for example, tiles in accordance with, for example, themeaning of an object in the picture. In this case, decoder 200 may decodeonly a partial region in a picture by selecting a tile to be decoded. Inaddition, an attribute of the object (person, car, ball, etc.) and a positionof the object in the picture (coordinates in identical images) may be storedas metadata. In this case, decoder 200 is capable of identifying theposition of a desired object based on the metadata, and determining thetile including the object. For example, as illustrated in FIG. 6, themetadata may be stored using a data storage structure different fromimage data, such as SEI in HEVC. This metadata indicates, for example,the position, size, or color of a main object.
[0116] Metadata may be stored in units of a plurality of pictures, such asa stream, a sequence, or a random access unit. In this way, decoder 200is capable of obtaining, for example, the time at which a specific personappears in the video, and by fitting the time information with picture unitinformation, is capable of identifying a picture in which the object ispresent and determining the position of the object in the picture.
[0117] [Encoder]Next, encoder 100 according to this embodiment is described.FIG. 7 is a block diagram illustrating one example of a functionalconfiguration of encoder 100 according to this embodiment. Encoder 100encodes an image in units of a block.
[0118] As illustrated in FIG. 7, encoder 100 is an apparatus whichencodes an image in units of a block, and includes splitter 102, subtractor104, transformer 106, quantizer 108, entropy encoder 110, inversequantizer 112, inverse transformer 114, adder 116, block memory 118,loop filter 120, frame memory 122, intra predictor 124, inter predictor 126,prediction controller 128, and prediction parameter generator 130. It isto be noted that intra predictor 124 and inter predictor 126 are configuredas part of a prediction executor.
[0119] [Mounting Example of Encoder]FIG. 8 is a block diagram illustrating a mounting example ofencoder 100. Encoder 100 includes processor a1 and memory a2. Forexample, the plurality of constituent elements of encoder 100 illustratedin FIG. 7 are mounted on processor a1 and memory a2 illustrated in FIG.8.
[0120] Processor a1 is circuitry which performs information processingand is accessible to memory a2. For example, processor a1 is dedicatedor general electronic circuitry which encodes an image. Processor a1may be a processor such as a CPU. In addition, processor a1 may be anaggregate of a plurality of electronic circuits. In addition, for example,processor a1 may take the roles of two or more constituent elements otherthan a constituent element for storing information out of the plurality ofconstituent elements of encoder 100 illustrated in FIG. 7, etc.
[0121] Memory a2 is dedicated or general memory for storing informationthat is used by processor a1 to encode the image. Memory a2 may beelectronic circuitry, and may be connected to processor a1. In addition,memory a2 may be included in processor a1. In addition, memory a2may be an aggregate of a plurality of electronic circuits. In addition,memory a2 may be a magnetic disc, an optical disc, or the like, or may berepresented as storage, a medium, or the like. In addition, memory a2may be non-volatile memory, or volatile memory.
[0122] For example, memory a2 may store an image to be encoded or astream corresponding to an encoded image. In addition, memory a2 maystore a program for causing processor a1 to encode an image.
[0123] In addition, for example, memory a2 may take the roles of two ormore constituent elements for storing information out of the plurality ofconstituent elements of encoder 100 illustrated in FIG. 7. Morespecifically, memory a2 may take the roles of block memory 118 and framememory 122 illustrated in FIG. 7. More specifically, memory a2 maystore a reconstructed image (specifically, a reconstructed block, areconstructed picture, or the like).
[0124] It is to be noted that, in encoder 100, not all of the plurality ofconstituent elements indicated in FIG. 7, etc. may be implemented, andnot all the processes described above may be performed. Part of theconstituent elements indicated in FIG. 7 may be included in anotherdevice, or part of the processes described above may be performed byanother device.
[0125] Hereinafter, an overall flow of processes performed by encoder 100is described, and then each of constituent elements included in encoder100 is described.
[0126] [Overall Flow of Encoding Process]FIG. 9 is a flow chart illustrating one example of an overallencoding process performed by encoder 100.
[0127] First, splitter 102 of encoder 100 splits each of pictures included inan original image into a plurality of blocks having a fixed size (128x128pixels) (Step Sa_1). Splitter 102 then selects a splitting pattern for thefixed-size block (Step Sa_2). In other words, splitter 102 further splitsthe fixed-size block into a plurality of blocks which form the selectedsplitting pattern. Encoder 100 performs, for each of the plurality ofblocks, Steps Sa_3 to Sa_9 for the block.
[0128] Prediction controller 128 and a prediction executor which isconfigured with intra predictor 124 and inter predictor 126 generate aprediction image of a current block (Step Sa_3). It is to be noted that theprediction image is also referred to as a prediction signal, a predictionblock, or prediction samples.
[0129] Next, subtractor 104 generates the difference between a currentblock and a prediction image as a prediction residual (Step Sa_4). It isto be noted that the prediction residual is also referred to as a predictionerror.
[0130] Next, transformer 106 transforms the prediction image andquantizer 108 quantizes the result, to generate a plurality of quantizedcoefficients (Step Sa_5).
[0131] Next, entropy encoder 110 encodes (specifically, entropy encodes)the plurality of quantized coefficients and a prediction parameter relatedto generation of a prediction image to generate a stream (Step Sa_6).
[0132] Next, inverse quantizer 112 performs inverse quantization of theplurality of quantized coefficients and inverse transformer 114 performsinverse transform of the result, to restore a prediction residual (StepSa_7).
[0133] Next, adder 116 adds the prediction image to the restoredprediction residual to reconstruct the current block (Step Sa_8). In thisway, the reconstructed image is generated. It is to be noted that thereconstructed image is also referred to as a reconstructed block, and, inparticular, that a reconstructed image generated by encoder 100 is alsoreferred to as a local decoded block or a local decoded image.
[0134] When the reconstructed image is generated, loop filter 120performs filtering of the reconstructed image as necessary (Step Sa_9).
[0135] Encoder 100 then determines whether encoding of the entirepicture has been finished (Step Sa_10). When determining that theencoding has not yet been finished (No in Step Sa_10), processes fromStep Sa_2 are executed repeatedly.
[0136] Although encoder 100 selects one splitting pattern for a fixed-sizeblock, and encodes each block according to the splitting pattern in theabove-described example, it is to be noted that each block may be encodedaccording to a corresponding one of a plurality of splitting patterns. Inthis case, encoder 100 may evaluate a cost for each of the plurality ofsplitting patterns, and, for example, may select the stream obtained byencoding according to the splitting pattern which yields the smallest costas a stream which is output finally.
[0137] Alternatively, the processes in Steps Sa_1 to Sa_10 may beperformed sequentially by encoder 100, or two or more of the processesmay be performed in parallel or may be reordered.
[0138] The encoding process by encoder 100 is hybrid encoding usingprediction encoding and transform encoding. In addition, predictionencoding is performed by an encoding loop configured with subtractor 104,transformer 106, quantizer 108, inverse quantizer 112, inversetransformer 114, adder 116, loop filter 120, block memory 118, framememory 122, intra predictor 124, inter predictor 126, and predictioncontroller 128. In other words, the prediction executor configured withintra predictor 124 and inter predictor 126 is part of the encoding loop.
[0139] [Splitter]Splitter 102 splits each of pictures included in the original imageinto a plurality of blocks, and outputs each block to subtractor 104. Forexample, splitter 102 first splits a picture into blocks of a fixed size (forexample, 128x128 pixels). The fixed-size block is also referred to as acoding tree unit (CTU). Splitter 102 then splits each fixed-size blockinto blocks of variable sizes (for example, 64x64 pixels or smaller), basedon recursive quadtree and / or binary tree block splitting. In other words,splitter 102 selects a splitting pattern. The variable-size block is alsoreferred to as a coding unit (CU), a prediction unit (PU), or a transformunit (TU). It is to be noted that, in various kinds of mounting examples,there is no need to differentiate between CU, PU, and TU; all or some ofthe blocks in a picture may be processed in units of a CU, a PU, or a TU.
[0140] FIG. 10 is a diagram illustrating one example of block splittingaccording to this embodiment. In FIG. 10, the solid lines represent blockboundaries of blocks split by quadtree block splitting, and the dashedlines represent block boundaries of blocks split by binary tree blocksplitting.
[0141] Here, block 10 is a square block having 128x128 pixels. Thisblock 10 is first split into four square 64x64 pixel blocks (quadtree blocksplitting).
[0142] The upper-left 64x64 pixel block is further vertically split into tworectangle 32x64 pixel blocks, and the left 32x64 pixel block is furthervertically split into two rectangle 16x64 pixel blocks (binary tree blocksplitting). As a result, the upper-left square 64x64 pixel block is splitinto two 16x64 pixel blocks 11 and 12 and one 32x64 pixel block 13.
[0143] The upper-right square 64x64 pixel block is horizontally split intotwo rectangle 64x32 pixel blocks 14 and 15 (binary tree block splitting).
[0144] The lower-left square 64x64 pixel block is first split into foursquare 32x32 pixel blocks (quadtree block splitting). The upper-leftblock and the lower-right block among the four square 32x32 pixel blocksare further split. The upper-left square 32x32 pixel block is verticallysplit into two rectangle 16x32 pixel blocks, and the right 16x32 pixelblock is further horizontally split into two 16x16 pixel blocks (binary treeblock splitting). The lower-right 32x32 pixel block is horizontally splitinto two 32x16 pixel blocks (binary tree block splitting). The upper-rightsquare 32x32 pixel block is horizontally split into two rectangle 32x16pixel blocks (binary tree block splitting). As a result, the lower-leftsquare 64x64 pixel block is split into rectangle16x32 pixel block 16, twosquare 16x16 pixel blocks 17 and 18, two square 32x32 pixel blocks 19and 20, and two rectangle 32x16 pixel blocks 21 and 22.
[0145] The lower-right 64x64 pixel block 23 is not split.
[0146] As described above, in FIG. 10, block 10 is split into thirteenvariable-size blocks 11 through 23 based on recursive quadtree andbinary tree block splitting. Such splitting is also referred to asquad-tree plus binary tree splitting (QTBT).
[0147] It is to be noted that, in FIG. 10, one block is split into four or twoblocks (quadtree or binary tree block splitting), but splitting is notlimited to these examples. For example, one block may be split intothree blocks (ternary block splitting). Splitting including such ternaryblock splitting is also referred to as multi type tree (MBT) splitting.
[0148] FIG. 11 is a diagram illustrating one example of a functionalconfiguration of splitter 102. As illustrated in FIG. 11, splitter 102 mayinclude block splitting determiner 102a. Block splitting determiner102a may perform the following processes as examples.
[0149] For example, block splitting determiner 102a collects blockinformation from either block memory 118 or frame memory 122, anddetermines the above-described splitting pattern based on the blockinformation. Splitter 102 splits the original image according to thesplitting pattern, and outputs at least one block obtained by the splittingto subtractor 104.
[0150] In addition, for example, block splitting determiner 102a outputs aparameter indicating the above-described splitting pattern totransformer 106, inverse transformer 114, intra predictor 124, interpredictor 126, and entropy encoder 110. Transformer 106 may transforma prediction residual based on the parameter. Intra predictor 124 andinter predictor 126 may generate a prediction image based on theparameter. In addition, entropy encoder 110 may entropy encodes theparameter.
[0151] The parameter related to the splitting pattern may be written in astream as indicated below as one example.
[0152] FIG. 12 is a diagram illustrating examples of splitting patterns.Examples of splitting patterns include: splitting into four regions (QT) inwhich a block is split into two regions both horizontally and vertically;splitting into three regions (HT or VT) in which a block is split in thesame direction in a ratio of 1:2:1; splitting into two regions (HB or VB) inwhich a block is split in the same direction in a ratio of 1:1; and nosplitting (NS).
[0153] It is to be noted that the splitting pattern does not have any blocksplitting direction in the case of splitting into four regions and nosplitting, and that the splitting pattern has splitting directioninformation in the case of splitting into two regions or three regions.
[0154] FIGs. 13A and 13B are each a diagram illustrating one example ofa syntax tree of a splitting pattern. In the example of FIG. 13A, first,information indicating whether to perform splitting (S: Split flag) ispresent, and information indicating whether to perform splitting into fourregions (QT: QT flag) is present next. Information indicating which oneof splitting into three regions and two regions is to be performed (TT: TTflag or BT: BT flag) is present next, and lastly, information indicating adivision direction (Ver: Vertical flag or Hor: Horizontal flag) is present.It is to be noted that each of at least one block obtained by splittingaccording to such a splitting pattern may be further split repeatedly in asimilar process. In other words, as one example, whether splitting isperformed, whether splitting into four regions is performed, which one ofthe horizontal direction and the vertical direction is the direction inwhich a splitting method is to be performed, which one of splitting intothree regions and splitting into two regions is to be performed may berecursively determined, and the determination results may be encoded ina stream according to the encoding order disclosed by the syntax treeillustrated in FIG. 13A.
[0155] In addition, although information items respectively indicating S,QT, TT, and Ver are arranged in the listed order in the syntax treeillustrated in FIG. 13A, information items respectively indicating S, QT,Ver, and BT may be arranged in the listed order. In other words, in theexample of FIG. 13B, first, information indicating whether to performsplitting (S: Split flag) is present, and information indicating whether toperform splitting into four regions (QT: QT flag) is present next.Information indicating the splitting direction (Ver: Vertical flag or Hor:Horizontal flag) is present next, and lastly, information indicating whichone of splitting into two regions and splitting into three regions is to beperformed (BT: BT flag or TT: TT flag) is present.
[0156] It is to be noted that the splitting patterns described above areexamples, and splitting patterns other than the described splittingpatterns may be used, or part of the described splitting patterns may beused.
[0157] [Subtractor]Subtractor 104 subtracts a prediction image (prediction image thatis input from prediction controller 128) from the original image in units ofa block input from splitter 102 and split by splitter 102. In other words,subtractor 104 calculates prediction residuals of a current block.Subtractor 104 then outputs the calculated prediction residuals totransformer 106.
[0158] The original signal is an input signal which has been input toencoder 100 and represents an image of each picture included in a video(for example, a luma signal and two chroma signals).
[0159] [Transformer]Transformer 106 transforms prediction residuals in spatial domaininto transform coefficients in frequency domain, and outputs thetransform coefficients to quantizer 108. More specifically, transformer106 applies, for example, a predefined discrete cosine transform (DCT) ordiscrete sine transform (DST) to prediction residuals in spatial domain.
[0160] It is to be noted that transformer 106 may adaptively select atransform type from among a plurality of transform types, and transformprediction residuals into transform coefficients by using a transformbasis function corresponding to the selected transform type. This sort oftransform is also referred to as explicit multiple core transform (EMT) oradaptive multiple transform (AMT). In addition, a transform basisfunction is also simply referred to as a basis.
[0161] The transform types include, for example, DCT-II, DCT-V,DCT-VIII, DST-I, and DST-VII. It is to be noted that these transformtypes may be represented as DCT2, DCT5, DCT8, DST1, and DST7. FIG.14 is a chart illustrating transform basis functions for each transformtype. In FIG. 14, N indicates the number of input pixels. For example,selection of a transform type from among the plurality of transform typesmay depend on a prediction type (one of intra prediction and interprediction), and may depend on an intra prediction mode.
[0162] Information indicating whether to apply such EMT or AMT(referred to as, for example, an EMT flag or an AMT flag) and informationindicating the selected transform type is normally signaled at the CUlevel. It is to be noted that the signaling of such information does notnecessarily need to be performed at the CU level, and may be performedat another level (for example, at the sequence level, picture level, slicelevel, brick level, or CTU level).
[0163] In addition, transformer 106 may re-transform the transformcoefficients (which are transform results). Such re-transform is alsoreferred to as adaptive secondary transform (AST) or non-separablesecondary transform (NSST). For example, transformer 106 performsre-transform in units of a sub-block (for example, 4x4 pixel sub-block)included in a transform coefficient block corresponding to an intraprediction residual. Information indicating whether to apply NSST andinformation related to a transform matrix for use in NSST are normallysignaled at the CU level. It is to be noted that the signaling of suchinformation does not necessarily need to be performed at the CU level,and may be performed at another level (for example, at the sequence level,picture level, slice level, brick level, or CTU level).
[0164] Transformer 106 may employ a separable transform and anon-separable transform. A separable transform is a method in which atransform is performed a plurality of times by separately performing atransform for each of directions according to the number of dimensions ofinputs. A non-separable transform is a method of performing a collectivetransform in which two or more dimensions in multidimensional inputsare collectively regarded as a single dimension.
[0165] In one example of the non-separable transform, when an input is a4x4 pixel block, the 4x4 pixel block is regarded as a single array includingsixteen elements, and the transform applies a 16x16 transform matrix tothe array.
[0166] In another example of the non-separable transform, an input blockof 4x4 pixels is regarded as a single array including sixteen elements, andthen a transform (hypercube givens transform) in which givens revolutionis performed on the array a plurality of times may be performed.
[0167] In the transform in transformer 106, the transform types oftransform basis functions to be transformed into the frequency domainaccording to regions in a CU can be switched. Examples include aspatially varying transform (SVT).
[0168] FIG. 15 is a diagram illustrating one example of SVT.
[0169] In SVT, as illustrated in FIG. 15, CUs are split into two equalregions horizontally or vertically, and only one of the regions istransformed into the frequency domain. A transform type can be set foreach region. For example, DST7 and DST8 are used. For example,among the two regions obtained by splitting a CU vertically into twoequal regions, DST7 and DCT8 may be used for the region at position 0.Alternatively, among the two regions, DST7 is used for the region atposition 1. Likewise, among the two regions obtained by splitting a CUhorizontally into two equal regions, DST7 and DCT8 are used for theregion at position 0. Alternatively, among the two regions, DST7 is usedfor the region at position 1. Although only one of the two regions in a CUis transformed and the other is not transformed in the exampleillustrated in FIG. 15, each of the two regions may be transformed. Inaddition, splitting method may include not only splitting into two regionsbut also splitting into four regions. In addition, the splitting method canbe more flexible. For example, information indicating the splittingmethod may be encoded and may be signaled in the same manner as theCU splitting. It is to be noted that SVT is also referred to as sub-blocktransform (SBT).
[0170] The AMT and EMT described above may be referred to as MTS(multiple transform selection). When MTS is applied, a transform typethat is DST7, DCT8, or the like can be selected, and the informationindicating the selected transform type may be encoded as indexinformation for each CU. There is another process referred to as IMTS(implicit MTS) as a process for selecting, based on the shape of a CU, atransform type to be used for orthogonal transform performed withoutencoding index information. When IMTS is applied, for example, when aCU has a rectangle shape, orthogonal transform of the rectangle shape isperformed using DST7 for the short side and DST2 for the long side. Inaddition, for example, when a CU has a square shape, orthogonaltransform of the rectangle shape is performed using DCT2 when MTS iseffective in a sequence and using DST7 when MTS is ineffective in thesequence. DCT2 and DST7 are mere examples. Other transform typesmay be used, and it is also possible to change the combination oftransform types for use to a different combination of transform types.IMTS may be used only for intra prediction blocks, or may be used forboth intra prediction blocks and inter prediction block.
[0171] The three processes of MTS, SBT, and IMTS have been describedabove as selection processes for selectively switching transform types foruse in orthogonal transform. However, all of the three selectionprocesses may be made effective, or only part of the selection processesmay be selectively made effective. Whether each of the selectionprocesses is made effective can be identified based on flag information orthe like in a header such as an SPS. For example, when all of the threeselection processes are effective, one of the three selection processes isselected for each CU and orthogonal transform of the CU is performed.It is to be noted that the selection processes for selectively switching thetransform types may be selection processes different from the above threeselection processes, or each of the three selection processes may bereplaced by another process as long as at least one of the following fourfunctions [1] to [4] can be achieved. Function [1] is a function forperforming orthogonal transform of the entire CU and encodinginformation indicating the transform type used in the transform.Function [2] is a function for performing orthogonal transform of theentire CU and determining the transform type based on a predeterminedrule without encoding information indicating the transform type.Function [3] is a function for performing orthogonal transform of a partialregion of a CU and encoding information indicating the transform typeused in the transform. Function [4] is a function for performingorthogonal transform of a partial region of a CU and determining thetransform type based on a predetermined rule without encodinginformation indicating the transform type used in the transform.
[0172] It is to be noted that whether each of MTS, IMTS, and SBT isapplied may be determined for each processing unit. For example,whether each of MTS, IMTS, and SBT is applied may be determined foreach sequence, picture, brick, slice, CTU, or CU.
[0173] It is to be noted that a tool for selectively switching transformtypes in the present disclosure may be rephrased by a method forselectively selecting a basis for use in a transform process, a selectionprocess, or a process for selecting a basis. In addition, the tool forselectively switching transform types may be rephrased by a mode foradaptively selecting a transform type.
[0174] FIG. 16 is a flow chart illustrating one example of a processperformed by transformer 106.
[0175] For example, transformer 106 determines whether to performorthogonal transform (Step St_1). Here, when determining to performorthogonal transform (Yes in Step St_1), transformer 106 selects atransform type for use in orthogonal transform from a plurality oftransform types (Step St_2). Next, transformer 106 performs orthogonaltransform by applying the selected transform type to the predictionresidual of a current block (Step St_3). Transformer 106 then outputsinformation indicating the selected transform type to entropy encoder 110,so as to allow entropy encoder 110 to encode the information (Step St_4).On the other hand, when determining not to perform orthogonaltransform (No in Step St_1), transformer 106 outputs informationindicating that no orthogonal transform is performed, so as to allowentropy encoder 110 to encode the information (Step St_5). It is to benoted that whether to perform orthogonal transform in Step St_1 may bedetermined based on, for example, the size of a transform block, aprediction mode applied to the CU, etc. Alternatively, orthogonaltransform may be performed using a predefined transform type withoutencoding information indicating the transform type for use in orthogonaltransform.
[0176] FIG. 17 is a flow chart illustrating another example of a processperformed by transformer 106. It is to be noted that the exampleillustrated in FIG. 17 is an example of orthogonal transform in the casewhere transform types for use in orthogonal transform are selectivelyswitched as in the case of the example illustrated in FIG. 16.
[0177] As one example, a first transform type group may include DCT2,DST7, and DCT8. As another example, a second transform type groupmay include DCT2. The transform types included in the first transformtype group and the transform types included in the second transform typegroup may partly overlap with each other, or may be totally different fromeach other.
[0178] More specifically, transformer 106 determines whether atransform size is smaller than or equal to a predetermined value (StepSu_1). Here, when determining that the transform size is smaller thanor equal to the predetermined value (Yes in Step Su_1), transformer 106performs orthogonal transform of the prediction residual of the currentblock using the transform type included in the first transform type group(Step Su_2). Next, transformer 106 outputs information indicating thetransform type to be used among at least one transform type included inthe first transform type group to entropy encoder 110, so as to allowentropy encoder 110 to encode the information (Step Su_3). On the otherhand, when determining that the transform size is not smaller than orequal to the predetermined value (No in Step Su_1), transformer 106performs orthogonal transform of the prediction residual of the currentblock using the second transform type group (Step Su_4).
[0179] In Step Su_3, the information indicating the transform type foruse in orthogonal transform may be information indicating a combinationof the transform type to be applied vertically in the current block and thetransform type to be applied horizontally in the current block. The firsttype group may include only one transform type, and the informationindicating the transform type for use in orthogonal transform may not beencoded. The second transform type group may include a plurality oftransform types, and information indicating the transform type for use inorthogonal transform among the one or more transform types included inthe second transform type group may be encoded.
[0180] Alternatively, a transform type may be determined based only on atransform size. It is to be noted that such determinations are not limitedto the determination as to whether the transform size is smaller than orequal to the predetermined value, and other processes are also possible aslong as the processes are for determining a transform type for use inorthogonal transform based on the transform size.
[0181] [Quantizer]Quantizer 108 quantizes the transform coefficients output fromtransformer 106. More specifically, quantizer 108 scans, in a determinedscanning order, the transform coefficients of the current block, andquantizes the scanned transform coefficients based on quantizationparameters (QP) corresponding to the transform coefficients. Quantizer108 then outputs the quantized transform coefficients (hereinafter alsoreferred to as quantized coefficients) of the current block to entropyencoder 110 and inverse quantizer 112.
[0182] A determined scanning order is an order for quantizing / inversequantizing transform coefficients. For example, a determined scanningorder is defined as ascending order of frequency (from low to highfrequency) or descending order of frequency (from high to low frequency).
[0183] A quantization parameter (QP) is a parameter defining aquantization step (quantization width). For example, when the value ofthe quantization parameter increases, the quantization step alsoincreases. In other words, when the value of the quantization parameterincreases, an error in quantized coefficients (quantization error)increases.
[0184] In addition, a quantization matrix may be used for quantization.For example, several kinds of quantization matrices may be usedcorrespondingly to frequency transform sizes such as 4x4 and 8x8,prediction modes such as intra prediction and inter prediction, and pixelcomponents such as luma and chroma pixel components. It is to be notedthat quantization means digitalizing values sampled at predeterminedintervals correspondingly to predetermined levels. In this technicalfield, quantization may be represented as other expressions such asrounding and scaling.
[0185] Methods using quantization matrices include a method using aquantization matrix which has been set directly at the encoder 100 sideand a method using a quantization matrix which has been set as a default(default matrix). At the encoder 100 side, a quantization matrix suitablefor features of an image can be set by directly setting a quantizationmatrix. This case, however, has a disadvantage of increasing a codingamount for encoding the quantization matrix. It is to be noted that aquantization matrix to be used to quantize the current block may begenerated based on a default quantization matrix or an encodedquantization matrix, instead of directly using the default quantizationmatrix or the encoded quantization matrix.
[0186] There is a method for quantizing a high-frequency coefficient and alow-frequency coefficient in the same manner without using aquantization matrix. It is to be noted that this method is equivalent to amethod using a quantization matrix (flat matrix) whose all coefficientshave the same value.
[0187] The quantization matrix may be encoded, for example, at thesequence level, picture level, slice level, brick level, or CTU level.
[0188] When using a quantization matrix, quantizer 108 scales, for eachtransform coefficient, for example a quantization width which can becalculated based on a quantization parameter, etc., using the value of thequantization matrix. The quantization process performed without usingany quantization matrix may be a process of quantizing transformcoefficients based on a quantization width calculated based on aquantization parameter, etc. It is to be noted that, in the quantizationprocess performed without using any quantization matrix, thequantization width may be multiplied by a predetermined value which iscommon for all the transform coefficients in a block.
[0189] FIG. 18 is a block diagram illustrating one example of a functionalconfiguration of quantizer 108.
[0190] For example, quantizer 108 includes difference quantizationparameter generator 108a, predicted quantization parameter generator108b, quantization parameter generator 108c, quantization parameterstorage 108d, and quantization executor 108e.
[0191] FIG. 19 is a flow chart illustrating one example of quantizationperformed by quantizer 108.
[0192] As one example, quantizer 108 may perform quantization for eachCU based on the flow chart illustrated in FIG. 19. More specifically,quantization parameter generator 108c determines whether to performquantization (Step Sv_1). Here, when determining to performquantization (Yes in Step Sv_1), quantization parameter generator 108cgenerates a quantization parameter for a current block (Step Sv_2), andstores the quantization parameter into quantization parameter storage108d (Step Sv_3).
[0193] Next, quantization executor 108e quantizes transform coefficientsof the current block using the quantization parameter generated in StepSv_2 (Step Sv_4). Predicted quantization parameter generator 108bthen obtains a quantization parameter for a processing unit differentfrom the current block from quantization parameter storage 108d (StepSv_5). Predicted quantization parameter generator 108b generates apredicted quantization parameter of the current block based on theobtained quantization parameter (Step Sv_6). Difference quantizationparameter generator 108a calculates the difference between thequantization parameter of the current block generated by quantizationparameter generator 108c and the predicted quantization parameter ofthe current block generated by predicted quantization parametergenerator 108b (Step Sv_7). The difference quantization parameter isgenerated by calculating the difference. Difference quantizationparameter generator 108a outputs the difference quantization parameterto entropy encoder 110, so as to allow entropy encoder 110 to encode thedifference quantization parameter (Step Sv_8).
[0194] It is to be noted that the difference quantization parameter may beencoded, for example, at the sequence level, picture level, slice level, bricklevel, or CTU level. In addition, the initial value of the quantizationparameter may be encoded at the sequence level, picture level, slice level,brick level, or CTU level. At this time, the quantization parameter maybe generated using the initial value of the quantization parameter andthe difference quantization parameter.
[0195] It is to be noted that quantizer 108 may include a plurality ofquantizers, and may apply dependent quantization in which transformcoefficients are quantized using a quantization method selected from aplurality of quantization methods.
[0196] [Entropy Encoder]FIG. 20 is a block diagram illustrating one example of a functionalconfiguration of entropy encoder 110.
[0197] Entropy encoder 110 generates a stream by entropy encoding thequantized coefficients input from quantizer 108 and a predictionparameter input from prediction parameter generator 130. For example,context-based adaptive binary arithmetic coding (CABAC) is used as theentropy encoding. More specifically, entropy encoder 110 includesbinarizer 110a, context controller 110b, and binary arithmetic encoder110c. Binarizer 110a performs binarization in which multi-level signalssuch as quantized coefficients and a prediction parameter aretransformed into binary signals. Examples of binarization methodsinclude truncated Rice binarization, exponential Golomb codes, and fixedlength binarization. Context controller 110b derives a context valueaccording to a feature or a surrounding state of a syntax element, that is,an occurrence probability of a binary signal. Examples of methods forderiving a context value include bypass, referring to a syntax element,referring to an upper and left adjacent blocks, referring to hierarchicalinformation, and others. Binary arithmetic encoder 110c arithmeticallyencodes the binary signal using the derived context value.
[0198] FIG. 21 is a diagram illustrating a flow of CABAC in entropyencoder 110.
[0199] First, initialization is performed in CABAC in entropy encoder 110.In the initialization, initialization in binary arithmetic encoder 110c andsetting of an initial context value are performed. For example, binarizer110a and binary arithmetic encoder 110c execute binarization andarithmetic encoding of a plurality of quantization coefficients in a CTUsequentially. At this time, context controller 110b updates the contextvalue each time arithmetic encoding is performed. Context controller110b then saves the context value as a post process. The saved contextvalue is used, for example, to initialize the context value for the nextCTU.
[0200] [Inverse Quantizer]Inverse quantizer 112 inverse quantizes quantized coefficientswhich have been input from quantizer 108. More specifically, inversequantizer 112 inverse quantizes, in a determined scanning order,quantized coefficients of the current block. Inverse quantizer 112 thenoutputs the inverse quantized transform coefficients of the current blockto inverse transformer 114.
[0201] [Inverse Transformer]Inverse transformer 114 restores prediction errors by inversetransforming the transform coefficients which have been input frominverse quantizer 112. More specifically, inverse transformer 114restores the prediction residuals of the current block by performing aninverse transform corresponding to the transform applied to thetransform coefficients by transformer 106. Inverse transformer 114 thenoutputs the restored prediction residuals to adder 116.
[0202] It is to be noted that since information is normally lost inquantization, the restored prediction residuals do not match theprediction errors calculated by subtractor 104. In other words, therestored prediction residuals normally include quantization errors.
[0203] [Adder]Adder 116 reconstructs the current block by adding the predictionresiduals which have been input from inverse transformer 114 andprediction images which have been input from prediction controller 128.Consequently, a reconstructed image is generated. Adder 116 thenoutputs the reconstructed image to block memory 118 and loop filter 120.
[0204] [Block Memory]Block memory 118 is storage for storing a block which is includedin a current picture and is referred to in intra prediction. Morespecifically, block memory 118 stores a reconstructed image output fromadder 116.
[0205] [Frame Memory]Frame memory 122 is, for example, storage for storing referencepictures for use in inter prediction, and is also referred to as a framebuffer. More specifically, frame memory 122 stores a reconstructedimage filtered by loop filter 120.
[0206] [Loop Filter]Loop filter 120 applies a loop filter to a reconstructed image outputby adder 116, and outputs the filtered reconstructed image to framememory 122. A loop filter is a filter used in an encoding loop (in-loopfilter). Examples of loop filters include, for example, an adaptive loopfilter (ALF), a deblocking filter (DF or DBF), a sample adaptive offset(SAO), etc.
[0207] FIG. 22 is a block diagram illustrating one example of a functionalconfiguration of loop filter 120.
[0208] For example, as illustrated in FIG. 22, loop filter 120 includesdeblocking filter executor 120a, SAO executor 120b, and ALF executor120c. Deblocking filter executor 120a performs a deblocking filterprocess of the reconstructed image. SAO executor 120b performs a SAOprocess of the reconstructed image after being subjected to the deblockingfilter process. ALF executor 120c performs an ALF process of thereconstructed image after being subjected to the SAO process. The ALFand deblocking filter processes are described later in detail. The SAOprocess is a process for enhancing image quality by reducing ringing (aphenomenon in which pixel values are distorted like waves around anedge) and correcting deviation in pixel value. Examples of SAOprocesses include an edge offset process and a band offset process. It isto be noted that loop filter 120 does not always need to include all theconstituent elements disclosed in FIG. 22, and may include only part ofthe constituent elements. In addition, loop filter 120 may be configuredto perform the above processes in a processing order different from theone disclosed in FIG. 22.
[0209] [Loop Filter > Adaptive Loop Filter]In an ALF, a least square error filter for removing compressionartifacts is applied. For example, one filter selected from among aplurality of filters based on the direction and activity of local gradients isapplied for each of 2x2 pixel sub-blocks in the current block.
[0210] More specifically, first, each sub-block (for example, each 2x2 pixelsub-block) is categorized into one out of a plurality of classes (for example,fifteen or twenty-five classes). The categorization of the sub-block isbased on, for example, gradient directionality and activity. In a specificexample, category index C (for example, C = 5D + A) is calculated basedon gradient directionality D (for example, 0 to 2 or 0 to 4) and gradientactivity A (for example, 0 to 4). Then, based on category index C, eachsub-block is categorized into one out of a plurality of classes.
[0211] For example, gradient directionality D is calculated by comparinggradients of a plurality of directions (for example, the horizontal, vertical,and two diagonal directions). Moreover, for example, gradient activity Ais calculated by adding gradients of a plurality of directions andquantizing the result of the addition.
[0212] The filter to be used for each sub-block is determined from amongthe plurality of filters based on the result of such categorization.
[0213] The filter shape to be used in an ALF is, for example, a circularsymmetric filter shape. FIG. 23A through FIG. 23C illustrate examplesof filter shapes used in ALFs. FIG. 23A illustrates a 5x5 diamond shapefilter, FIG. 23B illustrates a 7x7 diamond shape filter, and FIG. 23Cillustrates a 9x9 diamond shape filter. Information indicating the filtershape is normally signaled at the picture level. It is to be noted that thesignaling of such information indicating the filter shape does notnecessarily need to be performed at the picture level, and may beperformed at another level (for example, at the sequence level, slice level,brick level, CTU level, or CU level).
[0214] The ON or OFF of the ALF is determined, for example, at thepicture level or CU level. For example, the decision of whether to applythe ALF to luma may be made at the CU level, and the decision ofwhether to apply ALF to chroma may be made at the picture level.Information indicating ON or OFF of the ALF is normally signaled at thepicture level or CU level. It is to be noted that the signaling ofinformation indicating ON or OFF of the ALF does not necessarily need tobe performed at the picture level or CU level, and may be performed atanother level (for example, at the sequence level, slice level, brick level,or CTU level).
[0215] In addition, as described above, one filter is selected from theplurality of filters, and an ALF process of a sub-block is performed. Acoefficient set of coefficients to be used for each of the plurality of filters(for example, up to the fifteenth or twenty-fifth filter) is normallysignaled at the picture level. It is to be noted that the coefficient setdoes not always need to be signaled at the picture level, and may besignaled at another level (for example, the sequence level, slice level,brick level, CTU level, CU level, or sub-block level).
[0216] [Loop Filter > Cross Component Adaptive Loop Filter]FIG. 23D is a diagram illustrating an example where Y samples(first component) are used for a cross component ALF (CCALF) for Cb anda CCALF for Cr (components different from the first component). FIG.23E is a diagram illustrating a diamond shaped filter.
[0217] One example of CC-ALF operates by applying a linear, diamondshaped filter (FIGs. 23D, 23E) to a luma channel for each chromacomponent. The filter coefficients, for example, may be transmitted inthe APS, scaled by a factor of 2^10, and rounded for fixed pointrepresentation. The application of the filters is controlled on a variableblock size and signaled by a context-coded flag received for each block ofsamples. The block size along with a CC-ALF enabling flag is received atthe slice-level for each chroma component. Syntax and semantics forCC-ALF are provided in the Appendix. In the contribution, the followingblock sizes (in chroma samples) were supported: 16x16, 32x32, 64x64,and 128x128.
[0218] [Loop Filter > Joint Chroma Cross Component Adaptive Loop Filter]FIG. 23F is a diagram illustrating an example for a joint chromaCCALF (JC-CCALF).
[0219] One example of JC-CCALF, where only one CCALF filter will beused to generate one CCALF filtered output as a chroma refinementsignal for one color component only, while a properly weighted version ofthe same chroma refinement signal will be applied to the other colorcomponent. In this way, the complexity of existing CCALF is reducedroughly by half.
[0220] The weight value is coded into a sign flag and a weight index.The weight index (denoted as weight_index) is coded into 3 bits, andspecifies the magnitude of the JC-CCALF weight JcCcWeight. It cannotbe equal to 0. The magnitude of JcCcWeight is determined as follows.
[0221] - If weight_index is less than or equal to 4, JcCcWeight is equal toweight_index >> 2.
[0222] - Otherwise, JcCcWeight is equal to 4 / (weight_index - 4).
[0223] The block-level on / off control of ALF filtering for Cb and Cr areseparate. This is the same as in CCALF, and two separate sets ofblock-level on / off control flags will be coded. Different from CCALF,herein, the Cb, Cr on / off control block sizes are the same, and thus, onlyone block size variable is coded.
[0224] [Loop Filter > Deblocking Filter]In a deblocking filter process, loop filter 120 performs a filterprocess on a block boundary in a reconstructed image so as to reducedistortion which occurs at the block boundary.
[0225] FIG. 24 is a block diagram illustrating one example of a specificconfiguration of deblocking filter executor 120a.
[0226] For example, deblocking filter executor 120a includes: boundarydeterminer 1201; filter determiner 1203; filter executor 1205; processdeterminer 1208; filter characteristic determiner 1207; and switches 1202,1204, and 1206.
[0227] Boundary determiner 1201 determines whether a pixel to bedeblock filtered (that is, a current pixel) is present around a blockboundary. Boundary determiner 1201 then outputs the determinationresult to switch 1202 and process determiner 1208.
[0228] In the case where boundary determiner 1201 has determined thata current pixel is present around a block boundary, switch 1202 outputsan unfiltered image to switch 1204. In the opposite case where boundarydeterminer 1201 has determined that no current pixel is present around ablock boundary, switch 1202 outputs an unfiltered image to switch 1206.It is to be noted that the unfiltered image is an image configured with acurrent pixel and at least one surrounding pixel located around thecurrent pixel.
[0229] Filter determiner 1203 determines whether to perform deblockingfiltering of the current pixel, based on the pixel value of at least onesurrounding pixel located around the current pixel. Filter determiner1203 then outputs the determination result to switch 1204 and processdeterminer 1208.
[0230] In the case where filter determiner 1203 has determined toperform deblocking filtering of the current pixel, switch 1204 outputs theunfiltered image obtained through switch 1202 to filter executor 1205.In the opposite case where filter determiner 1203 has determined not toperform deblocking filtering of the current pixel, switch 1204 outputs theunfiltered image obtained through switch 1202 to switch 1206.
[0231] When obtaining the unfiltered image through switches 1202 and1204, filter executor 1205 executes, for the current pixel, deblockingfiltering having the filter characteristic determined by filtercharacteristic determiner 1207. Filter executor 1205 then outputs thefiltered pixel to switch 1206.
[0232] Under control by process determiner 1208, switch 1206 selectivelyoutputs a pixel which has not been deblock filtered and a pixel which hasbeen deblock filtered by filter executor 1205.
[0233] Process determiner 1208 controls switch 1206 based on the resultsof determinations made by boundary determiner 1201 and filterdeterminer 1203. In other words, process determiner 1208 causes switch1206 to output the pixel which has been deblock filtered when boundarydeterminer 1201 has determined that the current pixel is present aroundthe block boundary and filter determiner 1203 has determined to performdeblocking filtering of the current pixel. In addition, in a case otherthan the above case, process determiner 1208 causes switch 1206 tooutput the pixel which has not been deblock filtered. A filtered image isoutput from switch 1206 by repeating output of a pixel in this way. It isto be noted that the configuration illustrated in FIG. 24 is one example ofa configuration in deblocking filter executor 120a. Deblocking filterexecutor 120a may have another configuration.
[0234] FIG. 25 is a diagram illustrating an example of a deblocking filterhaving a symmetrical filtering characteristic with respect to a blockboundary.
[0235] In a deblocking filter process, one of two deblocking filters havingdifferent characteristics, that is, a strong filter and a weak filter isselected using pixel values and quantization parameters, for example.In the case of the strong filter, pixels p0 to p2 and pixels q0 to q2 arepresent across a block boundary as illustrated in FIG. 25, the pixel valuesof the respective pixels q0 to q2 are changed to pixel values q'0 to q'2 byperforming computations according to the expressions below.
[0236] q'0 = (p1 + 2 x p0 + 2 x q0 + 2 x q1 + q2 + 4) / 8q'1 = (p0 +q0 + q1 + q2 + 2) / 4q'2 = (p0 + q0 + q1 +3 x q2 + 2 x q3 +4) / 8
[0237] It is to be noted that, in the above expressions, p0 to p2 and q0 toq2 are the pixel values of respective pixels p0 to p2 and pixels q0 to q2.In addition, q3 is the pixel value of neighboring pixel q3 located at theopposite side of pixel q2 with respect to the block boundary. In addition,in the right side of each of the expressions, coefficients which aremultiplied with the respective pixel values of the pixels to be used fordeblocking filtering are filter coefficients.
[0238] Furthermore, in the deblocking filtering, clipping may beperformed so that the calculated pixel values do not change over athreshold value. In the clipping process, the pixel values calculatedaccording to the above expressions are clipped to a value obtainedaccording to "a pre-computation pixel value ± 2 x a threshold value" usingthe threshold value determined based on a quantization parameter. Inthis way, it is possible to prevent excessive smoothing.
[0239] FIG. 26 is a diagram for illustrating one example of a blockboundary on which a deblocking filter process is performed. FIG. 27 is adiagram illustrating examples of Bs values.
[0240] The block boundary on which the deblocking filter process isperformed is, for example, a boundary between CUs, PUs, or TUs having8x8 pixel blocks as illustrated in FIG. 26. The deblocking filter processis performed, for example, in units of four rows or four columns. First,boundary strength (Bs) values are determined as indicated in FIG. 27 forblock P and block Q illustrated in FIG. 26.
[0241] According to the Bs values in FIG. 27, whether to performdeblocking filter processes of block boundaries belonging to the sameimage using different strengths may be determined. The deblockingfilter process for a chroma signal is performed when a Bs value is 2. Thedeblocking filter process for a luma signal is performed when a Bs value is1 or more and a determined condition is satisfied. It is to be noted thatconditions for determining Bs values are not limited to those indicated inFIG. 27, and a Bs value may be determined based on another parameter.
[0242] [Predictor (Intra Predictor, Inter Predictor, Prediction Controller)]FIG. 28 is a flow chart illustrating one example of a processperformed by a predictor of encoder 100. It is to be noted that thepredictor, as one example, includes all or part of the following constituentelements: intra predictor 124; inter predictor 126; and predictioncontroller 128. The prediction executor includes, for example, intrapredictor 124 and inter predictor 126.
[0243] The predictor generates a prediction image of a current block (StepSb_1). It is to be noted that the prediction image is, for example, anintra prediction image (intra prediction signal) or an inter predictionimage (inter prediction signal). More specifically, the predictorgenerates the prediction image of the current block using a reconstructedimage which has been already obtained for another block throughgeneration of a prediction image, generation of a prediction residual,generation of quantized coefficients, restoring of a prediction residual,and addition of a prediction image.
[0244] The reconstructed image may be, for example, an image in areference picture or an image of an encoded block (that is, the other blockdescribed above) in a current picture which is the picture including thecurrent block. The encoded block in the current picture is, for example,a neighboring block of the current block.
[0245] FIG. 29 is a flow chart illustrating another example of a processperformed by the predictor of encoder 100.
[0246] The predictor generates a prediction image using a first method(Step Sc_1a), generates a prediction image using a second method (StepSc_1b), and generates a prediction image using a third method (StepSc_1c). The first method, the second method, and the third method maybe mutually different methods for generating a prediction image. Eachof the first to third methods may be an inter prediction method, an intraprediction method, or another prediction method. The above-describedreconstructed image may be used in these prediction methods.
[0247] Next, the predictor evaluates the prediction images generated inSteps Sc_1a, Sc_1b, and Sc_1c (Step Sc_2). For example, the predictorcalculates costs C for the prediction images generated in Step Sc_1a,Sc_1b, and Sc_1c, and evaluates the prediction images by comparing thecosts C of the prediction images. It is to be noted that cost C iscalculated according to an expression of an R-D optimization model, forexample, C = D + λ x R. In this expression, D indicates compressionartifacts of a prediction image, and is represented as, for example, a sumof absolute differences between the pixel value of a current block and thepixel value of a prediction image. In addition, R indicates a bit rate of astream. In addition, λ indicates, for example, a multiplier according tothe method of Lagrange multiplier.
[0248] The predictor then selects one of the prediction images generatedin Steps Sc_1a, Sc_1b, and Sc_1c (Step Sc_3). In other words, thepredictor selects a method or a mode for obtaining a final predictionimage. For example, the predictor selects the prediction image havingthe smallest cost C, based on costs C calculated for the prediction images.Alternatively, the evaluation in Step Sc_2 and the selection of theprediction image in Step Sc_3 may be made based on a parameter which isused in an encoding process. Encoder 100 may transform information foridentifying the selected prediction image, the method, or the mode into astream. The information may be, for example, a flag or the like. In thisway, decoder 200 is capable of generating a prediction image according tothe method or the mode selected by encoder 100, based on the information.It is to be noted that, in the example illustrated in FIG. 29, the predictorselects any of the prediction images after the prediction images aregenerated using the respective methods. However, the predictor mayselect a method or a mode based on a parameter for use in theabove-described encoding process before generating prediction images,and may generate a prediction image according to the method or modeselected.
[0249] For example, the first method and the second method may be intraprediction and inter prediction, respectively, and the predictor may selecta final prediction image for a current block from prediction imagesgenerated according to the prediction methods.
[0250] FIG. 30 is a flow chart illustrating another example of a processperformed by the predictor of encoder 100.
[0251] First, the predictor generates a prediction image using intraprediction (Step Sd_1a), and generates a prediction image using interprediction (Step Sd_1b). It is to be noted that the prediction imagegenerated by intra prediction is also referred to as an intra predictionimage, and the prediction image generated by inter prediction is alsoreferred to as an inter prediction image.
[0252] Next, the predictor evaluates each of the intra prediction imageand the inter prediction image (Step Sd_2). Cost C described above maybe used in the evaluation. The predictor may then select the predictionimage for which the smallest cost C has been calculated among the intraprediction image and the inter prediction image, as the final predictionimage for the current block (Step Sd_3). In other words, the predictionmethod or the mode for generating the prediction image for the currentblock is selected.
[0253] [Intra Predictor]Intra predictor 124 generates a prediction image (that is, intraprediction image) of a current block by performing intra prediction (alsoreferred to as intra frame prediction) of the current block by referring to ablock or blocks in the current picture which is or are stored in blockmemory 118. More specifically, intra predictor 124 generates an intraprediction image by performing intra prediction by referring to pixelvalues (for example, luma and / or chroma values) of a block or blocksneighboring the current block, and then outputs the intra predictionimage to prediction controller 128.
[0254] For example, intra predictor 124 performs intra prediction byusing one mode from among a plurality of intra prediction modes whichhave been predefined. The intra prediction modes normally include oneor more non-directional prediction modes and a plurality of directionalprediction modes.
[0255] The one or more non-directional prediction modes include, forexample, planar prediction mode and DC prediction mode defined in theH.265 / HEVC standard.
[0256] The plurality of directional prediction modes include, for example,the thirty-three directional prediction modes defined in the H.265 / HEVCstandard. It is to be noted that the plurality of directional predictionmodes may further include thirty-two directional prediction modes inaddition to the thirty-three directional prediction modes (for a total ofsixty-five directional prediction modes). FIG. 31 is a diagramillustrating sixty-seven intra prediction modes in total used in intraprediction (two non-directional prediction modes and sixty-fivedirectional prediction modes). The solid arrows represent thethirty-three directions defined in the H.265 / HEVC standard, and thedashed arrows represent the additional thirty-two directions (the twonon-directional prediction modes are not illustrated in FIG. 31).
[0257] In various kinds of mounting examples, a luma block may bereferred to in intra prediction of a chroma block. In other words, achroma component of the current block may be predicted based on a lumacomponent of the current block. Such intra prediction is also referred toas cross-component linear model (CCLM). The intra prediction mode fora chroma block in which such a luma block is referred to (also referred toas, for example, a CCLM mode) may be added as one of the intraprediction modes for chroma blocks.
[0258] Intra predictor 124 may correct intra-predicted pixel values basedon horizontal / vertical reference pixel gradients. The intra predictionwhich accompanies this sort of correcting is also referred to as positiondependent intra prediction combination (PDPC). Information indicatingwhether to apply PDPC (referred to as, for example, a PDPC flag) isnormally signaled at the CU level. It is to be noted that the signaling ofsuch information does not necessarily need to be performed at the CUlevel, and may be performed at another level (for example, at thesequence level, picture level, slice level, brick level, or CTU level).
[0259] FIG. 32 is a flow chart illustrating one example of a processperformed by intra predictor 124.
[0260] Intra predictor 124 selects one intra prediction mode from aplurality of intra prediction modes (Step Sw_1). Intra predictor 124 thengenerates a prediction image according to the selected intra predictionmode (Step Sw_2). Next, intra predictor 124 determines most probablemodes (MPMs) (Step Sw_3). MPMs include, for example, six intraprediction modes. Two modes among the six intra prediction modes maybe planar mode and DC prediction mode, and the other four modes may bedirectional prediction modes. Intra predictor 124 determines whetherthe intra prediction mode selected in Step Sw_1 is included in the MPMs(Step Sw_4).
[0261] Here, when determining that the intra prediction mode selected inStep Sw_1 is included in the MPMs (Yes in Step Sw_4), intra predictor124 sets an MPM flag to 1 (Step Sw_5), and generates informationindicating the selected intra prediction mode among the MPMs (StepSw_6). It is to be noted that the MPM flag set to 1 and the informationindicating the intra prediction mode are encoded as predictionparameters by entropy encoder 110.
[0262] When determining that the selected intra prediction mode is notincluded in the MPMs (No in Step Sw_4), intra predictor 124 sets theMPM flag to 0 (Step Sw_7). Alternatively, intra predictor 124 does notset any MPM flag. Intra predictor 124 then generates informationindicating the selected intra prediction mode among at least one intraprediction mode which is not included in the MPMs (Step Sw_8). It is tobe noted that the MPM flag set to 0 and the information indicating theintra prediction mode are encoded as prediction parameters by entropyencoder 110. The information indicating the intra prediction modeindicates, for example, any one of 0 to 60.
[0263] [Inter Predictor]Inter predictor 126 generates a prediction image (inter predictionimage) by performing inter prediction (also referred to as inter frameprediction) of the current block by referring to a block or blocks in areference picture which is different from the current picture and is storedin frame memory 122. Inter prediction is performed in units of a currentblock or a current sub-block in the current block. The sub-block isincluded in the block and is a unit smaller than the block. The size ofthe sub-block may be 4x4 pixels, 8x8 pixels, or another size. The size ofthe sub-block may be switched for a unit such as slice, brick, picture, etc.
[0264] For example, inter predictor 126 performs motion estimation in areference picture for a current block or a current sub-block, and finds outa reference block or a reference sub-block which best matches the currentblock or current sub-block. Inter predictor 126 then obtains motioninformation (for example, a motion vector) which compensates a motion ora change from the reference block or the reference sub-block to thecurrent block or the current sub-block. Inter predictor 126 generates aninter prediction image of the current block or the current sub-block byperforming motion compensation (or motion prediction) based on themotion information. Inter predictor 126 outputs the generated interprediction image to prediction controller 128.
[0265] The motion information used in motion compensation may besignaled as inter prediction images in various forms. For example, amotion vector may be signaled. As another example, the differencebetween a motion vector and a motion vector predictor may be signaled.
[0266] [Reference Picture List]FIG. 33 is a diagram illustrating examples of reference pictures.FIG. 34 is a conceptual diagram illustrating examples of reference picturelists. Each reference picture list is a list indicating at least onereference picture stored in frame memory 122. It is to be noted that, inFIG. 33, each of rectangles indicates a picture, each of arrows indicates apicture reference relationship, the horizontal axis indicates time, I, P, andB in the rectangles indicate an intra prediction picture, a uni-predictionpicture, and a bi-prediction picture, respectively, and numerals in therectangles indicate a decoding order. As illustrated in FIG. 33, thedecoding order of the pictures is an order of I0, P1, B2, B3, and B4, andthe display order of the pictures is an order of I0, B3, B2, B4, and P1. Asillustrated in FIG. 34, the reference picture list is a list representingreference picture candidates. For example, one picture (or a slice) mayinclude at least one reference picture list. For example, one referencepicture list is used when a current picture is a uni-prediction picture, andtwo reference picture lists are used when a current picture is abi-prediction picture. In the examples of FIGs. 33 and 34, picture B3which is current picture currPic has two reference picture lists which arethe L0 list and the L1 list. When current picture currPic is picture B3,reference picture candidates for current picture currPic are I0, P1, and B2,and the reference picture lists (which are the L0 list and the L1 list)indicate these pictures. Inter predictor 126 or prediction controller 128specifies which picture in each reference picture list is to be actuallyreferred to in form of a reference picture index refIdxLx. In FIG. 34,reference pictures P1 and B2 are specified by reference picture indicesrefIdxL0 and refIdxL1.
[0267] Such a reference picture list may be generated for each unit suchas a sequence, picture, slice, brick, CTU, or CU. In addition, amongreference pictures indicated in reference picture lists, a reference pictureindex indicating a reference picture to be referred to in inter predictionmay be signaled at the sequence level, picture level, slice level, brick level,CTU level, or CU level. In addition, a common reference picture list maybe used in a plurality of inter prediction modes.
[0268] [Basic Flow of Inter Prediction]FIG. 35 is a flow chart illustrating a basic processing flow of interprediction.
[0269] First, inter predictor 126 generates a prediction signal (Steps Se_1to Se_3). Next, subtractor 104 generates the difference between acurrent block and a prediction image as a prediction residual (Step Se_4).
[0270] Here, in the generation of the prediction image, inter predictor 126generates the prediction image through, for example, determination of amotion vector (MV) of the current block (Steps Se_1 and Se_2) and motioncompensation (Step Se_3). Furthermore, in determination of an MV,inter predictor 126 determines the MV through, for example, selection ofa motion vector candidate (MV candidate) (Step Se_1) and derivation ofan MV (Step Se_2). The selection of the MV candidate is made by meansof, for example, inter predictor 126 generating an MV candidate list andselecting at least one MV candidate from the MV candidate list. It is tobe noted that MVs derived in the past may be added to the MV candidatelist. Alternatively, in derivation of an MV, inter predictor 126 mayfurther select at least one MV candidate from the at least one MVcandidate, and determine the selected at least one MV candidate as theMV for the current block. Alternatively, inter predictor 126 maydetermine the MV for the current block by performing estimation in areference picture region specified by each of the selected at least one MVcandidate. It is to be noted that the estimation in the reference pictureregion may be referred to as motion estimation.
[0271] In addition, although Steps Se_1 to Se_3 are performed by interpredictor 126 in the above-described example, a process that is, forexample, Step Se_1, Step Se_2, or the like may be performed by anotherconstituent element included in encoder 100.
[0272] It is to be noted that an MV candidate list may be generated foreach process in inter prediction mode, or a common MV candidate list maybe used in a plurality of inter prediction modes. The processes in StepsSe_3 and Se_4 correspond to Steps Sa_3 and Sa_4 illustrated in FIG. 9,respectively. The process in Step Se_3 corresponds to the process in StepSd_1b in FIG. 30.
[0273] [MV Derivation Flow]FIG. 36 is a flow chart illustrating one example of MV derivation.
[0274] Inter predictor 126 may derive an MV for a current block in a modefor encoding motion information (for example, an MV). In this case, forexample, the motion information may be encoded as a predictionparameter, and may be signaled. In other words, the encoded motioninformation is included in a stream.
[0275] Alternatively, inter predictor 126 may derive an MV in a mode inwhich motion information is not encoded. In this case, no motioninformation is included in the stream.
[0276] Here, MV derivation modes include a normal inter mode, a normalmerge mode, a FRUC mode, an affine mode, etc. which are described later.Modes in which motion information is encoded among the modes includethe normal inter mode, the normal merge mode, the affine mode(specifically, an affine inter mode and an affine merge mode), etc. It is tobe noted that motion information may include not only an MV but alsoMV predictor selection information which is described later. Modes inwhich no motion information is encoded include the FRUC mode, etc.Inter predictor 126 selects a mode for deriving an MV of the current blockfrom the plurality of modes, and derives the MV of the current block usingthe selected mode.
[0277] FIG. 37 is a flow chart illustrating another example of MVderivation.
[0278] Inter predictor 126 may derive an MV for a current block in a modein which an MV difference is encoded. In this case, for example, the MVdifference is encoded as a prediction parameter, and is signaled. Inother words, the encoded MV difference is included in a stream. The MVdifference is the difference between the MV of the current block and theMV predictor. It is to be noted that the MV predictor is a motion vectorpredictor.
[0279] Alternatively, inter predictor 126 may derive an MV in a mode inwhich no MV difference is encoded. In this case, no encoded MVdifference is included in the stream.
[0280] Here, as described above, the MV derivation modes include thenormal inter mode, the normal merge mode, the FRUC mode, the affinemode, etc. which are described later. Modes in which an MV difference isencoded among the modes include the normal inter mode, the affine mode(specifically, the affine inter mode), etc. Modes in which no MVdifference is encoded include the FRUC mode, the normal merge mode,the affine mode (specifically, the affine merge mode), etc. Inter predictor126 selects a mode for deriving an MV of the current block from theplurality of modes, and derives the MV for the current block using theselected mode.
[0281] [MV Derivation Modes]FIGs. 38A and 38B are each a diagram illustrating one example ofcategorization of modes for MV derivation. For example, as illustratedin FIG. 38A, MV derivation modes are roughly categorized into threemodes according to whether to encode motion information and whether toencode MV differences. The three modes are inter mode, merge mode,and frame rate up-conversion (FRUC) mode. The inter mode is a mode inwhich motion estimation is performed, and in which motion informationand an MV difference are encoded. For example, as illustrated in FIG.38B, the inter mode includes affine inter mode and normal inter mode.The merge mode is a mode in which no motion estimation is performed,and in which an MV is selected from an encoded surrounding block and anMV for the current block is derived using the MV. The merge mode is amode in which, basically, motion information is encoded and no MVdifference is encoded. For example, as illustrated in FIG. 38B, themerge modes include normal merge mode (also referred to as normalmerge mode or regular merge mode), merge with motion vector difference(MMVD) mode, combined inter merge / intra prediction (CIIP) mode,triangle mode, ATMVP mode, and affine merge mode. Here, an MVdifference is encoded exceptionally in the MMVD mode among the modesincluded in the merge modes. It is to be noted that the affine mergemode and the affine inter mode are modes included in the affine modes.The affine mode is a mode for deriving, as an MV of a current block, anMV of each of a plurality of sub-blocks included in the current block,assuming affine transform. The FRUC mode is a mode which is forderiving an MV of the current block by performing estimation betweenencoded regions, and in which neither motion information nor any MVdifference is encoded. It is to be noted that the respective modes will bedescribed later in detail.
[0282] It is to be noted that the categorization of the modes illustrated inFIGs. 38A and 38B are examples, and categorization is not limited thereto.For example, when an MV difference is encoded in CIIP mode, the CIIPmode is categorized into inter modes.
[0283] [MV Derivation > Normal Inter Mode]The normal inter mode is an inter prediction mode for deriving anMV of a current block by finding out a block similar to the image of thecurrent block from a reference picture region specified by an MVcandidate. In this normal inter mode, an MV difference is encoded.
[0284] FIG. 39 is a flow chart illustrating an example of inter predictionby normal inter mode.
[0285] First, inter predictor 126 obtains a plurality of MV candidates fora current block based on information such as MVs of a plurality ofencoded blocks temporally or spatially surrounding the current block(Step Sg_1). In other words, inter predictor 126 generates an MVcandidate list.
[0286] Next, inter predictor 126 extracts N (an integer of 2 or larger) MVcandidates from the plurality of MV candidates obtained in Step Sg_1, asmotion vector predictor candidates according to a predetermined priorityorder (Step Sg_2). It is to be noted that the priority order is determinedin advance for each of the N MV candidates.
[0287] Next, inter predictor 126 selects one MV predictor candidate fromthe N MV predictor candidates as the MV predictor for the current block(Step Sg_3). At this time, inter predictor 126 encodes, in a stream, MVpredictor selection information for identifying the selected MV predictor.In other words, inter predictor 126 outputs the MV predictor selectioninformation as a prediction parameter to entropy encoder 110 throughprediction parameter generator 130.
[0288] Next, inter predictor 126 derives an MV of a current block byreferring to an encoded reference picture (Step Sg_4). At this time, interpredictor 126 further encodes, in the stream, the difference value betweenthe derived MV and the MV predictor as an MV difference. In otherwords, inter predictor 126 outputs the MV difference as a predictionparameter to entropy encoder 110 through prediction parametergenerator 130. It is to be noted that the encoded reference picture is apicture including a plurality of blocks which have been reconstructedafter being encoded.
[0289] Lastly, inter predictor 126 generates a prediction image for thecurrent block by performing motion compensation of the current blockusing the derived MV and the encoded reference picture (Step Sg_5).The processes in Steps Sg_1 to Sg_5 are executed on each block. Forexample, when the processes in Steps Sg_1 to Sg_5 are executed on eachof all the blocks in the slice, inter prediction of the slice using the normalinter mode finishes. For example, when the processes in Steps Sg_1 toSg_5 are executed on each of all the blocks in the picture, inter predictionof the picture using the normal inter mode finishes. It is to be noted thatnot all the blocks included in the slice may be subjected to the processesin Steps Sg_1 to Sg_5, and inter prediction of the slice using the normalinter mode may finish when part of the blocks are subjected to theprocesses. Likewise, inter prediction of the picture using the normalinter mode may finish when the processes in Steps Sg_1 to Sg_5 areexecuted on part of the blocks in the picture.
[0290] It is to be noted that the prediction image is an inter predictionsignal as described above. In addition, information indicating the interprediction mode (normal inter mode in the above example) used togenerate the prediction image is, for example, encoded as a predictionparameter in an encoded signal.
[0291] It is to be noted that the MV candidate list may be also used as alist for use in another mode. In addition, the processes related to the MVcandidate list may be applied to processes related to the list for use inanother mode. The processes related to the MV candidate list include,for example, extraction or selection of an MV candidate from the MVcandidate list, reordering of MV candidates, or deletion of an MVcandidate.
[0292] [MV Derivation > Normal Merge Mode]The normal merge mode is an inter prediction mode for selectingan MV candidate from an MV candidate list as an MV for a current block,thereby deriving the MV. It is to be noted that the normal merge mode isa merge mode in a narrow meaning and is also simply referred to as amerge mode. In this embodiment, the normal merge mode and the mergemode are distinguished, and the merge mode is used in a broad meaning.
[0293] FIG. 40 is a flow chart illustrating an example of inter predictionby normal merge mode.
[0294] First, inter predictor 126 obtains a plurality of MV candidates fora current block based on information such as MVs of a plurality ofencoded blocks temporally or spatially surrounding the current block(Step Sh_1). In other words, inter predictor 126 generates an MVcandidate list.
[0295] Next, inter predictor 126 selects one MV candidate from theplurality of MV candidates obtained in Step Sh_1, thereby deriving anMV for the current block (Step Sh_2). At this time, inter predictor 126encodes, in a stream, MV selection information for identifying theselected MV candidate. In other words, inter predictor 126 outputs theMV selection information as a prediction parameter to entropy encoder110 through prediction parameter generator 130.
[0296] Lastly, inter predictor 126 generates a prediction image for thecurrent block by performing motion compensation of the current blockusing the derived MV and the encoded reference picture (Step Sh_3).The processes in Steps Sh_1 to Sh_3 are executed, for example, on eachblock. For example, when the processes in Steps Sh_1 to Sh_3 areexecuted on each of all the blocks in the slice, inter prediction of the sliceusing the normal merge mode finishes. In addition, when the processesin Steps Sh_1 to Sh_3 are executed on each of all the blocks in the picture,inter prediction of the picture using the normal merge mode finishes. Itis to be noted that not all the blocks included in the slice may be subjectedto the processes in Steps Sh_1 to Sh_3, and inter prediction of the sliceusing the normal merge mode may finish when part of the blocks aresubjected to the processes. Likewise, inter prediction of the pictureusing the normal merge mode may finish when the processes in StepsSh_1 to Sh_3 are executed on part of the blocks in the picture.
[0297] In addition, information indicating the inter prediction mode(normal merge mode in the above example) used to generate theprediction image is, for example, encoded as a prediction parameter in astream.
[0298] FIG. 41 is a diagram for illustrating one example of an MVderivation process for a current picture by normal merge mode.
[0299] First, inter predictor 126 generates an MV candidate list in whichMV candidates are registered. Examples of MV candidates include:spatially neighboring MV candidates which are MVs of a plurality ofencoded blocks located spatially surrounding a current block; temporallyneighboring MV candidates which are MVs of surrounding blocks onwhich the position of a current block in an encoded reference picture isprojected; combined MV candidates which are MVs generated bycombining the MV value of a spatially neighboring MV predictor and theMV value of a temporally neighboring MV predictor; and a zero MVcandidate which is an MV having a zero value.
[0300] Next, inter predictor 126 selects one MV candidate from aplurality of MV candidates registered in an MV candidate list, anddetermines the MV candidate as the MV of the current block.
[0301] Furthermore, entropy encoder 110 writes and encodes, in a stream,merge_idx which is a signal indicating which MV candidate has beenselected.
[0302] It is to be noted that the MV candidates registered in the MVcandidate list described in FIG. 41 are examples. The number of MVcandidates may be different from the number of MV candidates in thediagram, the MV candidate list may be configured in such a manner thatsome of the kinds of the MV candidates in the diagram may not beincluded, or that one or more MV candidates other than the kinds of MVcandidates in the diagram are included.
[0303] A final MV may be determined by performing a dynamic motionvector refreshing (DMVR) to be described later using the MV of thecurrent block derived by normal merge mode. It is to be noted that, innormal merge mode, no MV difference is encoded, but an MV difference isencoded. In MMVD mode, one MV candidate is selected from an MVcandidate list as in the case of normal merge mode, an MV difference isencoded. As illustrated in FIG. 38B, MMVD may be categorized intomerge modes together with normal merge mode. It is to be noted thatthe MV difference in MMVD mode does not always need to be the same asthe MV difference for use in inter mode. For example, MV differencederivation in MMVD mode may be a process that requires a smalleramount of processing than the amount of processing required for MVdifference derivation in inter mode.
[0304] In addition, a combined inter merge / intra prediction (CIIP) modemay be performed. The mode is for overlapping a prediction imagegenerated in inter prediction and a prediction image generated in intraprediction to generate a prediction image for a current block.
[0305] It is to be noted that the MV candidate list may be referred to as acandidate list. In addition, merge_idx is MV selection information.
[0306] [MV Derivation > HMVP Mode]FIG. 42 is a diagram for illustrating one example of an MVderivation process for a current picture by HMVP merge mode.
[0307] In normal merge mode, an MV for, for example, a CU which is acurrent block is determined by selecting one MV candidate from an MVcandidate list generated by referring to an encoded block (for example, aCU). Here, another MV candidate may be registered in the MVcandidate list. The mode in which such another MV candidate isregistered is referred to as HMVP mode.
[0308] In HMVP mode, MV candidates are managed using a first-infirst-out (FIFO) buffer for HMVP, separately from the MV candidate listfor normal merge mode.
[0309] In FIFO buffer, motion information such as MVs of blocksprocessed in the past are stored newest first. In the management of theFIFO buffer, each time when one block is processed, the MV for thenewest block (that is the CU processed immediately before) is stored inthe FIFO buffer, and the MV of the oldest CU (that is, the CU processedearliest) is deleted from the FIFO buffer. In the example illustrated inFIG. 42, HMVP1 is the MV for the newest block, and HMVP5 is the MVfor the oldest MV.
[0310] Inter predictor 126 then, for example, checks whether each MVmanaged in the FIFO buffer is an MV different from all the MVcandidates which have been already registered in the MV candidate listfor normal merge mode starting from HMVP1. When determining thatthe MV is different from all the MV candidates, inter predictor 126 mayadd the MV managed in the FIFO buffer in the MV candidate list fornormal merge mode as an MV candidate. At this time, the MV candidateregistered from the FIFO buffer may be one or more.
[0311] By using the HMVP mode in this way, it is possible to add not onlythe MV of a block which neighbors the current block spatially ortemporally but also an MV for a block processed in the past. As a result,the variation of MV candidates for normal merge mode is expanded,which increases the probability that coding efficiency can be increased.
[0312] It is to be noted that the MV may be motion information. In otherwords, information stored in the MV candidate list and the FIFO buffermay include not only MV values but also reference picture information,reference directions, the numbers of pictures, etc. In addition, the blockis, for example, a CU.
[0313] It is to be noted that the MV candidate list and the FIFO bufferillustrated in FIG. 42 are examples. The MV candidate list and FIFObuffer may be different in size from those in FIG. 42, or may be configuredto register MV candidates in an order different from the one in FIG. 42.In addition, the process described here is common between encoder 100and decoder 200.
[0314] It is to be noted that the HMVP mode can be applied for modesother than the normal merge mode. For example, it is also excellent thatmotion information such as MVs of blocks processed in affine mode in thepast may be stored newest first, and may be used as MV candidates. Themode obtained by applying HMVP mode to affine mode may be referred toas history affine mode.
[0315] [MV Derivation > FRUC Mode]Motion information may be derived at the decoder 200 side withoutbeing signaled from the encoder 100 side. For example, motioninformation may be derived by performing motion estimation at thedecoder 200 side. At this time, at the decoder 200 side, motionestimation is performed without using any pixel value in a current block.Modes in which motion estimation is performed at the decoder 200 side inthis way include a frame rate up-conversion (FRUC) mode, a patternmatched motion vector derivation (PMMVD) mode, etc.
[0316] One example of a FRUC process is illustrated in FIG. 43. First, alist which indicates, as MV candidates, MVs for encoded blocks each ofwhich neighbors the current block spatially or temporally is generated byreferring to the MVs (the list may be an MV candidate list, and be alsoused as the MV candidate list for normal merge mode) (Step Si_1). Next,a best MV candidate is selected from the plurality of MV candidatesregistered in the MV candidate list (Step Si_2). For example, theevaluation values of the respective MV candidates included in the MVcandidate list are calculated, and one MV candidate is selected as the bestMV candidate based on the evaluation values. Based on the selectedbest MV candidate, a motion vector for the current block is then derived(Step Si_4). More specifically, for example, the selected best MVcandidate is directly derived as the MV for the current block. Inaddition, for example, the MV for the current block may be derived usingpattern matching in a surrounding region of a position which is includedin a reference picture and corresponds to the selected best MV candidate.In other words, estimation using the pattern matching in a referencepicture and the evaluation values may be performed in the surroundingregion of the best MV candidate, and when there is an MV that yields abetter evaluation value, the best MV candidate may be updated to the MVthat yields the better evaluation value, and the updated MV may bedetermined as the final MV for the current block. Update to the MV thatyields the better evaluation value may not be performed.
[0317] Lastly, inter predictor 126 generates a prediction image for thecurrent block by performing motion compensation of the current blockusing the derived MV and the encoded reference picture (Step Si_5). Theprocesses in Steps Si_1 to Si_5 are executed, for example, on each block.For example, when the processes in Steps Si_1 to Si_5 are executed oneach of all the blocks in the slice, inter prediction of the slice using theFRUC mode finishes. For example, when the processes in Steps Si_1 toSi_5 are executed on each of all the blocks in the picture, inter predictionof the picture using the FRUC mode finishes. It is to be noted that notall the blocks included in the slice may be subjected to the processes inSteps Si_1 to Si_5, and inter prediction of the slice using the FRUC modemay finish when part of the blocks are subjected to the processes.Likewise, inter prediction of the picture using the FRUC mode may finishwhen the processes in Steps Si_1 to Si_5 are executed on part of theblocks included in the picture.
[0318] Each sub-block may be processed similarly to the above-describedcase of processing each block.
[0319] Evaluation values may be calculated according to various kinds ofmethods. For example, a comparison is made between a reconstructedimage in a region in a reference picture corresponding to an MV and areconstructed image in a determined region (the region may be, forexample, a region in another reference picture or a region in aneighboring block of a current picture, as indicated below). Thedifference between the pixel values of the two reconstructed images maybe used for an evaluation value of the MV. It is to be noted that anevaluation value may be calculated using information other than thevalue of the difference.
[0320] Next, pattern matching is described in detail. First, one MVcandidate included in an MV candidate list (also referred to as a mergelist) is selected as a starting point for estimation by pattern matching.As the pattern matching, either a first pattern matching or a secondpattern matching may be used. The first pattern matching and thesecond pattern matching may be referred to as bilateral matching andtemplate matching, respectively.
[0321] [MV Derivation > FRUC > Bilateral Matching]In the first pattern matching, the pattern matching is performedbetween two blocks which are located along a motion trajectory of acurrent block and included in two different reference pictures.Accordingly, in the first pattern matching, a region in another referencepicture located along the motion trajectory of the current block is used asa determined region for calculating the evaluation value of theabove-described MV candidate.
[0322] FIG. 44 is a diagram for illustrating one example of the firstpattern matching (bilateral matching) between the two blocks in the tworeference pictures located along the motion trajectory. As illustrated inFIG. 44, in the first pattern matching, two motion vectors (MV0, MV1) arederived by estimating a pair which best matches among pairs of twoblocks which are included in the two different reference pictures (Ref0,Ref1) and located along the motion trajectory of the current block (Curblock). More specifically, a difference between the reconstructed imageat a specified position in the first encoded reference picture (Ref0)specified by an MV candidate and the reconstructed image at a specifiedposition in the second encoded reference picture (Ref1) specified by asymmetrical MV obtained by scaling the MV candidate at a display timeinterval is derived for the current block, and an evaluation value iscalculated using the value of the obtained difference. It is excellent toselect, as the best MV, the MV candidate which yields the best evaluationvalue among the plurality of MV candidates.
[0323] In the assumption of a continuous motion trajectory, the motionvectors (MV0, MV1) specifying the two reference blocks are proportionalto temporal distances (TD0, TD1) between the current picture (Cur Pic)and the two reference pictures (Ref0, Ref1). For example, when thecurrent picture is temporally located between the two reference picturesand the temporal distances from the current picture to the respective tworeference pictures are equal to each other, mirror-symmetricalbi-directional MVs are derived in the first pattern matching.
[0324] [MV Derivation > FRUC > Template Matching]In the second pattern matching (template matching), patternmatching is performed between a block in a reference picture and atemplate in the current picture (the template is a block neighboring thecurrent block in the current picture (the neighboring block is, for example,an upper and / or left neighboring block(s))). Accordingly, in the secondpattern matching, the block neighboring the current block in the currentpicture is used as the determined region for calculating the evaluationvalue of the above-described MV candidate.
[0325] FIG. 45 is a diagram for illustrating one example of patternmatching (template matching) between a template in a current pictureand a block in a reference picture. As illustrated in FIG. 45, in thesecond pattern matching, the MV for the current block (Cur block) isderived by estimating, in the reference picture (Ref0), the block whichbest matches the block neighboring the current block in the currentpicture (Cur Pic). More specifically, the difference between areconstructed image in an encoded region which neighbors both left andabove or either left or above and a reconstructed image which is in acorresponding region in the encoded reference picture (Ref0) and isspecified by an MV candidate is derived, and an evaluation value iscalculated using the value of the obtained difference. It is excellent toselect, as the best MV candidate, the MV candidate which yields the bestevaluation value among the plurality of MV candidates.
[0326] Such information indicating whether to apply the FRUC mode(referred to as, for example, a FRUC flag) may be signaled at the CU level.In addition, when the FRUC mode is applied (for example, when a FRUCflag is true), information indicating an applicable pattern matchingmethod (either the first pattern matching or the second pattern matching)may be signaled at the CU level. It is to be noted that the signaling ofsuch information does not necessarily need to be performed at the CUlevel, and may be performed at another level (for example, at thesequence level, picture level, slice level, brick level, CTU level, orsub-block level).
[0327] [MV Derivation > Affine Mode]The affine mode is a mode for generating an MV using affinetransform. For example, an MV may be derived in units of a sub-blockbased on motion vectors of a plurality of neighboring blocks. This modeis also referred to as an affine motion compensation prediction mode.
[0328] FIG. 46A is a diagram for illustrating one example of MVderivation in units of a sub-block based on MVs of a plurality ofneighboring blocks. In FIG. 46A, the current block includes sixteen 4x4pixel sub-blocks. Here, motion vector v0 at an upper-left corner controlpoint in the current block is derived based on an MV of a neighboringblock, and likewise, motion vector v1 at an upper-right corner controlpoint in the current block is derived based on an MV of a neighboringsub-block. Two motion vectors v0 and v1 are projected according to anexpression (1A) indicated below, and motion vectors (vx, vy) for therespective sub-blocks in the current block are derived.
[0329] [MATH. 1](1A)
[0330] Here, x and y indicate the horizontal position and the verticalposition of the sub-block, respectively, and w indicates a predeterminedweighting coefficient.
[0331] Such information indicating the affine mode (for example, referredto as an affine flag) may be signaled at the CU level. It is to be notedthat the signaling of such information does not necessarily need to beperformed at the CU level, and may be performed at another level (forexample, at the sequence level, picture level, slice level, brick level, CTUlevel, or sub-block level).
[0332] In addition, the affine mode may include several modes fordifferent methods for deriving MVs at the upper-left and upper-rightcorner control points. For example, the affine modes include two modeswhich are the affine inter mode (also referred to as an affine normal intermode) and the affine merge mode.
[0333] FIG. 46B is a diagram for illustrating one example of MVderivation in units of a sub-block in affine mode in which three controlpoints are used. In FIG. 46B, the current block includes, for example,sixteen 4x4 pixel sub-blocks. Here, motion vector v0 at an upper-leftcorner control point in the current block is derived based on an MV of aneighboring block. Here, motion vector v1 at an upper-right cornercontrol point in the current block is derived based on an MV of aneighboring block, and likewise, motion vector v2 at a lower-left cornercontrol point for the current block is derived based on an MV of aneighboring block. Three motion vectors v0, v1, and v2 are projectedaccording to an expression (1B) indicated below, and motion vectors (vx,vy) for the respective sub-blocks in the current block are derived.
[0334] [MATH. 2](1B)
[0335] Here, x and y indicate the horizontal position and the verticalposition of the sub-block, respectively, and each of w and h indicates apredetermined weighting coefficient. Here, w may indicate the width ofa current block, and h may indicate the height of the current block.
[0336] Affine modes in which different numbers of control points (forexample, two and three control points) are used may be switched andsignaled at the CU level. It is to be noted that information indicatingthe number of control points in affine mode used at the CU level may besignaled at another level (for example, the sequence level, picture level,slice level, brick level, CTU level, or sub-block level).
[0337] In addition, such an affine mode in which three control points areused may include different methods for deriving MVs at the upper-left,upper-right, and lower-left corner control points. For example, theaffine modes in which three control points are used include two modeswhich are affine inter mode and affine merge mode, as in the case ofaffine modes in which two control points are used.
[0338] It is to be noted that, in the affine modes, the size of each sub-blockincluded in the current block may not be limited to 4x4 pixels, and may beanother size. For example, the size of each sub-block may be 8x8 pixels.
[0339] [MV Derivation > Affine Mode > Control Point]FIGs. 47A, 47B, and 47C are each a conceptual diagram forillustrating one example of MV derivation at control points in an affinemode.
[0340] As illustrated in FIG. 47A, in the affine mode, for example, MVpredictors at respective control points for a current block are calculatedbased on a plurality of MVs corresponding to blocks encoded according tothe affine mode among encoded block A (left), block B (upper), block C(upper-right), block D (lower-left), and block E (upper-left) whichneighbor the current block. More specifically, encoded block A (left),block B (upper), block C (upper-right), block D (lower-left), and block E(upper-left) are checked in the listed order, and the first effective blockencoded according to the affine mode is identified. The MV at eachcontrol point for the current block is calculated based on the plurality ofMVs corresponding to the identified block.
[0341] For example, as illustrated in FIG. 47B, when block A whichneighbors to the left of the current block has been encoded according to anaffine mode in which two control points are used, motion vectors v3 and v4projected at the upper-left corner position and the upper-right cornerposition of the encoded block including block A are derived. Motionvector v0 at the upper-left control point and motion vector v1 at theupper-right control point for the current block are then calculated fromderived motion vectors v3 and v4.
[0342] For example, as illustrated in FIG. 47C, when block A whichneighbors to the left of the current block has been encoded according to anaffine mode in which three control points are used, motion vectors v3, v4,and v5 projected at the upper-left corner position, the upper-right cornerposition, and the lower-left corner position of the encoded block includingblock A are derived. Motion vector v0 at the upper-left control point forthe current block, motion vector v1 at the upper-right control point for thecurrent block, and motion vector v2 at the lower-left control point for thecurrent block are then calculated from derived motion vectors v3, v4, andv5.
[0343] The MV derivation methods illustrated in FIGs. 47A to 47C may beused in the MV derivation at each control point for the current block inStep Sk_1 illustrated in FIG. 50 described later, or may be used for MVpredictor derivation at each control point for the current block in StepSj_1 illustrated in FIG. 51 described later.
[0344] FIGs. 48A and 48B are each a conceptual diagram for illustratinganother example of MV derivation at control points in affine mode.
[0345] FIG. 48A is a diagram for illustrating an affine mode in which twocontrol points are used.
[0346] In the affine mode, as illustrated in FIG. 48A, an MV selected fromMVs at encoded block A, block B, and block C which neighbor the currentblock is used as motion vector v0 at the upper-left corner control point forthe current block. Likewise, an MV selected from MVs of encoded blockD and block E which neighbor the current block is used as motion vectorv1 at the upper-right corner control point for the current block.
[0347] FIG. 48B is a diagram for illustrating an affine mode in whichthree control points are used.
[0348] In the affine mode, as illustrated in FIG. 48B, an MV selected fromMVs at encoded block A, block B, and block C which neighbor the currentblock is used as motion vector v0 at the upper-left corner control point forthe current block. Likewise, an MV selected from MVs of encoded blockD and block E which neighbor the current block is used as motion vectorv1 at the upper-right corner control point for the current block.Furthermore, an MV selected from MVs of encoded block F and block Gwhich neighbor the current block is used as motion vector v2 at thelower-left corner control point for the current block.
[0349] It is to be noted that the MV derivation methods illustrated inFIGs. 48A and 48B may be used in the MV derivation at each control pointfor the current block in Step Sk_1 illustrated in FIG. 50 described later,or may be used for MV predictor derivation at each control point for thecurrent block in Step Sj_1 illustrated in FIG. 51 described later.
[0350] Here, when affine modes in which different numbers of controlpoints (for example, two and three control points) are used may beswitched and signaled at the CU level, the number of control points for anencoded block and the number of control points for a current block may bedifferent from each other.
[0351] FIGs. 49A and 49B are each a conceptual diagram for illustratingone example of a method for MV derivation at control points when thenumber of control points for an encoded block and the number of controlpoints for a current block are different from each other.
[0352] For example, as illustrated in FIG. 49A, a current block has threecontrol points at the upper-left corner, the upper-right corner, and thelower-left corner, and block A which neighbors to the left of the currentblock has been encoded according to an affine mode in which two controlpoints are used. In this case, motion vectors v3 and v4 projected at theupper-left corner position and the upper-right corner position in theencoded block including block A are derived. Motion vector v0 at theupper-left corner control point and motion vector v1 at the upper-rightcorner control point for the current block are then calculated from derivedmotion vectors v3 and v4. Furthermore, motion vector v2 at the lower-leftcorner control point is calculated from derived motion vectors v0 and v1.
[0353] For example, as illustrated in FIG. 49B, a current block has twocontrol points at the upper-left corner and the upper-right corner, andblock A which neighbors to the left of the current block has been encodedaccording to an affine mode in which three control points are used. Inthis case, motion vectors v3, v4, and v5 projected at the upper-left cornerposition in the encoded block including block A, the upper-right cornerposition in the encoded block, and the lower-left corner position in theencoded block are derived. Motion vector v0 at the upper-left cornercontrol point for the current block and motion vector v1 at the upper-rightcorner control point for the current block are then calculated from derivedmotion vectors v3, v4, and v5.
[0354] It is to be noted that the MV derivation methods illustrated inFIGs. 49A and 49B may be used in the MV derivation at each control pointfor the current block in Step Sk_1 illustrated in FIG. 50 described later,or may be used for MV predictor derivation at each control point for thecurrent block in Step Sj_1 illustrated in FIG. 51 described later.
[0355] [MV Derivation > Affine Mode > Affine Merge Mode]FIG. 50 is a flow chart illustrating one example of the affine mergemode.
[0356] In the affine merge mode, first, inter predictor 126 derives MVs atrespective control points for a current block (Step Sk_1). The controlpoints are an upper-left corner point of the current block and anupper-right corner point of the current block as illustrated in FIG. 46A, oran upper-left corner point of the current block, an upper-right cornerpoint of the current block, and a lower-left corner point of the currentblock as illustrated in FIG. 46B. At this time, inter predictor 126 mayencode MV selection information for identifying two or three derived MVsin a stream.
[0357] For example, when MV derivation methods illustrated in FIGs.47A to 47C are used, as illustrated in FIG. 47A, inter predictor 126 checksencoded block A (left), block B (upper), block C (upper-right), block D(lower-left), and block E (upper-left) in the listed order, and identifies thefirst effective block encoded according to the affine mode.
[0358] Inter predictor 126 derives the MV at the control point using theidentified first effective block encoded according to the identified affinemode. For example, when block A is identified and block A has twocontrol points, as illustrated in FIG. 47B, inter predictor 126 calculatesmotion vector v0 at the upper-left corner control point of the current blockand motion vector v1 at the upper-right corner control point of the currentblock from motion vectors v3 and v4 at the upper-left corner of the encodedblock including block A and the upper-right corner of the encoded block.For example, inter predictor 126 calculates motion vector v0 at theupper-left corner control point of the current block and motion vector v1 atthe upper-right corner control point of the current block by projectingmotion vectors v3 and v4 at the upper-left corner and the upper-rightcorner of the encoded block onto the current block.
[0359] Alternatively, when block A is identified and block A has threecontrol points, as illustrated in FIG. 47C, inter predictor 126 calculatesmotion vector v0 at the upper-left corner control point of the current block,motion vector v1 at the upper-right corner control point of the currentblock, and motion vector v2 at the lower-left corner control point of thecurrent block from motion vectors v3, v4, and v5 at the upper-left corner ofthe encoded block including block A, the upper-right corner of the encodedblock, and the lower-left corner of the encoded block. For example, interpredictor 126 calculates motion vector v0 at the upper-left corner controlpoint of the current block, motion vector v1 at the upper-right cornercontrol point of the current block, and motion vector v2 at the lower-leftcorner control point of the current block by projecting motion vectors v3,v4, and v5 at the upper-left corner, the upper-right corner, and thelower-left corner of the encoded block onto the current block.
[0360] It is to be noted that, as illustrated in FIG. 49A described above,MVs at three control points may be calculated when block A is identifiedand block A has two control points, and that, as illustrated in FIG. 49Bdescribed above, MVs at two control points may be calculated when blockA is identified and block A has three control points.
[0361] Next, inter predictor 126 performs motion compensation of each ofa plurality of sub-blocks included in the current block. In other words,inter predictor 126 calculates an MV for each of the plurality ofsub-blocks as an affine MV, using either two motion vectors v0 and v1 andthe above expression (1A) or three motion vectors v0, v1, and v2 and theabove expression (1B) (Step Sk_2). Inter predictor 126 then performsmotion compensation of the sub-blocks using these affine MVs andencoded reference pictures (Step Sk_3). When the processes in StepsSk_2 and Sk_3 are executed for each of all the sub-blocks included in thecurrent block, the process for generating a prediction image using theaffine merge mode for the current block finishes. In other words, motioncompensation of the current block is performed to generate a predictionimage of the current block.
[0362] It is to be noted that the above-described MV candidate list may begenerated in Step Sk_1. The MV candidate list may be, for example, alist including MV candidates derived using a plurality of MV derivationmethods for each control point. The plurality of MV derivation methodsmay be any combination of the MV derivation methods illustrated in FIGs.47A to 47C, the MV derivation methods illustrated in FIGs. 48A and 48B,the MV derivation methods illustrated in FIGs. 49A and 49B, and otherMV derivation methods.
[0363] It is to be noted that MV candidate lists may include MVcandidates in a mode in which prediction is performed in units of asub-block, other than the affine mode.
[0364] It is to be noted that, for example, an MV candidate list includingMV candidates in an affine merge mode in which two control points areused and an affine merge mode in which three control points are used maybe generated as an MV candidate list. Alternatively, an MV candidatelist including MV candidates in the affine merge mode in which twocontrol points are used and an MV candidate list including MV candidatesin the affine merge mode in which three control points are used may begenerated separately. Alternatively, an MV candidate list including MVcandidates in one of the affine merge mode in which two control points areused and the affine merge mode in which three control points are usedmay be generated. The MV candidate(s) may be, for example, MVs forencoded block A (left), block B (upper), block C (upper-right), block D(lower-left), and block E (upper-left), or an MV for an effective blockamong the blocks.
[0365] It is to be noted that index indicating one of the MVs in an MVcandidate list may be transmitted as MV selection information.
[0366] [MV Derivation > Affine Mode > Affine Inter Mode]FIG. 51 is a flow chart illustrating one example of an affine intermode.
[0367] In the affine inter mode, first, inter predictor 126 derives MVpredictors (v0, v1) or (v0, v1, v2) of respective two or three control points fora current block (Step Sj_1). The control points are an upper-left cornerpoint for the current block, an upper-right corner point of the currentblock, and a lower-left corner point for the current block as illustrated inFIG. 46A or FIG. 46B.
[0368] For example, when the MV derivation methods illustrated in FIGs.48A and 48B are used, inter predictor 126 derives the MV predictors (v0,v1) or (v0, v1, v2) at respective two or three control points for the currentblock by selecting MVs of any of the blocks among encoded blocks in thevicinity of the respective control points for the current block illustrated ineither FIG. 48A or FIG. 48B. At this time, inter predictor 126 encodes,in a stream, MV predictor selection information for identifying theselected two or three MV predictors.
[0369] For example, inter predictor 126 may determine, using a costevaluation or the like, the block from which an MV as an MV predictor ata control point is selected from among encoded blocks neighboring thecurrent block, and may write, in a bitstream, a flag indicating which MVpredictor has been selected. In other words, inter predictor 126 outputs,as a prediction parameter, the MV predictor selection information such asa flag to entropy encoder 110 through prediction parameter generator130.
[0370] Next, inter predictor 126 performs motion estimation (Steps Sj_3and Sj_4) while updating the MV predictor selected or derived in StepSj_1 (Step Sj_2). In other words, inter predictor 126 calculates, as anaffine MV, an MV of each of sub-blocks which corresponds to an updatedMV predictor, using either the expression (1A) or expression (1B)described above (Step Sj_3). Inter predictor 126 then performs motioncompensation of the sub-blocks using these affine MVs and encodedreference pictures (Step Sj_4). The processes in Steps Sj_3 and Sj_4 areexecuted on all the blocks in the current block each time an MV predictoris updated in Step Sj_2. As a result, for example, inter predictor 126determines the MV predictor which yields the smallest cost as the MV ata control point in a motion estimation loop (Step Sj_5). At this time,inter predictor 126 further encodes, in the stream, the difference valuebetween the determined MV and the MV predictor as an MV difference.In other words, inter predictor 126 outputs the MV difference as aprediction parameter to entropy encoder 110 through predictionparameter generator 130.
[0371] Lastly, inter predictor 126 generates a prediction image for thecurrent block by performing motion compensation of the current blockusing the determined MV and the encoded reference picture (Step Sj_6).
[0372] It is to be noted that the above-described MV candidate list may begenerated in Step Sj_1. The MV candidate list may be, for example, alist including MV candidates derived using a plurality of MV derivationmethods for each control point. The plurality of MV derivation methodsmay be any combination of the MV derivation methods illustrated in FIGs.47A to 47C, the MV derivation methods illustrated in FIGs. 48A and 48B,the MV derivation methods illustrated in FIGs. 49A and 49B, and otherMV derivation methods.
[0373] It is to be noted that the MV candidate list may include MVcandidates in a mode in which prediction is performed in units of asub-block, other than the affine mode.
[0374] It is to be noted that, for example, an MV candidate list includingMV candidates in an affine inter mode in which two control points areused and an affine inter mode in which three control points are used maybe generated as an MV candidate list. Alternatively, an MV candidatelist including MV candidates in the affine inter mode in which two controlpoints are used and an MV candidate list including MV candidates in theaffine inter mode in which three control points are used may be generatedseparately. Alternatively, an MV candidate list including MV candidatesin one of the affine inter mode in which two control points are used andthe affine inter mode in which three control points are used may begenerated. The MV candidate(s) may be, for example, MVs for encodedblock A (left), block B (upper), block C (upper-right), block D (lower-left),and block E (upper-left), or an MV for an effective block among the blocks.
[0375] It is to be noted that index indicating one of the MV candidates inan MV candidate list may be transmitted as MV predictor selectioninformation.
[0376] [MV Derivation > Triangle Mode]Inter predictor 126 generates one rectangular prediction image fora rectangular current block in the above example. However, interpredictor 126 may generate a plurality of prediction images each having ashape different from a rectangle for the rectangular current block, andmay combine the plurality of prediction images to generate the finalrectangular prediction image. The shape different from a rectangle maybe, for example, a triangle.
[0377] FIG. 52A is a diagram for illustrating generation of two triangularprediction images.
[0378] Inter predictor 126 generates a triangular prediction image byperforming motion compensation of a first partition having a triangularshape in a current block by using a first MV of the first partition, togenerate a triangular prediction image. Likewise, inter predictor 126generates a triangular prediction image by performing motioncompensation of a second partition having a triangular shape in a currentblock by using a second MV of the second partition, to generate atriangular prediction image. Inter predictor 126 then generates aprediction image having the same rectangular shape as the rectangularshape of the current block by combining these prediction images.
[0379] It is to be noted that a first prediction image having a rectangularshape corresponding to a current block may be generated as a predictionimage for a first partition, using a first MV. In addition, a secondprediction image having a rectangular shape corresponding to a currentblock may be generated as a prediction image for a second partition, usinga second MV. A prediction image for the current block may be generatedby performing a weighted addition of the first prediction image and thesecond prediction image. It is to be noted that the part which issubjected to the weighted addition may be a partial region across theboundary between the first partition and the second partition.
[0380] FIG. 52B is a conceptual diagram for illustrating examples of afirst portion of a first partition which overlaps with a second partition,and first and second sets of samples which may be weighted as part of acorrection process. The first portion may be, for example, one fourth ofthe width or height of the first partition. In another example, the firstportion may have a width corresponding to N samples adjacent to an edgeof the first partition, where N is an integer greater than zero, and N maybe, for example, the integer 2. As illustrated, the left example of FIG.52B shows a rectangular partition having a rectangular portion with awidth which is one fourth of the width of the first partition, with the firstset of samples including samples outside of the first portion and samplesinside of the first portion, and the second set of samples includingsamples within the first portion. The center example of FIG. 52B showsa rectangular partition having a rectangular portion with a height whichis one fourth of the height of the first partition, with the first set ofsamples including samples outside of the first portion and samples insideof the first portion, and the second set of samples including sampleswithin the first portion. The right example of FIG. 52B shows atriangular partition having a polygonal portion with a height whichcorresponds to two samples, with the first set of samples includingsamples outside of the first portion and samples inside of the first portion,and the second set of samples including samples within the first portion.
[0381] The first portion may be a portion of the first partition whichoverlaps with an adjacent partition. FIG. 52C is a conceptual diagramfor illustrating a first portion of a first partition, which is a portion of thefirst partition that overlaps with a portion of an adjacent partition. Forease of illustration, a rectangular partition having an overlapping portionwith a spatially adjacent rectangular partition is shown. Partitionshaving other shapes, such as triangular partitions, may be employed, andthe overlapping portions may overlap with a spatially or temporallyadjacent partition.
[0382] In addition, although an example is given in which a predictionimage is generated for each of two partitions using inter prediction, aprediction image may be generated for at least one partition using intraprediction.
[0383] FIG. 53 is a flow chart illustrating one example of a triangle mode.
[0384] In the triangle mode, first, inter predictor 126 splits the currentblock into the first partition and the second partition (Step Sx_1). Atthis time, inter predictor 126 may encode, in a stream, partitioninformation which is information related to the splitting into thepartitions as a prediction parameter. In other words, inter predictor 126may output the partition information as the prediction parameter toentropy encoder 110 through prediction parameter generator 130.
[0385] First, inter predictor 126 obtains a plurality of MV candidates fora current block based on information such as MVs of a plurality ofencoded blocks temporally or spatially surrounding the current block(Step Sx_2). In other words, inter predictor 126 generates an MVcandidate list.
[0386] Inter predictor 126 then selects the MV candidate for the firstpartition and the MV candidate for the second partition as a first MV anda second MV, respectively, from the plurality of MV candidates obtainedin Step Sx_2 (Step Sx_3). At this time, inter predictor 126 encodes, in astream, MV selection information for identifying the selected MVcandidate, as a prediction parameter. In other words, inter predictor126 outputs the MV selection information as a prediction parameter toentropy encoder 110 through prediction parameter generator 130.
[0387] Next, inter predictor 126 generates a first prediction image byperforming motion compensation using the selected first MV and anencoded reference picture (Step Sx_4). Likewise, inter predictor 126generates a second prediction image by performing motion compensationusing the selected second MV and an encoded reference picture (StepSx_5).
[0388] Lastly, inter predictor 126 generates a prediction image for thecurrent block by performing a weighted addition of the first predictionimage and the second prediction image (Step Sx_6).
[0389] It is to be noted that, although the first partition and the secondpartition are triangles in the example illustrated in FIG. 52A, the firstpartition and the second partition may be trapezoids, or other shapesdifferent from each other. Furthermore, although the current blockincludes two partitions in the example illustrated in FIG. 52A, thecurrent block may include three or more partitions.
[0390] In addition, the first partition and the second partition mayoverlap with each other. In other words, the first partition and thesecond partition may include the same pixel region. In this case, aprediction image for a current block may be generated using a predictionimage in the first partition and a prediction image in the second partition.
[0391] In addition, although the example in which the prediction image isgenerated for each of the two partitions using inter prediction has beenillustrated, a prediction image may be generated for at least one partitionusing intra prediction.
[0392] It is to be noted that the MV candidate list for selecting the firstMV and the MV candidate list for selecting the second MV may bedifferent from each other, or the MV candidate list for selecting the firstMV may be also used as the MV candidate list for selecting the secondMV.
[0393] It is to be noted that partition information may include an indexindicating the splitting direction in which at least a current block is splitinto a plurality of partitions. The MV selection information may includean index indicating the selected first MV and an index indicating theselected second MV. One index may indicate a plurality of pieces ofinformation. For example, one index collectively indicating a part or theentirety of partition information and a part or the entirety of MVselection information may be encoded.
[0394] [MV Derivation > ATMVP Mode]FIG. 54 is a diagram illustrating one example of an ATMVP modein which an MV is derived in units of a sub-block.
[0395] The ATMVP mode is a mode categorized into the merge mode. Forexample, in the ATMVP mode, an MV candidate for each sub-block isregistered in an MV candidate list for use in normal merge mode.
[0396] More specifically, in the ATMVP mode, first, as illustrated in FIG.54, a temporal MV reference block associated with a current block isidentified in an encoded reference picture specified by an MV (MV0) of aneighboring block located at the lower-left position with respect to thecurrent block. Next, in each sub-block in the current block, the MV usedto encode the region corresponding to the sub-block in the temporal MVreference block is identified. The MV identified in this way is includedin an MV candidate list as an MV candidate for the sub-block in thecurrent block. When the MV candidate for each sub-block is selectedfrom the MV candidate list, the sub-block is subjected to motioncompensation in which the MV candidate is used as the MV for thesub-block. In this way, a prediction image for each sub-block isgenerated.
[0397] Although the block located at the lower-left position with respectto the current block is used as a surrounding MV reference block in theexample illustrated in FIG. 54, it is to be noted that another block may beused. In addition, the size of the sub-block may be 4x4 pixels, 8x8 pixels,or another size. The size of the sub-block may be switched for a unitsuch as a slice, brick, picture, etc.
[0398] [Motion Estimation > DMVR]FIG. 55 is a diagram illustrating a relationship between a mergemode and DMVR.
[0399] Inter predictor 126 derives an MV for a current block according tothe merge mode (Step Sl_1). Next, inter predictor 126 determineswhether to perform estimation of an MV that is motion estimation (StepSl_2). Here, when determining not to perform motion estimation (No inStep Sl_2), inter predictor 126 determines the MV derived in Step Sl_1 asthe final MV for the current block (Step Sl_4). In other words, in thiscase, the MV for the current block is determined according to the mergemode.
[0400] When determining to perform motion estimation in Step Sl_1 (Yesin Step Sl_2), inter predictor 126 derives the final MV for the currentblock by estimating a surrounding region of the reference picturespecified by the MV derived in Step Sl_1 (Step Sl_3). In other words, inthis case, the MV for the current block is determined according to theDMVR.
[0401] FIG. 56 is a conceptual diagram for illustrating another example ofDMVR for determining an MV.
[0402] First, in the merge mode for example, MV candidates (L0 and L1)are selected for the current block. A reference pixel is identified from afirst reference picture (L0) which is an encoded picture in the L0 listaccording to the MV candidate (L0). Likewise, a reference pixel isidentified from a second reference picture (L1) which is an encodedpicture in the L1 list according to the MV candidate (L1). A template isgenerated by calculating an average of these reference pixels.
[0403] Next, each of the surrounding regions of MV candidates of the firstreference picture (L0) and the second reference picture (L1) are estimatedusing the template, and the MV which yields the smallest cost isdetermined to be the final MV. It is to be noted that the cost may becalculated, for example, using a difference value between each of the pixelvalues in the template and a corresponding one of the pixel values in theestimation region, the values of MV candidates, etc.
[0404] Exactly the same processes described here do not always need tobe performed. Any process for enabling derivation of the final MV byestimation in surrounding regions of MV candidates may be used.
[0405] FIG. 57 is a conceptual diagram for illustrating another example ofDMVR for determining an MV. Unlike the example of DMVR illustratedin FIG. 56, in the example illustrated in FIG. 57, costs are calculatedwithout generating any template.
[0406] First, inter predictor 126 estimates a surrounding region of areference block included in each of reference pictures in the L0 list and L1list, based on an initial MV which is an MV candidate obtained from eachMV candidate list. For example, as illustrated in FIG. 57, the initial MVcorresponding to the reference block in the L0 list is InitMV_L0, and theinitial MV corresponding to the reference block in the L1 list isInitMV_L1. In motion estimation, inter predictor 126 firstly sets asearch position for the reference picture in the L0 list. Based on theposition indicated by the vector difference indicating the search positionto be set, specifically, the initial MV (that is, InitMV_L0), the vectordifference to the search position is MVd_L0. Inter predictor 126 thendetermines the estimation position in the reference picture in the L1 list.This search position is indicated by the vector difference to the searchposition from the position indicated by the initial MV (that is, InitMV_L1).More specifically, inter predictor 126 determines the vector difference asMVd_L1 by mirroring of MVd_L0. In other words, inter predictor 126determines the position which is symmetrical with respect to the positionindicated by the initial MV to be the search position in each referencepicture in the L0 list and the L1 list. Inter predictor 126 calculates, foreach search position, the total sum of the absolute differences (SADs)between values of pixels at search positions in blocks as a cost, and findsout the search position that yields the smallest cost.
[0407] FIG. 58A is a diagram illustrating one example of motionestimation in DMVR, and FIG. 58B is a flow chart illustrating oneexample of the motion estimation.
[0408] First, in Step 1, inter predictor 126 calculates the cost between thesearch position (also referred to as a starting point) indicated by theinitial MV and eight surrounding search positions. Inter predictor 126then determines whether the cost at each of the search positions otherthan the starting point is the smallest. Here, when determining that thecost at the search position other than the starting point is the smallest,inter predictor 126 changes a target to the search position at which thesmallest cost is obtained, and performs the process in Step 2. When thecost at the starting point is the smallest, inter predictor 126 skips theprocess in Step 2 and performs the process in Step 3.
[0409] In Step 2, inter predictor 126 performs the search similar to theprocess in Step 1, regarding, as a new starting point, the search positionafter the target change according to the result of the process in Step 1.Inter predictor 126 then determines whether the cost at each of thesearch positions other than the starting point is the smallest. Here,when determining that the cost at the search position other than thestarting point is the smallest, inter predictor 126 performs the process inStep 4. When the cost at the starting point is the smallest, interpredictor 126 performs the process in Step 3.
[0410] In Step 4, inter predictor 126 regards the search position at thestarting point as the final search position, and determines the differencebetween the position indicated by the initial MV and the final searchposition to be a vector difference.
[0411] In Step 3, inter predictor 126 determines the pixel position atsub-pixel accuracy at which the smallest cost is obtained, based on thecosts at the four points located at upper, lower, left, and right positionswith respect to the starting point in Step 1 or Step 2, and regards thepixel position as the final search position. The pixel position at thesub-pixel accuracy is determined by performing weighted addition of eachof the four upper, lower, left, and right vectors ((0, 1), (0, -1), (-1, 0), and (1,0)), using, as a weight, the cost at a corresponding one of the four searchpositions. Inter predictor 126 then determines the difference betweenthe position indicated by the initial MV and the final search position to bethe vector difference.
[0412] [Motion Compensation > BIO / OBMC / LIC]Motion compensation involves a mode for generating a predictionimage, and correcting the prediction image. The mode is, for example,BIO, OBMC, and LIC to be described later.
[0413] FIG. 59 is a flow chart illustrating one example of generation of aprediction image.
[0414] Inter predictor 126 generates a prediction image (Step Sm_1), andcorrects the prediction image according to any of the modes describedabove (Step Sm_2).
[0415] FIG. 60 is a flow chart illustrating another example of generationof a prediction image.
[0416] Inter predictor 126 derives an MV of a current block (Step Sn_1).Next, inter predictor 126 generates a prediction image using the MV(Step Sn_2), and determines whether to perform a correction process(Step Sn_3). Here, when determining to perform a correction process(Yes in Step Sn_3), inter predictor 126 generates the final predictionimage by correcting the prediction image (Step Sn_4). It is to be notedthat, in LIC described later, luminance and chrominance may be correctedin Step Sn_4. When determining not to perform a correction process (Noin Step Sn_3), inter predictor 126 outputs the prediction image as thefinal prediction image without correcting the prediction image (StepSn_5).
[0417] [Motion Compensation > OBMC]It is to be noted that an inter prediction image may be generatedusing motion information for a neighboring block in addition to motioninformation for the current block obtained by motion estimation. Morespecifically, an inter prediction image may be generated for eachsub-block in a current block by performing weighted addition of aprediction image based on the motion information obtained by motionestimation (in a reference picture) and a prediction image based on themotion information of the neighboring block (in the current picture).Such inter prediction (motion compensation) is also referred to asoverlapped block motion compensation (OBMC) or an OBMC mode.
[0418] In OBMC mode, information indicating a sub-block size for OBMC(referred to as, for example, an OBMC block size) may be signaled at thesequence level. Moreover, information indicating whether to apply theOBMC mode (referred to as, for example, an OBMC flag) may be signaledat the CU level. It is to be noted that the signaling of such informationdoes not necessarily need to be performed at the sequence level and CUlevel, and may be performed at another level (for example, at the picturelevel, slice level, brick level, CTU level, or sub-block level).
[0419] The OBMC mode will be described in further detail. FIGs. 61 and62 are a flow chart and a conceptual diagram for illustrating an outline ofa prediction image correction process performed by OBMC.
[0420] First, as illustrated in FIG. 62, a prediction image (Pred) bynormal motion compensation is obtained using an MV assigned to acurrent block. In FIG. 62, the arrow "MV" points a reference picture,and indicates what the current block of the current picture refers to inorder to obtain the prediction image.
[0421] Next, a prediction image (Pred_L) is obtained by applying a motionvector (MV_L) which has been already derived for the encoded blockneighboring to the left of the current block to the current block (re-usingthe motion vector for the current block). The motion vector (MV_L) isindicated by an arrow "MV_L" indicating a reference picture from acurrent block. A first correction of a prediction image is performed byoverlapping two prediction images Pred and Pred_L. This provides aneffect of blending the boundary between neighboring blocks.
[0422] Likewise, a prediction image (Pred_U) is obtained by applying anMV (MV_U) which has been already derived for the encoded blockneighboring above the current block to the current block (re-using the MVfor the current block). The MV (MV_U) is indicated by an arrow "MV_U"indicating a reference picture from a current block. A second correctionof a prediction image is performed by overlapping the prediction imagePred_U to the prediction images (for example, Pred and Pred_L) on whichthe first correction has been performed. This provides an effect ofblending the boundary between neighboring blocks. The predictionimage obtained by the second correction is the one in which the boundarybetween the neighboring blocks has been blended (smoothed), and thus isthe final prediction image of the current block.
[0423] Although the above example is a two-path correction method usingleft and upper neighboring blocks, it is to be noted that the correctionmethod may be three- or more-path correction method using also the rightneighboring block and / or the lower neighboring block.
[0424] It is to be noted that the region in which such overlapping isperformed may be only part of a region near a block boundary instead ofthe pixel region of the entire block.
[0425] It is to be noted that the prediction image correction processaccording to OBMC for obtaining one prediction image Pred from onereference picture by overlapping additional prediction images Pred_L andPred_U has been described above. However, when a prediction image iscorrected based on a plurality of reference images, a similar process maybe applied to each of the plurality of reference pictures. In such a case,after corrected prediction images are obtained from the respectivereference pictures by performing OBMC image correction based on theplurality of reference pictures, the obtained corrected prediction imagesare further overlapped to obtain the final prediction image.
[0426] It is to be noted that, in OBMC, a current block unit may be a PUor a sub-block unit obtained by further splitting the PU.
[0427] One example of a method for determining whether to apply OBMCis a method for using an obmc_flag which is a signal indicating whether toapply OBMC. As one specific example, encoder 100 may determinewhether the current block belongs to a region having complicated motion.Encoder 100 sets the obmc_flag to a value of "1" when the block belongs toa region having complicated motion and applies OBMC when encoding,and sets the obmc_flag to a value of "0" when the block does not belong toa region having complicated motion and encodes the block withoutapplying OBMC. Decoder 200 switches between application andnon-application of OBMC by decoding the obmc_flag written in a stream.
[0428] [Motion Compensation > BIO]Next, an MV derivation method is described. First, a mode forderiving an MV based on a model assuming uniform linear motion isdescribed. This mode is also referred to as a bi-directional optical flow(BIO) mode. In addition, this bi-directional optical flow may be writtenas BDOF instead of BIO.
[0429] FIG. 63 is a diagram for illustrating a model assuming uniformlinear motion. In FIG. 63, (vx, vy) indicates a velocity vector, and τ0 andτ1 indicate temporal distances between a current picture (Cur Pic) andtwo reference pictures (Ref0, Ref1). (MVx0, MVy0) indicates an MVcorresponding to reference picture Ref0, and (MVx1, MVy1) indicates anMV corresponding to reference picture Ref1.
[0430] Here, under the assumption of uniform linear motion exhibited bya velocity vector (vx, vy), (MVx0, MVy0) and (MVx1, MVy1) are representedas (vxτ0, vyτ0) and (-vxτ1, -vyτ1), respectively, and the following opticalflow equation (2) is given.
[0431] [MATH. 3]
[0432] Here, I(k) denotes a luma value from reference image k (k = 0, 1)after motion compensation. This optical flow equation shows that thesum of (i) the time derivative of the luma value, (ii) the product of thehorizontal velocity and the horizontal component of the spatial gradientof a reference image, and (iii) the product of the vertical velocity and thevertical component of the spatial gradient of a reference image is equal tozero. A motion vector of each block obtained from, for example, an MVcandidate list may be corrected in units of a pixel, based on a combinationof the optical flow equation and Hermite interpolation.
[0433] It is to be noted that a motion vector may be derived on the decoder200 side using a method other than deriving a motion vector based on amodel assuming uniform linear motion. For example, a motion vectormay be derived in units of a sub-block based on MVs of a plurality ofneighboring blocks.
[0434] FIG. 64 is a flow chart illustrating one example of inter predictionaccording to BIO. FIG. 65 is a diagram illustrating one example of afunctional configuration of inter predictor 126 which performs interprediction according to BIO.
[0435] As illustrated in FIG. 65, inter predictor 126 includes, for example,memory 126a, interpolated image deriver 126b, gradient image deriver126c, optical flow deriver 126d, correction value deriver 126e, andprediction image corrector 126f. It is to be noted that memory 126a maybe frame memory 122.
[0436] Inter predictor 126 derives two motion vectors (M0, M1), using tworeference pictures (Ref0, Ref1) different from the picture (Cur Pic)including a current block. Inter predictor 126 then derives a predictionimage for the current block using the two motion vectors (M0, M1) (StepSy_1). It is to be noted that motion vector M0 is motion vector (MVx0,MVy0) corresponding to reference picture Ref0, and motion vector M1 ismotion vector (MVx1, MVy1) corresponding to reference picture Ref1.
[0437] Next, interpolated image deriver 126b derives interpolated imageI0 for the current block, using motion vector M0 and reference picture L0by referring to memory 126a. Next, interpolated image deriver 126bderives interpolated image I1 for the current block, using motion vectorM1 and reference picture L1 by referring to memory 126a (Step Sy_2).Here, interpolated image I0 is an image included in reference picture Ref0and to be derived for the current block, and interpolated image I1 is animage included in reference picture Ref1 and to be derived for the currentblock. Each of interpolated image I0 and interpolated image I1 may bethe same in size as the current block. Alternatively, each of interpolatedimage I0 and interpolated image I1 may be an image larger than thecurrent block. Furthermore, interpolated image I0 and interpolatedimage I1 may include a prediction image obtained by using motion vectors(M0, M1) and reference pictures (L0, L1) and applying a motioncompensation filter.
[0438] In addition, gradient image deriver 126c derives gradient images(Ix0, Ix1, Iy0, Iy1) of the current block, from interpolated image I0 andinterpolated image I1. It is to be noted that the gradient images in thehorizontal direction are (Ix0, Ix1), and the gradient images in the verticaldirection are (Iy0, Iy1). Gradient image deriver 126c may derive eachgradient image by, for example, applying a gradient filter to theinterpolated images. It is only necessary that a gradient image indicatethe amount of spatial change in pixel value along the horizontal directionor the vertical direction.
[0439] Next, optical flow deriver 126d derives, for each sub-block of thecurrent block, an optical flow (vx, vy) which is a velocity vector, using theinterpolated images (I0, I1) and the gradient images (Ix0, Ix1, Iy0, Iy1).The optical flow indicates coefficients for correcting the amount of spatialpixel movement, and may be referred to as a local motion estimationvalue, a corrected motion vector, or a corrected weighting vector. As oneexample, a sub-block may be 4x4 pixel sub-CU. It is to be noted that theoptical flow derivation may be performed for each pixel unit, or the like,instead of being performed for each sub-block.
[0440] Next, inter predictor 126 corrects a prediction image for thecurrent block using the optical flow (vx, vy). For example, correctionvalue deriver 126e derives a correction value for the value of a pixelincluded in a current block, using the optical flow (vx, vy) (Step Sy_5).Prediction image corrector 126f may then correct the prediction image forthe current block using the correction value (Step Sy_6). It is to be notedthat the correction value may be derived in units of a pixel, or may bederived in units of a plurality of pixels or in units of a sub-block.
[0441] It is to be noted that the BIO process flow is not limited to theprocess disclosed in FIG. 64. Only part of the processes disclosed in FIG.64 may be performed, or a different process may be added or used as areplacement, or the processes may be executed in a different processingorder.
[0442] [Motion Compensation > LIC]Next, one example of a mode for generating a prediction image(prediction) using a local illumination compensation (LIC) is described.
[0443] FIG. 66A is a diagram for illustrating one example of a predictionimage generation method using a luminance correction process performedby LIC. FIG. 66B is a flow chart illustrating one example of a predictionimage generation method using the LIC.
[0444] First, inter predictor 126 derives an MV from an encoded referencepicture, and obtains a reference image corresponding to the current block(Step Sz_1).
[0445] Next, inter predictor 126 extracts, for the current block,information indicating how the luma value has changed between thecurrent block and the reference picture (Step Sz_2). This extraction isperformed based on the luma pixel values of the encoded left neighboringreference region (surrounding reference region) and the encoded upperneighboring reference region (surrounding reference region) in thecurrent picture, and the luma pixel values at the corresponding positionsin the reference picture specified by the derived MVs. Inter predictor126 calculates a luminance correction parameter, using the informationindicating how the luma value has changed (Step Sz_3).
[0446] Inter predictor 126 generates a prediction image for the currentblock by performing a luminance correction process in which theluminance correction parameter is applied to the reference image in thereference picture specified by the MV (Step Sz_4). In other words, theprediction image which is the reference i mage in the reference picturespecified by the MV is subjected to the correction based on the luminancecorrection parameter. In this correction, luminance may be corrected, orchrominance may be corrected. In other words, a chrominance correctionparameter may be calculated using information indicating howchrominance has changed, and a chrominance correction process may beperformed.
[0447] It is to be noted that the shape of the surrounding reference regionillustrated in FIG. 66A is one example; another shape may be used.
[0448] Moreover, although the process in which a prediction image isgenerated from a single reference picture has been described here, casesin which a prediction image is generated from a plurality of referencepictures can be described in the same manner. The prediction imagemay be generated after performing a luminance correction process of thereference images obtained from the reference pictures in the samemanner as described above.
[0449] One example of a method for determining whether to apply LIC isa method for using a lic_flag which is a signal indicating whether to applythe LIC. As one specific example, encoder 100 determines whether thecurrent block belongs to a region having a luminance change. Encoder100 sets the lic_flag to a value of "1" when the block belongs to a regionhaving a luminance change and applies LIC when encoding, and sets thelic_flag to a value of "0" when the block does not belong to a region havinga luminance change and performs encoding without applying LIC.Decoder 200 may decode the lic_flag written in the stream and decode thecurrent block by switching between application and non-application ofLIC in accordance with the flag value.
[0450] One example of a different method of determining whether toapply a LIC process is a determining method in accordance with whethera LIC process has been applied to a surrounding block. As one specificexample, when a current block has been processed in merge mode, interpredictor 126 determines whether an encoded surrounding block selectedin MV derivation in merge mode has been encoded using LIC. Interpredictor 126 performs encoding by switching between application andnon-application of LIC according to the result. It is to be noted that, alsoin this example, the same processes are applied to processes at thedecoder 200 side.
[0451] The luminance correction (LIC) process has been described withreference to FIGs. 66A and 66B, and is further described below.
[0452] First, inter predictor 126 derives an MV for obtaining a referenceimage corresponding to a current block from a reference picture which isan encoded picture.
[0453] Next, inter predictor 126 extracts information indicating how theluma value of the reference picture has been changed to the luma value ofthe current picture, using the luma pixel values of encoded surroundingreference regions which neighbor to the left of and above the currentblock and the luma pixel values in the corresponding positions in thereference pictures specified by MVs, and calculates a luminancecorrection parameter. For example, it is assumed that the luma pixelvalue of a given pixel in the surrounding reference region in the currentpicture is p0, and that the luma pixel value of the pixel corresponding tothe given pixel in the surrounding reference region in the referencepicture is p1. Inter predictor 126 calculates coefficients A and B foroptimizing A x p1 + B = p0 as the luminance correction parameter for aplurality of pixels in the surrounding reference region.
[0454] Next, inter predictor 126 performs a luminance correction processusing the luminance correction parameter for the reference image in thereference picture specified by the MV, to generate a prediction image forthe current block. For example, it is assumed that the luma pixel valuein the reference image is p2, and that the luminance-corrected luma pixelvalue of the prediction image is p3. Inter predictor 126 generates theprediction image after being subjected to the luminance correctionprocess by calculating A x p2 + B = p3 for each of the pixels in thereference image.
[0455] For example, a region having a determined number of pixelsextracted from each of an upper neighboring pixel and a left neighboringpixel may be used as a surrounding reference region. In addition, thesurrounding reference region is not limited to a region which neighborsthe current block, and may be a region which does not neighbor thecurrent block. In the example illustrated in FIG. 66A, the surroundingreference region in the reference picture may be a region specified byanother MV in a current picture, from a surrounding reference region inthe current picture. For example, the other MV may be an MV in asurrounding reference region in the current picture.
[0456] Although operations performed by encoder 100 have beendescribed here, it is to be noted that decoder 200 performs similaroperations.
[0457] It is to be noted that LIC may be applied not only to luma but alsoto chroma. At this time, a correction parameter may be derivedindividually for each of Y, Cb, and Cr, or a common correction parametermay be used for any of Y, Cb, and Cr.
[0458] In addition, the LIC process may be applied in units of a sub-block.For example, a correction parameter may be derived using a surroundingreference region in a current sub-block and a surrounding referenceregion in a reference sub-block in a reference picture specified by an MVof the current sub-block.
[0459] [Prediction Controller]Prediction controller 128 selects one of an intra prediction image(an image or a signal output from intra predictor 124) and an interprediction image (an image or a signal output from inter predictor 126),and outputs the selected prediction image to subtractor 104 and adder116.
[0460] [Prediction Parameter Generator]Prediction parameter generator 130 may output informationrelated to intra prediction, inter prediction, selection of a predictionimage in prediction controller 128, etc. as a prediction parameter toentropy encoder 110. Entropy encoder 110 may generate a stream, basedon the prediction parameter which is input from prediction parametergenerator 130 and quantized coefficients which are input from quantizer108. The prediction parameter may be used in decoder 200. Decoder200 may receive and decode the stream, and perform the same processesas the prediction processes performed by intra predictor 124, interpredictor 126, and prediction controller 128. The prediction parametermay include (i) a selection prediction signal (for example, an MV, aprediction type, or a prediction mode used by intra predictor 124 or interpredictor 126), or (ii) an optional index, a flag, or a value which is basedon a prediction process performed in each of intra predictor 124, interpredictor 126, and prediction controller 128, or which indicates theprediction process.
[0461] [Decoder]Next, decoder 200 capable of decoding a stream output fromencoder 100 described above is described. FIG. 67 is a block diagramillustrating a functional configuration of decoder 200 according to thisembodiment. Decoder 200 is an apparatus which decodes a stream thatis an encoded image in units of a block.
[0462] As illustrated in FIG. 67, decoder 200 includes entropy decoder202, inverse quantizer 204, inverse transformer 206, adder 208, blockmemory 210, loop filter 212, frame memory 214, intra predictor 216, interpredictor 218, prediction controller 220, prediction parameter generator222, and splitting determiner 224. It is to be noted that intra predictor216 and inter predictor 218 are configured as part of a predictionexecutor.
[0463] [Mounting Example of Decoder]FIG. 68 is a block diagram illustrating a mounting example ofdecoder 200. Decoder 200 includes processor b1 and memory b2. Forexample, the plurality of constituent elements of decoder 200 illustratedin FIG. 67 are mounted on processor b1 and memory b2 illustrated in FIG.68.
[0464] Processor b1 is circuitry which performs information processingand is accessible to memory b2. For example, processor b1 is a dedicatedor general electronic circuit which decodes a stream. Processor b1 maybe a processor such as a CPU. In addition, processor b1 may be anaggregate of a plurality of electronic circuits. In addition, for example,processor b1 may take the roles of two or more constituent elements otherthan a constituent element for storing information out of the plurality ofconstituent elements of decoder 200 illustrated in FIG. 67, etc.
[0465] Memory b2 is dedicated or general memory for storing informationthat is used by processor b1 to decode a stream. Memory b2 may beelectronic circuitry, and may be connected to processor b1. In addition,memory b2 may be included in processor b1. In addition, memory b2may be an aggregate of a plurality of electronic circuits. In addition,memory b2 may be a magnetic disc, an optical disc, or the like, or may berepresented as a storage, a medium, or the like. In addition, memory b2may be non-volatile memory, or volatile memory.
[0466] For example, memory b2 may store an image or a stream. Inaddition, memory b2 may store a program for causing processor b1 todecode a stream.
[0467] In addition, for example, memory b2 may take the roles of two ormore constituent elements for storing information out of the plurality ofconstituent elements of decoder 200 illustrated in FIG. 67, etc. Morespecifically, memory b2 may take the roles of block memory 210 and framememory 214 illustrated in FIG. 67. More specifically, memory b2 maystore a reconstructed image (specifically, a reconstructed block, areconstructed picture, or the like).
[0468] It is to be noted that, in decoder 200, not all of the plurality ofconstituent elements illustrated in FIG. 67, etc. may be implemented, andnot all the processes described above may be performed. Part of theconstituent elements indicated in FIG. 67, etc. may be included inanother device, or part of the processes described above may be performedby another device.
[0469] Hereinafter, an overall flow of the processes performed by decoder200 is described, and then each of the constituent elements included indecoder 200 is described. It is to be noted that, some of the constituentelements included in decoder 200 perform the same processes asperformed by some of the constituent elements included in encoder 100,and thus the same processes are not repeatedly described in detail. Forexample, inverse quantizer 204, inverse transformer 206, adder 208,block memory 210, frame memory 214, intra predictor 216, inter predictor218, prediction controller 220, and loop filter 212 included in decoder 200perform similar processes as performed by inverse quantizer 112, inversetransformer 114, adder 116, block memory 118, frame memory 122, intrapredictor 124, inter predictor 126, prediction controller 128, and loopfilter 120 included in encoder 100, respectively.
[0470] [Overall Flow of Decoding Process]FIG. 69 is a flow chart illustrating one example of an overalldecoding process performed by decoder 200.
[0471] First, splitting determiner 224 in decoder 200 determines asplitting pattern of each of a plurality of fixed-size blocks (128x128pixels) included in a picture, based on a parameter which is input fromentropy decoder 202 (Step Sp_1). This splitting pattern is a splittingpattern selected by encoder 100. Decoder 200 then performs processes ofSteps Sp_2 to Sp_6 for each of a plurality of blocks of the splittingpattern.
[0472] Entropy decoder 202 decodes (specifically, entropy decodes)encoded quantized coefficients and a prediction parameter of a currentblock (Step Sp_2).
[0473] Next, inverse quantizer 204 performs inverse quantization of theplurality of quantized coefficients and inverse transformer 206 performsinverse transform of the result, to restore prediction residuals of thecurrent block (Step Sp_3).
[0474] Next, the prediction executor including all or part of intrapredictor 216, inter predictor 218, and prediction controller 220 generatesa prediction image of the current block (Step Sp_4).
[0475] Next, adder 208 adds the prediction image to a prediction residualto generate a reconstructed image (also referred to as a decoded imageblock) of the current block (Step Sp_5).
[0476] When the reconstructed image is generated, loop filter 212performs filtering of the reconstructed image (Step Sp_6).
[0477] Decoder 200 then determines whether decoding of the entirepicture has been finished (Step Sp_7). When determining that thedecoding has not yet been finished (No in Step Sp_7), decoder 200repeatedly executes the processes starting with Step Sp_1.
[0478] It is to be noted that the processes of these Steps Sp_1 to Sp_7 maybe performed sequentially by decoder 200, or two or more of the processesmay be performed in parallel. The processing order of the two or more ofthe processes may be modified.
[0479] [Splitting Determiner]FIG. 70 is a diagram illustrating a relationship between splittingdeterminer 224 and other constituent elements. Splitting determiner224 may perform the following processes as examples.
[0480] For example, splitting determiner 224 collects block informationfrom block memory 210 or frame memory 214, and furthermore obtains aparameter from entropy decoder 202. Splitting determiner 224 may thendetermine the splitting pattern of a fixed-size block, based on the blockinformation and the parameter. Splitting determiner 224 may thenoutput information indicating the determined splitting pattern to inversetransformer 206, intra predictor 216, and inter predictor 218. Inversetransformer 206 may perform inverse transform of transform coefficients,based on the splitting pattern indicated by the information from splittingdeterminer 224. Intra predictor 216 and inter predictor 218 maygenerate a prediction image, based on the splitting pattern indicated bythe information from splitting determiner 224.
[0481] [Entropy Decoder]FIG. 71 is a block diagram illustrating one example of a functionalconfiguration of entropy decoder 202.
[0482] Entropy decoder 202 generates quantized coefficients, a predictionparameter, and a parameter related to a splitting pattern, by entropydecoding the stream. For example, CABAC is used in the entropydecoding. More specifically, entropy decoder 202 includes, for example,binary arithmetic decoder 202a, context controller 202b, and debinarizer202c. Binary arithmetic decoder 202a arithmetically decodes the streamusing a context value derived by context controller 202b to a binary signal.Context controller 202b derives a context value according to a feature or asurrounding state of a syntax element, that is, an occurrence probabilityof a binary signal, in the same manner as performed by context controller110b of encoder 100. Debinarizer 202c performs debinarization fortransforming the binary signal output from binary arithmetic decoder202a to a multi-level signal indicating quantized coefficients as describedabove. This binarization is performed according to the binarizationmethod described above.
[0483] With this, entropy decoder 202 outputs quantized coefficients ofeach block to inverse quantizer 204. Entropy decoder 202 may output aprediction parameter included in a stream (see FIG. 1) to intra predictor216, inter predictor 218, and prediction controller 220. Intra predictor216, inter predictor 218, and prediction controller 220 are capable ofexecuting the same prediction processes as those performed by intrapredictor 124, inter predictor 126, and prediction controller 128 at theencoder 100 side.
[0484] [Entropy Decoder]FIG. 72 is a diagram illustrating a flow of CABAC in entropydecoder 202.
[0485] First, initialization is performed in CABAC in entropy decoder 202.In the initialization, initialization in binary arithmetic decoder 202a andsetting of an initial context value are performed. Binary arithmeticdecoder 202a and debinarizer 202c then execute arithmetic decoding anddebinarization of, for example, encoded data of a CTU. At this time,context controller 202b updates the context value each time arithmeticdecoding is performed. Context controller 202b then saves the contextvalue as a post process. The saved context value is used, for example, toinitialize the context value for the next CTU.
[0486] [Inverse Quantizer]Inverse quantizer 204 inverse quantizes quantized coefficients of acurrent block which are inputs from entropy decoder 202. Morespecifically, inverse quantizer 204 inverse quantizes the quantizedcoefficients of the current block, based on quantization parameterscorresponding to the quantized coefficients. Inverse quantizer 204 thenoutputs the inverse quantized transform coefficients (that are transformcoefficients) of the current block to inverse transformer 206.
[0487] FIG. 73 is a block diagram illustrating one example of a functionalconfiguration of inverse quantizer 204.
[0488] Inverse quantizer 204 includes, for example, quantizationparameter generator 204a, predicted quantization parameter generator204b, quantization parameter storage 204d, and inverse quantizationexecutor 204e.
[0489] FIG. 74 is a flow chart illustrating one example of inversequantization performed by inverse quantizer 204.
[0490] Inverse quantizer 204 may perform an inverse quantizationprocess as one example for each CU based on the flow illustrated in FIG.74. More specifically, quantization parameter generator 204adetermines whether to perform inverse quantization (Step Sv_11). Here,when determining to perform inverse quantization (Yes in Step Sv_11),quantization parameter generator 204a obtains a difference quantizationparameter for the current block from entropy decoder 202 (Step Sv_12).
[0491] Next, predicted quantization parameter generator 204b thenobtains a quantization parameter for a processing unit different from thecurrent block from quantization parameter storage 204d (Step Sv_13).Predicted quantization parameter generator 204b generates a predictedquantization parameter of the current block based on the obtainedquantization parameter (Step Sv_14).
[0492] Quantization parameter generator 204a then adds the differencequantization parameter for the current block obtained from entropydecoder 202 and the predicted quantization parameter for the currentblock generated by predicted quantization parameter generator 204b(Step Sv_15). This addition generates a quantization parameter for thecurrent block. In addition, quantization parameter generator 204astores the quantization parameter for the current block in quantizationparameter storage 204d (Step Sv_16).
[0493] Next, inverse quantization executor 204e inverse quantizes thequantized coefficients of the current block into transform coefficients,using the quantization parameter generated in Step Sv_15 (Step Sv_17).
[0494] It is to be noted that the difference quantization parameter may bedecoded at the bit sequence level, picture level, slice level, brick level, orCTU level. In addition, the initial value of the quantization parametermay be decoded at the sequence level, picture level, slice level, brick level,or CTU level. At this time, the quantization parameter may begenerated using the initial value of the quantization parameter and thedifference quantization parameter.
[0495] It is to be noted that inverse quantizer 204 may include a pluralityof inverse quantizers, and may inverse quantize the quantizedcoefficients using an inverse quantization method selected from aplurality of inverse quantization methods.
[0496] [Inverse Transformer]Inverse transformer 206 restores prediction residuals by inversetransforming the transform coefficients which are inputs from inversequantizer 204.
[0497] For example, when information parsed from a stream indicatesthat EMT or AMT is to be applied (for example, when an AMT flag is true),inverse transformer 206 inverse transforms the transform coefficients ofthe current block based on information indicating the parsed transformtype.
[0498] Moreover, for example, when information parsed from a streamindicates that NSST is to be applied, inverse transformer 206 applies asecondary inverse transform to the transform coefficients.
[0499] FIG. 75 is a flow chart illustrating one example of a processperformed by inverse transformer 206.
[0500] For example, inverse transformer 206 determines whetherinformation indicating that no orthogonal transform is performed ispresent in a stream (Step St_11). Here, when determining that no suchinformation is present (No in Step St_11), inverse transformer 206obtains information indicating the transform type decoded by entropydecoder 202 (Step St_12). Next, based on the information, inversetransformer 206 determines the transform type used for the orthogonaltransform in encoder 100 (Step St_13). Inverse transformer 206 thenperforms inverse orthogonal transform using the determined transformtype (Step St_14).
[0501] FIG. 76 is a flow chart illustrating another example of a processperformed by inverse transformer 206.
[0502] For example, inverse transformer 206 determines whether atransform size is smaller than or equal to a predetermined value (StepSu_11). Here, when determining that the transform size is smaller thanor equal to a predetermined value (Yes in Step Su_11), inversetransformer 206 obtains, from entropy decoder 202, informationindicating which transform type has been used by encoder 100 among atleast one transform type included in the first transform type group (StepSu_12). It is to be noted that such information is decoded by entropydecoder 202 and output to inverse transformer 206.
[0503] Based on the information, inverse transformer 206 determines thetransform type used for the orthogonal transform in encoder 100 (StepSu_13). Inverse transformer 206 then inverse orthogonal transforms thetransform coefficients of the current block using the determinedtransform type (Step Su_14). When determining that a transform size isnot smaller than or equal to the predetermined value (No in Step Su_11),inverse transformer 206 inverse transforms the transform coefficients ofthe current block using the second transform type group (Step Su_15).
[0504] It is to be noted that the inverse orthogonal transform by inversetransformer 206 may be performed according to the flow illustrated inFIG. 75 or FIG. 76 for each TU as one example. In addition, inverseorthogonal transform may be performed by using a predefined transformtype without decoding information indicating a transform type used fororthogonal transform. In addition, the transform type is specificallyDST7, DCT8, or the like. In inverse orthogonal transform, an inversetransform basis function corresponding to the transform type is used.
[0505] [Adder]Adder 208 reconstructs the current block by adding a predictionresidual which is an input from inverse transformer 206 and a predictionimage which is an input from prediction controller 220. In other words,a reconstructed image of the current block is generated. Adder 208 thenoutputs the reconstructed image of the current block to block memory 210and loop filter 212.
[0506] [Block Memory]Block memory 210 is storage for storing a block which is includedin a current picture and is referred to in intra prediction. Morespecifically, block memory 210 stores a reconstructed image output fromadder 208.
[0507] [Loop Filter]Loop filter 212 applies a loop filter to the reconstructed imagegenerated by adder 208, and outputs the filtered reconstructed image toframe memory 214 and a display device, etc.
[0508] When information indicating ON or OFF of an ALF parsed from astream indicates that an ALF is ON, one filter from among a plurality offilters is selected based on the direction and activity of local gradients,and the selected filter is applied to the reconstructed image.
[0509] FIG. 77 is a block diagram illustrating one example of a functionalconfiguration of loop filter 212. It is to be noted that loop filter 212 has aconfiguration similar to the configuration of loop filter 120 of encoder 100.
[0510] For example, as illustrated in FIG. 77, loop filter 212 includesdeblocking filter executor 212a, SAO executor 212b, and ALF executor212c. Deblocking filter executor 212a performs a deblocking filterprocess of the reconstructed image. SAO executor 212b performs a SAOprocess of the reconstructed image after being subjected to the deblockingfilter process. ALF executor 212c performs an ALF process of thereconstructed image after being subjected to the SAO process. It is to benoted that loop filter 212 does not always need to include all theconstituent elements disclosed in FIG. 77, and may include only part ofthe constituent elements. In addition, loop filter 212 may be configuredto perform the above processes in a processing order different from theone disclosed in FIG. 77.
[0511] [Frame Memory]Frame memory 214 is, for example, storage for storing referencepictures for use in inter prediction, and is also referred to as a framebuffer. More specifically, frame memory 214 stores a reconstructedimage filtered by loop filter 212.
[0512] [Predictor (Intra Predictor, Inter Predictor, Prediction Controller)]FIG. 78 is a flow chart illustrating one example of a processperformed by a predictor of decoder 200. It is to be noted that theprediction executor includes all or part of the following constituentelements: intra predictor 216; inter predictor 218; and predictioncontroller 220. The prediction executor includes, for example, intrapredictor 216 and inter predictor 218.
[0513] The predictor generates a prediction image of a current block (StepSq_1). This prediction image is also referred to as a prediction signal ora prediction block. It is to be noted that the prediction signal is, forexample, an intra prediction signal or an inter prediction signal. Morespecifically, the predictor generates the prediction image of the currentblock using a reconstructed image which has been already obtained foranother block through generation of a prediction image, restoration of aprediction residual, and addition of a prediction image. The predictor ofdecoder 200 generates the same prediction image as the prediction imagegenerated by the predictor of encoder 100. In other words, the predictionimages are generated according to a method common between thepredictors or mutually corresponding methods.
[0514] The reconstructed image may be, for example, an image in areference picture, or an image of a decoded block (that is, the other blockdescribed above) in a current picture which is the picture including thecurrent block. The decoded block in the current picture is, for example, aneighboring block of the current block.
[0515] FIG. 79 is a flow chart illustrating another example of a processperformed by the predictor of decoder 200.
[0516] The predictor determines either a method or a mode for generatinga prediction image (Step Sr_1). For example, the method or mode may bedetermined based on, for example, a prediction parameter, etc.
[0517] When determining a first method as a mode for generating aprediction image, the predictor generates a prediction image according tothe first method (Step Sr_2a). When determining a second method as amode for generating a prediction image, the predictor generates aprediction image according to the second method (Step Sr_2b). Whendetermining a third method as a mode for generating a prediction image,the predictor generates a prediction image according to the third method(Step Sr_2c).
[0518] The first method, the second method, and the third method may bemutually different methods for generating a prediction image. Each ofthe first to third methods may be an inter prediction method, an intraprediction method, or another prediction method. The above-describedreconstructed image may be used in these prediction methods.
[0519] FIG. 80A and FIG. 80B illustrate a flow chart illustrating anotherexample of a process performed by a predictor of decoder 200.
[0520] The predictor may perform a prediction process according to theflow illustrated in FIG. 80A and FIG. 80B as one example. It is to benoted that intra block copy illustrated in FIG. 80A and FIG. 80B is onemode which belongs to inter prediction, and in which a block included in acurrent picture is referred to as a reference image or a reference block.In other words, no picture different from the current picture is referred toin intra block copy. In addition, the PCM mode illustrated in FIG. 80A isone mode which belongs to intra prediction, and in which no transformand quantization is performed.
[0521] [Intra Predictor]Intra predictor 216 performs intra prediction by referring to ablock in a current picture stored in block memory 210, based on the intraprediction mode parsed from the stream, to generate a prediction image ofa current block (that is, an intra prediction image). More specifically,intra predictor 216 performs intra prediction by referring to pixel values(for example, luma and / or chroma values) of a block or blocks neighboringthe current block to generate an intra prediction image, and then outputsthe intra prediction image to prediction controller 220.
[0522] It is to be noted that when an intra prediction mode in which aluma block is referred to in intra prediction of a chroma block is selected,intra predictor 216 may predict the chroma component of the currentblock based on the luma component of the current block.
[0523] Moreover, when information parsed from a stream indicates thatPDPC is to be applied, intra predictor 216 corrects intra predicted pixelvalues based on horizontal / vertical reference pixel gradients.
[0524] FIG. 81 is a diagram illustrating one example of a processperformed by intra predictor 216 of decoder 200.
[0525] Intra predictor 216 firstly determines whether an MPM flagindicating 1 is present in the stream (Step Sw_11). Here, whendetermining that the MPM flag indicating 1 is present (Yes in StepSw_11), intra predictor 216 obtains, from entropy decoder 202,information indicating the intra prediction mode selected in encoder 100among MPMs (Step Sw_12). It is to be noted that such information isdecoded by entropy decoder 202 and output to intra predictor 216. Next,intra predictor 216 determines an MPM (Step Sw_13). MPMs include,for example, six intra prediction modes. Intra predictor 216 thendetermines the intra prediction mode which is included in a plurality ofintra prediction modes included in the MPMs and is indicated by theinformation obtained in Step Sw_12 (Step Sw_14).
[0526] When determining that no MPM flag indicating 1 is present (No inStep Sw_11), intra predictor 216 obtains information indicating the intraprediction mode selected in encoder 100 (Step Sw_15). In other words,intra predictor 216 obtains, from entropy decoder 202, informationindicating the intra prediction mode selected in encoder 100 from amongat least one intra prediction mode which is not included in the MPMs. Itis to be noted that such information is decoded by entropy decoder 202and output to intra predictor 216. Intra predictor 216 then determinesthe intra prediction mode which is not included in a plurality of intraprediction modes included in the MPMs and is indicated by theinformation obtained in Step Sw_15 (Step Sw_17).
[0527] Intra predictor 216 generates a prediction image according to theintra prediction mode determined in Step Sw_14 or Step Sw_17 (StepSw_18).
[0528] [Inter Predictor]Inter predictor 218 predicts the current block by referring to areference picture stored in frame memory 214. Prediction is performedin units of a current block or a current sub-block in the current block. Itis to be noted that the sub-block is included in the block and is a unitsmaller than the block. The size of the sub-block may be 4x4 pixels, 8x8pixels, or another size. The size of the sub-block may be switched for aunit such as a slice, brick, picture, etc.
[0529] For example, inter predictor 218 generates an inter predictionimage of a current block or a current sub-block by performing motioncompensation using motion information (for example, an MV) parsed froma stream (for example, a prediction parameter output from entropydecoder 202), and outputs the inter prediction image to predictioncontroller 220.
[0530] When the information parsed from the stream indicates that theOBMC mode is to be applied, inter predictor 218 generates the interprediction image using motion information of a neighboring block inaddition to motion information of the current block obtained throughmotion estimation.
[0531] Moreover, when the information parsed from the stream indicatesthat the FRUC mode is to be applied, inter predictor 218 derives motioninformation by performing motion estimation in accordance with thepattern matching method (bilateral matching or template matching)parsed from the stream. Inter predictor 218 then performs motioncompensation (prediction) using the derived motion information.
[0532] Moreover, when the BIO mode is to be applied, inter predictor 218derives an MV based on a model assuming uniform linear motion. Inaddition, when the information parsed from the stream indicates that theaffine mode is to be applied, inter predictor 218 derives an MV for eachsub-block, based on the MVs of a plurality of neighboring blocks.
[0533] [MV Derivation Flow]FIG. 82 is a flow chart illustrating one example of MV derivationin decoder 200.
[0534] Inter predictor 218 determines, for example, whether to decodemotion information (for example, an MV). For example, inter predictor218 may make the determination according to the prediction modeincluded in the stream, or may make the determination based on otherinformation included in the stream. Here, when determining to decodemotion information, inter predictor 218 derives an MV for a current blockin a mode in which the motion information is decoded. Whendetermining not to decode motion information, inter predictor 218 derivesan MV in a mode in which no motion information is decoded.
[0535] Here, MV derivation modes include a normal inter mode, a normalmerge mode, a FRUC mode, an affine mode, etc. which are described later.Modes in which motion information is decoded among the modes includethe normal inter mode, the normal merge mode, the affine mode(specifically, an affine inter mode and an affine merge mode), etc. It is tobe noted that motion information may include not only an MV but alsoMV predictor selection information which is described later. Modes inwhich no motion information is decoded include the FRUC mode, etc.Inter predictor 218 selects a mode for deriving an MV for the currentblock from the plurality of modes, and derives the MV for the currentblock using the selected mode.
[0536] FIG. 83 is a flow chart illustrating another example of MVderivation in decoder 200.
[0537] For example, inter predictor 218 may determine whether to decodean MV difference, that is for example, may make the determinationaccording to the prediction mode included in the stream, or may make thedetermination based on other information included in the stream. Here,when determining to decode an MV difference, inter predictor 218 mayderive an MV for a current block in a mode in which the MV difference isdecoded. In this case, for example, the MV difference included in thestream is decoded as a prediction parameter.
[0538] When determining not to decode any MV difference, inter predictor218 derives an MV in a mode in which no MV difference is decoded. Inthis case, no encoded MV difference is included in the stream.
[0539] Here, as described above, the MV derivation modes include thenormal inter mode, the normal merge mode, the FRUC mode, the affinemode, etc. which are described later. Modes in which an MV difference isencoded among the modes include the normal inter mode and the affinemode (specifically, the affine inter mode), etc. Modes in which no MVdifference is encoded include the FRUC mode, the normal merge mode,the affine mode (specifically, the affine merge mode), etc. Inter predictor218 selects a mode for deriving an MV for the current block from theplurality of modes, and derives the MV for the current block using theselected mode.
[0540] [MV Derivation > Normal Inter Mode]For example, when information parsed from a stream indicatesthat the normal inter mode is to be applied, inter predictor 218 derives anMV based on the information parsed from the stream and performsmotion compensation (prediction) using the MV.
[0541] FIG. 84 is a flow chart illustrating an example of inter predictionby normal inter mode in decoder 200.
[0542] Inter predictor 218 of decoder 200 performs motion compensationfor each block. At this time, first, inter predictor 218 obtains a pluralityof MV candidates for a current block based on information such as MVs ofa plurality of decoded blocks temporally or spatially surrounding thecurrent block (Step Sg_11). In other words, inter predictor 218generates an MV candidate list.
[0543] Next, inter predictor 218 extracts N (an integer of 2 or larger) MVcandidates from the plurality of MV candidates obtained in Step Sg_11, asmotion vector predictor candidates (also referred to as MV predictorcandidates) according to the predetermined ranks in priority order (StepSg_12). It is to be noted that the ranks in priority order are determinedin advance for the respective N MV predictor candidates.
[0544] Next, inter predictor 218 decodes the MV predictor selectioninformation from the input stream, and selects one MV predictorcandidate from the N MV predictor candidates as the MV predictor for thecurrent block using the decoded MV predictor selection information (StepSg_13).
[0545] Next, inter predictor 218 decodes an MV difference from the inputstream, and derives an MV for the current block by adding a differencevalue which is the decoded MV difference and the selected MV predictor(Step Sg_14).
[0546] Lastly, inter predictor 218 generates a prediction image for thecurrent block by performing motion compensation of the current blockusing the derived MV and the decoded reference picture (Step Sg_15).The processes in Steps Sg_11 to Sg_15 are executed on each block. Forexample, when the processes in Steps Sg_11 to Sg_15 are executed oneach of all the blocks in the slice, inter prediction of the slice using thenormal inter mode finishes. For example, when the processes in StepsSg_11 to Sg_15 are executed on each of all the blocks in the picture, interprediction of the picture using the normal inter mode finishes. It is to benoted that not all the blocks included in the slice may be subjected to theprocesses in Steps Sg_11 to Sg_15, and inter prediction of the slice usingthe normal inter mode may finish when part of the blocks are subjected tothe processes. Likewise, inter prediction of the picture using the normalinter mode may finish when the processes in Steps Sg_11 to Sg_15 areexecuted on part of the blocks in the picture.
[0547] [MV Derivation > Normal Merge Mode]For example, when information parsed from a stream indicatesthat the normal merge mode is to be applied, inter predictor 218 derivesan MV and performs motion compensation (prediction) using the MV.
[0548] FIG. 85 is a flow chart illustrating an example of inter predictionby normal merge mode in decoder 200.
[0549] At this time, first, inter predictor 218 obtains a plurality of MVcandidates for a current block based on information such as MVs of aplurality of decoded blocks temporally or spatially surrounding thecurrent block (Step Sh_11). In other words, inter predictor 218generates an MV candidate list.
[0550] Next, inter predictor 218 selects one MV candidate from theplurality of MV candidates obtained in Step Sh_11, thereby deriving anMV for the current block (Step Sh_12). More specifically, inter predictor218 obtains MV selection information included as a prediction parameterin a stream, and selects the MV candidate identified by the MV selectioninformation as the MV for the current block.
[0551] Lastly, inter predictor 218 generates a prediction image for thecurrent block by performing motion compensation of the current blockusing the derived MV and the decoded reference picture (Step Sh_13).The processes in Steps Sh_11 to Sh_13 are executed, for example, on eachblock. For example, when the processes in Steps Sh_11 to Sh_13 areexecuted on each of all the blocks in the slice, inter prediction of the sliceusing the normal merge mode finishes. In addition, when the processesin Steps Sh_11 to Sh_13 are executed on each of all the blocks in thepicture, inter prediction of the picture using the normal merge modefinishes. It is to be noted that not all the blocks included in the slice aresubjected to the processes in Steps Sh_11 to Sh_13, and inter prediction ofthe slice using the normal merge mode may finish when part of the blocksare subjected to the processes. Likewise, inter prediction of the pictureusing the normal merge mode may finish when the processes in StepsSh_11 to Sh_13 are executed on part of the blocks in the picture.
[0552] [MV Derivation > FRUC Mode]For example, when information parsed from a stream indicatesthat the FRUC mode is to be applied, inter predictor 218 derives an MV inthe FRUC mode and performs motion compensation (prediction) using theMV. In this case, the motion information is derived at the decoder 200side without being signaled from the encoder 100 side. For example,decoder 200 may derive the motion information by performing motionestimation. In this case, decoder 200 performs motion estimationwithout using any pixel value in a current block.
[0553] FIG. 86 is a flow chart illustrating an example of inter predictionby FRUC mode in decoder 200.
[0554] First, inter predictor 218 generates a list indicating MVs ofdecoded blocks spatially or temporally neighboring the current block byreferring to the MVs as MV candidates (the list is an MV candidate list,and may be used also as an MV candidate list for normal merge mode(Step Si_11). Next, a best MV candidate is selected from the plurality ofMV candidates registered in the MV candidate list (Step Si_12). Forexample, inter predictor 218 calculates the evaluation value of each MVcandidate included in the MV candidate list, and selects one of the MVcandidates as the best MV candidate based on the evaluation values.Based on the selected best MV candidate, inter predictor 218 then derivesan MV for the current block (Step Si_14). More specifically, for example,the selected best MV candidate is directly derived as the MV for thecurrent block. In addition, for example, the MV for the current blockmay be derived using pattern matching in a surrounding region of aposition which is included in a reference picture and corresponds to theselected best MV candidate. In other words, estimation using thepattern matching in a reference picture and the evaluation values may beperformed in the surrounding region of the best MV candidate, and whenthere is an MV that yields a better evaluation value, the best MVcandidate may be updated to the MV that yields the better evaluationvalue, and the updated MV may be determined as the final MV for thecurrent block. Update to the MV that yields the better evaluation valuemay not be performed.
[0555] Lastly, inter predictor 218 generates a prediction image for thecurrent block by performing motion compensation of the current blockusing the derived MV and the decoded reference picture (Step Si_15).The processes in Steps Si_11 to Si_15 are executed, for example, on eachblock. For example, when the processes in Steps Si_11 to Si_15 areexecuted on each of all the blocks in the slice, inter prediction of the sliceusing the FRUC mode finishes. For example, when the processes inSteps Si_11 to Si_15 are executed on each of all the blocks in the picture,inter prediction of the picture using the FRUC mode finishes. Eachsub-block may be processed similarly to the above-described case ofprocessing each block.
[0556] [MV Derivation > Affine Merge Mode]For example, when information parsed from a stream indicatesthat the affine merge mode is to be applied, inter predictor 218 derives anMV in the affine merge mode and performs motion compensation(prediction) using the MV.
[0557] FIG. 87 is a flow chart illustrating an example of inter predictionby the affine merge mode in decoder 200.
[0558] In the affine merge mode, first, inter predictor 218 derives MVs atrespective control points for a current block (Step Sk_11). The controlpoints are an upper-left corner point of the current block and anupper-right corner point of the current block as illustrated in FIG. 46A, oran upper-left corner point of the current block, an upper-right cornerpoint of the current block, and a lower-left corner point of the currentblock as illustrated in FIG. 46B.
[0559] For example, when the MV derivation methods illustrated in FIGs.47A to 47C are used, as illustrated in FIG. 47A, inter predictor 218 checksdecoded block A (left), block B (upper), block C (upper-right), block D(lower-left), and block E (upper-left) in this order, and identifies the firsteffective block decoded according to the affine mode.
[0560] Inter predictor 218 derives the MV at the control point using theidentified first effective block decoded according to the affine mode. Forexample, when block A is identified and block A has two control points, asillustrated in FIG. 47B, inter predictor 218 calculates motion vector v0 atthe upper-left corner control point of the current block and motion vectorv1 at the upper-right corner control point of the current block byprojecting motion vectors v3 and v4 at the upper-left corner and theupper-right corner of the decoded block including block A onto the currentblock. In this way, the MV at each control point is derived.
[0561] It is to be noted that, as illustrated in FIG. 49A, MVs at threecontrol points may be calculated when block A is identified and block Ahas two control points, and that, as illustrated in FIG. 49B, MVs at twocontrol points may be calculated when block A is identified and whenblock A has three control points.
[0562] In addition, when MV selection information is included as aprediction parameter in a stream, inter predictor 218 may derive the MVat each control point for the current block using the MV selectioninformation.
[0563] Next, inter predictor 218 performs motion compensation of each ofa plurality of sub-blocks included in the current block. In other words,inter predictor 218 calculates an MV for each of the plurality ofsub-blocks as an affine MV, using either two motion vectors v0 and v1 andthe above expression (1A) or three motion vectors v0, v1, and v2 and theabove expression (1B) (Step Sk_12). Inter predictor 218 then performsmotion compensation of the sub-blocks using these affine MVs anddecoded reference pictures (Step Sk_13). When the processes in StepsSk_12 and Sk_13 are executed for each of all the sub-blocks included inthe current block, the inter prediction using the affine merge mode for thecurrent block finishes. In other words, motion compensation of thecurrent block is performed to generate a prediction image of the currentblock.
[0564] It is to be noted that the above-described MV candidate list may begenerated in Step Sk_11. The MV candidate list may be, for example, alist including MV candidates derived using a plurality of MV derivationmethods for each control point. The plurality of MV derivation methodsmay be any combination of the MV derivation methods illustrated in FIGs.47A to 47C, the MV derivation methods illustrated in FIGs. 48A and 48B,the MV derivation methods illustrated in FIGs. 49A and 49B, and otherMV derivation methods.
[0565] It is to be noted that an MV candidate list may include MVcandidates in a mode in which prediction is performed in units of asub-block, other than the affine mode.
[0566] It is to be noted that, for example, an MV candidate list includingMV candidates in an affine merge mode in which two control points areused and an affine merge mode in which three control points are used maybe generated as an MV candidate list. Alternatively, an MV candidatelist including MV candidates in the affine merge mode in which twocontrol points are used and an MV candidate list including MV candidatesin the affine merge mode in which three control points are used may begenerated separately. Alternatively, an MV candidate list including MVcandidates in one of the affine merge mode in which two control points areused and the affine merge mode in which three control points are usedmay be generated.
[0567] [MV Derivation > Affine Inter Mode]For example, when information parsed from a stream indicatesthat the affine inter mode is to be applied, inter predictor 218 derives anMV in the affine inter mode and performs motion compensation(prediction) using the MV.
[0568] FIG. 88 is a flow chart illustrating an example of inter predictionby the affine inter mode in decoder 200.
[0569] In the affine inter mode, first, inter predictor 218 derives MVpredictors (v0, v1) or (v0, v1, v2) of respective two or three control points fora current block (Step Sj_11). The control points are an upper-left cornerpoint of the current block, an upper-right corner point of the current block,and a lower-left corner point of the current block as illustrated in FIG.46A or FIG. 46B.
[0570] Inter predictor 218 obtains MV predictor selection informationincluded as a prediction parameter in the stream, and derives the MVpredictor at each control point for the current block using the MVidentified by the MV predictor selection information. For example, whenthe MV derivation methods illustrated in FIGs. 48A and 48B are used,inter predictor 218 derives the motion vector predictors (v0, v1) or (v0, v1,v2) at control points for the current block by selecting the MV of the blockidentified by the MV predictor selection information among decodedblocks in the vicinity of the respective control points for the current blockillustrated in either FIG. 48A or FIG. 48B.
[0571] Next, inter predictor 218 obtains each MV difference included as aprediction parameter in the stream, and adds the MV predictor at eachcontrol point for the current block and the MV difference corresponding tothe MV predictor (Step Sj_12). In this way, the MV at each control pointfor the current block is derived.
[0572] Next, inter predictor 218 performs motion compensation of each ofa plurality of sub-blocks included in the current block. In other words,inter predictor 218 calculates an MV for each of the plurality ofsub-blocks as an affine MV, using either two motion vectors v0 and v1 andthe above expression (1A) or three motion vectors v0, v1, and v2 and theabove expression (1B) (Step Sj_13). Inter predictor 218 then performsmotion compensation of the sub-blocks using these affine MVs anddecoded reference pictures (Step Sj_14). When the processes in StepsSj_13 and Sj_14 are executed for each of all the sub-blocks included in thecurrent block, the inter prediction using the affine merge mode for thecurrent block finishes. In other words, motion compensation of thecurrent block is performed to generate a prediction image of the currentblock.
[0573] It is to be noted that the above-described MV candidate list may begenerated in Step Sj_11 as in Step Sk_11.
[0574] [MV Derivation > Triangle Mode]For example, when information parsed from a stream indicatesthat the triangle mode is to be applied, inter predictor 218 derives an MVin the triangle mode and performs motion compensation (prediction)using the MV.
[0575] FIG. 89 is a flow chart illustrating an example of inter predictionby the triangle mode in decoder 200.
[0576] In the triangle mode, first, inter predictor 218 splits the currentblock into a first partition and a second partition (Step Sx_11). At thistime, inter predictor 218 may obtain, from the stream, partitioninformation which is information related to the splitting as a predictionparameter. Inter predictor 218 may then split a current block into a firstpartition and a second partition according to the partition information.
[0577] Next, first, inter predictor 218 obtains a plurality of MVcandidates for a current block based on information such as MVs of aplurality of decoded blocks temporally or spatially surrounding thecurrent block (Step Sx_12). In other words, inter predictor 218generates an MV candidate list.
[0578] Inter predictor 218 then selects the MV candidate for the firstpartition and the MV candidate for the second partition as a first MV anda second MV, respectively, from the plurality of MV candidates obtainedin Step Sx_11 (Step Sx_13). At this time, inter predictor 218 may obtain,from the stream, MV selection information for identifying each selectedMV candidate, as a prediction parameter. Inter predictor 218 may thenselect the first MV and the second MV according to the MV selectioninformation.
[0579] Next, inter predictor 218 generates a first prediction image byperforming motion compensation using the selected first MV and adecoded reference picture (Step Sx_14). Likewise, inter predictor 218generates a second prediction image by performing motion compensationusing the selected second MV and a decoded reference picture (StepSx_15).
[0580] Lastly, inter predictor 218 generates a prediction image for thecurrent block by performing a weighted addition of the first predictionimage and the second prediction image (Step Sx_16).
[0581] [Motion Estimation > DMVR]For example, information parsed from a stream indicates thatDMVR is to be applied, inter predictor 218 performs motion estimationusing DMVR.
[0582] FIG. 90 is a flow chart illustrating an example of motionestimation by DMVR in decoder 200.
[0583] Inter predictor 218 derives an MV for a current block according tothe merge mode (Step Sl_11). Next, inter predictor 218 derives the finalMV for the current block by searching the region surrounding thereference picture indicated by the MV derived in Sl_11 (Step Sl_12). Inother words, the MV of the current block is determined according to theDMVR.
[0584] FIG. 91 is a flow chart illustrating a specific example of motionestimation by DMVR in decoder 200.
[0585] First, in Step 1 illustrated in FIG. 58A, inter predictor 218calculates the cost between the search position (also referred to as astarting point) indicated by the initial MV and eight surrounding searchpositions. Inter predictor 218 then determines whether the cost at eachof the search positions other than the starting point is the smallest.Here, when determining that the cost at one of the search positions otherthan the starting point is the smallest, inter predictor 218 changes atarget to the search position at which the smallest cost is obtained, andperforms the process in Step 2 illustrated in FIG. 58A. When the cost atthe starting point is the smallest, inter predictor 218 skips the process inStep 2 illustrated in FIG. 58A and performs the process in Step 3.
[0586] In Step 2 illustrated in FIG. 58A, inter predictor 218 performssearch similar to the process in Step 1, regarding the search position afterthe target change as a new starting point according to the result of theprocess in Step 1. Inter predictor 218 then determines whether the costat each of the search positions other than the starting point is thesmallest. Here, when determining that the cost at one of the searchpositions other than the starting point is the smallest, inter predictor 218performs the process in Step 4. When the cost at the starting point is thesmallest, inter predictor 218 performs the process in Step 3.
[0587] In Step 4, inter predictor 218 regards the search position at thestarting point as the final search position, and determines the differencebetween the position indicated by the initial MV and the final searchposition to be a vector difference.
[0588] In Step 3 illustrated in FIG. 58A, inter predictor 218 determinesthe pixel position at sub-pixel accuracy at which the smallest cost isobtained, based on the costs at the four points located at upper, lower, left,and right positions with respect to the starting point in Step 1 or Step 2,and regards the pixel position as the final search position. The pixelposition at the sub-pixel accuracy is determined by performing weightedaddition of each of the four upper, lower, left, and right vectors ((0, 1), (0,-1), (-1, 0), and (1, 0)), using, as a weight, the cost at a corresponding oneof the four search positions. Inter predictor 218 then determines thedifference between the position indicated by the initial MV and the finalsearch position to be the vector difference.
[0589] [Motion Compensation > BIO / OBMC / LIC]For example, when information parsed from a stream indicatesthat correction of a prediction image is to be performed, upon generatinga prediction image, inter predictor 218 corrects the prediction imagebased on the mode for the correction. The mode is, for example, one ofBIO, OBMC, and LIC described above.
[0590] FIG. 92 is a flow chart illustrating one example of generation of aprediction image in decoder 200.
[0591] Inter predictor 218 generates a prediction image (Step Sm_11),and corrects the prediction image according to any of the modes describedabove (Step Sm_12).
[0592] FIG. 93 is a flow chart illustrating another example of generationof a prediction image in decoder 200.
[0593] Inter predictor 218 derives an MV for a current block (Step Sn_11).Next, inter predictor 218 generates a prediction image using the MV(Step Sn_12), and determines whether to perform a correction process(Step Sn_13). For example, inter predictor 218 obtains a predictionparameter included in the stream, and determines whether to perform acorrection process based on the prediction parameter. This predictionparameter is, for example, a flag indicating whether each of theabove-described modes is to be applied. Here, when determining toperform a correction process (Yes in Step Sn_13), inter predictor 218generates the final prediction image by correcting the prediction image(Step Sn_14). It is to be noted that, in LIC, the luminance andchrominance of the prediction image may be corrected in Step Sn_14.When determining not to perform a correction process (No in Step Sn_13),inter predictor 218 outputs the final prediction image without correctingthe prediction image (Step Sn_15).
[0594] [Motion Compensation > OBMC]For example, when information parsed from a stream indicatesthat OBMC is to be performed, upon generating a prediction image, interpredictor 218 corrects the prediction image according to the OBMC.
[0595] FIG. 94 is a flow chart illustrating an example of correction of aprediction image by OBMC in decoder 200. It is to be noted that the flowchart in FIG. 94 indicates the correction flow of a prediction image usingthe current picture and the reference picture illustrated in FIG. 62.
[0596] First, as illustrated in FIG. 62, inter predictor 218 obtains aprediction image (Pred) by normal motion compensation using an MVassigned to the current block.
[0597] Next, inter predictor 218 obtains a prediction image (Pred_L) byapplying a motion vector (MV_L) which has been already derived for thedecoded block neighboring to the left of the current block to the currentblock (re-using the motion vector for the current block). Inter predictor218 then performs a first correction of a prediction image by overlappingtwo prediction images Pred and Pred_L. This provides an effect ofblending the boundary between neighboring blocks.
[0598] Likewise, inter predictor 218 obtains a prediction image (Pred_U)by applying an MV (MV_U) which has been already derived for thedecoded block neighboring above the current block to the current block(re-using the motion vector for the current block). Inter predictor 218then performs a second correction of the prediction image by overlappingthe prediction image Pred_U to the prediction images (for example, Predand Pred_L) on which the first correction has been performed. Thisprovides an effect of blending the boundary between neighboring blocks.The prediction image obtained by the second correction is the one inwhich the boundary between the neighboring blocks has been blended(smoothed), and thus is the final prediction image of the current block.
[0599] [Motion Compensation > BIO]For example, when information parsed from a stream indicatesthat BIO is to be performed, upon generating a prediction image, interpredictor 218 corrects the prediction image according to the BIO.
[0600] FIG. 95 is a flow chart illustrating an example of correction of aprediction image by the BIO in decoder 200.
[0601] As illustrated in FIG. 63, inter predictor 218 derives two motionvectors (M0, M1), using two reference pictures (Ref0, Ref1) different fromthe picture (Cur Pic) including a current block. Inter predictor 218 thenderives a prediction image for the current block using the two motionvectors (M0, M1) (Step Sy_11). It is to be noted that motion vector M0 isa motion vector (MVx0, MVy0) corresponding to reference picture Ref0, andmotion vector M1 is a motion vector (MVx1, MVy1) corresponding toreference picture Ref1.
[0602] Next, inter predictor 218 derives interpolated image I0 for thecurrent block using motion vector M0 and reference picture L0. Inaddition, inter predictor 218 derives interpolated image I1 for the currentblock using motion vector M1 and reference picture L1 (Step Sy_12).Here, interpolated image I0 is an image included in reference picture Ref0and to be derived for the current block, and interpolated image I1 is animage included in reference picture Ref1 and to be derived for the currentblock. Each of interpolated image I0 and interpolated image I1 may bethe same in size as the current block. Alternatively, each of interpolatedimage I0 and interpolated image I1 may be an image larger than thecurrent block. Furthermore, interpolated image I0 and interpolatedimage I1 may include a prediction image obtained by using motion vectors(M0, M1) and reference pictures (L0, L1) and applying a motioncompensation filter.
[0603] In addition, inter predictor 218 derives gradient images (Ix0, Ix1,Iy0, Iy1) of the current block, from interpolated image I0 and interpolatedimage I1 (Step Sy_13). It is to be noted that the gradient images in thehorizontal direction are (Ix0, Ix1), and the gradient images in the verticaldirection are (Iy0, Iy1). Inter predictor 218 may derive the gradientimages by, for example, applying a gradient filter to the interpolatedimages. The gradient images may be the ones each of which indicatesthe amount of spatial change in pixel value along the horizontal directionor the amount of spatial change in pixel value along the vertical direction.
[0604] Next, inter predictor 218 derives, for each sub-block of the currentblock, an optical flow (vx, vy) which is a velocity vector, using theinterpolated images (I0, I1) and the gradient images (Ix0, Ix1, Iy0, Iy1).As one example, a sub-block may be 4x4 pixel sub-CU.
[0605] Next, inter predictor 218 corrects a prediction image for thecurrent block using the optical flow (vx, vy). For example, interpredictor 218 derives a correction value for the value of a pixel includedin a current block, using the optical flow (vx, vy) (Step Sy_15). Interpredictor 218 may then correct the prediction image for the current blockusing the correction value (Step Sy_16). It is to be noted that thecorrection value may be derived in units of a pixel, or may be derived inunits of a plurality of pixels or in units of a sub-block.
[0606] It is to be noted that the BIO process flow is not limited to theprocess disclosed in FIG. 95. Only part of the processes disclosed in FIG.95 may be performed, or a different process may be added or used as areplacement, or the processes may be executed in a different processingorder.
[0607] [Motion Compensation > LIC]For example, when information parsed from a stream indicatesthat LIC is to be performed, upon generating a prediction image, interpredictor 218 corrects the prediction image according to the LIC.
[0608] FIG. 96 is a flow chart illustrating an example of correction of aprediction image by the LIC in decoder 200.
[0609] First, inter predictor 218 obtains a reference image correspondingto a current block from a decoded reference picture using an MV (StepSz_11).
[0610] Next, inter predictor 218 extracts, for the current block,information indicating how the luma value has changed between thecurrent picture and the reference picture (Step Sz_12). This extractionis performed based on the luma pixel values for the decoded leftneighboring reference region (surrounding reference region) and thedecoded upper neighboring reference region (surrounding referenceregion), and the luma pixel values at the corresponding positions in thereference picture specified by the derived MVs. Inter predictor 218calculates a luminance correction parameter, using the informationindicating how the luma value changed (Step Sz_13).
[0611] Inter predictor 218 generates a prediction image for the currentblock by performing a luminance correction process in which theluminance correction parameter is applied to the reference image in thereference picture specified by the MV (Step Sz_14). In other words, theprediction image which is the reference image in the reference picturespecified by the MV is subjected to the correction based on the luminancecorrection parameter. In this correction, luminance may be corrected, orchrominance may be corrected.
[0612] [Prediction Controller]Prediction controller 220 selects either an intra prediction imageor an inter prediction image, and outputs the selected image to adder 208.As a whole, the configurations, functions, and processes of predictioncontroller 220, intra predictor 216, and inter predictor 218 at the decoder200 side may correspond to the configurations, functions, and processes ofprediction controller 128, intra predictor 124, and inter predictor 126 atthe encoder 100 side.
[0613] [Specification of Interlayer Reference Pictures]In the multilayer encoding such as spatially scalable encoding andmultiview encoding, interlayer prediction using interlayer correlation isperformed, and thus one or more interlayer reference pictures may beincluded in a reference picture list. It is to be noted that multilayerencoding may be represented as a multilayer coding scheme. That themultilayer encoding is performed corresponds to encoding and decoding ofone or more pictures are performed using the multilayer coding scheme.
[0614] FIG. 97 is a diagram indicating one example of a syntax structurefor notifying of information used to configure a reference picture list.Specifically, ref_pic_list_struct(listIdx, rplsIdx) in FIG. 97 represents thesyntax structure for notifying of the information used to configure thereference picture list that is identified based on listldx and rplsIdx.
[0615] For example, in a sequence parameter set (SPS), one or morereference picture lists may be defined for each of L0 and L1. In at leastone of a picture parameter set, a picture header, or a slice header,whether to refer to the one or more reference picture lists defined in theSPS and to newly define a reference picture list may be selectable.
[0616] Specifically, in the SPS, [i] number of reference picture listsindicated by sps_num_ref_pic_lists[i] may be defined(sps_num_ref_pic_lists[i] is not illustrated). In this case, the referencepicture list which is newly defined in the at least one of the pictureparameter set, the picture header, or the slice header may be handled assps_num_ref_pic_lists[i]+1-th reference picture list.
[0617] In addition, for example, listIdx may be an index indicating eitherL0 or L1. In addition, rplsIdx may be an index of the reference picturelist corresponding to the one or more reference picture lists defined for L0or L1.
[0618] In addition, inter_layer_ref_pic_flag[listIdx][rplsIdx][i] indicateswhether the i-th entry in the reference picture list identified based onlistIdx and rplsIdx is an interlayer reference picture. This flag is alsocalled as an interlayer reference picture flag.
[0619] For example, when the interlayer reference picture flag indicatestrue, an interlayer reference picture (ILRP) is specified by an interlayerreference picture index.
[0620] Specifically, when inter_layer_ref_pic_flag[listIdx][rplsIdx][i] is 1,an interlayer reference picture is specified as an i-th reference picture byilrp_idx[listIdx][rplsIdx][i]. Here, ilrp_idx[listIdx][rplsIdx][i] specifiesan index that is assigned for the interlayer reference picture in a directreference layer when the i-th reference picture in the reference picturelist identified based on listIdx and rplsIdx is an interlayer referencepicture.
[0621] In other words, when inter_layer_ref_pic_flag[listIdx][rplsIdx][i]is 1, the i-th reference picture in the reference picture list identifiedbased on listIdx and rplsIdx is an interlayer reference picture. On theother hand, when inter_layer_ref_pic_flag[listIdx][rplsIdx][i] is 0, the i-threference picture in the reference picture list identified based on listIdxand rplsIdx is not an interlayer reference picture.
[0622] In addition, when inter_layer_ref_pic_flag[listIdx][rplsIdx][i] isnot preset (is not encoded), inter_layer_ref_pic_flag[listIdx][rplsIdx][i] isassumed to be 0.
[0623] It is to be noted that the interlayer reference picture index is in arange from 0 to NumDirectRefLayers[GeneralLayerIdx[nuh_layer_id]] - 1.Here, NumDirectRefLayers[GeneralLayerIdx[nuh_layer_id]] correspondsto the number of referable reference layers.
[0624] In addition, the above-described direct reference layer means alayer that is directly referred to in interlayer reference. For example, inthe interlayer reference, a dependency relationship in which a secondlayer refers to a first layer and a third layer refers to the second layermay be present. The third layer indirectly depends on the first layer, butdirectly refers to the second layer only. The layer that is indirectlyreferred to and the layer that is directly referred to are distinguishedfrom each other, and the layer that is directly referred to is called as adirect reference layer.
[0625] In FIG. 97, when inter_layer_ref_pics_present_flag is 0,notification of inter_layer_ref_pic_flag[listIdx][rplsIdx][i] is skipped. Inthis case, inter_layer_ref_pic_flag[listIdx][rplsIdx][i] is regarded as 0.As a result, no interlayer reference picture is specified.
[0626] Here, inter_layer_ref_pics_present_flag is an interlayer referencepicture present flag included in a sequence parameter set (SPS), andindicates, for example, whether use of the one or more interlayerreference pictures is allowed.
[0627] When the interlayer reference picture present flag is 1 thatindicates true, the use of the one or more interlayer reference pictures isallowed. In this case, the one or more interlayer reference pictures maybe used for inter prediction regarding one or more of the pictures includedin an encoded layer video stream.
[0628] When the interlayer reference picture present flag is 0 thatindicates false, the use of the one or more interlayer reference pictures isnot allowed. In this case, no interlayer reference pictures is used forinter prediction regarding any one of the pictures included in the encodedlayer video stream.
[0629] For example, inter_layer_ref_pic_flag[listIdx][rplsIdx][i] describedabove is an interlayer reference picture flag and indicates whether theone or more interlayer reference pictures are used. More specifically, theinterlayer reference picture indicates whether each of the referencepictures in the reference picture list that is used to encode and decode acurrent picture is one of the one or more interlayer reference pictures.
[0630] In addition, when the interlayer reference picture present flagindicates that the use of the one or more interlayer reference pictures isallowed, the interlayer reference picture flag indicating whether the oneor more interlayer reference pictures are used is encoded and decoded.When the interlayer reference picture flag indicates that the interlayerreference picture is used, the interlayer reference picture is used as areference picture in the reference picture list.
[0631] On the other hand, when encoding and decoding of the interlayerreference picture flag is not performed or the interlayer reference pictureflag indicates that the one or more interlayer reference pictures are notused, a reference pictures different from the one or more interlayerreference pictures is used.
[0632] It is to be noted that multilayer encoding is disabled whensps_video_parameter_set_id that is a video parameter set ID notified of inthe SPS is 0. For this reason, in this case, no interlayer prediction isperformed, and no interlayer reference picture is specified. For example,in this case, inter_layer_ref_pics_present_flag is assumed to be 0.
[0633] In addition, whenvps_independent_layer_flag[GeneralLayerIdx[nuh_layer_id]] (notillustrated) is 1, inter_layer_ref_pics_present_flag is 0. In other words,when interlayer prediction is not used, the use of the one or moreinterlayer reference pictures is not allowed.
[0634] Here,vps_independent_layer_flag[GeneralLayerIdx[nuh_layer_id]] is anindependent layer flag included in a video parameter set (VPS), andindicates, for example, whether interlayer prediction is used. When theindependent layer flag is 1, interlayer prediction is not used. When theindependent layer flag is 0, interlayer prediction may be used.
[0635] [Aspect 1 regarding Notification of Interlayer Reference Picture PresentFlags]FIG. 98 is a diagram indicating one example of an SPS syntaxstructure for notifying of an interlayer reference picture present flag inthis aspect.
[0636] In this example, sps_video_parameter_set_id that is a videoparameter set ID is notified of. Subsequently, whether to notify ofinter_layer_ref_pics_present_flag that is an interlayer reference picturepresent flag is switched according to the value ofsps_video_parameter_set_id.
[0637] Specifically, sps_video_parameter_set_id is a video parameter setID that is notified of in a sequence parameter set (SPS), as an identifier ofthe video parameter set that is applied to a sequence. In addition,sps_video_parameter_set_id is represented as an integer of 0 or greater,and corresponds to whether multilayer encoding is performed.
[0638] More specifically, multilayer encoding is performed whensps_video_parameter_set_id is greater than 0. In this case, the videoparameter set ID represents an identifier of the video parameter set forperforming the multilayer encoding. On the other hand, multilayerencoding is not performed when sps_video_parameter_set_id is 0.
[0639] For example, when sps_video_parameter_set_id has a valueindicating that multilayer encoding is performed,inter_layer_ref_pics_present_flag is notified of. Specifically, whensps_video_parameter_set_id is greater than 0,inter_layer_ref_pics_present_flag is notified of.
[0640] For example, when sps_video_parameter_set_id has a valueindicating that multilayer encoding is not performed, notification ofinter_layer_ref_pics_present_flag is skipped. Specifically, whensps_video_parameter_set_id is 0, inter_layer_ref_pics_present_flag is notnotified of. In this case, inter_layer_ref_pics_present_flag can beregarded as 0.
[0641] FIG. 99 is a diagram indicating one example of an operation fornotifying of the interlayer reference picture present flag.
[0642] As in FIG. 99, when sps_video_parameter_set_id is greater than 0(Yes in S101), inter_layer_ref_pics_present_flag is notified of (S102).When sps_video_parameter_set_id is not greater than 0 (No in S101),inter_layer_ref_pics_present_flag is not notified of.
[0643] In other words, when multilayer encoding is used, the interlayerreference picture preset flag indicating whether the use of the one or moreinterlayer reference pictures is allowed is encoded and decoded. On theother hand, when multilayer encoding is not used, the interlayerreference picture preset flag indicating whether the use of the one or moreinterlayer reference pictures is allowed is not encoded and decoded.When encoding and decoding of the interlayer reference picture presentflag is not performed, the use of the one or more interlayer referencepictures is not allowed.
[0644] As described above, with the configuration in this aspect, there is apossibility that notification of inter_layer_ref_pics_present_flag isskipped and the amount of codes is reduced according to a use case, whenmultilayer encoding is not performed.
[0645] It is to be noted that in the above-described syntax structure,encoding and decoding of sps_video_parameter_set_id is performed.However, encoding and decoding of sps_video_parameter_set_id may notbe performed. For example, sps_video_parameter_set_id may be definedaccording to another coding parameter, or the like.
[0646] In addition, in the above description, whether multilayer encodingis performed is determined using sps_video_parameter_set_id. However,sps_video_parameter_set_id may not be used to determine whethermultilayer encoding is performed. For example, whether to performmultilayer encoding may be determined according to another codingparameter, or the like.
[0647] In addition, this aspect may be combined with at least part ofanother aspect and performed. In addition, a part of this aspect may becombined with another aspect and performed. In addition, the syntaxesand values described above are examples, and similar processes may beperformed using syntaxes and values different from the syntaxes andvalues described above. Not all the constituent elements in this aspectare always necessary, and only a part of the constituent elements in thisaspect may be used.
[0648] [Aspect 2 regarding Notification of Interlayer Reference Picture PresentFlags]FIG. 100 is a diagram indicating one example of an SPS syntax fornotifying of an interlayer reference picture present flag in this aspect.
[0649] In this example, sps_video_parameter_set_id that is a videoparameter set ID is notified of. Subsequently, regardless of the value ofsps_video_parameter_set_id, inter_layer_ref_pics_present_flag that is aninterlayer reference picture present flag is always notified of.
[0650] For example, when sps_video_parameter_set_id has a valueindicating that multilayer encoding is not performed,inter_layer_ref_pics_present_flag is fixed to 0. Specifically, whensps_video_parameter_set_id is 0, inter_layer_ref_pics_present_flag isfixed to 0.
[0651] In other words, inter_layer_ref_pics_present_flag that is notifiedof when multilayer encoding is not performed indicates that the use of theone or more interlayer reference pictures is not allowed.
[0652] On the other hand, when sps_video_parameter_set_id has a valueindicating that multilayer encoding is performed,inter_layer_ref_pics_present_flag may be one of 0 and 1. Specifically,when sps_video_parameter_set_id is 0, inter_layer_ref_pics_present_flagmay be one of 0 and 1.
[0653] In other words, inter_layer_ref_pics_present_flag that is notifiedof when multilayer encoding is performed indicates that the use of the oneor more interlayer reference pictures is not allowed or that the use isallowed.
[0654] As described above, with the configuration in this aspect,inter_layer_ref_pics_present_flag is fixed to 0 when multilayer encodingis not performed. Accordingly, in the syntax structure for notifyinginformation that is used to configure a reference picture list as indicatedin FIG. 97, there is a possibility that occurrence of mismatch ofinformation related to an interlayer reference picture is prevented.
[0655] It is to be noted that this aspect may be combined with at leastpart of another aspect and performed. In addition, this aspect may becombined with at least part of another aspect and performed. Inaddition, the syntaxes and values described above are examples, andsimilar processes may be performed using syntaxes and values differentfrom the syntaxes and values described above. Not all the constituentelements in this aspect are always necessary, and only a part of theconstituent elements in this aspect may be used.
[0656] [Representative Examples of Configurations and Processes]Representative examples of configurations of and processesperformed by encoder 100 and decoder 200 described above are indicatedbelow.
[0657] FIG. 101 is a flow chart indicating an operation that is performedby encoder 100. For example, encoder 100 includes circuitry andmemory coupled to the circuitry. The circuitry and memory included inencoder 100 may correspond to processor a1 and memory a2 illustrated inFIG. 8. In operation, the circuitry of encoder 100 performs the following.
[0658] For example, the circuitry of encoder 100 switches whether toencode the first flag, according to whether a multilayer coding scheme isused to encode one or more pictures (S201). Here, the first flag indicateswhether use of one or more interlayer reference pictures for interlayerprediction is allowed.
[0659] In this way, there is a possibility that encoder 100 is capable ofskipping encoding of the first flag indicating whether the use of the one ormore interlayer reference pictures is allowed, according to whether themultilayer coding scheme is used. Accordingly, there is a possibility thatencoder 100 is capable of contributing reduction in the amount of codesregarding information for controlling the interlayer prediction.
[0660] FIG. 102 is a flow chart indicating a more detailed operation thatis performed by encoder 100. In operation, the circuitry of encoder 100may perform the following.
[0661] For example, when the multilayer coding scheme is used to encodethe one or more pictures (Yes in S301), the circuitry of encoder 100encodes the first flag (S302). In addition, when the multilayer codingscheme is not used to encode the one or more pictures (No in S301), thecircuitry of encoder 100 does not encode the first flag.
[0662] In this way, there is a possibility that encoder 100 is capable ofadaptively controlling whether the use of the one or more interlayerreference pictures is allowed, when the multilayer coding scheme is used.In other words, in this case, there is a possibility that encoder 100 iscapable of adaptively controlling whether the use of the interlayerprediction is allowed.
[0663] Furthermore, there is a possibility that encoder 100 is capable ofskipping encoding of the first flag because the one or more interlayerreference pictures are not used when the multilayer coding scheme is notused. Accordingly, there is a possibility that encoder 100 is capable ofcontributing reduction in the amount of codes. It is to be noted thatunuse of the multilayer coding scheme corresponds to use of a single layercoding scheme.
[0664] In addition, for example, when (i) the first flag is encoded and (ii)the first flag indicates that the use of the one or more interlayer referencepictures is allowed, the use of the one or more interlayer referencepictures may be allowed. In addition, for example, when (i) the first flagis encoded and (ii) the first flag indicates that the use of the one or moreinterlayer reference pictures is not allowed, the use of the one or moreinterlayer reference pictures may not be allowed. In addition, forexample, when the first flag is not encoded, the use of the one or moreinterlayer reference pictures may not be allowed.
[0665] In this way, there is a possibility that encoder 100 is capable ofcontrolling whether the one or more interlayer reference pictures areused so as to conform to the first flag when the first flag is encoded. Inaddition, there is a possibility that encoder 100 is capable of simplifyingthe process because the use of the one or more interlayer referencepictures is not allowed when the first flag is not encoded.
[0666] In addition, for example, the circuitry of encoder 100 may encode asecond flag when the use of the one or more interlayer reference picturesis allowed. The second flag indicates whether any one of the one or moreinterlayer reference pictures is used as a reference picture that isincluded in a reference picture list.
[0667] In this way, there is a possibility that encoder 100 is capable ofadaptively controlling whether the use of the one or more interlayerreference pictures is allowed as the reference pictures in the referencepicture list, when the use of the one or more interlayer reference picturesis allowed. Accordingly, in this case, there is a possibility that encoder100 is capable of appropriately encoding a current picture either usingthe interlayer reference pictures or without using the interlayer referencepictures.
[0668] In addition, for example, when (i) the second flag that is encodedindicates that any of the one or more interlayer reference pictures is notused or (ii) the second flag is not encoded, a reference picture differentfrom the one or more interlayer reference pictures may be used.
[0669] In this way, there is a possibility that encoder 100 is capable ofusing the reference picture other than the one or more interlayerreference pictures either when the use of the one or more interlayerreference pictures is not allowed or when the one or more interlayerreference pictures are not used. Accordingly, there is a possibility thatencoder 200 is capable of appropriately encoding the current pictureusing the reference picture other than the one or more interlayerreference pictures.
[0670] In addition, for example, the circuitry of encoder 100 may switchwhether to encode the first flag, according to a video parameter set IDcorresponding to whether the multilayer coding scheme is used to encodethe one or more pictures.
[0671] Here, the video parameter set ID is 0 or greater. When the videoparameter set ID is 0, the multilayer coding scheme is not used to encodethe one or more pictures. The multilayer coding scheme is used toencode the one or more pictures when the video parameter set ID isgreater than 0. In addition, in this case, the video parameter set IDindicates an identifier of a video parameter set for encoding the one ormore pictures using the multilayer coding scheme.
[0672] In this way, there is a possibility that encoder 100 is capable ofskipping encoding of the first flag according to the video parameter set ID.Accordingly, there is a possibility that encoder 100 is capable ofcontributing reduction in the amount of codes regarding information forcontrolling the interlayer prediction. Furthermore, there is a possibilitythat encoder 100 is capable of appropriately encoding the one or morepictures using the video parameter set identified according to the videoparameter set ID and the multilayer coding scheme, when the videoparameter set ID is greater than 0.
[0673] It is to be noted that inter_layer_ref_pics_present_flag is anexample of a first flag. In addition, inter_layer_ref_pics_present_flag isan example of a second flag. In addition, sps_video_parameter_set_id isan example of a video parameter set ID.
[0674] Alternatively, entropy encoder 110 of encoder 100 may perform theabove-described operation as the circuitry of encoder 100.
[0675] FIG. 103 is a flow chart indicating an operation that is performedby decoder 200. For example, decoder 200 includes circuitry and memorycoupled to the circuitry. The circuitry and memory included in decoder200 may correspond to processor b1 and memory b2 illustrated in FIG. 68.In operation, the circuitry of decoder 200 may perform the following.
[0676] For example, the circuitry of decoder 200 switches whether todecode the first flag, according to whether a multilayer coding scheme isused to decode one or more pictures (S401). Here, the first flag indicateswhether use of one or more interlayer reference pictures for interlayerprediction is allowed.
[0677] In this way, there is a possibility that decoder 200 is capable ofskipping decoding of the first flag indicating whether the use of the one ormore interlayer reference pictures is allowed, according to whether themultilayer coding scheme is used. Accordingly, there is a possibility thatdecoder 200 is capable of contributing reduction in the amount of codesregarding information for controlling the interlayer prediction.
[0678] FIG. 104 is a flow chart indicating a more detailed operation thatis performed by decoder 200. In operation, the circuitry of decoder 200may perform the following.
[0679] For example, when the multilayer coding scheme is used to decodethe one or more picture (Yes in S501), the circuitry of decoder 200 decodesthe first flag (S502). In addition, when the multilayer coding scheme isnot used to decode the one or more pictures (No in S501), the circuitry ofdecoder 200 does not decode the first flag.
[0680] In this way, there is a possibility that decoder 200 is capable ofadaptively controlling whether the use of the one or more interlayerreference pictures is allowed, when the multilayer coding scheme is used.In other words, in this case, there is a possibility that decoder 200 iscapable of adaptively controlling whether the use of the interlayerprediction is allowed.
[0681] Furthermore, there is a possibility that decoder 200 is capable ofskipping decoding of the first flag because the one or more interlayerreference pictures are not used when the multilayer coding scheme is notused. Accordingly, there is a possibility that decoder 200 is capable ofcontributing reduction in the amount of codes. It is to be noted thatunuse of the multilayer coding scheme corresponds to use of a single layercoding scheme.
[0682] In addition, for example, when (i) the first flag is decoded and (ii)the first flag indicates that the use of the one or more interlayer referencepictures is allowed, the use of the one or more interlayer referencepictures may be allowed. In addition, for example, when (i) the first flagis decoded and (ii) the first flag indicates that the use of the one or moreinterlayer reference pictures is not allowed, the use of the one or moreinterlayer reference pictures may not be allowed. In addition, forexample, when the first flag is not decoded, the use of the one or moreinterlayer reference pictures may not be allowed.
[0683] In this way, there is a possibility that decoder 200 is capable ofcontrolling whether the one or more interlayer reference pictures areused so as to conform to the first flag when the first flag is decoded. Inaddition, there is a possibility that decoder 200 is capable of simplifyingthe process because the use of the one or more interlayer referencepictures is not allowed when the first flag is not decoded.
[0684] In addition, for example, the circuitry of decoder 200 may decode asecond flag when the use of the one or more interlayer reference picturesis allowed. The second flag indicates whether any one of the one or moreinterlayer reference pictures is used as a reference picture that isincluded in a reference picture list.
[0685] In this way, there is a possibility that decoder 200 is capable ofadaptively controlling whether the use of the one or more interlayerreference pictures is allowed as the reference pictures in the referencepicture list, when the use of the one or more interlayer reference picturesis allowed. Accordingly, in this case, there is a possibility that decoder200 is capable of appropriately decoding a current picture either using theone or more interlayer reference pictures or without using the one or moreinterlayer reference pictures.
[0686] In addition, for example, when (i) the second flag that is decodedindicates that any of the one or more interlayer reference pictures is notused or (ii) the second flag is not decoded, a reference picture differentfrom the one or more interlayer reference pictures may be used.
[0687] In this way, there is a possibility that decoder 200 is capable ofusing the reference picture other than the one or more interlayerreference pictures either when the use of the one or more interlayerreference pictures is not allowed or when the one or more interlayerreference pictures are not used. Accordingly, there is a possibility thatdecoder 200 is capable of appropriately decoding the current picture usingthe reference picture other than the one or more interlayer referencepictures.
[0688] In addition, for example, the circuitry of decoder 200 may switchwhether to decode the first flag, according to a video parameter set IDcorresponding to whether the multilayer coding scheme is used to decodethe one or more pictures.
[0689] Here, the video parameter set ID is 0 or greater. When the videoparameter set ID is 0, the multilayer coding scheme is not used to decodethe one or more pictures. The multilayer coding scheme is used todecode the one or more pictures when the video parameter set ID isgreater than 0. In addition, in this case, the video parameter set IDindicates an identifier of a video parameter set for decoding the one ormore pictures using the multilayer coding scheme.
[0690] In this way, there is a possibility that decoder 200 is capable ofskipping decoding of the first flag according to the video parameter set ID.Accordingly, there is a possibility that decoder 200 is capable ofcontributing reduction in the amount of codes regarding information forcontrolling the interlayer prediction. Furthermore, there is a possibilitythat decoder 200 is capable of appropriately decoding the one or morepictures using the video parameter set identified according to the videoparameter set ID and the multilayer coding scheme, when the videoparameter set ID is greater than 0.
[0691] It is to be noted that inter_layer_ref_pics_present_flag is anexample of a first flag. In addition, inter_layer_ref_pics_present_flag isan example of a second flag. In addition, sps_video_parameter_set_id isan example of a video parameter set ID.
[0692] In addition, entropy decoder 202 of decoder 200 may perform theabove-described operation as the circuitry of decoder 200.
[0693] [Other Examples]Encoder 100 and decoder 200 in each of the above-describedexamples may be used as an image encoder and an image decoder,respectively, or may be used as a video encoder and a video decoder,respectively.
[0694] Alternatively, encoder 100 and decoder 200 may be used as anentropy encoder and an entropy decoder, respectively. In other words,encoder 100 and decoder 200 may correspond only to entropy encoder 110and entropy decoder 202, respectively. The other constituent elementsmay be included in other devices.
[0695] In addition, encoder 100 may include an inputter and an outputter.For example, one or more pictures are input to an inputter of encoder 100,and an encoded bitstream is output from an outputter of encoder 100. Inaddition, decoder 200 may include an inputter and an outputter. Forexample, an encoded bitstream is input to an inputter of decoder 200, andone or more pictures are output from an outputter of decoder 200. Theencoded bitstream may include quantized coefficients to which variablelength encoding has been applied and control information.
[0696] In addition, in the above-described description, the multilayercoding scheme corresponds to a scheme for encoding and decoding the oneor more pictures using a multilayer structure. In other words, themultilayer coding scheme corresponds to a scheme for classifying each ofthe pictures into any one of layers and encoding and decoding the picture.In addition, although each of the scheme corresponding to encoding andthe scheme corresponding to decoding is represented as the multilayercoding scheme in the above-described description, it is to be noted thatthe multilayer coding scheme corres...
Claims
1. An encoder comprising: circuitry; and memory coupled to the circuitry, wherein, in operation, the circuitry: switches whether to encode a first flag, according to whether a multilayer coding scheme is used to encode one or more pictures, the first flag indicating whether use of one or more interlayer reference pictures for interlayer prediction is allowed, and wherein the circuitry encodes a second flag which indicates whether any of the one or more interlayer reference pictures is used as a reference picture that is included in a reference picture list, when the use of the one or more interlayer reference pictures is allowed.
2. The encoder according to claim 1, wherein a reference picture different from any of the one or more interlayer reference pictures is used as the reference picture that is included in the reference picture list, when (I-i) the second flag is encoded and (I-ii) the second flag indicates that any of the one or more interlayer reference pictures is not used as the reference picture that is included in the reference picture list, or (II) the second flag is not to be encoded.
3. A decoder comprising: circuitry; and memory coupled to the circuitry, wherein, in operation, the circuitry: switches whether to decode a first flag, according to whether a multilayer coding scheme is used to decode one or more pictures, the first flag indicating whether use of one or more interlayer reference pictures for interlayer prediction is allowed, and wherein the circuitry decodes a second flag which indicates whether any of the one or more interlayer reference pictures is used as a reference picture that is included in a reference picture list, when the use of the one or more interlayer reference pictures is allowed.
4. The decoder according to claim 3, wherein a reference picture different from any of the one or more interlayer reference pictures is used as the reference picture that is included in the reference picture list, when (I-i) the second flag is decoded and (I-ii) the second flag indicates that any of the one or more interlayer reference pictures is not used as the reference picture that is included in the reference picture list, or (II) the second flag is not to be decoded.
5. An apparatus comprising a non-transitory computer readable medium storing a bitstream, the bitstream comprising: information which corresponds to whether a multilayer coding scheme is used to decode one or more pictures, and according to which a decoder performs a decoding process; and in the decoding process, whether to decode a first flag indicating whether use of one or more interlayer reference pictures for interlayer prediction is allowed is switched according to the information, wherein, in the decoding process, a second flag which indicates whether any of the one or more interlayer reference pictures is used as a reference picture that is included in a reference picture list is decoded when the use of the one or more interlayer reference pictures is allowed.
6. The apparatus according to claim 5, wherein a reference picture different from any of the one or more interlayer reference pictures is used as the reference picture that is included in the reference picture list, when (I-i) the second flag is decoded and (I-ii) the second flag indicates that any of the one or more interlayer reference pictures is not used as the reference picture that is included in the reference picture list, or (II) the second flag is not to be decoded.
