Integrating generative neural networks in video encoding

Abstract

Various embodiments provide methods, apparatuses, and computer program products. An example apparatus includes at least one processor; and at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform: receiving a first input picture; encoding the first input picture into a first coded picture, wherein the first coded picture comprises a coded base picture or a coded driving picture, and wherein a first picture unit comprises the first coded picture; receiving a second input picture; extracting a feature parameter from the second input picture; encoding the feature parameter into an encoded feature parameter; and including the encoded feature parameter in the first picture unit or a subsequent picture unit, wherein the subsequent picture unit comprises a subsequent coded picture, wherein the subsequent coded picture comprises a second coded driving picture or a second coded base picture.

Description

Technical Field

[0001] The exemplary and non-limiting embodiments generally relate to multimedia coding, and more specifically to integrating generative neural networks in multimedia coding. Background Technology

[0002] Standardized formats are known to be provided for encoding, signaling, or decoding media data. Summary of the Invention

[0003] Example 1: An apparatus comprising at least one processor; and at least one non-transitory memory storing instructions which, when executed by the at least one processor, cause the apparatus to at least: encode suffix enhancement information units to generate an encoded picture, the suffix enhancement information units including a facial video enhancement information message in a picture unit, the picture unit including a base or driving picture referenced by the facial video enhancement information message; and signal the encoded picture.

[0004] Example 2: The apparatus according to Example 1, wherein the facial video enhancement information message includes a generative video supplementation enhancement information message.

[0005] Example 3. The apparatus according to any one of Examples 1 or 2, wherein the suffix enhancement information unit includes a suffix supplemental enhancement information network abstraction layer unit.

[0006] Example 4. An apparatus comprising at least one processor; and at least one non-transitory memory storing instructions, which, when executed by the at least one processor, cause the apparatus to at least: receive a first input image; encode the first input image into a first encoded image, wherein the first encoded image includes an encoded base image or an encoded driving image, and wherein a first image unit includes the first encoded image; receive a second input image; extract feature parameters from the second input image; encode the feature parameters into encoded feature parameters; and include the encoded feature parameters in a first image unit or a subsequent image unit, wherein the subsequent image unit includes a subsequent encoded image, wherein the subsequent encoded image includes a second encoded driving image or a second encoded base image.

[0007] Example 5. The apparatus according to Example 4, wherein the apparatus is further configured to include encoded feature parameters in a Generative Video (GV) Supplemental Enhancement Information (SEI) message, the GV SEI message being decoded after the first encoded image.

[0008] Example 6. The apparatus according to Example 5, wherein the apparatus is further configured to: include a generative video SEI message in a suffix SEI network abstraction layer (NAL) unit within a first picture unit.

[0009] Example 7. The apparatus according to any one of Examples 4 to 6, wherein when the encoded feature parameters are decoded, the encoded feature parameters invoke generative neural network inference.

[0010] Example 8. An apparatus according to any one of Examples 4 to 7, wherein the apparatus is further caused to perform: including activation signals to invoke generative neural network inference along encoded feature parameters.

[0011] Example 9. An apparatus according to any one of Examples 4 to 8, wherein the apparatus is further configured to: include an identifier or counter value in a generative video SEI message, wherein the identifier or counter value identifies the generative video SEI message from other generative video SEI messages in a first picture unit or a subsequent picture unit.

[0012] Example 10. The apparatus according to Example 4, wherein the apparatus is further configured to perform: generating a plurality of generative face video (GFV) supplemental enhancement information (SEI) messages with different payloads in a first picture unit or a subsequent picture unit, wherein each instance of the GFV SEI message invokes generative neural network (NN) inference.

[0013] Example 11. The apparatus according to Example 10, wherein the apparatus is caused to perform: using a suffix SEI NAL unit in a first picture unit that includes a GFV SEI message, the first picture unit including a base picture or driving picture referenced by the GFV SEI message.

[0014] Example 12. The apparatus according to Example 11, wherein the apparatus is caused to perform: adding a counter syntax element in a GFV SEI message.

[0015] Example 13. An apparatus comprising at least one processor; and at least one non-transitory memory storing instructions, which, when executed by the at least one processor, cause the apparatus to at least: receive a first encoded image from a first image unit, wherein the first encoded image includes an encoded base image or an encoded driving image; decode the first encoded image to generate a first decoded image; receive encoded feature parameters from the first image unit or from a subsequent image unit; create a feature input tensor based on the encoded feature parameters; and invoke generative neural network inference with the first decoded image and the feature input tensor as input to generate an image.

[0016] Example 14. The apparatus according to Example 13, wherein the apparatus is made to perform: accepting less than one image unit through an interface for being passed for decoding, and wherein the decoding of the first encoded image is invoked before receiving or creating the feature input tensor.

[0017] Example 15. The apparatus according to Example 13, wherein the apparatus is made to: accept less than one picture unit through an interface for being passed for decoding, and wherein the creation of a feature input tensor is performed before receiving or decoding a subsequent encoded picture unit.

[0018] Example 16. An apparatus according to any one of Examples 13 to 15, wherein the apparatus is further caused to perform: decoding the encoded feature parameters; and invoking generative neural network inference based on the decoding of the encoded feature parameters.

[0019] Example 17. An apparatus according to any one of Examples 13 to 16, wherein the apparatus is further configured to perform: decoding an identifier or counter value from a generative video SEI message, wherein the identifier or counter value identifies the generative video SEI message from other generative video SEI messages in a first picture unit or a subsequent picture unit.

[0020] Example 18. The apparatus according to Example 17, wherein when an identifier or counter value is the same as the identifier or counter value of a previous generative video SEI message in a first picture unit or a subsequent picture unit, the current generative video SEI message is inferred to be a copy of the previous generative video SEI message.

[0021] Example 19. The apparatus according to Example 13, wherein the apparatus is further configured to perform: decoding a plurality of generative face video (GFV) supplemental enhancement information (SEI) messages with different payloads from a first picture unit or a subsequent picture unit, wherein each instance of the GFV SEI message invokes generative neural network (NN) inference.

[0022] Example 20. The apparatus according to Example 19, wherein the apparatus is caused to perform: decoding a suffix SEI NAL unit, the suffix SEI NAL unit including a GFV SEI message from a first picture unit, the first picture unit including a base picture or driving picture referenced by the GFV SEI message.

[0023] Example 21. The apparatus according to Example 20, wherein the apparatus is caused to perform: decoding a counter syntax element from a GFV SEI message.

[0024] Example 22. An apparatus comprising at least one processor; and at least one non-transitory memory storing instructions, which, when executed by the at least one processor, cause the apparatus to at least: receive a first input image; encode the first input image into a first encoded image, wherein the first encoded image is an encoded base image, and wherein the first encoded image resides in a separate layer, and wherein a first access unit includes the first encoded image; receive a second input image; encode a second encoded image, wherein the second encoded image includes an encoded virtual image or an encoded driving image based on the second input image, and wherein the second encoded image resides in a dependency layer, and wherein a second image unit includes the second encoded image, and wherein a second access unit includes the second encoded image; extract feature parameters from the second input image; encode the feature parameters into encoded feature parameters; and include the encoded feature parameters in the second image unit.

[0025] Example 23. The apparatus according to Example 22, wherein the apparatus is further configured to: encode an encoded copy image in a first access unit, wherein the encoded copy image resides in a dependency layer and is predicted according to an independent layer, and wherein the encoded copy image does not include prediction residuals.

[0026] Example 24. The apparatus according to Example 22 or 23, wherein the apparatus is further configured to: include encoded feature parameters in a generative video supplementation enhancement information (SEI) message in a second picture unit.

[0027] Example 25. An apparatus according to any one of Examples 22 to 24, wherein the apparatus is further configured to perform: predict an encoded virtual image based on previous images in decoding order within a dependency layer.

[0028] Example 26. An apparatus comprising at least one processor; and at least one non-transitory memory storing instructions, which, when executed by the at least one processor, cause the apparatus to at least: receive a first input image; encode the first input image into a first encoded base layer image, wherein the first encoded base layer image is an encoded base image, and wherein the first encoded base layer image resides in an independent layer, and wherein a first access unit includes the first encoded base layer image; encode a first encoded dependency layer image, wherein the first encoded dependency layer image resides in a dependency layer, and wherein the first access unit includes the first encoded dependency layer image; receive a second input image; encode a second encoded base layer image, wherein the second encoded base layer image includes an encoded virtual... The second input image is used to generate a simulated image, or an encoded driving image based on a second input image, wherein the second encoded base layer image resides in an independent layer, and wherein the second access unit includes the second encoded base layer image; the second encoded dependency layer image is encoded, wherein the second encoded dependency layer image resides in a dependency layer, and the second access unit includes the second encoded dependency layer image; the second encoded dependency layer image is instructed to use the first encoded dependency layer image and the second encoded base layer image as references for prediction in or along the second encoded dependency layer image; feature parameters are extracted from the second input image; the feature parameters are encoded into encoded feature parameters; and the encoded feature parameters are indicated to be used for decoding the second encoded dependency layer image.

[0029] Example 27. The apparatus according to Example 26, wherein the apparatus is further configured to: encode a first dependent layer image into an encoded copy image, wherein the encoded copy image is predicted according to an independent layer, and wherein the encoded copy image has no prediction residual.

[0030] Example 28. An apparatus according to any one of Examples 26 or 27, wherein the apparatus is further configured to: encode an indication in or along the second encoded base layer image to omit the output of a reconstructed image decoded from the second encoded base layer image.

[0031] Example 29. An apparatus according to any one of Examples 26 or 27, wherein the apparatus is further caused to perform: instructing a first reconstructed base layer image decoded from a first encoded base layer image to be output; instructing a first reconstructed dependent layer image decoded from a first encoded dependent layer image to be output; instructing a second reconstructed dependent layer image decoded from a second encoded dependent layer image to be output; instructing independent layers and dependent layers to form an output layer set, wherein the dependent layers are output layers of the output layer set; or instructing a profile to the output layer set, wherein the profile includes generative neural network capabilities.

[0032] Example 30. An apparatus according to any one of Examples 26 to 28, wherein the apparatus is further caused to perform: creating a feature input tensor based on encoded feature parameters.

[0033] Example 31. An apparatus according to any one of Examples 26 to 30, wherein the apparatus is further configured to: include encoded feature parameters or feature input tensors in an adaptive parameter set having an indication of the type of generative video; or include encoded feature parameters or feature input tensors in the image header of a second encoded dependent layer image.

[0034] Example 32. An apparatus comprising at least one processor; and at least one non-transitory memory storing instructions which, when executed by the at least one processor, cause the apparatus to at least: reconstruct a first decoded dependency layer image from a first encoded dependency layer image; and reconstruct a second decoded dependency layer image by invoking generative neural network inference to generate an image with the first decoded dependency layer image and a feature input tensor as input.

[0035] Example 33. The apparatus according to Example 32, wherein the grammatical exclusion of the second encoded dependency layer image is irrelevant to the grammatical elements of the second decoded dependency layer image reconstructed using a generative neural network.

[0036] Example 34. According to the apparatus described in Example 32, grammatical exclusion of one or more grammatical structures of the second dependency layer and reconstruction of grammatical elements of the second decoded dependency layer image using a generative neural network are irrelevant.

[0037] Example 35. An apparatus comprising at least one processor; and at least one non-transitory memory storing instructions which, when executed by the at least one processor, cause the apparatus to at least: reconstruct a first decoded dependency layer image from a first encoded dependency layer image; reconstruct one or more prediction units of a second decoded dependency layer image by invoking generative neural network inference with the first decoded dependency layer image and a feature input tensor as input; and encode the difference between the one or more prediction units and co-located units of a second input image as a prediction residual of the second encoded dependency layer image.

[0038] Example 36. An apparatus comprising at least one processor; and at least one non-transitory memory storing instructions which, when executed by the at least one processor, cause the apparatus to at least: reconstruct a first decoded dependency layer image from a first encoded dependency layer image; invoke generative neural network inference with the first decoded dependency layer image and a feature input tensor as input to generate an inter-layer reference image; and encode a second encoded dependency layer image with reference to the inter-layer reference image.

[0039] Example 37. The apparatus according to Example 36, wherein the apparatus is further configured to: encode the difference between the second input image and the interlayer reference image as a prediction residual of the second encoded dependent layer image.

[0040] Example 38. An apparatus comprising at least one processor; and at least one non-transitory memory storing instructions, which, when executed by the at least one processor, cause the apparatus to at least: receive a first encoded base layer image, wherein the first encoded base layer image is an encoded base image residing in an independent layer, and a first access unit includes the first encoded base layer image; decode the first encoded base layer image into a first decoded base layer image; receive a first encoded dependency layer image, wherein the first encoded dependency layer image resides in a dependency layer, and a first access unit includes the first encoded dependency layer image; decode the first encoded dependency layer image into a first decoded dependency layer image; receive a second encoded base layer image, wherein the second encoded base layer image may be an encoded virtual image or an encoded driving image based on a second input image, the second... An encoded base layer image resides in an independent layer, and a second access unit includes a second encoded base layer image; in response to the second encoded base layer image being an encoded driving image, the second encoded base layer image is decoded into a second decoded base layer image; a second encoded dependency layer image is received, wherein the second encoded dependency layer image resides in a dependency layer, and a second access unit includes the second encoded dependency layer image; from or along the second encoded dependency layer image, it is decoded that the second encoded dependency layer image can use the first encoded dependency layer image and the second encoded base layer image as references for prediction; a feature input tensor for decoding the second encoded dependency layer image is obtained; and the second decoded dependency layer image is reconstructed by invoking generative neural network inference with the first decoded dependency layer image and the feature input tensor as input to generate an image.

[0041] Example 39. An apparatus comprising at least one processor; and at least one non-transitory memory storing instructions, which, when executed by the at least one processor, cause the apparatus to at least: receive a first input image; encode the first input image into a first encoded image, wherein the first encoded image is an encoded base image or an encoded driving image, and wherein the first encoded image resides in an independent layer, and wherein a first access unit includes the first encoded image; receive a second input image; extract feature parameters from the second input image; encode the feature parameters into encoded feature parameters; encode a second encoded image, wherein the second encoded image resides in a dependency layer, and wherein a second access unit includes the second encoded image; and instruct the encoded feature parameters to be used to decode the second encoded dependency layer image.

[0042] Example 40. The apparatus according to Example 39, wherein the apparatus is further configured to perform: reconstructing a first decoded image from a first encoded image; creating a feature input tensor based on encoded feature parameters; invoking generative neural network inference with the first decoded image and the feature input tensor as input to generate a first interlayer reference image; and encoding a second encoded image with reference to the first interlayer reference image.

[0043] Example 41. The apparatus according to Example 39 or 40, wherein the apparatus is further configured to perform: omit the encoding of the prediction residual for the second encoded image.

[0044] Example 42. An apparatus according to any one of Examples 39 to 41, wherein the apparatus is further configured to: encode the difference between the second input image and the first interlayer reference image as a prediction residual of the second encoded image.

[0045] Example 43. An apparatus according to any one of Examples 39 to 41, wherein the apparatus is further configured to: indicate that the first encoded image is an output in response to the first encoded image being an encoded base image; indicate that the first encoded image is not an output in response to the first encoded image being an encoded driving image; indicate that the independent layer and the dependent layer form an output layer set, and the dependent layer is an output layer of the output layer set; or indicate that the second encoded image is output.

[0046] Example 44. An apparatus according to any one of Examples 39 to 43, wherein the apparatus is further caused to perform: indicating in or along the first encoded image that the first encoded image is an encoded base image or the first encoded image is an encoded driving image; or indicating with additional bits in the image header that the first encoded image is an encoded base image or the first encoded image is an encoded driving image.

[0047] Example 45. An apparatus according to any one of Examples 39 to 44, wherein the apparatus is further caused to perform: instructing a second encoded image to use interlayer prediction from independent layers.

[0048] Example 46. The apparatus according to any one of Examples 39 to 44, wherein the apparatus is further caused to perform: inferring an interlayer prediction reference from an independent layer to a first interlayer reference picture.

[0049] Example 47. An apparatus according to any one of Examples 39 to 44, wherein the apparatus is further configured to perform: receiving a third input image; and encoding the third input image into a third encoded image, wherein the third encoded image is an encoded driving image, and wherein the third encoded image resides in an independent layer, and the third encoded image is after the first encoded image in decoding order and before the second encoded image in decoding order.

[0050] Example 48. The apparatus according to Example 47, wherein the apparatus is further configured to perform: in response to the third encoded image residing in the second access unit, omitting an indication that the third encoded image is an encoded driving image, and inferring that the third encoded image is an encoded driving image.

[0051] Example 49. The apparatus according to Example 47, wherein the apparatus is further configured to: indicate in or along the third encoded image a directive indicating that the third encoded image is an encoded driving image.

[0052] Example 50. The apparatus according to Example 47, wherein the apparatus is further configured to: indicate using additional bits in the image header that the third encoded image is an encoded driving image.

[0053] Example 51. An apparatus according to any one of Examples 47 to 50, wherein the apparatus is further caused to perform: reconstruct a third decoded image from a third encoded image; and use the third decoded image as additional input in a call to generative neural network inference.

[0054] Example 52. An apparatus according to any one of Examples 47 to 51, wherein the apparatus is further configured to perform: inferring when an interlayer prediction from an independent layer references a first interlayer reference image or a third decoded image.

[0055] Example 53. An apparatus according to any one of Examples 47 to 51, wherein the apparatus is further configured to: indicate syntax structure information in a list of reference images, the syntax structure information indicating that a first encoded image is used as input for generating an interlayer reference image.

[0056] Example 54. The apparatus according to Example 53, wherein the information indicating that the first encoded image is used as input for generating an interlayer reference image includes the image order count difference between the current image and the first encoded image.

[0057] Example 55. A method comprising: encoding a suffix enhancement information unit to generate an encoded picture, the suffix enhancement information unit including a facial video enhancement information message in a picture unit, the picture unit including a base or driving picture referenced by the facial video enhancement information message; and signaling the encoded picture.

[0058] Example 56. The method according to Example 55, wherein the facial video enhancement information message includes a generative video supplementation enhancement information message.

[0059] Example 57. The method according to any one of Examples 55 or 56, wherein the suffix enhancement information unit includes a suffix supplemental enhancement information network abstraction layer unit.

[0060] Example 58. A method comprising: receiving a first input image; encoding the first input image into a first encoded image, wherein the first encoded image includes an encoded base image or an encoded driving image, and wherein a first image unit includes the first encoded image; receiving a second input image; extracting feature parameters from the second input image; encoding the feature parameters into encoded feature parameters; and including the encoded feature parameters in a first image unit or a subsequent image unit, wherein the subsequent image unit includes a subsequent encoded image, wherein the subsequent encoded image includes a second encoded driving image or a second encoded base image.

[0061] Example 59. The method according to Example 58 further includes: including encoded feature parameters in a generative video (GV) supplemental enhancement information (SEI) message, the GV SEI message being decoded after the first encoded image.

[0062] Example 60. The method according to Example 59 further includes: including the generative video SEI message in a suffix SEI Network Abstraction Layer (NAL) unit within the first picture unit.

[0063] Example 61. The method according to any one of Examples 58 to 60, wherein when the encoded feature parameters are decoded, the encoded feature parameters invoke generative neural network inference.

[0064] Example 62. The method according to any one of Examples 58 to 61 further includes: including an activation signal to invoke generative neural network inference along encoded feature parameters.

[0065] Example 63. The method according to any one of Examples 58 to 62 further includes: including an identifier or counter value in the generative video SEI message, wherein the identifier or counter value identifies the generative video SEI message from other generative video SEI messages in the first picture unit or subsequent picture units.

[0066] Example 64. The method according to Example 58 further includes: generating multiple generative face video (GFV) supplemental enhancement information (SEI) messages with different payloads in a first picture unit or a subsequent picture unit, wherein each instance of the GFV SEI message invokes generative neural network (NN) inference.

[0067] Example 65. The method according to Example 64 further includes: using a suffix SEI NAL unit in a first picture unit that includes a GFV SEI message, the first picture unit including a base picture or driving picture referenced by the GFV SEI message.

[0068] Example 66. The method described in Example 65 further includes adding a counter syntax element to the GFV SEI message.

[0069] Example 67. A method comprising: receiving a first encoded image from a first image unit, wherein the first encoded image includes an encoded base image or an encoded driving image; decoding the first encoded image to generate a first decoded image; receiving encoded feature parameters from the first image unit or from a subsequent image unit; creating a feature input tensor based on the encoded feature parameters; and invoking a generative neural network inference with the first decoded image and the feature input tensor as input to generate the image.

[0070] Example 68. The method according to Example 67 further includes: accepting less than one image unit via an interface to be passed for decoding, and wherein the decoding of the first encoded image is invoked before receiving or creating the feature input tensor.

[0071] Example 69. According to the method described in Example 67, less than one picture unit is accepted through an interface to be passed for decoding, and wherein the creation of a feature input tensor is performed before receiving or decoding subsequent encoded pictures from subsequent picture units.

[0072] Example 70. The method according to any one of Examples 67 to 69 further includes: decoding the encoded feature parameters; and invoking generative neural network inference based on the decoding of the encoded feature parameters.

[0073] Example 71. The method according to any one of Examples 67 to 70 further includes: decoding an identifier or counter value from a generative video SEI message, wherein the identifier or counter value identifies the generative video SEI message from other generative video SEI messages in a first picture unit or a subsequent picture unit.

[0074] Example 72. According to the method of Example 71, wherein when the identifier or counter value is the same as the identifier or counter value of a previous generative video SEI message in a first picture unit or a subsequent picture unit, the current generative video SEI message is inferred to be a copy of the previous generative video SEI message.

[0075] Example 73. The method according to Example 67 further includes: decoding multiple generative face video (GFV) supplemental enhancement information (SEI) messages with different payloads from a first picture unit or a subsequent picture unit, wherein each instance of the GFV SEI message invokes generative neural network (NN) inference.

[0076] Example 74. The method according to Example 73 further includes: decoding a suffix SEI NAL unit, the suffix SEINAL unit including a GFV SEI message from a first picture unit, the first picture unit including a base picture or driving picture referenced by the GFV SEI message.

[0077] Example 75. The method described in Example 74 further includes: decoding the counter syntax element from the GFV SEI message.

[0078] Example 76. A method comprising: receiving a first input image; encoding the first input image into a first encoded image, wherein the first encoded image is an encoded base image, and wherein the first encoded image resides in a separate layer, and wherein a first access unit includes the first encoded image; receiving a second input image; encoding a second encoded image, wherein the second encoded image includes an encoded virtual image or an encoded driving image based on the second input image, and wherein the second encoded image resides in a dependency layer, and wherein a second image unit includes the second encoded image, and wherein a second access unit includes the second encoded image; extracting feature parameters from the second input image; encoding the feature parameters into encoded feature parameters; and including the encoded feature parameters in the second image unit.

[0079] Example 77. The method according to Example 76 further includes: encoding the encoded copy image in a first access unit, wherein the encoded copy image resides in a dependency layer and is predicted according to an independent layer, and wherein the encoded copy image does not include prediction residuals.

[0080] Example 78. The method according to any one of Examples 76 or 77 further includes: including the encoded feature parameters in a generative video supplementation enhancement information (SEI) message in the second picture unit.

[0081] Example 79. The method according to any one of Examples 76 to 78 further includes: predicting the encoded virtual image based on previous images in decoding order within the dependency layer.

[0082] Example 80. A method comprising: receiving a first input image; encoding the first input image into a first encoded base layer image, wherein the first encoded base layer image is an encoded base image, and wherein the first encoded base layer image resides in an independent layer, and wherein a first access unit includes the first encoded base layer image; encoding a first encoded dependency layer image, wherein the first encoded dependency layer image resides in a dependency layer, and wherein the first access unit includes the first encoded dependency layer image; receiving a second input image; encoding a second encoded base layer image, wherein the second encoded base layer image includes an encoded virtual image or an encoded driving image based on the second input image, and wherein... The second encoded base layer image resides in an independent layer, and the second access unit includes the second encoded base layer image; the second encoded dependency layer image is encoded, wherein the second encoded dependency layer image resides in a dependency layer, and the second access unit includes the second encoded dependency layer image; the second encoded dependency layer image is instructed, either in or along the second encoded dependency layer image, to use the first encoded dependency layer image and the second encoded base layer image as references for prediction; feature parameters are extracted from the second input image; the feature parameters are encoded into encoded feature parameters; and the encoded feature parameters are indicated to be used for decoding the second encoded dependency layer image.

[0083] Example 81. The method according to Example 80 further includes: encoding a first dependency layer image into an encoded copy image, wherein the encoded copy image is predicted according to an independent layer, and wherein the encoded copy image has no prediction residual.

[0084] Example 82. The method according to any one of Examples 80 or 81 further includes: encoding an indication in or along the second encoded base layer image to omit the output of the reconstructed image decoded from the second encoded base layer image.

[0085] Example 83. The method according to any one of Examples 80 or 81 further includes: instructing a first reconstructed base layer image decoded from a first encoded base layer image to be output; instructing a first reconstructed dependent layer image decoded from a first encoded dependent layer image to be output; instructing a second reconstructed dependent layer image decoded from a second encoded dependent layer image to be output; instructing independent layers and dependent layers to form an output layer set, wherein the dependent layers are output layers of the output layer set; or instructing a profile to the output layer set, wherein the profile includes generative neural network capabilities.

[0086] Example 84. The method according to any one of Examples 80 to 82 further includes: creating a feature input tensor based on encoded feature parameters.

[0087] Example 85. The method according to any one of Examples 80 to 84 further includes: including the encoded feature parameters or feature input tensors in an adaptive parameter set having an indication of the type of generative video; or including the encoded feature parameters or feature input tensors in the image header of the second encoded dependent layer image.

[0088] Example 86. A method comprising: reconstructing a first decoded dependency layer image from a first encoded dependency layer image; and reconstructing a second decoded dependency layer image by invoking generative neural network inference to generate an image with the first decoded dependency layer image and a feature input tensor as input.

[0089] Example 87. The method described in Example 86, wherein the grammatical exclusion of the second encoded dependency layer image is irrelevant to the grammatical elements of the second decoded dependency layer image reconstructed using a generative neural network.

[0090] Example 88. According to the method described in Example 86, grammatical exclusion of one or more grammatical structures of the second dependency layer and reconstruction of irrelevant grammatical elements of the second decoded dependency layer image using a generative neural network.

[0091] Example 89. A method comprising: reconstructing a first decoded dependency layer image from a first encoded dependency layer image; reconstructing one or more prediction units of a second decoded dependency layer image by invoking generative neural network inference with the first decoded dependency layer image and a feature input tensor as input; and encoding the difference between the one or more prediction units and co-located units of a second input image as a prediction residual of the second encoded dependency layer image.

[0092] Example 90. A method comprising: reconstructing a first decoded dependency layer image from a first encoded dependency layer image; invoking generative neural network inference with the first decoded dependency layer image and a feature input tensor as input to generate an inter-layer reference image; and encoding a second encoded dependency layer image with reference to the inter-layer reference image.

[0093] Example 91. The method according to Example 90 further includes: encoding the difference between the second input image and the interlayer reference image as a prediction residual of the second encoded dependent layer image.

[0094] Example 92. A method comprising: receiving a first encoded base layer image, wherein the first encoded base layer image is an encoded base image residing in an independent layer, and a first access unit including the first encoded base layer image; decoding the first encoded base layer image into a first decoded base layer image; receiving a first encoded dependency layer image, wherein the first encoded dependency layer image residing in a dependency layer, and a first access unit including the first encoded dependency layer image; decoding the first encoded dependency layer image into a first decoded dependency layer image; receiving a second encoded base layer image, wherein the second encoded base layer image may be an encoded virtual image or an encoded driving image based on a second input image, the second encoded base layer image residing in an independent layer, and a second access unit including... A second encoded base layer image; in response that the second encoded base layer image is an encoded driving image, the second encoded base layer image is decoded into a second decoded base layer image; a second encoded dependency layer image is received, wherein the second encoded dependency layer image resides in a dependency layer, and a second access unit includes the second encoded dependency layer image; from or along the second encoded dependency layer image, it is decoded that the second encoded dependency layer image can use the first encoded dependency layer image and the second encoded base layer image as references for prediction; a feature input tensor for decoding the second encoded dependency layer image is obtained; and the second decoded dependency layer image is reconstructed by invoking generative neural network inference with the first decoded dependency layer image and the feature input tensor as input to generate an image.

[0095] Example 93. A method comprising: receiving a first input image; encoding the first input image into a first encoded image, wherein the first encoded image is an encoded base image or an encoded driving image, and wherein the first encoded image resides in a separate layer, and wherein a first access unit includes the first encoded image; receiving a second input image; extracting feature parameters from the second input image; encoding the feature parameters into encoded feature parameters; encoding a second encoded image, wherein the second encoded image resides in a dependency layer, and wherein a second access unit includes the second encoded image; and instructing that the encoded feature parameters are used to decode the second encoded dependency layer image.

[0096] Example 94. The method according to Example 93 further includes: reconstructing a first decoded image from a first encoded image; creating a feature input tensor based on encoded feature parameters; invoking generative neural network inference with the first decoded image and the feature input tensor as input to generate a first interlayer reference image; and encoding a second encoded image with reference to the first interlayer reference image.

[0097] Example 95. The method according to any one of Examples 93 or 94 further includes: omitting the encoding of the prediction residual for the second encoded image.

[0098] Example 96. The method according to any one of Examples 93 to 95 further includes: encoding the difference between the second input image and the first inter-layer reference image as a prediction residual of the second encoded image.

[0099] Example 97. The method according to any one of Examples 93 to 95 further includes: indicating that the first encoded image is an output in response to the first encoded image being an encoded base image; indicating that the first encoded image is not an output in response to the first encoded image being an encoded driving image; indicating that the independent layer and the dependent layer form an output layer set, and the dependent layer is an output layer of the output layer set; or indicating that the second encoded image is output.

[0100] Example 98. The method according to any one of Examples 93 to 97 further includes: indicating in or along the first encoded image that the first encoded image is an encoded base image or the first encoded image is an encoded driving image; or indicating with additional bits in the image header that the first encoded image is an encoded base image or the first encoded image is an encoded driving image.

[0101] Example 99. The method according to any one of Examples 93 to 98 further includes: instructing the second encoded image to use inter-layer predictions from independent layers.

[0102] Example 100. The method according to any one of Examples 93 to 98 further includes: inferring an interlayer prediction reference from an independent layer to a first interlayer reference picture.

[0103] Example 101. The method according to any one of Examples 93 to 98 further includes: receiving a third input image; and encoding the third input image into a third encoded image, wherein the third encoded image is an encoded driving image, and wherein the third encoded image resides in an independent layer, and the third encoded image is after the first encoded image in decoding order and before the second encoded image in decoding order.

[0104] Example 102. The method according to Example 101 further includes: in response to the third encoded image residing in the second access unit, omitting the indication that the third encoded image is an encoded driving image, and inferring that the third encoded image is an encoded driving image.

[0105] Example 103. The method according to Example 101 further includes: indicating, in or along the third encoded image, an indication that the third encoded image is an encoded driving image.

[0106] Example 104. The method according to Example 101 further includes: using additional bits in the image header to indicate that the third encoded image is an encoded driving image.

[0107] Example 105. The method according to any one of Examples 101 to 104 further includes: reconstructing a third decoded image from a third encoded image; and using the third decoded image as additional input in a call to generative neural network inference.

[0108] Example 106. The method according to any one of Examples 101 to 105 further includes: inferring when an interlayer prediction from an independent layer references a first interlayer reference image or a third decoded image.

[0109] Example 107. The method according to any one of Examples 101 to 105 further includes: indicating syntax structure information in a list of reference images, the syntax structure information indicating that a first encoded image is used as input for generating interlayer reference images.

[0110] Example 108. The method according to Example 107, wherein the information indicating that the first encoded image is used as input for generating an interlayer reference image includes the image order count difference between the current image and the first encoded image.

[0111] Example 109. An apparatus comprising components for performing the method according to any one of Examples 55 to 108.

[0112] Example 110. A computer-readable medium comprising program instructions that, when executed by a device, cause the device to perform the method according to any one of Examples 55 to 108.

[0113] Example 111. A computer-readable medium according to Example 110, wherein the computer-readable medium includes a non-transitory computer-readable medium. Attached Figure Description

[0114] The above embodiments and other features are explained in the following description taken in conjunction with the accompanying drawings, in which:

[0115] Figure 1 An electronic device employing an embodiment of the examples described herein is illustrated schematically.

[0116] Figure 2 A user equipment suitable for employing an embodiment of the examples described herein is illustrated schematically.

[0117] Figure 3An electronic device employing an embodiment of the examples described herein is further illustrated schematically using wireless and wired network connections.

[0118] Figure 4 A block diagram showing the general structure of a video encoder is provided.

[0119] Figure 5 The illustration shows the image unit.

[0120] Figure 6 The diagram illustrates a general block diagram of a generative facial video codec.

[0121] Figure 7 The illustration shows an image unit generated by the encoder.

[0122] Figure 8 The illustration shows a bitstream generated by an encoder according to an embodiment.

[0123] Figure 9 The illustration shows a bitstream generated by an encoder according to another embodiment.

[0124] Figure 10 An example bitstream generated by an encoder according to yet another embodiment is illustrated.

[0125] Figure 11 This is an example device that can be implemented in hardware and is made to implement the examples described herein.

[0126] Figure 12 A representation of an example of a non-volatile storage medium for storing instructions that implement the examples described herein is shown.

[0127] Figure 13 These are example methods for implementing the embodiments described herein, based on the embodiments.

[0128] Figure 14 This is another example method for implementing the embodiments described herein, according to the embodiments.

[0129] Figure 15 This is yet another example method for implementing the embodiments described herein, according to the embodiments.

[0130] Figure 16 This is yet another example method for implementing the embodiments described herein, according to the embodiments.

[0131] Figure 17 This is yet another example method for implementing the embodiments described herein, according to the embodiments.

[0132] Figure 18 This is yet another example method for implementing the embodiments described herein, according to the embodiments.

[0133] Figure 19 This is yet another example method for implementing the embodiments described herein, according to the embodiments.

[0134] Figure 20 This is yet another example method for implementing the embodiments described herein, according to the embodiments.

[0135] Figure 21 This is yet another example method for implementing the embodiments described herein, according to the embodiments.

[0136] Figure 22 This is yet another example method for implementing the embodiments described herein, according to the embodiments. Detailed Implementation

[0137] The following abbreviations and acronyms that may appear in the instruction manual and / or drawings are defined as follows: 4CC: Four-character code 5G: Fifth Generation Cellular Network Technology 5GC: 5G Core Network aka: also known as AVC: Advanced Video Coding CU: Central Unit DSP: Digital Signal Processor DU: Distributed Unit eNB (or eNodeB): Evolved Node B (e.g., LTE base station) EN-DC: E-UTRA-NR Dual Connection en-gNB or En-gNB: A node that provides NR user plane and control plane protocol termination to the UE and acts as a secondary node in the EN-DC. E-UTRA: Evolved Universal Terrestrial Radio Access, such as LTE radio access technology F1 or F1-C: Interface between CU and DU control interfaces gNB (or gNodeB): A 5G / NR base station, for example, a node that provides NR user plane and control plane protocol termination to the UE and connects to the 5GC via the NG interface. IEC: International Electrotechnical Commission IoT: Internet of Things ISO: International Organization for Standardization ISOBMFF: ISO Basic Media File Format JPEG: Joint Photo Experts Group LTE: Long Term Evolution mdat:MediaDataBox MIME: Multipurpose Internet Mail Extensions MME: Mobility Management Entity moov: MovieBox MP4: The file format of MPEG-4 Part 14 files. MPEG: Moving Pictures Experts Group MPEG-2: H.222 / H.262 as defined by the ITU. MPEG-4: The audio and video coding standard of ISO / IEC 14496 ng or NG: the new generation ng-eNB or NG-eNB: Next-generation eNB NR: New Wireless (5G Wireless) N / W or NW: Network PDCP: Packet Data Convergence Protocol PHY: Physical Layer PNG: Portable Network Graphics RAN: Radio Access Network RFC: Comment Request RLC: Radio Link Control RRC: Radio Resource Control RRH: Remote Wireless Header RU: Wireless Unit Rx: Receiver SDAP: Service Data Adaptation Protocol SGW: Service Gateway SMF: Session Management Function SPS: Sequence Parameter Set SVC: Scalable Video Codec S1: Interface between eNodeB and EPC trak: TrackBox Tx: Transmitter UE: User Equipment UICC: General Purpose Integrated Circuit Card UPF: User-Face Functionality URL: Uniform Resource Locator X2: Interconnection interface between two eNodeBs in an LTE network Xn: Interface between two NG-RAN nodes

[0138] Some embodiments will now be described more fully with reference to the accompanying drawings, which illustrate some, but not all, of the embodiments. In fact, various embodiments of the invention may be embodied in many different forms and should not be construed as limited to the embodiments described herein; rather, these embodiments are provided so that this disclosure will meet applicable legal requirements. The same reference numerals throughout refer to the same elements. As used herein, the terms 'data,' 'content,' 'information,' and similar terms are used interchangeably to refer to data that can be transmitted, received, and / or stored according to embodiments of the invention. Therefore, the use of any such terms should not be considered as limiting the spirit and scope of the embodiments.

[0139] Furthermore, as used herein, the term 'circuit system' refers to (a) a purely hardware circuit implementation (e.g., an implementation in an analog circuit and / or digital circuit system); (b) a combination of circuitry and (multiple) computer program products, including software and / or firmware instructions stored on one or more computer-readable storage media, which cooperate to cause a device to perform one or more functions described herein; and (c) a circuit, such as (multiple) microprocessors or a portion thereof, which requires software or firmware to function even if the software or firmware is not physically present. This definition of 'circuit system' applies to all uses of the term herein, including in any claim. As another example, as used herein, the term 'circuit system' also includes an implementation comprising one or more processors and / or portions thereof, along with accompanying software and / or firmware. As yet another example, for instance, the term 'circuit system' as used herein also includes, for example, baseband integrated circuits or application processor integrated circuits for mobile phones, or similar integrated circuits in servers, cellular network devices, other network devices, and / or other computing devices.

[0140] As defined herein, a 'computer-readable storage medium' (which refers to a non-transitory physical storage medium (e.g., a volatile or non-volatile storage device)) can be distinguished from a 'computer-readable transmission medium' (which refers to an electromagnetic signal).

[0141] According to an example embodiment, a method, apparatus, and computer program product for integrating generative neural networks in multimedia coding are provided.

[0142] In one example, suitable apparatus and possible mechanisms for integrating generative neural networks in multimedia coding are described in detail below. In this regard, reference is made first. Figure 1 and Figure 2 ,in Figure 1An example block diagram of device 50 is shown. This device can be an Internet of Things (IoT) device configured to perform various functions, such as collecting, receiving, or transmitting information via one or more sensors, analyzing information collected or received by the device, and so on. The device may include a video encoding system that can incorporate a codec. Figure 2 The layout of the device according to an example embodiment is shown. The following will explain... Figure 1 and Figure 2 Components.

[0143] Device 50 may be, for example, a mobile terminal or user equipment of a wireless communication system, a sensor device, a tag, or a low-power device. However, it should be understood that the exemplary embodiments described herein can be implemented in any electronic device or apparatus that can process data via a neural network.

[0144] Device 50 may include a housing 30 for accommodating and protecting the device. Device 50 may also include a display 32, such as a liquid crystal display, a light-emitting diode display, an organic light-emitting diode display, or the like. In other embodiments of the examples described herein, the display may be any suitable display technology suitable for displaying media or multimedia content (e.g., images or videos). Device 50 may also include a keypad 34. In other embodiments of the examples described herein, any suitable data or user interface mechanism may be employed. For example, the user interface may be implemented as a virtual keyboard or data input system as part of a touch-sensitive display.

[0145] The device may include a microphone 36 or any suitable audio input, which may be a digital or analog signal input. The device 50 may also include an audio output device, which in the example embodiments described herein may be a headset 38, a speaker, or any analog or digital audio output connection. The device 50 may also include a battery (or, in other example embodiments described herein, the device may be powered by any suitable mobile energy device, such as a solar cell, fuel cell, or clockwork generator). The device may also include a camera 42 capable of recording or capturing images and / or video. The device 50 may also include an infrared port for short-range line-of-sight communication with other devices. In other embodiments, the device 50 may also include any suitable short-range communication technology, such as Bluetooth wireless connectivity or USB / FireWire wired connectivity.

[0146] The device 50 may include a controller 56, a processor, or a processor circuitry for controlling the device 50. The controller 56 may be connected to a memory 58, which, in the example embodiments described herein, may store data in the form of image, audio, and video data, and / or may also store instructions for implementation on the controller 56. The controller 56 may also be connected to a codec circuitry 54 adapted to perform encoding and / or decoding of audio, image, and / or video data, or to assist in encoding and / or decoding performed by the controller.

[0147] The device 50 may also include a card reader 48 and a smart card 46 for providing user information and suitable for providing authentication information for authenticating and authorizing the user at the network, such as a Universal Integrated Circuit Card (UICC) and a UICC reader.

[0148] Device 50 may include a wireless interface circuitry system 52 connected to a controller and adapted to generate wireless communication signals, such as communication with a cellular communication network, a wireless communication system, or a wireless local area network. Device 50 may also include an antenna 44 connected to the wireless interface circuitry system 52 for transmitting radio frequency signals generated at the wireless interface circuitry system 52 to and / or receiving radio frequency signals from the other devices.

[0149] The device 50 may include a camera 42 capable of recording or detecting individual frames, which are then passed to a codec circuitry system 54 or a controller for processing. The device may receive video image data for processing from another device before transmission and / or storage. The device 50 may also receive images wirelessly or via a wired connection for encoding / decoding. The structural elements of the device 50 described above represent examples of components used to perform the corresponding functions.

[0150] about Figure 3 This document illustrates an example of a system in which embodiments of the examples described herein can be utilized. System 10 includes multiple communication devices that can communicate over one or more networks. System 10 may include any combination of wired or wireless networks, including but not limited to wireless cellular telephone networks (such as GSM, UMTS, CDMA, LTE, 4G, 5G networks, etc.), wireless local area networks (WLANs) such as those defined by the IEEE 802.x standard, Bluetooth personal area networks, Ethernet LANs, token ring LANs, wide area networks, and the Internet.

[0151] System 10 may include wired and wireless communication devices and / or apparatus 50 suitable for implementing embodiments of the examples described herein.

[0152] For example, Figure 3The system shown illustrates a representation of mobile phone network 11 and Internet 28. Connection to Internet 28 can include, but is not limited to, long-range wireless connections, short-range wireless connections, and various wired connections, including but not limited to telephone lines, power lines, and similar communication paths.

[0153] The example communication device shown in System 10 may include, but is not limited to: electronic device or apparatus 50, a combination of personal digital assistant (PDA) and mobile phone 14, PDA 16, integrated messaging device (IMD) 18, desktop computer 20, laptop computer 22, or head-mounted device 21 (which may be a head-mounted display (HMD), or glasses with a camera or other device for processing images and / or video). Apparatus 50 may be stationary or mobile (when carried by a moving individual). Apparatus 50 may also be located in a vehicle, including but not limited to cars, trucks, taxis, buses, trains, ships, airplanes, bicycles, motorcycles, or any similar suitable means of transport.

[0154] Implementations may also be: implemented in a set-top box (e.g., a digital television receiver) which may or may not have a display or wireless capability; implemented in a tablet or (laptop) personal computer (PC) with hardware and / or software for processing neural network data; implemented in various operating systems; and implemented in chipsets, processors, DSPs, and / or embedded systems that provide hardware / software-based coding.

[0155] Some or additional devices can send and receive calls and messages, and communicate with service providers via a wireless connection 25 to base station 24. Base station 24 can be connected to a network server 26, which allows communication between mobile phone network 11 and the Internet 28. The system may include additional communication equipment and various types of communication devices.

[0156] Communication devices can communicate using various transmission technologies, including but not limited to Code Division Multiple Access (CDMA), Global System for Mobile Communications (GSM), Universal Mobile Telecommunications System (UMTS), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Transmission Control Protocol / Internet Protocol (TCP-IP), Short Message Service (SMS), Multimedia Messaging Service (MMS), Email, Instant Messaging Service (IMS), Bluetooth, IEEE 802.11, 3GPP Narrowband IoT, and any similar wireless communication technologies. Communication devices implementing various embodiments of the examples described herein can communicate using various media, including but not limited to wireless, infrared, laser, cable connections, and any suitable connection.

[0157] In telecommunications and data networks, a channel can refer to a physical channel or a logical channel. A physical channel can refer to a physical transmission medium such as a wired connection, while a logical channel can refer to a logical connection on a multiplexed medium capable of transmitting multiple logical channels. A channel can be used to transmit information signals (e.g., bit streams) from one or more transmitters (or transmitters) to one or more receivers.

[0158] These embodiments can also be implemented in so-called IoT devices. For example, the Internet of Things (IoT) can be defined as the interconnection of uniquely identifiable embedded computing devices within existing internet infrastructure. The convergence of various technologies has enabled and can enable many embedded systems domains, such as wireless sensor networks, control systems, home / building automation, etc., to be incorporated into the IoT. To utilize the internet, IoT devices are provided with IP addresses as unique identifiers. IoT devices can be provided with wireless transmitters, such as WLAN or Bluetooth transmitters or RFID tags. Alternatively, IoT devices can access IP-based networks via wired networks, such as Ethernet-based networks or power line connections (PLC).

[0159] Figure 4 A block diagram showing the general structure of a video encoder is provided. Figure 4 An encoder for two layers is presented, but it should be understood that the encoder presented can be similarly extended to encode more than two layers. Figure 4 The illustration shows a video encoder including a first encoder portion 401 for a base layer and a second encoder portion 451 for an enhancement layer. Each of the first encoder portion 401 and the second encoder portion 451 may include similar elements for encoding incoming images. Encoder portions 401 and 451 may include pixel predictors 402 and 452, prediction error encoders 403 and 453, and prediction error decoders 404 and 454. Figure 4An embodiment of pixel predictors 402, 452 is also shown, which includes inter-frame predictors 406, 456, intra-frame predictors 408, 458, mode selectors 410, 460, filters 416, 466, and reference frame memories 418, 468. The pixel predictor 402 of the first encoder section 401 receives multiple base layer pictures / images of the video stream to be encoded at both the inter-frame predictor 406 (which determines the difference between the image and the motion-compensated reference frame) and the intra-frame predictor 408 (which determines the prediction of image patches based only on the processed portion of the current frame or picture). The outputs of both the inter-frame predictor and the intra-frame predictor are passed to the mode selector 410. The intra-frame predictor 408 may have more than one intra-frame prediction mode. Thus, each mode can perform intra-frame prediction and provide a prediction signal to the mode selector 410. The mode selector 410 also receives copies of multiple base layer images 400. Accordingly, the pixel predictor 452 of the second encoder section 451 receives the enhancement layer pictures / images 450 of the video stream to be encoded at the inter-frame predictor 456 (which determines the difference between the image and the motion-compensated reference frame) and the intra-frame predictor 458 (which determines the prediction of image patches based only on the processed portion of the current frame or picture). The outputs of both the inter-frame predictor and the intra-frame predictor are passed to the mode selector 460. The intra-frame predictor 458 may have more than one intra-frame prediction mode. Therefore, each mode can perform intra-frame prediction and provide the prediction signal to the mode selector 460. The mode selector 460 also receives a copy of the enhancement layer picture 450.

[0160] Depending on the encoding mode selected for encoding the current block, the outputs of inter-frame predictors 406 and 456, or the output of one of the optional intra-frame predictor modes, or the output of the surface encoder within the mode selector, are passed to the outputs of mode selectors 410 and 460. The outputs of mode selectors 410 and 460 are then passed to first summing devices 421 and 471. The first summing devices can subtract the outputs of pixel predictors 402 and 452 from the base layer image(s) 400 and enhancement layer image(s) 450 to generate first prediction error signals 420 and 470, which are input to prediction error encoders 403 and 453.

[0161] Pixel predictors 402 and 452 also receive a combination of the predicted representations of image blocks 412 and 462 and the outputs 438 and 488 of the prediction error decoders 404 and 454 from the preliminary reconstructors 439 and 489. The preliminary reconstructed images 414 and 464 can be passed to intra-frame predictors 408 and 458 and filters 416 and 466. Filters 416 and 466, receiving the preliminary representations, can filter the preliminary representations and output final reconstructed images 440 and 490, which can be stored in reference frame memories 418 and 468. Reference frame memory 418 can be connected to inter-frame predictor 406 to serve as a reference image for comparison with the future base layer image 400 during inter-frame prediction operations. According to some embodiments, when the base layer is selected and indicated as the source for inter-layer sample prediction and / or inter-layer motion information prediction of the enhancement layer, the reference frame memory 418 may also be connected to the inter-frame predictor 456 to serve as a reference image for comparison with (multiple) future enhancement layer images 450 during inter-frame prediction operations. Furthermore, the reference frame memory 468 may be connected to the inter-frame predictor 456 to serve as a reference image for comparison with (multiple) future enhancement layer images 450 during inter-frame prediction operations.

[0162] According to some embodiments, the filtering parameters of the filter 416 from the first encoder section 401 can be provided to the second encoder section 451, provided that the base layer is selected and indicated as the source for predicting the filtering parameters of the enhancement layer.

[0163] The prediction error encoders 403 and 453 include transform units 442 and 492 and quantizers 444 and 494. Transform units 442 and 492 transform the first prediction error signals 420 and 470 to the transform domain. For example, this transform is a DCT transform. Quantizers 444 and 494 quantize the transform domain signals (e.g., DCT coefficients) to form quantized coefficients.

[0164] Prediction error decoders 404 and 454 receive outputs from prediction error encoders 403 and 453 and perform the reverse processing of the prediction error encoders 403 and 453 to generate decoded prediction error signals 438 and 488. These signals, when combined with the prediction representations of image blocks 412 and 462 at second summing devices 439 and 489, produce preliminary reconstructed images 414 and 464. The prediction error decoders can be considered to include dequantizers 446 and 496 and inverse transform units 448 and 498. The dequantizers dequantize quantization coefficient values ​​(e.g., DCT coefficients) to reconstruct the transform signal, and the inverse transform units perform an inverse transform on the reconstructed transform signal. The outputs of the inverse transform units 448 and 498 include multiple reconstructed blocks. The prediction error decoders may also include block filters that can filter the multiple reconstructed blocks based on further decoded information and filter parameters.

[0165] Entropy encoders 430 and 480 receive the outputs of prediction error encoders 403 and 453 and can perform appropriate entropy coding / variable length coding on the signal to provide a compressed signal. The outputs of entropy encoders 430 and 480 can be inserted into the bitstream, for example, via multiplexer 465.

[0166] Some video encoding and video metadata specifications

[0167] The Advanced Video Coding Standard (which can be abbreviated as H.264, AVC, or H.264 / AVC) was developed by the Video Coding Experts Group (VCEG) of the Telecommunication Standardization Sector of the International Telecommunication Union (ITU-T) and the Joint Video Team (JVT) of the Moving Picture Experts Group (MPEG) of the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). The H.264 / AVC standard was published by the two higher-level standardization organizations as ITU-T Recommendation H.264 and ISO / IEC International Standard 14496-10, also known as MPEG-4 Part 10 Advanced Video Coding (AVC). The H.264 / AVC standard has several versions, each integrating new extensions or features into the specification. These extensions include Scalable Video Coding (SVC) and Multi-View Video Coding (MVC).

[0168] The High Efficiency Video Coding Standard (which may be abbreviated as H.265, HEVC, or H.265 / HEVC) was developed by the Joint Collaboration Team on Video Coding (JCT-VC) of VCEG and MPEG. This standard was published by two higher-level standardization organizations as the ITU-T H.265 Recommendation and ISO / IEC International Standard 23008-2, also known as MPEG-H Part 2 High Efficiency Video Coding (HEVC). Extensions to H.265 / HEVC include scalable, multi-view, 3D, and fidelity range extensions, which may be referred to as SHVC, MV-HEVC, 3D-HEVC, and REXT, respectively. Unless otherwise stated, H.265 / HEVC, SHVC, MV-HEVC, 3D-HEVC, and REXT referenced in this specification for understanding the definitions, structure, or concepts of these standard specifications should be understood as the latest versions of these standards available prior to the date of reference.

[0169] Multifunctional Video Coding (which can be abbreviated as VVC, H.266, or H.266 / VVC) is a video compression standard developed as a successor to HEVC. VVC is specified in the ITU-T H.266 Recommendation and is equivalently specified in ISO / IEC 23090-3, which is also known as MPEG-I Part 3.

[0170] The Alliance for Open Media (AOM) developed the specifications for the AV1 bitstream format and decoding process. The AV1 specification was released in 2018. AOM is reportedly working on the AV2 specification.

[0171] ITU-T Recommendation H.274 (equivalent to ISO / IEC 23002-7) can be referred to as "General Supplemental Enhancement Information Messages for Encoded Video Bitstreams" and is known as "General Supplemental Enhancement Information" or VSEI. The VSEI standard specifies the syntax and semantics of Video Availability Information (VUI) parameters and Supplemental Enhancement Information (SEI) messages. The VUI parameters and SEI messages defined in the VSEI standard are designed to be transmitted within an encoded video bitstream in the manner specified in the video coding specification, or in other ways determined by the specifications of systems using such encoded video bitstreams. The VSEI standard is intended for use with VVC encoded video bitstreams, although it was drafted in a manner intended to be sufficiently general to allow its use with other types of encoded video bitstreams. For example, VUI parameters and SEI messages can assist in processes related to decoding, display, or other purposes.

[0172] Video encoding

[0173] In most cases, the basic units used as input to the encoder and output to the decoder are images. The image used as input to the encoder can also be called the source image, and the image decoded by the decoder can be called the decoded image or the reconstructed image.

[0174] Both the source image and the decoded image consist of one or more sample arrays, such as one of the following sample arrays: - Luminance (Y) only (monochrome). - Brightness and two chromaticities (YCbCr or YCgCo). - Green, blue, and red (GBR, also known as RGB). - Indicates an array of other unspecified monochromatic or tristimulus color samples (e.g., YZX, also known as XYZ).

[0175] In the following text, these arrays may be referred to as luminance (or L or Y) and chrominance, where the two chrominance arrays may be referred to as Cb and Cr or Cg and Co; regardless of the actual color representation used. For example, the actual color representation used may be indicated in the encoded bitstream using Video Availability Information (VUI) syntax. A component may be defined as: an array or a single sample from one of the three sample arrays (luminance and two chrominance), or an array or a single sample of an array that makes up an image in monochrome format.

[0176] An image can be defined as a frame or a field. A frame consists of a matrix of luminance samples and possible corresponding chrominance samples. A field is a set of alternating sample rows of frames and can be used as encoder input when the source signals are interleaved. The chrominance sample array may not exist (and therefore monochrome sampling can be used), or the chrominance sample array may be subsampled compared to the luminance sample array.

[0177] Some chroma formats can be summarized as follows: - In monochrome sampling, there is only one sample array, which can be nominally considered as a brightness array. In 4:2:0 sampling, each of the two chroma arrays has half the height and width of the luminance array. In 4:2:2 sampling, each of the two chroma arrays has the same height and half the width of the luminance array. - In 4:4:4 sampling, when no separate color plane is used, each of the two chroma arrays has the same height and width as the luminance array.

[0178] Encoding formats or standards can allow sample arrays to be encoded as individual color planes into a bitstream, and the individual encoded color planes to be decoded from the bitstream accordingly. When using individual color planes, each of them is processed separately (by the encoder and / or decoder) as a monochrome sampled image.

[0179] A video codec can include an encoder that converts input video into a compressed representation suitable for storage / transmission and a decoder that can decompress the compressed video representation back into a visual form. The compressed representation can be called a bitstream or video bitstream. The video encoder and / or video decoder can also be separate entities, i.e., they do not need to form a codec. The encoder can discard some information from the original video sequence to represent the video in a more compact form (i.e., at a lower bit rate).

[0180] A video encoder can encode video information in two stages. First, it predicts the pixel values ​​in a region (or “block”) of a picture, for example, through motion compensation methods (finding and indicating the region closely corresponding to the block being encoded in previously encoded video frames) or spatial methods (using the pixel values ​​around the block to be encoded in a specified manner). Second, it encodes the prediction error (i.e., the difference between the predicted pixel block and the original pixel block). This can be achieved by transforming the difference in pixel values ​​using a specified transform (e.g., Discrete Cosine Transform (DCT) or a variant thereof), quantizing the coefficients, and entropy encoding the quantization coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the precision of the pixel representation (picture quality) and the size of the final encoded video representation (file size or transmission bitrate).

[0181] In inter-frame prediction, the prediction source is a previously decoded image (also known as a reference image). Temporal inter-frame prediction (also known as temporal prediction) can refer to inter-frame prediction where the reference image is a previous image in the same layer as the current image being encoded or decoded, in the order of decoding.

[0182] In intra-block copy (IBC; also known as intra-block copy prediction or current picture reference), prediction can be applied similarly to inter-frame prediction, but the reference picture is the current picture, and only previously decoded samples can be referenced during the prediction process.

[0183] Inter-layer or inter-view prediction can be applied similarly to inter-temporal frame prediction, but the reference image is a decoded image from another scalable layer or another view.

[0184] In some cases, inter-frame prediction may refer only to temporal inter-frame prediction, while in others, inter-frame prediction may be collectively referred to as temporal inter-frame prediction, as well as any of intra-block copying, inter-layer prediction, and inter-view prediction, provided that these predictions are performed using the same or similar procedures as temporal inter-frame prediction.

[0185] Inter-frame prediction or temporal prediction can sometimes be referred to as motion compensation or motion-compensated prediction.

[0186] Intra-frame prediction leverages the fact that adjacent pixels within the same image may be correlated. Intra-frame prediction can be performed in the spatial domain or the transform domain; that is, it can predict sample values ​​or transform coefficients. Intra-frame prediction is typically used in intra-frame coding, where inter-frame prediction is not applied.

[0187] One outcome of the encoding process is a set of encoding parameters, such as motion vectors and quantization transform coefficients. These parameters can be entropy-coded more efficiently when many parameters are first predicted from spatially or temporally adjacent ones. For example, motion vectors can be predicted based on spatially adjacent motion vectors, and only the differences between the predicted values ​​and the motion vectors can be encoded. The prediction of encoding parameters and intra-frame prediction can be collectively referred to as intra-picture prediction.

[0188] A video decoder can reconstruct the output video by applying an encoder-like prediction method to form a predictive representation of pixel blocks (using motion or spatial information created by the encoder and stored in the compressed representation) and prediction error decoding (the inverse operation of prediction error coding, used to recover the quantized prediction error signal in the spatial pixel domain). After applying the prediction and prediction error decoding methods, the decoder adds the prediction and prediction error signals (pixel values) to form the output video frame. The decoder (and encoder) can also apply additional filtering methods to improve the quality of the output video before sending it to display and / or storing it as a prediction reference for upcoming frames in a video sequence.

[0189] Partitioning can be defined as dividing a set into subsets such that each element of the set is located in exactly one of these subsets.

[0190] In some video codecs, a picture can be divided into coding units (CUs) covering a region of the picture. A CU includes one or more prediction units (PUs) defining the prediction process for samples within the CU, and one or more transform units (TUs) defining the prediction error coding process for samples within the CU. A CU can include rectangular sample blocks whose size can be selected from a predefined set of possible CU sizes. The CU with the largest allowed size can be referred to as an LCU (Maximum Coding Unit) or a coding tree unit (CTU), and video pictures are divided into non-overlapping LCUs. An LCU can be further subdivided into combinations of smaller CUs, for example, by recursively splitting the LCU and the resulting CU. Each resulting CU can have at least one PU and at least one TU associated with it. Each PU and TU can be further subdivided into smaller PUs and TUs to improve the granularity of the prediction and prediction error coding processes, respectively. Each PU has prediction information associated with it that defines which prediction is applied to the pixels within that PU (e.g., motion vector information for inter-frame prediction PUs and intra-frame prediction directionality information for intra-frame prediction PUs).

[0191] Each TU can be associated with information describing the prediction error decoding process of samples within that TU (including, for example, DCT coefficient information). Prediction error coding can be signaled at the CU level, regardless of whether it is applied to each CU. In the absence of prediction error residuals associated with a CU, the CU can be considered to have no TU. Dividing the image into CUs and CUs into PUs and TUs can be signaled in the bitstream so that the decoder can reproduce the expected structure of these units.

[0192] In some encoding formats (such as VVC), samples are processed in units of Coding Tree Blocks (CTBs). In some encoding formats, the encoder can choose the size of the CTB. The array size (in samples) of each Luminance CTB in width and height can be represented as CtbSizeY. The VVC encoder can select CtbSizeY based on the sequence from values ​​supported by the VVC standard (32, 64, 128), or the VVC encoder can be configured to use a specific CtbSizeY value. The width and height (in samples) of the array for each Chroma CTB can be represented as CtbWidthC and CtbHeightC, respectively.

[0193] In some coding formats, each CTB can be assigned a partition signaling to identify the block size used for intra-frame or inter-frame prediction and for transform coding. A partition is a recursive quadtree partition. The root of the quadtree is associated with the CTB. The quadtree is split until a leaf is reached, which is called a quadtree leaf. When the component width is not an integer multiple of the CTB size, the CTB at the right component boundary is incomplete. When the component height is not an integer multiple of the CTB size, the CTB at the bottom component boundary is incomplete.

[0194] In some encoding formats, the coded block is the root node of both the prediction tree and the transform tree. The prediction tree specifies the position and size of the prediction block. The transform tree specifies the position and size of the transform block. The luma and chroma splitting information is the same for the prediction tree, but can be the same or different for the transform tree.

[0195] In some encoding formats, blocks and associated syntax structures can be grouped into "unit" structures as follows: - A transform unit is associated with one transform block (monochrome image) or three transform blocks (luminance and chrominance components of an image in 4:2:0, 4:2:2 or 4:4:4 color format) and the associated transform syntax structure unit. - A coding block (monochrome image) or three coding blocks (luminance and chrominance), the associated coding syntax structure, and the associated transformation unit are associated with a coding unit. - A CTU is associated with a CTB (monochrome image) or three CTBs (luminance and chrominance), the associated coding tree syntax structure, and the associated coding unit.

[0196] In AV1, a superblock is similar to a CTU in VVC. A superblock can be considered the largest coded block supported by the AV1 specification. The size of the superblock is signaled in the sequence header as 128×128 or 64×64 luminance samples. A superblock can be recursively partitioned into smaller coded blocks. Each coded block can have its own prediction and transform modes, independent of those of other coded blocks.

[0197] In some coding schemes of certain video coding formats, reference pictures used for inter-frame prediction are indicated by indices in a list of reference pictures. These indices can be encoded using variable-length coding, which typically results in smaller indices having shorter values ​​for the corresponding syntax elements. In some coding formats, two lists of reference pictures (reference picture list 0 and reference picture list 1) are generated for each double-prediction (B) slice, and one list of reference pictures (reference picture list 0) is formed for each single-prediction (P) slice.

[0198] In some encoding formats, a list of reference images can exist within a parameter set, such as a sequence parameter set, and the list of (multiple) reference images in use can be indicated, for example, in a slice header or image header. When the encoder deems that none of the reference image lists in the parameter set are suitable for the current image or slice, the reference image list can be directly included in the corresponding syntax structure, such as an image header or slice header.

[0199] Out-of-band transmission, signaling, or storage can refer to the ability to send, signal, or store information in a manner that associates information with a video bitstream. Out-of-band transmission can use more reliable transmission mechanisms compared to protocols used to carry encoded video data, such as slices. Out-of-band transmission, signaling, or storage can be used alternatively or separately, for example, to facilitate access or session negotiation. For example, sample entries for tracks in a file conforming to the ISO Basic Media File Format may include parameter sets, while the encoded data in the bitstream is stored elsewhere in the file or in another file. Another example of out-of-band transmission, signaling, or storage includes including information such as NN and / or NN updates in a file format track separate from the track(s) containing encoded video data.

[0200] In the embodiments described in the claims, the phrase "along the bitstream" (e.g., indicating along the bitstream) or "along the encoded unit of the bitstream" (e.g., indicating along the encoded tile) can be used to refer to transmissions, signaling, or storage that are associated with, but not included in, the bitstream or encoded unit, respectively, as 'out-of-band' data. Decoding of the phrase "along the bitstream" or "along the encoded unit of the bitstream," etc., can refer to decoding the referenced out-of-band data (which may be obtained from out-of-band transmission, signaling, or storage) associated with the bitstream or encoded unit, respectively. For example, the phrase "along the bitstream" can be used when the bitstream is included in a container file, such as a file conforming to the ISO Basic Media File Format, and some file metadata is stored in the file in a manner that associates metadata with the bitstream, such as boxes in sample entries for tracks that include the bitstream, sample groups for tracks that include the bitstream, or timing metadata tracks associated with tracks that include the bitstream. In another example, the phrase "along the bitstream" can be used when the bitstream is available as a stream over a communication protocol and a media description, such as a stream list, is provided to describe the stream.

[0201] A bitstream can be defined as a sequence of bits or a sequence of syntax structures. The bitstream format can constrain the order of the syntax structures in the bitstream.

[0202] Syntax elements can be defined as data elements represented in a bitstream. Syntax structures can be defined as zero or more syntax elements that appear together in a bitstream in a specified order.

[0203] For example, syntax constructs can be specified using arithmetic, logical, relational, bitwise, and assignment operators similar to those available in many programming languages. For instance, '&' can indicate a bitwise AND operation. Furthermore, syntax constructs can be specified by referencing mathematical functions.

[0204] Syntactic structures and semantics can use the values ​​of variables derived from the values ​​of syntactic elements. Naming conventions can be defined for variables. For example, variables can be named using a mix of lowercase and uppercase letters, without any underscore characters. Variables beginning with an uppercase letter can be derived for decoding the current syntactic structure and all dependent syntactic structures. In some cases, variables beginning with an uppercase letter can be used in subsequent syntactic structures during decoding without mentioning the variable's original syntactic structure. Variables beginning with a lowercase letter can only be used within syntactic structures or functions where they have already been defined.

[0205] The basic units used for the output of a video encoder and the input of a video decoder can be Network Abstraction Layer (NAL) units. For transmission over packet-oriented networks or storage in structured files, NAL units can be encapsulated within packets or similar structures. Byte stream formats encapsulating NAL units can be used in transmission or storage environments that do not provide frame structures. Byte stream formats can separate NAL units by appending start codes to each NAL unit. To avoid erroneous detection of NAL unit boundaries, the encoder can run a byte-oriented start code anti-racement algorithm, which adds anti-racement bytes to the NAL unit payload where start codes would normally be present. To enable direct gateway operation between packet-oriented and stream-oriented systems, start code emulation prevention can be performed regardless of whether a byte stream format is used. A NAL unit can be defined as a syntax structure that includes an indication of the data type to be followed, and bytes that include that data in the form of a raw byte sequence payload, interspersed with anti-racement bytes where necessary. A raw byte sequence payload (RBSP) can be defined as a syntax structure that includes an integer number of bytes encapsulated within the NAL unit. RBSP can be empty or in the form of a data bit string, including syntax elements, followed by RBSP stop bits, followed by zero or more subsequent bits equal to 0.

[0206] A bitstream can be defined as logically including syntactic structures, such as NAL units, when the syntactic structure is sent along the bitstream but can be included within the bitstream according to the bitstream format. A bitstream can also be defined as natively including syntactic structures when it includes them.

[0207] In some encoding formats or standards, a bitstream can be in the form of a Network Abstraction Layer (NAL) unit stream or a byte stream, which forms a representation of encoded images and associated data to form one or more encoded video sequences.

[0208] In some formats or standards, a first bitstream may be followed by a second bitstream in the same logical channel, such as within the same file or on the same connection of a communication protocol. A primary stream (in the context of video coding) can be defined as a sequence of one or more bitstreams.

[0209] In some encoding formats (such as AV1), a bitstream may include a sequence of Open Bitstream Units (OBUs). An OBU includes a header and a payload, where the header identifies the type of OBU. Additionally, an OBU may include an OBU size, which may follow the OBU header and precede the OBU payload within the OBU, where the OBU size may indicate the size of the payload (in bytes).

[0210] The OBU bitstream can be formatted as a so-called low-overhead bitstream format, which consists of a sequence of OBUs, where the OBU header indicates the OBU size (without a size field) in bytes. Alternatively, the OBU bitstream can be formatted as a so-called length-delimited bitstream format, where each time unit begins with a size field (temporal_unit_size) indicating the size of the time unit payload in bytes, and each frame unit begins with a size field (frame_unit_size) indicating the size of the frame unit payload in bytes, and each OBU is preceded by a size field indicating the size of the OBU in bytes.

[0211] OBU types can include the following. - The sequence header includes information applicable to the entire sequence and whether certain encoding tools are enabled. - The time delimiter indicates the frame's presentation timestamp. All displayable frame OBUs after the time delimiter will use that timestamp until the next time delimiter OBU arrives. The time delimiter and its subsequent OBUs with the same timestamp are referred to as time units. In the context of scalable encoding, compressed data associated with all representations of frames across various spatial and fidelity resolutions will reside in the same time unit. - The frame header sets up the encoding information for a given frame, including the signaling inter-frame or intra-frame type, which indicates the reference frame and the signaling probability model update method. - A tile group comprises tile data associated with a frame. Each tile can be decoded independently. After possible loop filtering, they are collectively reconstructed to form a reconstructed frame. - A frame includes a frame header and tile data. The frame OBU is largely equivalent to the frame header OBU and the tile group OBU, but allows for lower overhead costs. - Metadata carries information such as high dynamic range, scalability, and timecode. - The tile list includes tile data similar to that of a tile group (OBU). However, each tile here has an additional header indicating its reference frame index and position within the current frame. This allows the decoder to process a subset of tiles and display the corresponding portion of the frame without needing to fully decode all tiles in the frame.

[0212] In some coding standards, a NAL unit includes a header and a payload. The NAL unit header indicates the type of NAL unit. In some coding standards, the NAL unit header indicates a scalability layer identifier (e.g., nuh_layer_id in H.265 / HEVC and H.266 / VVC), which can be used to indicate, for example, a spatial or quality layer, a view or auxiliary layer (such as a depth map or alpha plane) in multi-view video. In some coding standards, the NAL unit header includes a temporal sublayer identifier, which can be used to indicate a temporal subset of the bitstream, such as a 30-frame-per-second subset of a 60-frame-per-second bitstream.

[0213] Bitstreams or encoded video sequences can be encoded to be temporally scalable, as shown below. Each picture can be assigned to a specific temporal sublayer. Temporal sublayers can be equivalently referred to as sublayers, temporal second layers, sublayers, or temporal levels. Temporal sublayers can be enumerated, for example, starting from 0. The lowest temporal sublayer (sublayer 0) can be decoded independently. The picture at temporal sublayer 1 can be predicted based on the reconstructed pictures at temporal sublayers 0 and 1. The picture at temporal sublayer 2 can be predicted based on the reconstructed pictures at temporal sublayers 0, 1, and 2, and so on. In other words, the picture at temporal sublayer N does not use any pictures at temporal sublayers greater than N as a reference for inter-frame prediction. A bitstream created by excluding all pictures greater than or equal to the selected sublayer value and including the remaining pictures is still compliant with the specification.

[0214] Each picture in a time-scalable bitstream can be assigned a time identifier (also known as a TID, time layer identifier, time sub-layer identifier, or time layer ID), for example, this identifier can be assigned to the variable TemporalId. The time identifier can be indicated in the NAL cell header or OBU extension header, for example. A TemporalId equal to 0 corresponds to the lowest time level. A bitstream created by excluding all encoded pictures with a TemporalId greater than or equal to the selected value and including all other encoded pictures is still compliant with the specification. Therefore, a picture with a TemporalId equal to tid_value does not use any picture with a TemporalId greater than tid_value as a prediction reference.

[0215] NAL units can be divided into Video Coding Layer (VCL) NAL units and non-VCL NAL units. VCL NAL units are typically encoded slice NAL units.

[0216] For example, a non-VCL NAL unit can be one of the following types: Video Parameter Set (VPS), Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Adaptive Parameter Set (APS), Supplemental Enhancement Information (SEI) NAL unit, Access Unit Delimiter, Sequence End NAL unit, Bitstream End NAL unit, or Padding Data NAL unit. Reconstruction of a decoded image may require a parameter set, while many other non-VCL NAL units are not necessary for reconstructing decoded sample values.

[0217] Video coding standards enable the use of Supplemental Enhancement Information (SEI) messages. Some video coding standards include SEI NAL units, while others include both prefix SEI NAL units and suffix SEI NAL units, where the former can begin a picture unit, and the latter can end a picture unit. An SEI NAL unit can include one or more SEI messages that are not essential for decoding the output picture but can assist in related processes such as picture output timing, post-processing of decoded pictures, rendering, error detection, error hiding, and resource reservation. Several SEI messages are specified in the H.264 / AVC, H.265 / HEVC, H.266 / VVC, and H.274 / VSEI standards, and user data SEI messages allow organizations and companies to specify SEI messages for their own use. These standards may include the syntax and semantics for specifying SEI messages, but may not define the procedures for processing the messages in the receiver. Therefore, encoders may need to follow standards for specifying SEI messages when creating (multiple) SEI messages, while decoders may not need to process SEI messages to maintain output order consistency. One example reason for including the syntax and semantics of SEI messages in standards is to allow different system specifications to interpret supplementary information in the same way, thus enabling interoperability. System specifications can require the use of specific SEI messages at both the encoder and decoder ends, and can also specify the procedure for processing specific SEI messages at the receiver.

[0218] Some video coding standards enable the metadata OBU. The metadata OBU includes a type field that specifies the type of metadata.

[0219] A video sequence encoded (CVS) can be defined as a sequence of encoded images in decoding order, which is independently decodeable and followed by the end of another encoded video sequence or bitstream.

[0220] A coded layer video sequence (CLVS) can be defined as a sequence of images and other associated data within the same scalable layer (e.g., having the same nuh_layer_id value in a VVC) that can be decoded independently of other images in the same layer.

[0221] Some codecs use the concept of Picture Order Count (POC). A POC value is derived for each picture, and the POC value increases rather than decreases as the pictures are positioned in the output order. Therefore, the POC indicates the output order of the pictures. The POC can be used in the decoding process, such as for implicit scaling of motion vectors and initialization of a list of reference pictures. Furthermore, the POC can be used to verify the consistency of the output order. The variable that includes the POC value of a picture can be called PicOrderCntVal.

[0222] Decoded image buffers (DPBs) can be used in encoders and / or decoders. Buffering decoded images can serve two purposes: as a reference in inter-frame prediction and to reorder decoded images for output order. Some encoding formats (such as HEVC) offer great flexibility for both reference image tagging and output reordering, and separate buffers for reference and output images could waste memory resources. Therefore, a DPB can include a unified decoded image buffering process for both reference and output reordering. Decoded images can be removed from the DPB when they are no longer used as a reference and are no longer needed for output.

[0223] The output order can be defined as the order in which decoded images are output from the decoded image buffer (for decoded images that will be output from the decoded image buffer).

[0224] Output time can be defined as the time it takes for a decoded image to be output from the decoder or the decoder's DPB (for a decoded image to be output from the DPB), for example, as specified by the assumed reference decoder specification based on the output timing of the DPB operation.

[0225] Images with the same output order can be defined as images with the same output time.

[0226] Decoding order can be defined as the order in which syntax elements are processed during the decoding process. It may be necessary to sort the syntax elements in the bitstream according to their decoding order.

[0227] The decoder and / or HRD may include an image output process. The output process can be considered as the process by which the decoder provides a decoded and cropped image as the output of the decoding process. The output process is typically part of the video coding standard, usually a part of the assumed reference decoder specification. In output cropping, sample rows and / or columns are removed from the decoded image according to a cropping rectangle to form the output image. The cropped decoded image can be defined as the result of cropping the decoded image based on a consistent cropping window, which is specified, for example, in the sequence parameter set referenced to the corresponding encoded image.

[0228] In some video coding standards, the encoder can control whether the decoded (and cropped) images are output by the image output process or a similar decoder-side process. The encoder may include one or more syntax elements in or along the bitstream to control image output. For example, the bitstream syntax may include flags (e.g., called pic_output_flag or ph_pic_output_flag) in the image header and / or image segment header (e.g., slice header). The semantics of pic_output_flag can be specified such that when pic_output_flag equals 0, the corresponding decoded image is not output (via the image output process, etc.), while when pic_output_flag equals 1, the corresponding decoded image is output unless otherwise concluded during the decoding process.

[0229] Identifiers can be defined as syntactic elements that identify syntactic structures. For example, the value of an identifier can differ in different instances of the same syntactic structure, such as a parameter set. A specific instance of a syntactic structure can be referenced by its identifier value. For example, the parameter set referenced by the (de)coding of an encoded video slice can be identified by providing the identifier value of the parameter set in the header of the encoded video slice.

[0230] An indicator (idc) can be defined as a syntax element whose value indicates a choice between more than two values ​​(for which semantics have already been assigned). The name of an indicator syntax element may have the suffix _idc.

[0231] An image can be broken down into independent, encodeable, and decodeable image segments (e.g., slices, tiles, or groups of tiles). Such image segments enable parallel processing. Image segments can be encoded as individual units in a bitstream, such as VCL NAL units in H.264 / AVC, HEVC, and VVC. Encoded image segments may include a header and a payload, where the header contains the parameter values ​​needed to decode the payload.

[0232] In some video coding formats (such as HEVC and VVC), images are divided into one or more tile rows and one or more tile columns. A tile is a sequence of Coded Tree Units (CTUs) that covers a rectangular area of ​​the image. Partitioning into tiles can form a grid, characterized by a list of tile column widths (in CTUs) and a list of tile row heights (in CTUs). For encoding and / or decoding, the CTUs within a tile are scanned in raster scan order within that tile. In HEVC, tiles are sequentially ordered in the bitstream according to the raster scan order of the tile grid.

[0233] In some video coding formats (such as AV1), images can be partitioned into tiles, and each tile comprises an integer number of complete superblocks that together form a complete rectangular region of the image. Intra-image prediction across tile boundaries can be disabled. The minimum tile size can be a superblock, and the maximum tile size for the currently specified level in AV1, in terms of luminance sample count, is 4096 × 2304. Images are partitioned into a tile grid consisting of one or more tile rows and one or more tile columns. The tile grid can be signaled in the image header to have uniform or non-uniform tile sizes, where, in the latter case, the tile row height and tile column width are signaled. Superblocks within a tile are scanned in raster scan order within that tile.

[0234] In some video coding formats (such as VVC), a slice consists of: an integer number of complete tiles, or an integer number of consecutive complete CTU rows within a tile of an image. Therefore, each vertical slice boundary is always also a vertical tile boundary. The horizontal boundary of a slice may not be a tile boundary, but it includes the horizontal CTU boundaries within a tile; this occurs when a tile is split into multiple rectangular slices, each of which includes an integer number of consecutive complete CTU rows within the tile.

[0235] Some video coding formats (such as VVC) support two slicing modes: raster scan slicing mode and rectangular slicing mode. In raster scan slicing mode, a slice consists of a complete sequence of tiles in a raster scan of an image. In rectangular slicing mode, a slice consists of multiple complete tiles that together form a rectangular area of ​​the image, or multiple consecutive complete CTU rows of a tile (these CTU rows together form a rectangular area within the rectangular area corresponding to the slice).

[0236] In HEVC, a slice comprises an integer number of CTUs. When a tile is not in use, the CTUs are scanned within the tile or image in the order of their raster scans. A slice can comprise an integer number of tiles, or a slice can be included within a tile.

[0237] In some video encoding formats (such as AV1), a tile group OBU carries one or more complete tiles. The first and last tiles in the tile group OBU can be indicated in the tile group OBU before the encoded tile data. Tiles within the tile group OBU can appear in a tile raster scan of an image.

[0238] An encoded image can be defined as the encoded representation of an image. In some video coding formats, an encoded image can include VCL NAL units. In other coding formats, an encoded image can be defined as the encoded representation of an image that includes all coding tree units of the image.

[0239] In some encoding formats, an Access Unit (AU) can be defined as a set of NAL units that are associated with each other according to a specified classification rule, are sequential in decoding order, and include at most one encoded picture in any scalable layer (e.g., any specific value of nuh_layer_id in some encoding formats such as HEVC or VVC). In some encoding formats, an Access Unit includes one or more complete picture units. In some encoding formats, in addition to VCL NAL units that include encoded pictures, Access Units may also include non-VCL NAL units associated with encoded pictures. For example, the specified classification rule can associate pictures with the same output time or picture order count value into the same Access Unit.

[0240] A picture unit can be defined as a set of NAL units that are related to each other according to a specified classification rule, are consecutive in decoding order, and include exactly one encoded picture. Some non-VCL NAL units (such as prefix SEINAL units) are allowed to precede all VCL NAL units of the same picture unit in decoding order, and therefore can begin a picture unit, but are not allowed to follow all VCL NAL units of the same picture unit in decoding order. Some non-VCLNAL units (such as suffix SEI NAL units) are allowed to follow all VCL NAL units of the same picture unit in decoding order, but are not allowed to precede all VCL NAL units of the same picture unit in decoding order. The simplified structure of a picture unit is given below, where the decoding order of NAL units is from top to bottom, dashed boxes indicate optional syntax structures or concepts, and solid boxes indicate mandatory syntax structures or concepts. Figure 5 The illustration shows image unit 500. Image unit 500 needs to include exactly one encoded image 502, and the encoded image needs to have at least one encoded slice, such as 504-1 and 504-2.

[0241] In some encoding formats (such as AV1), an encoded video sequence comprises one or more time units. A time unit consists of a series of OBUs, beginning with a time delimiter, an optional sequence header, an optional metadata OBU, and a sequence of one or more frame headers (each followed by zero or more tile group OBUs and optional padding OBUs). A time unit can be defined to include all OBUs associated with a specific, distinct moment. A time unit may include the time delimiter OBU and all subsequent OBUs up to but not including the next time delimiter. The time delimiter OBU can be defined as an indication that subsequent OBUs will have a different rendering / decoding timestamp than the last frame preceding the time delimiter.

[0242] Video coding standards can specify profiles and levels. A profile can be considered a subset of the standard's algorithmic features. Alternatively, a profile can be defined as a specified subset of the coding standard's syntax. A level can be defined as a set of constraints on coding parameters that impose constraints on decoder resource consumption. Alternatively, a level can be defined as a set of constraints defining the values ​​that syntax elements and variables of the coding standard can take. The same set of levels can be defined for all profiles, where most aspects of the definition of each level are common across different profiles, although aspects can differ between profiles. Profiles and levels can be used to signal attributes of the media stream and capabilities of the media decoder. Each pair of profiles and levels can be considered as forming an "interoperability point".

[0243] By combining profiles and levels, a decoder can declare whether it can decode a stream without actually attempting the decoding process. When a decoder cannot decode a stream, it may cause the decoder to crash, run slower than real-time, and / or drop data due to buffer overflow.

[0244] The concept of levels has been specified and used in HEVC and can be similarly specified and used in other codecs. A level can be defined as a level constraint of a specified category imposed on the values ​​of syntax elements in a bitstream, where level constraints are nested within levels, and a decoder that conforms to a certain level will be able to decode all bitstreams that conform to the same level or a lower level or any level below that level.

[0245] Scalable video encoding

[0246] Scalable video coding can refer to a coding structure where a bitstream can include multiple representations of content, for example, at different bitrates, resolutions, or frame rates. In these cases, a receiver can extract the desired representation based on its characteristics (e.g., the resolution best matched to the display device). Alternatively, a server or network element can extract portions of the bitstream to be sent to the receiver based on, for example, the receiver's network characteristics or processing capabilities. Meaningful decoded representations can be produced by decoding only certain portions of the scalable bitstream. Scalable bitstreams typically include a 'base layer' that provides the lowest available quality video, and one or more enhancement layers that can improve video quality when received and decoded together with the lower layers. To improve the coding efficiency of the enhancement layers, the encoded representation of the enhancement layers often depends on the lower layers. For example, motion and pattern information of the enhancement layers can be predicted from the lower layers. Similarly, pixel data from the lower layers can be used to create predictions for the enhancement layers.

[0247] A scalable bitstream may include a 'base layer' that provides the lowest available quality video, and one or more enhancement layers that can improve video quality when received and decoded together with the lower layer. To improve the coding efficiency of the enhancement layer, its encoded representation may depend on the lower layer. For example, motion and pattern information for the enhancement layer can be predicted from the lower layer. Similarly, pixel data from the lower layer can be used to create predictions for the enhancement layer.

[0248] Scalable video codecs for quality scalability (also known as signal-to-noise ratio or SNR) and / or spatial scalability can be implemented as follows. For the base layer, a conventional non-scalable video encoder and decoder are used. The reconstructed / decoded image of the base layer is included in a reference image buffer for the enhancement layer. In H.264 / AVC, HEVC, and similar codecs that use a list of reference images for inter-frame prediction, the base layer decoded image can be inserted into the list of reference images for encoding / decoding the enhancement layer image, similar to the decoded reference image for the enhancement layer. Therefore, the encoder can select the base layer reference image as an inter-frame prediction reference and indicate its use, e.g., using a reference image index in the encoded bitstream. The decoder decodes the base layer image from the bitstream (e.g., based on the reference image index) to use as an inter-frame prediction reference for the enhancement layer. When a decoded base layer image is used as a prediction reference for the enhancement layer, it is called an inter-layer reference image.

[0249] It is important to understand that the description of scalable video coding can be generalized to any scalable hierarchy with more than two layers. In this case, during encoding and / or decoding, a second enhancement layer may depend on a first enhancement layer, and therefore the first enhancement layer can be considered as the base layer for encoding and / or decoding the second enhancement layer. Furthermore, it is important to understand that inter-layer reference images from more than one layer may exist in the reference image buffer or reference image list of the enhancement layer, and each of these inter-layer reference images can be considered to reside in the base layer or reference layer of the enhancement layer being encoded and / or decoded. Additionally, it is important to understand that other types of inter-layer processing can be performed besides or in lieu of upsampling the reference layer images. For example, the bit depth of samples from the reference layer images can be converted to the bit depth of the enhancement layer, and / or sample values ​​can undergo a mapping from the color space of the reference layer to the color space of the enhancement layer.

[0250] Scalable video encoding and / or decoding schemes can use multi-ring encoding and / or decoding, characterized as follows: During encoding / decoding, the base layer image can be reconstructed / decoded to serve as a motion compensation reference image for subsequent images within the same layer in encoding / decoding order, or as a reference for inter-layer (or inter-view or inter-component) prediction. The reconstructed / decoded base layer image can be stored in a decoded image buffer (DPB). Similarly, enhancement layer images can be reconstructed / decoded to serve as a motion compensation reference image for subsequent images within the same layer in encoding / decoding order, or as a reference for inter-layer (or inter-view or inter-component) prediction of higher enhancement layers (if any). In addition to the reconstructed / decoded sample values, syntax element values ​​of the base / reference layer or variables derived from the syntax element values ​​of the base / reference layer can also be used in inter-layer / inter-component / inter-view predictions.

[0251] Inter-layer prediction can be defined as prediction that relies on data elements (e.g., sample values ​​or motion vectors) from a reference image that is different from the layer being encoded or decoded. Many types of inter-layer prediction exist and can be applied to scalable video encoders / decoders.

[0252] Inter-layer prediction can be of one or more, including but not limited to, inter-layer sample prediction, inter-layer motion prediction, and intra-layer residual prediction. In inter-layer sample prediction, at least a subset of the reconstructed sample values ​​of the source images used for inter-layer prediction is used as a reference for predicting the sample values ​​of the current image. In inter-layer motion prediction, at least a subset of the motion vectors of the source images used for inter-layer prediction is used as a reference for predicting the motion vectors of the current image. Typically, prediction information about which reference images are associated with the motion vectors is also included in inter-layer motion prediction. For example, the reference index of the reference images used for the motion vectors can be inter-layer predicted, and / or the image sequence count or any other identifier of the reference images can be intra-layer predicted. In some cases, inter-layer motion prediction may also include prediction of block coding modes, header information, block partitions, and / or other similar parameters. In some cases, coding parameter prediction (such as inter-layer prediction of block partitions) can be considered as another type of inter-layer prediction. In inter-layer residual prediction, the prediction error or residual of selected blocks of the source images used for inter-layer prediction is used to predict the current image.

[0253] A direct reference layer can be defined as a layer that can be used for inter-layer prediction of another layer, and that layer is the direct reference layer of that other layer. A direct prediction layer can be defined as a layer whose other layer is a direct reference layer. An indirect reference layer can be defined as: a layer that is not a direct reference layer of the second layer, but is a direct reference layer of the third layer, the third layer being a direct reference layer of the second layer, or an indirect reference layer of the direct reference layer of the second layer, and for the second layer, this layer is an indirect reference layer. An indirect prediction layer can be defined as a layer whose other layer is its indirect reference layer. A dependent layer can be a direct prediction layer or an indirect prediction layer. An independent layer can be defined as a layer without a direct reference layer. In other words, an independent layer does not use inter-layer prediction for prediction. A non-fundamental layer can be defined as any layer other than the fundamental layer, and the fundamental layer can be defined as the lowest layer in the bitstream. An independent non-fundamental layer can be defined as a layer that is both an independent layer and a non-fundamental layer.

[0254] A multi-layer bitstream is a bitstream that comprises multiple layers, which can be, but are not limited to, the base layer and enhancement layer used for scalable video coding as described above. A multi-layer bitstream may additionally or alternatively include independent layers that do not have inter-layer prediction relationships with each other, and can even represent different types of content.

[0255] Scalability patterns or scalability dimensions may include, but are not limited to, one or more of the following: - Quality Scalability: The encoding quality of the base layer image is lower than that of the enhancement layer image. This can be achieved by using larger quantization parameter values ​​in the base layer than in the enhancement layer (e.g., a larger quantization step size for transform coefficient quantization). - Spatial scalability: The encoding resolution of the base layer image is lower than that of the enhancement layer image (e.g., fewer samples). Spatial scalability and quality scalability can sometimes be considered as the same type of scalability. - Bit depth scalability: The encoding bit depth of the base layer image (e.g., 8 bits) is lower than that of the enhancement layer image (e.g., 10 or 12 bits). - Dynamic range scalability: Scalable layers represent images acquired with different dynamic ranges and / or using different tone mapping functions and / or different optical transfer functions. - Chroma format scalability: The spatial resolution provided by the base layer image in the chroma sample array (e.g., encoded in 4:2:0 chroma format) is lower than that of the enhancement layer image (e.g., 4:4:4 format). - Color gamut scalability: The color representation range of the enhancement layer image is richer / wider than that of the base layer image. For example, the enhancement layer can have the UHDTV (ITU-R BT.2020) color gamut, while the base layer can have the ITU-R BT.709 color gamut. - Region of Interest (ROI) Scalability: The enhancement layer represents a spatial subset of the base layer. ROI scalability can be used in conjunction with other types of scalability, such as quality or spatial scalability, so that the enhancement layer can provide higher subjective quality for the spatial subset. - View scalability, also known as multi-view encoding. In one example, the base layer represents the first view or the first camera, while the enhancement layer represents the second view or the second camera. In another example, the base layer represents the first set of views, which can be packed into frames, while the enhancement layer represents the second set of views. - Depth scalability, also known as depth-enhanced coding. One or more layers of a bitstream can represent texture views, while other layers can represent depth views.

[0256] In the scalability example above, base layer information can be used to encode enhancement layers to minimize additional bit rate overhead.

[0257] Scalability can be enabled in two exemplary ways: by introducing new coding schemes to perform pixel value or syntax prediction from lower layers of the scalable representation; or by placing lower-layer images into a reference image buffer (decoded image buffer (DPB)) at a higher layer. The first method (coding scheme) is more flexible and therefore provides better coding efficiency in most cases. Compared to the first method, reference frame-based scalability can be implemented more efficiently with minimal changes to the single-layer codec, while still achieving most of the available coding efficiency gains. For example, a reference frame-based scalable codec can be implemented using the same hardware or software for all layers, with DPB management easily managed externally.

[0258] An Output Layer Set (OLS) can be defined as a set of layers, where one or more layers in the set are designated as output layers. The output layer's image can be determined by the decoder, similar to determining whether an image is output in a single-layer bitstream as described earlier. The decoder does not output images that are in layers of the OLS but not in the output layers. When multiple OLSs are indicated for a bitstream, the decoder can be instructed, for example, via an interface, which OLS to use for decoding and output. OLSs can be indicated, for example, within a VPS.

[0259] In some video encoding formats, the encoder can specify the profile and level for the output layer set, such as in a VPS. The ability to associate the specified profile and level with the output layer set may be required for decoding. The output layer set may be subject to constraints defined for the specified profile and level.

[0260] The time sub-layer can be used for any type of scalability, as proposed in, for example, JVET-O1150, which is available at [https: / / www.jvet-experts.org / doc_end_user / documents / 15_Gothenburg / wg11 / JVET-O1150-v2.zip (last accessed January 3, 2024)]. For example, the mapping from scalability dimensions to sub-layer identifiers can be provided in the VPS or SEI message.

[0261] Additional enhancements regarding Neural Network Post-Filter Properties (NNPFC) and Neural Network Post-Filter Activation (NNPFA). SEI (Search Engine Information) message information

[0262] The NNPFC SEI message and the NNPFA SEI message have been described in version 3 of the General Supplemental Enhancement Information (VSEI) standard.

[0263] The syntax structure for specifying an NNPFC SEI message can be called nn_post_filter_characteristics. The syntax structure for specifying an NNPFA SEI message can be called nn_post_filter_activation.

[0264] The NNPFC SEI message includes an `nnpfc_id` syntax element, which contains an identifier that can be used to identify post-processing filters. The basic post-processing filter is a filter included in or identified by the first NNPFC SEI message in decoding order, and has a specific `nnpfc_id` value within the Coordinated Layer Video Sequence (CLVS). When a second NNPFC SEI message exists whose `nnpfc_id` value is the same as the one defining the basic post-processing filter, an update relative to the basic post-processing filter is applied to obtain the post-processing filter associated with the `nnpfc_id` value. This update can be obtained by decoding the encoded neural network bitstream in the second NNPFC SEI message. Otherwise, the post-processing filter associated with the `nnpfc_id` value is assigned the same as the basic post-processing filter.

[0265] The NNPFC SEI message includes the nnpfc_mode_idc syntax element, the semantics of which can be defined as follows:

[0266] An nnpfc_mode_idc value of 1 indicates that the base post-processing filter associated with the nnpfc_id value, or the update relative to that base post-processing filter, is a neural network identified by a Uniform Resource Identifier (URI) nnpfc_uri, the format of which is identified by the tag URI nnpfc_tag_uri.

[0267] The nnpfc_mode_idc value being 0 indicates that the SEI message includes an ISO / IEC 15938-17 bitstream that specifies a base post-processing filter with the same nnpfc_id value or an update relative to that base post-processing filter.

[0268] NNPFC SEI messages may also include: - The purpose of the post-processing filter may include, but is not limited to, one or more of the following: Improved visual quality; Chromaticity upsampling from 4:2:0 chroma format to 4:2:2 or 4:4:4 chroma format, or from 4:2:2 chroma format to 4:4:4 chroma format; Increase the width or height of the input image; Frame rate upsampling; Bit depth upsampling; or Coloring. - The format of the input tensor given as input to neural network inference - Formatting of the output tensors generated by neural network inference; and - A representation of the complexity of neural networks.

[0269] The NNPFC SEI message syntax includes nnpfc_base_flag. An nnpfc_base_flag equal to 1 specifies that the SEI message specifies the base NNPF. An nnpfc_base_flag equal to 0 specifies that the SEI message specifies an update relative to the base NNPF.

[0270] When nnpfc_base_flag equals 0, the following applies: - This SEI message defines an update relative to the previous base NNPF in decoding order, with the same nnpfc_id value. The updates are not cumulative; instead, each update is applied to the base NNPF, i.e., the NNPF specified in decoding order by the first NNPFC SEI message, which has a specific nnpfc_id value within the current CLVS. The NNPF defined by this SEI message is obtained by applying the update defined by this SEI message relative to the base NNPF with the same nnpfc_id value. - This SEI message relates to the currently decoded image and all subsequent decoded images in the current layer in output order, up to the end of the current CLVS, or up to (but not including) the decoded image that is in the current CLVS after the currently decoded image in output order and is associated with a subsequent NNPFC SEI message in decoding order (that NNPFC SEI message has nnpfc_base_flag equal to 0 and has a specific nnpfc_id value in the current CLVS), whichever is earlier.

[0271] The NNPFC SEI message syntax includes the syntax element nnpfc_num_input_pics_minus1. nnpfc_num_input_pics_minus1 plus 1 specifies the number of images used as input to NNPF. The variable numInputPics can be set to equal nnpfc_num_input_pics_minus1 + 1.

[0272] Frame rate upsampling filters can be interchangeably called image rate upsampling filters. Such filters generate or interpolate one or more images between a pair of images that serve as filter inputs. Frame rate upsampling filters can also be used where the number of input images can be greater than two. Such frame rate upsampling filters can generate images between more than one pair of input images. Frame rate upsampling filters can include neural networks, in which case the interpolated images between a pair of input images are generated through inference from the neural network. A frame rate upsampling filter can exist that extrapolates images before or after the input images(s), as an alternative or supplement to extrapolation between input images.

[0273] When the filtering objective includes frame rate upsampling, the NNPFC SEI message includes the nnpfc_interpolated_pics[i] syntax element for a value i in the range of 0 (inclusive) to nnpfc_num_input_pics_minus1 (exclusive). nnpfc_interpolated_pics[i] specifies the number of interpolated pictures generated by NNPF between the i-th picture used as input to NNPF and the (i+1)-th picture.

[0274] The NNPFC SEI message syntax can include an indication of how images not originating from the current bitstream are expected to be replaced in the input tensor; this can be referred to as `nnpfc_absent_input_pic_zero_flag`. `nnpfc_absent_input_pic_zero_flag` equal to 1 indicates that NNPF expects input images not present in the current bitstream to be represented by an array of samples with a value of 0. `nnpfc_absent_input_pic_flag` equal to 0 indicates that NNPF expects input images not present in the current bitstream to be represented by the closest input image in the current bitstream in output order.

[0275] The NNPFC SEI message syntax can include an indication, which may be referred to as nnpfc_auxiliary_inp_idc, indicating whether auxiliary input data exists in the NNPF input tensor in addition to the sample array(s) of the input images(s). A nnpfc_auxiliary_inp_idc greater than 0 indicates that auxiliary input data exists in the NNPF input tensor. Specific semantics can be specified for specific non-zero values ​​of nnpfc_auxiliary_inp_idc. A nnpfc_auxiliary_inp_idc equal to 0 indicates that auxiliary input data does not exist in the input tensor.

[0276] The NNPFA SEI message specifies a Neural Network Post-Processing Filter (NNPF) that can be used for post-processing filtering of the current image, or for post-processing filtering of the current image and one or more other images. The NNPFA SEI message includes the nnpfa_target_id syntax element, which indicates that a neural network post-processing filter with nnpfc_id equal to nnpfa_target_id can be used for post-processing filtering of the indicated persistence. The indicated persistence can be limited to the current image only (indicated by nnpfa_persistence_flag equal to 0). Alternatively, NNPF activations can be indicated as persistent by nnpfa_persistence_flag equal to 1, in which case the persistence of the NNPF activation can persist until the end of the current CLVS, or until the next image in the current layer (excluding that image) associated with an NNPFA SEI message having the same nnpfa_target_id as the current SEI message, in the output order.

[0277] The NNPFA SEI message syntax may include syntax elements indicating when the base post-processing filter or the latest post-processing filter is activated, where the latest post-processing filter is defined by the base post-processing filter that has been updated (if any) with the latest filter update applied. This syntax element may be referred to as `nnpfa_target_base_flag`. `nnpfa_target_base_flag` equal to 1 indicates that the target NNPF is the base NNPF, where `nnpfc_id` equals `nnpfa_target_id`. `nnpfa_target_base_flag` equal to 0 indicates that the target NNPF is the NNPF specified by the last NNPFC SEI message, where `nnpfc_id` equals `nnpfa_target_id`, which is decoded before the first VCL NAL unit of the current image and is not a duplicate of an NNPFC SEI message that includes the base NNPF.

[0278] The NNPFA SEI message syntax can include an indication of which images from the filtered images corresponding to the input images are output by the NNPF process. For the i-th input image filtered by NNPF, the NNPFA SEI message syntax can include the nnpfa_output_flag[i] syntax element, which specifies that the filtered image is not output by the NNPF process when it is equal to 0, and specifies that the filtered image is output by the NNPF process when it is equal to 1.

[0279] Regarding NNPFA SEI messages, two picture sets can be defined: nnpfcTargetPictures and nnpfaTargetPictures. nnpfcTargetPictures can be defined as the picture set belonging to the last NNPFC SEI message preceding the current NNPFASEI message in decoding order, where nnpfc_id equals nnpfa_target_id. nnpfaTargetPictures can be defined as the picture set of the target NNPF activated by the current NNPFA SEI message. For a compliant bitstream, it can be required that any pictures included in nnpfaTargetPictures should also be included in nnpfcTargetPictures.

[0280] The NNPF process involves performing NNPF inference on a given input image. NNPF inference can be performed block-by-block, filtering the entire image region. After NNPF inference, the NNPF generated images can be output in ascending index order, where all NNPF generated images interpolated by NNPF are output, and those NNPF generated images corresponding to any input image of NNPF are output according to the semantics specified in the NNPFA SEI message.

[0281] The general post-processing filtering procedure using NNPF can be described as follows. The input to this procedure is a bitstream BitstreamToFilter. The output of this procedure is a list of NNPF output images, ListNnpfOutputPics. First, BitstreamToFilter is decoded, and the list CroppedDecodedPictures is set as a list of cropped, decoded images obtained from decoding BitstreamToFilter, in output order. Second, for each cropped, decoded image in CroppedDecodedPictures (where one or more NNPFs are activated), the filtering procedure for one image is repeated in output order, as described below. The order of the images in ListNnpfOutputPics is consistent with the output order. It may be necessary that there should not be more than one image associated with any particular output time instance within ListNnpfOutputPics. When multiple NNPFs are activated for any particular image in CroppedDecodedPictures, and only one NNPF is allowed to be selected for application, the above constraints should apply regardless of which NNPF is selected for a particular image, although any NNPF can be selected.

[0282] The filtering process for an image using NNPF can be described as follows. The NNPF filtering process can be applied to each cropped, decoded image, referred to as the current image (i.e., the image in CroppedDecodedPictures), where one or more NNPFs are activated. When NNPF is applied to the current image, NNPF generates filtered and / or interpolated images by applying the NNPF process to the current image. When NNPF is applied to the current image, the images generated by NNPF through the application process are stored in the NNPF output tensor in the output order. When the applied NNPF is the last NNPF applied to the current image, the images generated by NNPF and output by the NNPF process are included in ListNnpfOutputPics in the same order as when the images were stored in the NNPF output tensor.

[0283] The use of NNPFC and NNPFA SEI messages for VVC is described in version 3 of the Multifunctional Video Coding (VVC) standard. It should be understood that NNPFC and NNPFA SEI messages can be used similarly for any other video coding specification.

[0284] When NNPFC and NNPFA SEI messages are used in VVC, the decoder selects an input image for NNPF. Input images can be selected in reverse output order, starting with the image for which NNPF was activated via the NNPFA SEI message. Input images can be indexed, starting with index 0, which is assigned to the image for which NNPF was activated via the NNPFA SEI message. In one example, the decoder selects the input image at index i (where i is greater than 0) as the latest cropped decoded output image preceding the input image at index i-1 in output order. When the result of decoding the bitstream is that there is no cropped decoded output image preceding the input image at index i-1 in the output order, the input image at index i can be considered not present in the current bitstream (e.g., lost), and subsequent input images from indices i+1 to numInputPics-1 (if any) are also lost. Lost input images can be handled as described above regarding the nnpfc_absent_input_pic_zero_flag syntax element.

[0285] When NNPFC and NNPFA SEI messages are used in VVC, and NNPF is continuously activated at the image rate upsampling between single pairs of input images until the end of the bitstream, NNPF is repeatedly applied at the end of the bitstream to different sets of input images, up to but not including the following sets of input images that would cause any interpolated images to be created after the last image of the bitstream in output order. In these sets of input images, some images may be lost and, for example, may be replaced with the last image of the bitstream in output order.

[0286] Visual time extrapolation

[0287] It should be understood that, in various embodiments, the terms visual temporal extrapolation, temporal extrapolation, and video prediction are used interchangeably. Visual temporal extrapolation can be defined as a method, algorithm, or process that, given one or more past images as input, generates one or more future images. Visual temporal extrapolation can be implemented through neural network inference, but is not necessarily based on or limited to neural network inference.

[0288] Use cases for visual time extrapolation

[0289] Use cases for visual temporal extrapolation include: - Extremely low latency computer vision in fields such as robotics and autonomous driving, where extrapolated future images help make predictive decisions. - In very low-latency applications (such as cloud gaming), the increase in the rate of rendering images relative to the rate of decoding images. - Reduce end-to-end latency in low-latency applications by extrapolating and displaying future images before they are received. - Generative facial videos for very low bitrate video encoding.

[0290] Information about generative AI, including temporal extrapolation and generative facial videos.

[0291] The terms generative artificial intelligence (AI), generative modeling, or generative machine learning (and other similar terms) are generally used to refer to a class of models that learn from data and are able to generate new data. Some example generative models are based on neural networks. The basic components or layers of a generative neural network are often indistinguishable from those of a non-generative neural network. Examples of such components include non-linear layers, fully connected layers, normalization layers, attention layers, etc.

[0292] A typical example of a neural network architecture that allows the generation of text data is a transformer-based "decoder," where the "decoder" doesn't necessarily refer to part of a codec that compresses the input data into a small bitstream. Instead, the decoder is a neural network that takes an input word set, a portion of an input word, or a token extracted from an input word, and outputs an output word set, a portion of a word, or a token. During inference, such a NN operates in an autoregressive mode, where the generated word(s) or lexical(s) are provided as part of the input word(s) or lexical(s). To enable such a NN to generate data, it is trained to predict the next word(s) (or an estimate of the probability distribution of the next word) given an input word set. The NN can be based on a transformer architecture that includes the use of a self-attention mechanism, where an attention score is assigned to each input lexical or word based on all other input lexical or word, including previously generated words or lexical(s). During the training of a decoder-like transformer architecture, future data items (words or lexical(s)) are masked to avoid revealing future information. In some cases, decoder-style transformer architectures are called "unidirectional" because they use or process information from left to right, while in contrast, some encoder-style transformer architectures are called "bidirectional" (because they use or process information from left to right and from right to left).

[0293] Another example of generative modeling is visual temporal extrapolation, where an image is generated by a neural network based on one or more previously decoded or generated images and one or more other data items. The one or more previously decoded or generated images can be images decoded without involving a generative modeling process, such as traditional codecs, like VVC-compliant codecs. The one or more data items can include parameters or features describing the differences between the one or more previously decoded images and the current image to be temporally extrapolated. Examples of such parameters are facial parameters (such as facial landmarks and their position or difference relative to the previous image) or parameters of other objects. The one or more data items can be signaled from the encoder to the decoder.

[0294] Figure 6A general block diagram of a generative face video codec (GFVC) is illustrated. An input base image 602 is encoded into a bitstream 606 by any video encoder 604 (such as a VVC encoder). Subsequent input images (e.g., subsequent images 608-1, 608-2) are fed into an analysis model 610 to extract feature parameters. The feature parameters are encoded 612 to generate a feature bitstream 614. The feature parameters can be encoded independently of the feature parameters of any other image, e.g., for the first subsequent image following the base image in decoding order. Alternatively, the feature parameters can be encoded predictively 612 with reference to earlier encoded feature parameters in decoding order, where feature residuals can be encoded relative to the predicted feature parameters. The feature residuals can be quantized and entropy-encoded as part of the feature encoding. The encoded feature parameters are included in the feature bitstream 614 or along the feature bitstream, e.g., in the generative face video SEI message. For subsequent images, the video encoder can encode virtual or driving images, as described in later sections. Encoded feature parameters of subsequent images can be included in the corresponding encoded virtual image or driving image, or included along the corresponding encoded virtual image or driving image, for example, in the same image unit.

[0295] Decoder 616 decodes the encoded base image from bitstream 606. For subsequent encoded virtual or driving images, feature decoder 618 decodes the encoded feature parameters. The decoded feature parameters, the decoded base image, and optionally the decoded driving image are input into generative neural network model inference 620, which generates an output image sequence 622.

[0296] Information from GFV SEI messages

[0297] In JVET, a generative face video (GFV) SEI message has been proposed for the new version of the General Supplemental Enhancement Information (VSEI) standard to support visual temporal extrapolation of faces in videos. One of the main functions of the GFV SEI message is to signal face parameters that represent auxiliary inputs to the neural network performing the temporal extrapolation. The most recent document describing the GFV SEI message is JVET-AF0234, which is available at [https: / / jvet-experts.org / doc_end_user / documents / 32_Hannover / wg11 / JVET-AF0234-v2.zip, last accessed on January 3, 2024](https: / / jvet-experts.org / doc_end_user / documents / 32_Hannover / wg11 / JVET-AF0234-v2.zip, last accessed on January 3, 2024). An example overview is provided below.

[0298] The GFV SEI message defines the interface between the video decoder and the generator NN, which performs visual temporal extrapolation of faces in the video.

[0299] The following are the image types considered in GFV SEI messages: - Base Image: The decoded output image that generative networks can use to generate new facial images. It is encoded by a traditional codec such as a VVC-compatible codec. - Virtual image: Only the image unit with the minimum allowed resolution of the SEI message is included. - (Optional) Master image (also known as driving image), which may be optionally fed into the generative neural network to improve background texture and / or facial details. This driving image may also be encoded by a conventional codec, such as a VVC-compatible codec.

[0300] The second neural network (called the converter NN) is used to convert the facial parameters (signaled in the GFV SEI message) into the following conversion parameters: - 15 3D key points - 3×3 matrix - A 1×3 matrix.

[0301] The input to the generator NN can be one or more of the following: - (Previously decoded) base image - Transformed parameters (output of the converter NN) - (Optional) Master / Driver Image.

[0302] The Generative Face Video (GFV) SEI message proposed in JVET-AF0234 uses dummy images, such as images with the minimum allowed resolution, which are only used to include the GFV SEI message. These dummy images are allegedly redundant and cause unnecessary bitrate overhead.

[0303] Traditional decoders will display the driving image and the virtual image as presented in JVET-AF0234, which is annoying to human observers because they are not intended for display and may not represent image content similar to the underlying image.

[0304] Some argue that generative video SEI messages (such as GFV SEI messages) cannot achieve interoperability between different implementations because support for SEI messages in the decoder may be optional, and the generative neural network associated with generative video SEI messages may not be specified in a precise manner.

[0305] Some embodiments are described with respect to generative video SEI messages. The generative face video (GFV) SEI message described above is an example of a generative video SEI message. A generative video SEI message can be defined as an SEI message that may include syntax elements from which input(s) for generative neural network inference can be derived, and / or the message may induce a call to generative neural network inference. It should be understood that the embodiments are not limited to GFV SEI messages, but are applicable to any generative video SEI message.

[0306] Some of the embodiments described herein avoid encoding virtual images and make the decoded output of conventional decoders (e.g., those that cannot run generative neural networks) more suitable for display.

[0307] In some embodiments, generative neural networks are integrated into scalable video coding.

[0308] Multiple generative face video SEI messages are enabled within a single image unit, and each generative face video SEI message... Inference of generative neural networks using information calls

[0309] This section describes an embodiment for avoiding the encoding of virtual images. Instead of encoding the virtual image containing the GFV SEI message, the encoder encodes a suffix SEI NAL unit that includes the GFV SEI message in the image unit, which includes the base or driving image referenced by the GFV SEI message. Multiple GFV SEI messages with different payloads are allowed within the same image unit. Each new instance of the GFV SEI message is intended to invoke generative neural network (NN) inference.

[0310] Figure 7 The illustration depicts an image unit generated by an encoder according to the embodiments described in this section. Image unit 700 includes an encoded image 702, and the encoded image includes at least one encoded slice NAL unit, such as 704-1 and 704-2. Image unit 700 also includes at least one suffix SEI NAL unit, such as 706-1 and 706-2. Each suffix SEI NAL unit includes a generative video SEI message.

[0311] Including the suffix SEI NAL unit of the image unit containing the encoded base or driving image, the generative video SEI is eliminated. Information encoding

[0312] In one embodiment, an encoder: - Receive the first input image; - Encode the first input image into a first encoded image, wherein the first encoded image may be an encoded base image or an encoded driving image, and the first image unit includes the first encoded image; - Receive the second input image; - Extract feature parameters from the second input image; - Encode the feature parameters into encoded feature parameters; and - Include the encoded feature parameters in the first image unit.

[0313] In another embodiment, the encoder: - Encoded feature parameters are included in the Generative Video Supplemental Enhancement Information (SEI) message, which is generated after the first encoded image in the decoding order.

[0314] In another embodiment, the encoder: - Include the generative video SEI message in a suffix SEI Network Abstraction Layer (NAL) unit within the first image unit.

[0315] Generate in the prefix SIE NAL unit of the image unit following the encoded base or driving image, according to the decoding order. Encoding of SEI messages in video format

[0316] In one embodiment, an encoder: - Receive the first input image; - Encode the first input image into a first encoded image, wherein the first encoded image may be an encoded base image or an encoded driving image; - Receive the second input image; - Extract feature parameters from the second input image; - Encode the feature parameters into encoded feature parameters; and - Encoded feature parameters are included in subsequent picture units, wherein the subsequent picture unit is intended to include a subsequent encoded picture, which may be a second encoded driving picture or a second encoded base picture.

[0317] In another embodiment, the encoder: - Encoded feature parameters are included in the Generative Video Supplemental Enhancement Information (SEI) message, which is placed before subsequent encoded images in the decoding order.

[0318] In another embodiment, the encoder: - Include the generative video SEI message in the prefix SEI Network Abstraction Layer (NAL) unit within the subsequent image unit.

[0319] Calling Generative Neural Network Inference

[0320] In one embodiment, when the encoded feature parameters are decoded, they are intended to invoke the inference of the generative neural network.

[0321] In one embodiment, the inference of the generative neural network is performed on the decoder side, such as by signaling with generative video SEI messages (e.g., GFV SEI messages). When there are multiple generative video SEI messages (such as multiple GFV SEI messages) with different content in the same image unit, the inference of the generative neural network can be performed separately for each of these generative video SEI messages.

[0322] In one embodiment, the encoder also includes activation signals to invoke generative neural network inference along encoded feature parameters. For example, the encoder may include neural network post-filter activation (NNPFA) SEI messages immediately after the generative video SEI messages in decoding order to invoke generative neural network inference.

[0323] In one embodiment, the encoder further includes an activation signal to identify the generative neural network to be used for inference, which is invoked in other ways, such as through a generative video SEI message. For example, the encoder may encode a Neural Network Post-Filter Activation (NNPFA) SEI message preceding the generative video SEI message in decoding order to invoke inference of the generative neural network. The NNPFA SEI message identifies the NNPF (e.g., whether a base NNPF or an updated NNPF is in use given an nnpfa_target_id) and does not represent inference. Inference for the identified NNPF is performed on the decoder side, as signaled by the generative video SEI message(s).

[0324] An example of a generative video SEI message

[0325] In one embodiment, the encoder includes an identifier or counter value for each generative video SEI message, wherein the identifier or counter value identifies the generative video SEI from other generative video SEI messages in the same picture unit. When the identifier or counter value is the same as a previous generative video SEI message in the same picture unit, the current generative video SEI message can be inferred to be a copy of the previous generative video SEI message.

[0326] Example Implementations Related to Generative Face Video (GFV) SEI Messages

[0327] In an example embodiment, the encoder generates multiple GFV SEI messages with different payloads within the same picture unit. Each new instance of the GFV SEI message invokes generative neural network (NN) inference. Instead of using a virtual picture that includes the GFV SEI message, the encoder uses a suffix SEI NAL unit that includes the GFV SEI message in the picture unit, which includes a base or driving picture referenced by the GFV SEI message.

[0328] In another embodiment, the encoder adds a counter syntax element (gfv_cnt) to the GFV SEI message. Using the counter syntax element, duplicate and new instances of the GFV SEI message within a picture unit can be distinguished.

[0329] In the first example embodiment, the counter syntax element can use the following syntax and semantics:

[0330] gfv_cnt specifies the GFV SEI message instance count value for the gfv_id value within the image unit.

[0331] In one example, the value of gfv_cnt can be constrained as follows: Within a picture unit, the first GFV SEI message with a specific gfv_id value should have a gfv_cnt equal to 0 in decoding order. When the gfv_cnt assigned to currGfvCnt is greater than 0, GFV SEI messages with the same gfv_id value and a gfv_cnt equal to currGfvCnt-1 should precede the current GFV SEI message in decoding order within the same picture unit. The value of gfv_cnt should be in the range of 0 to 65,535, inclusive.

[0332] It may be necessary that when multiple GFVSEI messages with the same gfv_id and gfv_cnt values ​​exist in the same image unit, the SEI messages should have the same SEI payload content.

[0333] Generative neural network inference can be called in ascending order of gfv_cnt to generate video images based on each GFV SEI message, where gfv_base_pic_flag is equal to 0 and gfv_cnt has a unique value within the image unit.

[0334] In one embodiment, `gfv_base_pic_flag` equal to 1 indicates that the currently decoded output image corresponds to a base image. `gfv_base_pic_flag` equal to 0 indicates that the currently decoded output image does not correspond to a base image, or that the SEI message does not specify syntax elements for a base image. When `gfv_cnt` is greater than 0, it may be necessary for `gfv_base_pic_flag` to be equal to 0.

[0335] In another embodiment, gfv_base_pic_flag exists only when gfv_cnt equals 0, as shown in the following syntax. When gfv_base_pic_flag does not exist, it is inferred that it equals 0.

[0336] In the second example embodiment, the counter syntax element can use the following syntax and semantics:

[0337] gfv_cnt specifies the GFV SEI message instance count value for the gfv_id value within the image unit.

[0338] In one example, the value of gfv_cnt can be constrained as follows: the gfv_cnt of the first GFV SEI message in decoding order within a picture unit, having a specific gfv_id value and gfv_base_pic_flag equal to 0, should be equal to 0. When the gfv_cnt assigned to currGfvCnt is greater than 0, GFV SEI messages with the same gfv_id value and gfv_cnt equal to currGfvCnt-1 should precede the current GFV SEI message in decoding order within the same picture unit. The value of gfv_cnt should be in the range of 0 to 65,535, inclusive.

[0339] It may be necessary that when multiple GFVSEI messages with the same gfv_id and gfv_cnt values ​​exist in the same image unit, the SEI messages should have the same SEI payload content.

[0340] Generative neural network inference can be called in ascending order of gfv_cnt to generate video images based on each GFV SEI message, where gfv_base_pic_flag is equal to 0 and gfv_cnt has a unique value within the image unit.

[0341] Decoding generative video from image units with the suffix SEI NAL, which includes encoded base or driving images. SEI news

[0342] In one embodiment, a decoder: - Receive a first encoded image from the first image unit, wherein the first encoded image may be an encoded base image or an encoded driving image; - Decode the first encoded image into a first decoded image; - Receive encoded feature parameters from the first image unit; - Create a feature input tensor based on the encoded feature parameters; and - Use the first decoded image and the feature input tensor as input to invoke generative neural network inference to generate an image.

[0343] In one embodiment, the decoder is characterized by receiving fewer than one image unit to be passed to the interface for decoding, and the decoding of the first encoded image is invoked before receiving or creating the feature input tensor.

[0344] Decode from the prefix SIE NAL unit of the image unit following the encoded base or driving image in decoding order. Generative video SEI messages

[0345] In one embodiment, a decoder: - Receive a first encoded image, wherein the first encoded image may be an encoded base image or an encoded driving image; - Decode the first encoded image into a first decoded image; - Receive encoded feature parameters from subsequent image units; - Create a feature input tensor based on the encoded feature parameters; and - Use the first decoded image and the feature input tensor as input to invoke generative neural network inference to generate an image.

[0346] In one embodiment, the decoder is characterized by receiving fewer than one picture unit to be passed as an interface for decoding, and the creation of the feature input tensor is performed before receiving or decoding subsequent encoded pictures from subsequent picture units.

[0347] A method based on multiple layers and(multiple) generative video SEI messages

[0348] This section describes an implementation where the driver and virtual image reside in a dependency layer. Therefore, a conventional decoder can decode only the independent layer and avoid accidentally displaying the driver and virtual image.

[0349] The base image resides in an independent layer. When an access unit has a base image in an independent layer, it can also have an image in a dependent layer that is predicted solely based on the independent layer and has no residuals. This image in the dependent layer can be referred to as the copy image because the decoded base image and the decoded copy image are typically identical. The copy image can be used as a reference for inter-frame prediction within the dependent layer.

[0350] The driving images can be labeled as output images (e.g., pic_output_flag equals 1 in VVC) because they are needed as input in the generative NN that is performed as post-processing.

[0351] Virtual images can be predicted based on previous images in the decoding order within a dependency layer, can be encoded without prediction residuals, and can be marked as output images (e.g., ph_pic_output_flag equals 1 in VVC). Alternatively, virtual images can be encoded in any way that results in low bit-rate overhead, regardless of the content of the decoded virtual images, and can be marked as not being output (e.g., ph_pic_output_flag equals 0 in VVC).

[0352] Any dependent layer image unit, including virtual or driving images, includes generative video SEI messages (such as GFVSEI messages).

[0353] Traditional decoders (such as the VVC Main 10 profile decoder) will only decode individual layers, such as displaying only the base image.

[0354] Figure 8 The illustration shows a bitstream generated by an encoder according to the embodiments described in this section. NAL units (e.g., encoded sliced ​​NAL units or non-VCL NAL units) belong to a layer and are contained in boxes 804, 806 with rounded corners. Figure 8 This includes an independent layer 804 and a dependent layer 806. The decoding order of the access units (e.g., the first access unit 802 and the second access unit 803) is from left to right. Within an access unit, the independent layer 804 is decoded before the dependent layer 806. Within picture units (e.g., the second picture unit 808) and encoded pictures (e.g., encoded picture 810, the first encoded picture 812, and the second encoded picture 814), the NAL units are decoded from top to bottom.

[0355] In one embodiment, an encoder: - Receive the first input image; - Encode the first input image into a first encoded image, wherein the first encoded image is an encoded base image, the first encoded image resides in an independent layer, and the first access unit includes the first encoded image; - Receive the second input image; - Encode a second encoded image, wherein the second encoded image may be an encoded virtual image or an encoded driving image based on a second input image, the second encoded image resides in the dependency layer, the second image unit includes the second encoded image, and the second access unit includes the second encoded image; - Extract feature parameters from the second input image; - Encode the feature parameters into encoded feature parameters; and - Include the encoded feature parameters in the second image unit.

[0356] In one embodiment, the encoder encodes the encoded copy image in a first access unit, wherein the encoded copy image resides in a dependent layer, and is characterized by being predicted only according to an independent layer and without prediction residuals.

[0357] In one embodiment, encoded feature parameters are included in the generative video supplementation and enhancement information (SEI) message in the second picture unit.

[0358] In one example, the decoder may only be able to decode individual layers, and therefore only the encoded underlying image.

[0359] In one embodiment, the encoder predicts the encoded virtual image based on the previous images in the decoding order within the dependency layer.

[0360] A multi-layered approach eliminates the need for (multiple) generative video SEI messages.

[0361] This section describes an embodiment where decoding of an encoded image in a dependency layer requires generative neural network inference. Decoding of the dependency layer is indicated to be possible under a profile that includes generative neural network capabilities, so conventional decoders will not attempt to decode the dependency layer.

[0362] In the embodiments described in this section, a reference image is conventionally indicated. Therefore, inter-layer prediction is performed between images within the same access unit.

[0363] The independent layer contains a base image, a virtual image, and a driver image.

[0364] When an independent layer has a base image, the same access unit has an inter-layer image predicted based on the base image in the dependent layer, and can have zero residuals, and both the independent layer image and the dependent layer image are marked as output (e.g., ph_pic_output_flag equals 1 according to VVC).

[0365] The driving images for independent layers are marked as not to be output (e.g., ph_pic_output_flag equals 0 in VVC) because they are only used to carry content that will be used as input to the generative NN, rather than being displayed directly.

[0366] Virtual images in independent layers are marked as not to be output (e.g., according to VVC, ph_pic_output_flag equals 0).

[0367] When an independent layer has a virtual or driving image, the same access unit includes an image in the dependent layer that uses exactly two images as references, i.e., an inter-layer reference image, which can be a virtual or driving image, and a base image of the dependent layer, which can be indicated as a short-term or long-term reference image.

[0368] In some embodiments, the decoding process of the dependency layer image includes running inference of a generative neural network.

[0369] In some embodiments, the output layer set, including both dependent and independent layers, is indicated to conform to a new profile, which may be referred to as Scalable Generative Master 10. The conventional decoder is unaware of the new profile and therefore will not attempt to decode either layer. In this output layer set, only enhancement layers can be output layers.

[0370] In some embodiments, the output layer set consisting only of independent layers is indicated to conform to an existing profile, such as Master 10. Conventional decoders can be able to decode independent layers and display the base image, but not the virtual or driving image, which is a desirable behavior.

[0371] Figure 9 The illustration shows a bitstream generated by an encoder according to the embodiments described in this section. Arrows 902, 904, and 906 indicate inter-frame prediction, wherein the horizontal arrow(s)906 indicates temporal inter-frame prediction, and the vertical arrows 902 and 904 indicate inter-layer prediction. The arrows point to the images being predicted, such as the first encoded DL image 908 and the second encoded DL image 910, and the arrow sources indicate images used as prediction references, such as the first encoded BL image (basic) 912 and the second encoded BL image (basic) 914.

[0372] In one embodiment, an encoder: - Receive the first input image; - Encode the first input image into a first encoded base layer image, wherein the first encoded base layer image is an encoded base image, the first encoded base layer image resides in an independent layer, and the first access unit includes the first encoded base layer image; - Encode a first encoded dependency layer image, wherein the first encoded dependency layer image resides in a dependency layer, and the first access unit includes the first encoded dependency layer image; - Receive the second input image; - Encode a second encoded base layer image, wherein the second encoded base layer image may be an encoded virtual image or an encoded driving image based on a second input image, the second encoded base layer image resides in an independent layer, and the second access unit includes the second encoded base layer image; - Encode a second encoded dependency layer image, wherein the second encoded dependency layer image resides in a dependency layer, and the second access unit includes the second encoded dependency layer image; - In the second encoded dependency layer image or along the second encoded dependency layer image, it is indicated that the second encoded dependency layer image can use the first encoded dependency layer image and the second encoded base layer image as references for prediction; - Extract feature parameters from the second input image; - Encode the feature parameters into encoded feature parameters; and - Indicates the encoded feature parameters used for decoding the second encoded dependency layer image.

[0373] In one embodiment, the encoder encodes the first dependent layer image into an encoded copy image, wherein the encoded copy image is characterized by being predicted only based on the independent layer and having no prediction residual.

[0374] Indicates output status

[0375] In one embodiment, the encoder encodes the indication of the omitted output in or along the second encoded base layer image.

[0376] In one embodiment, the encoder instructs the first encoded base layer image to be output.

[0377] In one embodiment, the encoder instructs the first encoded dependency layer image to be output.

[0378] In one embodiment, the encoder instructs the first encoded dependency layer image and the second encoded dependency layer image to be output.

[0379] In one embodiment, the encoder instructs independent layers and dependent layers to form an output layer set, and the dependent layer is the output layer of the output layer set.

[0380] In one embodiment, the encoder indicates a profile of the output layer set, wherein the profile includes generative neural network capabilities.

[0381] Derive the feature input tensor

[0382] In one embodiment, the encoder creates a feature input tensor based on encoded feature parameters.

[0383] Alternatives for indicating encoded feature parameters or feature input tensors

[0384] In one embodiment, the encoder includes encoded feature parameters or feature input tensors in an adaptive parameter set that indicates the type of generative video, such as generative facial video.

[0385] In one embodiment, the encoder includes encoded feature parameters or feature input tensors in the image header of the second encoded dependency layer image. For example, the image header extension of the VVC image header syntax may include encoded feature parameters.

[0386] Generative neural networks are used to generate decoded dependency layer images, and irrelevant images generated during reconstruction are excluded. Syntax elements

[0387] In one embodiment, the encoder: - Reconstruct the first decoded dependency layer image from the first encoded dependency layer image; and - The second decoded dependency layer image is reconstructed by invoking generative neural network inference to generate an image using the first decoded dependency layer image and feature input tensor as input.

[0388] In one embodiment, the syntax exclusion of the second encoded dependency layer image is used to reconstruct irrelevant syntax elements of the second decoded dependency layer image using a generative neural network.

[0389] In one embodiment, the syntax of one or more syntactic structures of the second dependency layer excludes syntactic elements irrelevant to the reconstruction of the second decoded dependency layer image using a generative neural network. For example, the syntax of the dependency layer's parameter set, image header, slice header, and / or slice data syntactic structures can exclude syntactic elements irrelevant to the reconstruction of the second decoded dependency layer image using a generative neural network.

[0390] Reconstructing prediction units using generative neural networks

[0391] In one embodiment, the encoder: - Reconstruct the first decoded dependency layer image from the first encoded dependency layer image; - By invoking generative neural network inference with the first decoded dependency layer image and feature input tensor as input, one or more prediction units of the second decoded dependency layer image are reconstructed; and - Encode the difference between one or more prediction units and co-located units of the second input image as the prediction residual of the second encoded dependency layer image.

[0392] Generative neural networks are used to generate interlayer reference images.

[0393] In one embodiment, the encoder: - Reconstruct the first decoded dependency layer image from the first encoded dependency layer image; - Using the first decoded dependency layer image and the feature input tensor as input, a generative neural network inference is invoked to generate inter-layer reference images; and - Reference images between reference layers, encoding the second already encoded dependent layer image.

[0394] In one embodiment, the encoder: - Encode the difference between the second input image and the interlayer reference image into the prediction residual of the second encoded dependent layer image.

[0395] decoding

[0396] In one embodiment, a decoder: - Receive a first encoded base layer image, wherein the first encoded base layer image is an encoded base image, the first encoded base layer image resides in an independent layer, and the first access unit includes the first encoded base layer image; - Decode the first encoded base layer image into a first decoded base layer image; - Receive a first encoded dependency layer image, wherein the first encoded dependency layer image resides in a dependency layer, and the first access unit includes the first encoded dependency layer image; - Decode the first encoded dependency layer image into a first decoded dependency layer image; - Receive a second encoded base layer image, wherein the second encoded base layer image may be an encoded virtual image or an encoded driving image based on the second input image, the second encoded base layer image resides in an independent layer, and the second access unit includes the second encoded base layer image; - In response that the second encoded base layer image is an encoded driving image, decode the second encoded base layer image into a second decoded base layer image; - Receive a second encoded dependency layer image, wherein the second encoded dependency layer image resides in a dependency layer, and the second access unit includes the second encoded dependency layer image; - Decoded from or along the second encoded dependency layer image: The second encoded dependency layer image can be used as a reference for prediction using the first encoded dependency layer image and the second encoded base layer image; - Obtain the feature input tensor for decoding the second encoded dependency layer image; and - The second decoded dependency layer image is reconstructed by invoking generative neural network inference to generate an image using the first decoded dependency layer image and feature input tensor as input.

[0397] It should be noted that the corresponding encoder embodiment is applicable to most decoder embodiments.

[0398] A method based on multi-layer and diagonal inter-layer prediction eliminates the need for (multiple) generative video SEI messages.

[0399] This section describes an embodiment where decoding of the encoded image in the dependency layer requires generative neural network inference. Decoding of the dependency layer is indicated to be possible under a profile that includes generative neural network capabilities, therefore conventional decoders will not attempt to decode the dependency layer.

[0400] In the embodiments described in this section, inter-layer prediction can be performed "diagonally," for example, starting with the base or driving picture present in a first access unit that precedes the second access unit, which includes the picture of the currently dependent layer, in the decoding order.

[0401] The base image and the driving image exist in a separate layer.

[0402] The base image is marked as to be output (e.g., according to VVC, ph_pic_output_flag equals 1).

[0403] The driving images are marked as not to be output (e.g., ph_pic_output_flag equals 0 according to VVC) because they are only used to carry content that will be used as input to the generative NN, rather than being displayed directly.

[0404] The dependency layer comprises images at each time point where the decoder runs the generative neural network. In some embodiments, the dependency layer images are dummy images, e.g., they do not include residual information associated with the inter-layer reference images. In some embodiments, the dependency layer images may include residuals relative to the inter-layer reference images.

[0405] In some embodiments, instead of using traditional interlayer reference images (e.g., upsampled independent layer images), interlayer reference images are generated by a generative neural network, i.e., generative neural network inference is treated as an interlayer process.

[0406] Various embodiments are proposed to indicate the base image and / or driving image used as input to a generative neural network.

[0407] Figure 10 An example bitstream generated by an encoder according to some embodiments described in this section is illustrated. A first encoded image 1002 may be a base image, and optionally a third encoded image 1004 may be present as a driving image. In some embodiments, the base image 1002, or both the base image 1002 and the driving image 1004, are used as input to a generative neural network to generate interlayer reference images. In the illustration, the driving image 1004 is present in a second access unit 1006, but it can typically be present in any access unit, as long as it precedes the second encoded image 1008 in the decoding order.

[0408] In one embodiment, an encoder: - Receive the first input image; - Encode the first input image into a first encoded image, wherein the first encoded image is an encoded base image or an encoded driver image, the first encoded image resides in an independent layer, and the first access unit includes the first encoded image; - Receive the second input image; - Extract feature parameters from the second input image; - Encode the feature parameters into encoded feature parameters; - Encode a second encoded image, wherein the second encoded image resides in the dependency layer, and the second access unit includes the second encoded image; and - Indicates the encoded feature parameters used for decoding the second encoded image.

[0409] In one embodiment, the encoder: - Reconstruct the first decoded image from the first encoded image; - Create a feature input tensor based on the encoded feature parameters; - Using the first decoded image and the feature input tensor as input, invoke generative neural network inference to generate the first inter-layer reference image; and - Refer to the first layer reference image and encode the second already encoded image.

[0410] In one embodiment, the encoder: - The encoding of the prediction residual for the second encoded image is omitted.

[0411] In one embodiment, the encoder: - Encode the difference between the second input image and the first inter-layer reference image into the prediction residual of the second encoded image.

[0412] Indicates output status

[0413] In one embodiment, in response to the first encoded image being an encoded base image, the encoder indicates that the first encoded image is the output.

[0414] In one embodiment, in response to the first encoded image being an encoded driving image, the encoder indicates that the first encoded image is not the output.

[0415] In one embodiment, the encoder instructs independent layers and dependent layers to form an output layer set, and the dependent layer is the output layer of the output layer set.

[0416] In one embodiment, the encoder instructs a second encoded image to be output.

[0417] Indicates when an independent layer image is the base image or driving image used for decoding dependent layers.

[0418] In one embodiment, the encoder: - In or along the first encoded image, an indication is given that: the first encoded image is an encoded base image or the first encoded image is an encoded driving image.

[0419] In one embodiment, the encoder uses additional bits in the image header to indicate whether the first encoded image is an encoded base image or an encoded driving image. For example, the image header syntax may conform to VVC.

[0420] Indicates interlayer reference images

[0421] In one embodiment, the encoder: - Instructs the second encoded image to use inter-layer predictions from independent layers.

[0422] In one embodiment, the encoder: - Inferring interlayer predictions from independent layers using the first interlayer reference image.

[0423] In one embodiment, the encoder: - Receive a third input image; and - Encode the third input image into a third encoded image, wherein the third encoded image is an encoded driving image, the third encoded image resides in an independent layer, and the third encoded image is decoded after the first encoded image and before the second encoded image in the decoding order.

[0424] In one embodiment, in response to the third encoded image residing in the second access unit, the encoder omits an indication that the third encoded image is an encoded driving image, and infers that the third encoded image is an encoded driving image.

[0425] In one embodiment, the encoder: - In the third encoded image or along the third encoded image, an indication is given that the third encoded image is an encoded driving image.

[0426] In one embodiment, the encoder uses extra bits in the image header to indicate that the third encoded image is an encoded driving image. For example, the image header syntax may conform to VVC.

[0427] In one embodiment: - The encoder reconstructs a third decoded image from a third encoded image; and - The third decoded image is used as additional input to invoke generative neural network inference.

[0428] In one embodiment, the encoder: - Infer when interlayer predictions from independent layers refer to the first interlayer reference image or the third decoded image.

[0429] In an example embodiment using VVC syntax, the first occurrence of a specific ilrp_idx value in the ref_pic_list_struct references an interlayer reference picture (i.e., an interlayer reference picture generated with a previously decoded base picture as input), and when a second occurrence of the same ilrp_idx value exists in the same ref_pic_list_struct, it references a third decoded picture (i.e., a previously decoded driving picture). In some embodiments, the third decoded picture may need to reside in the second access unit.

[0430] In the section “Multi-layered approach without multiple generative video SEI messages”, the embodiments related to the syntax including encoded feature parameters and / or feature input tensors are also applicable to the embodiments in this section.

[0431] In one embodiment, the encoder: - The reference image list indicates syntax structure information, which indicates that the first encoded image was used as input to generate interlayer reference images.

[0432] In one embodiment, the information indicating that the first encoded image is used as input to generate an interlayer reference image is the image order count difference between the current image and the first encoded image.

[0433] In one embodiment, the reference image list syntax structure of VVC is expanded as follows, where sps_diagonal_inter_layer_pred_enabled_flag may exist in the SPS extension and is inferred to be equal to 0 when the SPS extension does not exist:

[0434] abs_delta_poc_ilrp[listIdx][rplsIdx][i] specifies the absolute picture order count (POC) difference between the POC of the picture in which the reference picture list is applied and the POC of the indicated interlayer reference picture.

[0435] `delta_poc_ilrp_sign_flag[listIdx][rplsIdx][i]` equal to 1 indicates that the POC of the indicated interlayer reference image is greater than the POC of the images in which the reference image list is applied. `delta_poc_ilrp_sign_flag[listIdx][rplsIdx][i]` equal to 0 indicates that the POC of the indicated interlayer reference image is less than the POC of the images in which the reference image list is applied.

[0436] Figure 11 This is an example device, which can be implemented in hardware, and is configured to perform the integration of generative neural networks in multimedia encoding. Device 1100 includes at least one processor 1102 (e.g., an FPGA and / or a CPU), and one or more memories 1104 including computer program code 1105 having instructions for performing the methods described herein, wherein at least one memory 1104 and computer program code 1105 are configured, together with at least one processor 1102, to cause device 1100 to implement circuitry, processes, components, modules, or functions (implemented by control module 1106) to implement the examples described herein, including the integration of generative neural networks in multimedia encoding. Optionally, encoder 1180 of control module 1106 performs encoding, and optionally, decoder 1190 performs decoding. Memory 1104 may be non-transitory memory, transient memory, volatile memory (e.g., RAM), or non-volatile memory (e.g., ROM).

[0437] Device 1100 includes a display and / or I / O interface 1108, which includes user interface (UI) circuitry and components. This display and / or I / O interface can be used to display features or states of the methods described herein (e.g., while one of these methods is being performed or at a subsequent time), or to receive input from a user, such as using a keypad, camera, touchscreen, touch area, microphone, biometrics, one or more sensors, etc. Device 1100 includes one or more communication (e.g., network (N / W)) interfaces (I / F) 1110. The communication I / Fs 1110 can be wired and / or wireless and communicate over the Internet / (multiple) other networks via any communication technology, including via one or more links 1124. The communication I / Fs 1110 can include one or more transmitters or one or more receivers.

[0438] Transceiver 1116 includes one or more transmitters 1118 and one or more receivers 1120. Transceiver 1116 and / or (multiple) communication I / F 1110 may include standard, well-known components such as amplifiers, filters, frequency converters, modulators (demodulators), and encoder / decoder circuitry, as well as one or more antennas, such as antenna 1114 for communication via wireless link 1124.

[0439] The control module 1106 of device 1100 includes one or both of components 1106-1 and / or 1106-2, which can be implemented in various ways. Control module 1106 can be implemented in hardware as control module 1106-1, such as being implemented as part of one or more processors 1102. Control module 1106-1 can also be implemented as an integrated circuit or by other hardware such as a programmable gate array. In another example, control module 1106 can be implemented as control module 1106-2, which is implemented as computer program code (with corresponding instructions) 1105 and executed by one or more processors 1102. For example, one or more memories 1104 store instructions that, when executed by one or more processors 1102, cause device 1100 to perform one or more operations as described herein. Furthermore, one or more processors 1102, one or more memories 1104, and example algorithms encoded as instructions, programs, or code (e.g., as flowcharts and / or signaling diagrams) are components for performing the operations described herein.

[0440] The means 1100 used to implement the functions of the control module 1106 may correspond to any means described herein. Alternatively, the means 1100 and its elements may not correspond to any other means described herein, as the means 1100 may be part of a self-organizing / optimized network (SON) node or other node, such as a node in the cloud.

[0441] Device 1100 can also be distributed throughout the network (e.g., Internet 28), including within and between device 1100 and any network elements (such as base station 24 and / or device 90).

[0442] like Figure 11As shown, interface 1112 enables data communication and signaling between the various components of device 1100. For example, interface 1112 may be one or more buses, such as address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optic cables, or other optical communication devices. Computer program code (e.g., instructions) 1105, including control 1106, may include object-oriented software configured to pass data or messages between objects within computer program code 1105. Device 1100 does not need to include each of the features described above, or may include other features. The various components of device 1100 may be at least partially located in a common housing 1128, or a subset of the various components of device 1100 may be at least partially located in different housings, which may include housing 1128.

[0443] Figure 12 A schematic diagram is shown of a non-volatile storage medium 1200a (e.g., a computer / optical disc (CD) or digital universal optical disc (DVD)) and 1200b (e.g., a Universal Serial Bus (USB) Memory Stick) and 1200c (e.g., cloud storage for downloading instructions and / or parameters 1202 or receiving email instructions and / or parameters 1202), which, when executed by a processor, allow the processor to perform one or more operations of the methods described herein.

[0444] Figure 13 This is an example method 1300 for implementing the embodiments described herein, according to embodiments. At 1302, method 1300 includes: encoding suffix enhancement information units to generate an encoded picture, the suffix enhancement information units including a facial video enhancement information message in a picture unit, the picture unit including: a base or driving picture referenced by the facial video enhancement information message. At 1304, method 1300 includes: signaling the encoded picture.

[0445] Method 1300 can be performed using the apparatus described herein, for example, Figures 1 to 4 Any device Figure 11 Any device thereof, or any other device described herein.

[0446] Figure 14This is another example method 1400 for implementing the embodiments described herein, according to embodiments. At 1402, method 1400 includes: receiving a first input image. At 1404, method 1400 includes: encoding the first input image into a first encoded image, wherein the first encoded image includes an encoded base image or an encoded driving image, and wherein a first image unit includes the first encoded image. At 1406, method 1400 includes: receiving a second input image. At 1408, method 1400 includes: extracting feature parameters from the second input image. At 1410, method 1400 includes: encoding the feature parameters into encoded feature parameters. At 1412, method 1400 includes: including the encoded feature parameters in a first image unit or a subsequent image unit, wherein a subsequent image unit includes a subsequent encoded image, wherein the subsequent encoded image includes a second encoded driving image or a second encoded base image.

[0447] Method 1400 can be performed using the apparatus described herein, for example, Figures 1 to 4 Any device Figure 11 Any device thereof, or any other device described herein.

[0448] Figure 15 This is yet another example method 1500 for implementing the embodiments described herein, according to embodiments. At 1502, method 1500 includes: receiving a first encoded image from a first image unit, wherein the first encoded image includes an encoded base image or an encoded driving image. At 1504, method 1500 includes: decoding the first encoded image to generate a first decoded image. At 1506, method 1500 includes: receiving encoded feature parameters from the first image unit or from a subsequent image unit. At 1508, method 1500 includes: creating a feature input tensor based on the encoded feature parameters. At 1510, method 1500 includes: invoking generative neural network inference with the first decoded image and the feature input tensor as input to generate an image.

[0449] Method 1500 can be performed using the apparatus described herein, for example, Figures 1 to 4 Any device Figure 11 Any device thereof, or any other device described herein.

[0450] Figure 16This is yet another example method 1600 for implementing the embodiments described herein, according to embodiments. At 1602, method 1600 includes: receiving a first input image. At 1604, method 1600 includes: encoding the first input image into a first encoded image, wherein the first encoded image is an encoded base image, and wherein the first encoded image resides in a separate layer, and wherein a first access unit includes the first encoded image. At 1606, method 1600 includes: receiving a second input image. At 1608, method 1600 includes: encoding a second encoded image, wherein the second encoded image includes an encoded virtual image or an encoded driving image based on the second input image, and wherein the second encoded image resides in a dependency layer, and wherein a second image unit includes the second encoded image, and wherein a second access unit includes the second encoded image. At 1610, method 1600 includes: extracting feature parameters from the second input image. At 1612, method 1600 includes: encoding the feature parameters into encoded feature parameters. At 1614, method 1600 includes: incorporating encoded feature parameters into the second picture unit.

[0451] Method 1600 can be performed using the apparatus described herein, for example, Figures 1 to 4 Any device Figure 11 Any device thereof, or any other device described herein.

[0452] Figure 17This is another example method 1700 for implementing the embodiments described herein, according to embodiments. At 1702, method 1700 includes: receiving a first input image. At 1704, method 1700 includes: encoding the first input image into a first encoded base layer image, wherein the first encoded base layer image is an encoded base image, and wherein the first encoded base layer image resides in an independent layer, and wherein a first access unit includes the first encoded base layer image. At 1706, method 1700 includes: encoding a first encoded dependency layer image, wherein the first encoded dependency layer image resides in a dependency layer, and wherein a first access unit includes the first encoded dependency layer image. At 1708, method 1700 includes: receiving a second input image. At 1710, method 1700 includes: encoding a second encoded base layer image, wherein the second encoded base layer image includes an encoded virtual image or an encoded driving image based on the second input image, and wherein the second encoded base layer image resides in an independent layer, and wherein a second access unit includes the second encoded base layer image. At 1712, method 1700 includes: encoding a second encoded dependency layer image, wherein the second encoded dependency layer image resides in a dependency layer, and the second access unit includes the second encoded dependency layer image. At 1714, method 1700 includes: instructing the second encoded dependency layer image, or along the second encoded dependency layer image, to use the first encoded dependency layer image and the second encoded base layer image as references for prediction. At 1716, method 1700 includes: extracting feature parameters from a second input image. At 1718, method 1700 includes: encoding the feature parameters into encoded feature parameters. At 1720, method 1700 includes: instructing that the encoded feature parameters will be used for decoding the second encoded dependency layer image.

[0453] Method 1700 can be performed using the apparatus described herein, for example, Figures 1 to 4 Any device Figure 11 Any device thereof, or any other device described herein.

[0454] Figure 18 This is yet another example method 1800 for implementing the embodiments described herein, according to an embodiment. At 1802, method 1800 includes: reconstructing a first decoded dependency layer image from a first encoded dependency layer image. At 1804, method 1800 includes: reconstructing a second decoded dependency layer image by invoking generative neural network inference to generate an image using the first decoded dependency layer image and a feature input tensor as input.

[0455] Method 1800 can be performed using the apparatus described herein, for example, Figures 1 to 4 Any device Figure 11Any device thereof, or any other device described herein.

[0456] Figure 19 This is yet another example method 1900 for implementing the embodiments described herein, according to embodiments. At 1902, method 1900 includes: reconstructing a first decoded dependency layer image from a first encoded dependency layer image. At 1904, method 1900 includes: reconstructing one or more prediction units of a second decoded dependency layer image by invoking generative neural network inference with the first decoded dependency layer image and a feature input tensor as input. At 1906, method 1900 includes: encoding the difference between the one or more prediction units and co-located units of the second input image as a prediction residual of the second encoded dependency layer image.

[0457] Method 1900 can be performed using the apparatus described herein, for example, Figures 1 to 4 Any device Figure 11 Any device thereof, or any other device described herein.

[0458] Figure 20 This is yet another example method 2000 for implementing the embodiments described herein, according to embodiments. At 2002, method 2000 includes: reconstructing a first decoded dependency layer image from a first encoded dependency layer image. At 2004, method 2000 includes: invoking generative neural network inference with the first decoded dependency layer image and a feature input tensor as input to generate an inter-layer reference image. At 2006, method 2000 includes: encoding a second encoded dependency layer image, referencing the inter-layer reference image.

[0459] Method 2000 can be performed using the apparatus described herein, for example, Figures 1 to 4 Any device Figure 11 Any device thereof, or any other device described herein.

[0460] Figure 21This is yet another example method 2100 for implementing the embodiments described herein, according to embodiments. At 2102, method 2100 includes: receiving a first encoded base layer image, wherein the first encoded base layer image is an encoded base image residing in an independent layer, and a first access unit includes the first encoded base layer image. At 2104, method 2100 includes: decoding the first encoded base layer image into a first decoded base layer image. At 2106, method 2100 includes: receiving a first encoded dependency layer image, wherein the first encoded dependency layer image resides in a dependency layer, and a first access unit includes the first encoded dependency layer image. At 2108, method 2100 includes: decoding the first encoded dependency layer image into a first decoded dependency layer image. At 2110, method 2100 includes: receiving a second encoded base layer image, wherein the second encoded base layer image may be an encoded virtual image or an encoded driving image based on a second input image, the second encoded base layer image residing in an independent layer, and a second access unit including the second encoded base layer image. At 2112, method 2100 includes: decoding the second encoded base layer image into a second decoded base layer image in response to the second encoded base layer image being an encoded driving image. At 2114, method 2100 includes: receiving a second encoded dependency layer image, wherein the second encoded dependency layer image resides in a dependency layer, and a second access unit including the second encoded dependency layer image. At 2116, method 2100 includes: decoding from or along the second encoded dependency layer image that the second encoded dependency layer image can use the first encoded dependency layer image and the second encoded base layer image as references for prediction. At 2118, method 2100 includes: obtaining a feature input tensor for decoding the second encoded dependency layer image. At 2120, method 2100 includes: reconstructing the second decoded dependency layer image by invoking generative neural network inference to generate an image with the first decoded dependency layer image and the feature input tensor as input.

[0461] Method 2100 can be performed using the apparatus described herein, for example, Figures 1 to 4 Any device Figure 11 Any device thereof, or any other device described herein.

[0462] Figure 22This is yet another example method 2200 for implementing the embodiments described herein, according to embodiments. At 2202, method 2200 includes: receiving a first input image. At 2204, method 2200 includes: encoding the first input image into a first encoded image, wherein the first encoded image is an encoded base image or an encoded driving image, and wherein the first encoded image resides in a separate layer, and wherein a first access unit includes the first encoded image. At 2206, method 2200 includes: receiving a second input image. At 2208, method 2200 includes: extracting feature parameters from the second input image. At 2210, method 2200 includes: encoding the feature parameters into encoded feature parameters, encoding a second encoded image, wherein the second encoded image resides in a dependency layer, and wherein a second access unit includes the second encoded image. At 2212, method 2200 includes: instructing the encoded feature parameters to be used for decoding the second encoded dependency layer image.

[0463] Method 2200 can be performed using the apparatus described herein, for example, Figures 1 to 4 Any device Figure 11 Any device thereof, or any other device described herein.

[0464] As mentioned above, Figures 13 to 22Flowcharts include apparatus (e.g., 50, 1100, or any other apparatus described herein), methods, and computer program products according to certain example embodiments. It should be understood that each block of the flowchart, and combinations of blocks in the flowchart, can be implemented in various ways, such as hardware, firmware, processors, circuit systems, and / or other devices associated with the execution of software including one or more computer program instructions. For example, one or more processes described above can be embodied by computer program instructions. In this regard, computer program instructions embodying the processes described above can be stored in the memory (e.g., 58 or 1104) of an apparatus employing embodiments of the invention and executed by the processing circuitry of that apparatus (e.g., 56 or 1102). It should be understood that any such computer program instructions can be loaded onto a computer or other programmable device (e.g., hardware) to produce a machine that causes the resulting computer or other programmable device to perform the functions specified in the flowchart blocks. These computer program instructions can also be stored in a computer-readable storage medium that can instruct a computer or other programmable device to operate in a particular manner, such that the instructions stored in the computer programmable memory produce an article of art whose execution performs the functions specified in the flowchart blocks. Computer program instructions may also be loaded onto a computer or other programmable device to cause a series of operations to be performed on the computer or other programmable device, thereby producing a computer-implemented process, such that the instructions, which execute in the computer or other programmable device, provide operations for implementing the function specified in the flowchart box.

[0465] Therefore, when computer program instructions (such as computer-readable program code portions) are stored by at least one non-transitory computer-readable storage medium, a computer program product is defined, wherein the computer program instructions (such as computer-readable program code portions) are configured, upon execution, to perform the aforementioned functions, such as combining... Figures 13 to 22 The flowchart(s) are as follows. In other embodiments, computer program instructions (such as computer-readable program code portions) do not need to be stored or otherwise embodied by a non-transitory computer-readable storage medium, but may be embodied by a transient medium, wherein the computer program instructions (such as computer-readable program code portions) are still configured to perform the above-described functions when executed.

[0466] Therefore, the boxes in a flowchart support combinations of components used to perform a specified function and combinations of operations used to perform a specific function. It should also be understood that one or more boxes in a flowchart, as well as combinations of boxes in a flowchart, can be implemented by a dedicated hardware-based computer system or a combination of dedicated hardware and computer instructions that performs the specified function.

[0467] In some embodiments, certain operations described above may be modified or further extended. Furthermore, in some embodiments, additional optional operations may be included. Modifications, additions, or extensions to the operations described above may be made in any order and in any combination.

[0468] Some embodiments have already been described regarding the basic and driving images. It should be understood that embodiments can similarly be implemented using only the base image(s).

[0469] Some embodiments have been described with respect to a single base image. It should be understood that embodiments can be similarly implemented using multiple base images.

[0470] Some embodiments have been described with respect to a single driver image. It should be understood that embodiments can be similarly implemented using multiple driver images.

[0471] Some embodiments have been described with respect to one or more generative neural networks. It should be understood that the embodiments can also be implemented with any neural network, such as a convolutional neural network. For example, in some embodiments, the inference of a convolutional neural network can be viewed as an inter-layer process for generating inter-layer reference images.

[0472] Several embodiments have been described with respect to specific SEI messages, such as the GFV SEI message and / or the NNPFA SEI message. It should be understood that the embodiments are not limited to these specific SEI messages, but can be implemented with any similar SEI message.

[0473] Some embodiments of SEI messages have already been described. It should be understood that the embodiments are not limited to SEI messages, but can be implemented using any similar syntax structure, such as metadata OBU.

[0474] Some embodiments have been described regarding certain terms (such as picture units) applicable to some video coding formats (such as VVC). It should be understood that the embodiments are not limited to the video coding formats to which the terms apply, but can be implemented with any video coding format having its respective terms. Some examples of corresponding terms in different video coding formats have been described above. For example, an access unit in one video coding format may be referred to as a time unit in another video coding format. In another example, a picture unit may be considered to include encoded frames and associated metadata.

[0475] In the preceding text, some example embodiments have been described using the syntax of bitstreams. However, it should be understood that corresponding structures and / or computer programs may reside at the encoder used to generate the bitstream and / or the decoder used to decode the bitstream.

[0476] In the foregoing description of an example embodiment with reference to the encoder, it should be understood that the resulting bitstream and decoder have corresponding elements. Similarly, in the description of an example embodiment with reference to the decoder, it should be understood that the encoder has a structure and / or computer program for generating the bitstream to be decoded by the decoder.

[0477] Benefiting from the teachings given in the foregoing description and the accompanying drawings, those skilled in the art will be able to conceive of many modifications and other embodiments of the invention described herein. Therefore, it should be understood that the invention is not limited to the specific embodiments disclosed, and that modifications and other embodiments are intended to be included within the scope of the appended claims. Furthermore, although the foregoing description and accompanying drawings describe exemplary embodiments in the context of certain example combinations of elements and / or functions, it should be understood that alternative embodiments may provide different combinations of elements and / or functions without departing from the scope of the appended claims. In this regard, for example, as described in some of the appended claims, combinations of elements and / or functions different from those explicitly described above may also be contemplated. Therefore, this specification is intended to cover all such alternatives, modifications, and variations that fall within the scope of the appended claims. Although specific terms are used herein, they are used only in a general and descriptive sense and not for limiting purposes.

[0478] It should be understood that the above description is illustrative only. Those skilled in the art can devise various alternatives and modifications. For example, the features recited in the various dependent claims can be combined with each other in any suitable combination(s). Furthermore, features of the different embodiments described above can be selectively combined to form new embodiments. Therefore, this specification is intended to cover all such alternatives, modifications, and variations falling within the scope of the appended claims.

[0479] References to 'computer', 'processor', etc., should be understood to include not only computers with different architectures, such as single / multiprocessor architectures and sequential (von Neumann) / parallel architectures, but also special-purpose circuits, such as field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), signal processing devices, and other processing circuit systems. References to computer programs, instructions, code, etc., should be understood to include the software or firmware of programmable processors, such as the programmable content of hardware devices, such as processor instructions, or the configuration settings of fixed-function devices, gate arrays, or programmable logic devices.

[0480] As used herein, the term 'circuit system' can refer to any of the following: (a) a hardware circuit implementation, such as an implementation in an analog and / or digital circuit system; and (b) a combination of circuitry and software (and / or firmware), such as (if applicable): (i) a combination of (multiple) processors, or (ii) portions of (multiple) processors / software (including (multiple) digital signal processors), software, and (multiple) memories, which work together to enable a device to perform various functions; and (c) circuitry that requires software or firmware to operate (even if the software or firmware does not physically exist), such as (multiple) microprocessors or portions of (multiple) microprocessors. This description of 'circuit system' applies to the use of the term in this application. As another example, as used herein, the term 'circuit system' also covers an implementation of only one processor (or multiple processors) or a portion of a processor and its accompanying software and / or firmware. For example, if applicable to a particular element, the term 'circuit system' also covers a baseband integrated circuit or application processor integrated circuit for a mobile phone, or a similar integrated circuit in a server, cellular network device, or another network device.

[0481] Circuit system or circuit: As used in this application, the terms "circuit system" or "circuit" may refer to one or more or all of the following: (a) Hardware circuit implementation only (such as implementation only in analog and / or digital circuit systems); and (b) A combination of hardware circuitry and software, such as (if applicable): (i) A combination of (multiple) analog and / or digital hardware circuits with software / firmware; and (ii) Any part of a hardware processor (including digital signal processors), software, and memory (including multiple memory) that works together to enable a device such as a mobile phone or server to perform various functions; and (c) (Multiple) hardware circuits and / or (multiple) processors, such as (multiple) microprocessors or a portion thereof, which require software (e.g., firmware) to function, but may be absent when operation does not require the software to function.

[0482] This definition of circuit system applies to all uses of the term in this application, including in any claim. As another example, as used in this application, the term circuit system also covers only hardware circuitry or a processor (or multiple processors) or a portion of hardware circuitry or a processor and its accompanying software and / or firmware. For example, if applicable to a particular claim element, the term circuit also covers baseband integrated circuits or processor integrated circuits for mobile devices, or similar integrated circuits in servers, cellular network devices, or other computing or network devices.

Claims

1. An apparatus comprising at least one processor; and at least one non-transitory memory storing instructions, said instructions, when executed by said at least one processor, causing the apparatus to perform at least: Encoding suffix enhancement information units to generate encoded images, the suffix enhancement information units including facial video enhancement information messages in image units, the image units including base or driving images referenced by the facial video enhancement information messages; and The encoded image is signaled.

2. The apparatus of claim 1, wherein the facial video enhancement information message includes a generative video supplementation enhancement information message.

3. The apparatus according to any one of claims 1 or 2, wherein the suffix enhancement information unit comprises a suffix supplementary enhancement information network abstraction layer unit.

4. An apparatus comprising at least one processor; and at least one non-transitory memory storing instructions, said instructions, when executed by said at least one processor, causing the apparatus to perform at least: Receive the first input image; The first input image is encoded into a first encoded image, wherein the first encoded image includes an encoded base image or an encoded driving image, and wherein the first image unit includes the first encoded image; Receive the second input image; Extract feature parameters from the second input image; The feature parameters are encoded into encoded feature parameters; as well as The encoded feature parameters are included in the first image unit or a subsequent image unit, wherein the subsequent image unit includes a subsequent encoded image, and the subsequent encoded image includes a second encoded driving image or a second encoded base image.

5. The apparatus of claim 4, wherein the apparatus is further configured to: include the encoded feature parameters in a Generative Video (GV) Supplemental Enhancement Information (SEI) message, the GV SEI message being in the decoding order after the first encoded image.

6. The apparatus of claim 5, wherein the apparatus is further configured to: include the generative video SEI message in a suffix SEI Network Abstraction Layer (NAL) unit within the first image unit.

7. The apparatus according to any one of claims 4 to 6, wherein when the encoded feature parameters are decoded, the encoded feature parameters invoke generative neural network inference.

8. The apparatus according to any one of claims 4 to 7, wherein the apparatus is further configured to perform: including an activation signal to invoke the generative neural network inference along the encoded feature parameters.

9. The apparatus according to any one of claims 4 to 8, wherein the apparatus is further configured to: include an identifier or a counter value in the generative video SEI message, wherein the identifier or the counter value identifies the generative video SEI message from other generative video SEI messages in the first picture unit or the subsequent picture unit.

10. The apparatus of claim 4, wherein the apparatus is further configured to perform: generating a plurality of generative face video (GFV) supplemental enhancement information (SEI) messages with different payloads in the first picture unit or the subsequent picture unit, wherein each instance of the GFV SEI message invokes generative neural network (NN) inference.

11. The apparatus of claim 10, wherein the apparatus is configured to perform: using a suffix SEI NAL unit comprising a GFV SEI message in the first image unit, the first image unit comprising a base image or driving image referenced by the GFV SEI message.

12. The apparatus of claim 11, wherein the apparatus is configured to perform: adding a counter syntax element to the GFV SEI message.

13. An apparatus comprising at least one processor; and at least one non-transitory memory storing instructions, said instructions, when executed by said at least one processor, causing the apparatus to perform at least: Receive a first encoded image from a first image unit, wherein the first encoded image includes an encoded base image or an encoded driving image; The first encoded image is decoded to generate a first decoded image; Receive encoded feature parameters from the first image unit or from a subsequent image unit; Create a feature input tensor based on the encoded feature parameters; as well as The generative neural network is invoked to generate an image by using the first decoded image and the feature input tensor as input.

14. The apparatus of claim 13, wherein the apparatus is configured to: accept less than one image unit via an interface for transmission for decoding, and wherein the decoding of the first encoded image is invoked before receiving or creating the feature input tensor.

15. The apparatus of claim 13, wherein the apparatus is configured to: accept less than one image unit via an interface for transmission for decoding, and wherein the creation of the feature input tensor is performed before receiving or decoding a subsequent encoded image from the subsequent image unit.

16. The apparatus according to any one of claims 13 to 15, wherein the apparatus is further caused to perform: Decode the encoded feature parameters; and Based on decoding the encoded feature parameters, the generative neural network is invoked for inference.

17. The apparatus according to any one of claims 13 to 16, wherein the apparatus is further configured to: decode an identifier or counter value from the generative video SEI message, wherein the identifier or counter value identifies the generative video SEI message from other generative video SEI messages in the first picture unit or the subsequent picture unit.

18. The apparatus of claim 17, wherein when the identifier or the counter value is the same as the identifier or counter value of a previous generative video SEI message in the first picture unit or the subsequent picture unit, the current generative video SEI message is inferred to be a copy of the previous generative video SEI message.

19. The apparatus of claim 13, wherein the apparatus is further configured to perform: decoding a plurality of generative face video (GFV) supplemental enhancement information (SEI) messages with different payloads from the first picture unit or the subsequent picture unit, wherein each instance of the GFV SEI message invokes generative neural network (NN) inference.

20. The apparatus of claim 19, wherein the apparatus is configured to: decode a suffix SEI NAL unit, the suffix SEI NAL unit comprising a GFV SEI message from the first picture unit, the first picture unit comprising a base picture or a driving picture referenced by the GFV SEI message.

21. The apparatus of claim 20, wherein the apparatus is configured to: decode a counter syntax element from the GFV SEI message.

22. An apparatus comprising at least one processor; and at least one non-transitory memory storing instructions, said instructions, when executed by said at least one processor, causing the apparatus to perform at least: Receive the first input image; The first input image is encoded into a first encoded image, wherein the first encoded image is an encoded base image, and wherein the first encoded image resides in an independent layer, and wherein the first access unit includes the first encoded image; Receive the second input image; Encode a second encoded image, wherein the second encoded image includes an encoded virtual image or an encoded driving image based on the second input image, and wherein the second encoded image resides in a dependency layer, and wherein the second image unit includes the second encoded image, and wherein the second access unit includes the second encoded image; Extract feature parameters from the second input image; The feature parameters are encoded into encoded feature parameters; as well as The encoded feature parameters are included in the second image unit.

23. The apparatus of claim 22, wherein the apparatus is further configured to: encode an encoded copy image in the first access unit, wherein the encoded copy image resides in the dependency layer and is predicted according to the independent layer, and wherein the encoded copy image does not include prediction residuals.

24. The apparatus of claim 22 or 23, wherein the apparatus is further configured to: include the encoded feature parameters in a generative video supplementation enhancement information (SEI) message in the second picture unit.

25. The apparatus according to any one of claims 22 to 24, wherein the apparatus is further configured to perform: predicting the encoded virtual image based on previous images in decoding order within the dependency layer.

26. An apparatus comprising at least one processor; and at least one non-transitory memory storing instructions, said instructions, when executed by said at least one processor, causing the apparatus to perform at least: Receive the first input image; The first input image is encoded into a first encoded base layer image, wherein the first encoded base layer image is an encoded base image, wherein the first encoded base layer image resides in an independent layer, and wherein the first access unit includes the first encoded base layer image. Encode a first encoded dependency layer image, wherein the first encoded dependency layer image resides in a dependency layer, and wherein the first access unit includes the first encoded dependency layer image; Receive the second input image; Encode a second encoded base layer image, wherein the second encoded base layer image includes an encoded virtual image or an encoded driving image based on the second input image, and wherein the second encoded base layer image resides in the independent layer, and wherein the second access unit includes the second encoded base layer image; Encode a second encoded dependency layer image, wherein the second encoded dependency layer image resides in the dependency layer, and the second access unit includes the second encoded dependency layer image; In or along the second encoded dependency layer image, the second encoded dependency layer image is instructed to use the first encoded dependency layer image and the second encoded base layer image as references for prediction; Extract feature parameters from the second input image; The feature parameters are encoded into encoded feature parameters; as well as The encoded feature parameters are indicated to be used for decoding the second encoded dependency layer image.

27. The apparatus of claim 26, wherein the apparatus is further configured to: encode the first dependent layer image into an encoded copy image, wherein the encoded copy image is predicted according to the independent layer, and wherein the encoded copy image has no prediction residual.

28. The apparatus of any one of claims 26 or 27, wherein the apparatus is further configured to: encode an indication in or along the second encoded base layer image to omit the output of a reconstructed image decoded from the second encoded base layer image.

29. The apparatus according to any one of claims 26 or 27, wherein the apparatus is further caused to perform: The instruction is that the first reconstructed base layer image, decoded from the first encoded base layer image, is output; The instruction is that the first reconstructed dependency layer image, decoded from the first encoded dependency layer image, is output; The instruction is that the second reconstructed dependency layer image, decoded from the second encoded dependency layer image, is output; The independent layer and the dependent layer form an output layer set, and the dependent layer is the output layer of the output layer set; or A profile is indicated to the output layer set, wherein the profile includes generative neural network capabilities.

30. The apparatus according to any one of claims 26 to 28, wherein the apparatus is further configured to perform: creating a feature input tensor based on the encoded feature parameters.

31. The apparatus according to any one of claims 26 to 30, wherein the apparatus is further caused to perform: Include the encoded feature parameters or the feature input tensor in an adaptive parameter set that indicates the type of generative video; or The encoded feature parameters or the feature input tensor are included in the image header of the second encoded dependency layer image.

32. An apparatus comprising at least one processor; and at least one non-transitory memory storing instructions, said instructions, when executed by said at least one processor, causing the apparatus to perform at least: Reconstruct the first decoded dependency layer image from the first encoded dependency layer image; and The second decoded dependency layer image is reconstructed by invoking generative neural network inference to generate an image using the first decoded dependency layer image and feature input tensor as input.

33. The apparatus of claim 32, wherein the grammatical exclusion of the second encoded dependency layer image is irrelevant to the grammatical elements of the second decoded dependency layer image reconstructed using the generative neural network.

34. The apparatus of claim 32, wherein the grammatical exclusion of one or more grammatical structures of the second dependency layer is irrelevant to the reconstructing of the second decoded dependency layer image using the generative neural network.

35. An apparatus comprising at least one processor; and at least one non-transitory memory storing instructions, said instructions, when executed by said at least one processor, causing the apparatus to perform at least: Reconstruct the first decoded dependency layer image from the first encoded dependency layer image; By invoking generative neural network inference with the first decoded dependency layer image and feature input tensor as input, one or more prediction units of the second decoded dependency layer image are reconstructed; and The difference between the one or more prediction units and the co-located units of the second input image is encoded as the prediction residual of the second encoded dependency layer image.

36. An apparatus comprising at least one processor; and at least one non-transitory memory storing instructions, said instructions, when executed by said at least one processor, causing the apparatus to perform at least: Reconstruct the first decoded dependency layer image from the first encoded dependency layer image; Using the first decoded dependency layer image and the feature input tensor as input, a generative neural network inference is invoked to generate inter-layer reference images; and Referring to the interlayer reference image, the second encoded dependent layer image is encoded.

37. The apparatus of claim 36, wherein the apparatus is further configured to: encode the difference between the second input image and the interlayer reference image as a prediction residual of the second encoded dependent layer image.

38. An apparatus comprising at least one processor; and at least one non-transitory memory storing instructions, said instructions, when executed by said at least one processor, causing the apparatus to perform at least: Receive a first encoded base layer image, wherein the first encoded base layer image is an encoded base image, the first encoded base layer image resides in an independent layer, and the first access unit includes the first encoded base layer image; Decode the first encoded base layer image into a first decoded base layer image; Receive a first encoded dependency layer image, wherein the first encoded dependency layer image resides in a dependency layer, and the first access unit includes the first encoded dependency layer image; Decode the first encoded dependency layer image into a first decoded dependency layer image; Receive a second encoded base layer image, wherein the second encoded base layer image may be an encoded virtual image or an encoded driving image based on the second input image, the second encoded base layer image resides in the independent layer, and the second access unit includes the second encoded base layer image; In response to the fact that the second encoded base layer image is the encoded driving image, the second encoded base layer image is decoded into a second decoded base layer image; Receive a second encoded dependency layer image, wherein the second encoded dependency layer image resides in the dependency layer, and the second access unit includes the second encoded dependency layer image; Decoded from or along the second encoded dependency layer image: The second encoded dependency layer image can use the first encoded dependency layer image and the second encoded base layer image as references for prediction; Obtain the feature input tensor for the decoding of the second encoded dependency layer image; as well as The second decoded dependency layer image is reconstructed by invoking generative neural network inference to generate an image using the first decoded dependency layer image and the feature input tensor as input.

39. An apparatus comprising at least one processor; and at least one non-transitory memory storing instructions, said instructions, when executed by said at least one processor, causing the apparatus to perform at least: Receive the first input image; The first input image is encoded into a first encoded image, wherein the first encoded image is an encoded base image or an encoded driving image, and wherein the first encoded image resides in an independent layer, and wherein the first access unit includes the first encoded image; Receive the second input image; Extract feature parameters from the second input image; The feature parameters are encoded into encoded feature parameters; Encode a second encoded image, wherein the second encoded image resides in a dependency layer, and wherein the second access unit includes the second encoded image; and The encoded feature parameters are used to decode the second encoded dependency layer image.

40. The apparatus of claim 39, wherein the apparatus is further caused to perform: Reconstruct the first decoded image from the first encoded image; Create a feature input tensor based on the encoded feature parameters; Using the first decoded image and the feature input tensor as input, a generative neural network inference is invoked to generate a first inter-layer reference image; and The second encoded image is encoded by referring to the first interlayer reference image.

41. The apparatus of claim 39 or 40, wherein the apparatus is further configured to perform: omit encoding of the prediction residual for the second encoded image.

42. The apparatus according to any one of claims 39 to 41, wherein the apparatus is further configured to: encode the difference between the second input image and the first interlayer reference image as a prediction residual of the second encoded image.

43. The apparatus according to any one of claims 39 to 41, wherein the apparatus is further caused to perform: In response to the first encoded image being the encoded base image, the first encoded image is indicated to be the output; In response to the first encoded image being the encoded driving image, it is indicated that the first encoded image is not the output; The independent layer and the dependent layer are instructed to form an output layer set, and the dependent layer is the output layer of the output layer set; or This indicates that the second encoded image is output.

44. The apparatus according to any one of claims 39 to 43, wherein the apparatus is further caused to perform: The first encoded image is indicated in or along the first encoded image, the indication indicating that: the first encoded image is the encoded base image or the first encoded image is the encoded driving image; or The extra bits in the image header indicate whether the first encoded image is the encoded base image or the first encoded image is the encoded driving image.

45. The apparatus according to any one of claims 39 to 44, wherein the apparatus is further configured to: instruct the second encoded image to use interlayer prediction from the independent layer.

46. ​​The apparatus according to any one of claims 39 to 44, wherein the apparatus is further configured to perform: inferring the interlayer prediction reference from the independent layer to the first interlayer reference image.

47. The apparatus according to any one of claims 39 to 44, wherein the apparatus is further caused to perform: Receive a third input image; and The third input image is encoded into a third encoded image, wherein the third encoded image is the encoded driving image, and wherein the third encoded image resides in the independent layer, and the third encoded image is after the first encoded image in the decoding order and before the second encoded image in the decoding order.

48. The apparatus of claim 47, wherein the apparatus is further configured to: omit indicating that the third encoded image is the encoded driving image in response to the third encoded image residing in the second access unit, and infer that the third encoded image is the encoded driving image.

49. The apparatus of claim 47, wherein the apparatus is further configured to: indicate in or along the third encoded image a reference, the reference indicating that the third encoded image is the encoded driving image.

50. The apparatus of claim 47, wherein the apparatus is further configured to: indicate, using additional bits in the image header, that the third encoded image is the encoded driving image.

51. The apparatus according to any one of claims 47 to 50, wherein the apparatus is further caused to perform: Reconstructing the third decoded image from the third encoded image; and The third decoded image is used as additional input in the invocation of the generative neural network inference.

52. The apparatus according to any one of claims 47 to 51, wherein the apparatus is further configured to perform: inferring when the interlayer prediction from the independent layer references the first interlayer reference image or the third decoded image.

53. The apparatus according to any one of claims 47 to 51, wherein the apparatus is further configured to: indicate syntax structure information in a list of reference images, the syntax structure information indicating that the first encoded image is used as input for generating the interlayer reference image.

54. The apparatus of claim 53, wherein the information indicating that the first encoded image is used as input for generating the interlayer reference image includes the image order count difference between the current image and the first encoded image.

55. A method comprising: Encoding suffix enhancement information units to generate encoded images, the suffix enhancement information units including facial video enhancement information messages in image units, the image units including base or driving images referenced by the facial video enhancement information messages; and The encoded image is signaled.

56. The method of claim 55, wherein the facial video enhancement information message includes a generative video supplementation enhancement information message.

57. The method according to any one of claims 55 or 56, wherein the suffix enhancement information unit comprises a suffix supplementary enhancement information network abstraction layer unit.

58. A method comprising: Receive the first input image; The first input image is encoded into a first encoded image, wherein the first encoded image includes an encoded base image or an encoded driving image, and wherein the first image unit includes the first encoded image; Receive the second input image; Extract feature parameters from the second input image; The feature parameters are encoded into encoded feature parameters; as well as The encoded feature parameters are included in the first image unit or a subsequent image unit, wherein the subsequent image unit includes a subsequent encoded image, and the subsequent encoded image includes a second encoded driving image or a second encoded base image.

59. The method of claim 58, further comprising: The encoded feature parameters are included in a Generative Video (GV) Supplemental Enhancement Information (SEI) message, which is placed after the first encoded image in the decoding order.

60. The method of claim 59, further comprising: The generative video SEI message is included in the suffix SEI Network Abstraction Layer (NAL) unit within the first image unit.

61. The method according to any one of claims 58 to 60, wherein when the encoded feature parameters are decoded, the encoded feature parameters invoke generative neural network inference.

62. The method according to any one of claims 58 to 61, further comprising: This includes activation signals to invoke the generative neural network inference along the encoded feature parameters.

63. The method according to any one of claims 58 to 62, further comprising: The generative video SEI message includes an identifier or a counter value, wherein the identifier or the counter value identifies the generative video SEI message from other generative video SEI messages in the first picture unit or the subsequent picture unit.

64. The method of claim 58, further comprising: In the first image unit or the subsequent image unit, a plurality of generative face video (GFV) supplemental enhancement information (SEI) messages with different payloads are generated, wherein each instance of the GFVSEI message invokes generative neural network (NN) inference.

65. The method of claim 64, further comprising: The first image unit uses a suffix SEI NAL unit that includes a GFV SEI message, and the first image unit includes a base image or driving image referenced by the GFV SEI message.

66. The method of claim 65, further comprising: Add a counter syntax element to the GFV SEI message.

67. A method comprising: Receive a first encoded image from a first image unit, wherein the first encoded image includes an encoded base image or an encoded driving image; The first encoded image is decoded to generate a first decoded image; Receive encoded feature parameters from the first image unit or from a subsequent image unit; Create a feature input tensor based on the encoded feature parameters; as well as The generative neural network is invoked to generate an image by using the first decoded image and the feature input tensor as input.

68. The method of claim 67, further comprising: The interface accepts fewer than one image unit to be passed for decoding, and the decoding of the first encoded image is invoked before the feature input tensor is received or created.

69. The method of claim 67, wherein less than one image unit is accepted via an interface to be passed for decoding, and wherein the creation of the feature input tensor is performed before receiving or decoding a subsequent encoded image from the subsequent image unit.

70. The method according to any one of claims 67 to 69, further comprising: Decode the encoded feature parameters; as well as Based on decoding the encoded feature parameters, the generative neural network is invoked for inference.

71. The method according to any one of claims 67 to 70, further comprising: Decode an identifier or counter value from the generative video SEI message, wherein the identifier or counter value identifies the generative video SEI message from other generative video SEI messages in the first picture unit or the subsequent picture unit.

72. The method of claim 71, wherein when the identifier or the counter value is the same as the identifier or counter value of a previous generative video SEI message in the first picture unit or the subsequent picture unit, the current generative video SEI message is inferred to be a copy of the previous generative video SEI message.

73. The method of claim 67, further comprising: Multiple generative face video (GFV) supplemental enhancement information (SEI) messages with different payloads from the first image unit or the subsequent image unit are decoded, wherein each instance of the GFV SEI message invokes generative neural network (NN) inference.

74. The method of claim 73, further comprising: The suffix SEI NAL unit is decoded, the suffix SEINAL unit includes the GFV SEI message from the first image unit, the first image unit includes the base image or driving image referenced by the GFV SEI message.

75. The method of claim 74, further comprising: Decode the counter syntax element from the GFV SEI message.

76. A method comprising: Receive the first input image; The first input image is encoded into a first encoded image, wherein the first encoded image is an encoded base image, and wherein the first encoded image resides in an independent layer, and wherein the first access unit includes the first encoded image; Receive the second input image; Encode a second encoded image, wherein the second encoded image includes an encoded virtual image or an encoded driving image based on the second input image, and wherein the second encoded image resides in a dependency layer, and wherein the second image unit includes the second encoded image, and wherein the second access unit includes the second encoded image; Extract feature parameters from the second input image; The feature parameters are encoded into encoded feature parameters; as well as The encoded feature parameters are included in the second image unit.

77. The method of claim 76, further comprising: In the first access unit, an encoded copy image is encoded, wherein the encoded copy image resides in the dependency layer and is predicted according to the independent layer, and wherein the encoded copy image does not include prediction residuals.

78. The method according to any one of claims 76 or 77, further comprising: The encoded feature parameters are included in the Generative Video Supplement Enhancement Information (SEI) message in the second picture unit.

79. The method according to any one of claims 76 to 78, further comprising: The encoded virtual image is predicted based on the previous images in the decoding order within the dependency layer.

80. A method comprising: Receive the first input image; The first input image is encoded into a first encoded base layer image, wherein the first encoded base layer image is an encoded base image, wherein the first encoded base layer image resides in an independent layer, and wherein the first access unit includes the first encoded base layer image. Encode a first encoded dependency layer image, wherein the first encoded dependency layer image resides in a dependency layer, and wherein the first access unit includes the first encoded dependency layer image; Receive the second input image; Encode a second encoded base layer image, wherein the second encoded base layer image includes an encoded virtual image or an encoded driving image based on the second input image, and wherein the second encoded base layer image resides in the independent layer, and wherein the second access unit includes the second encoded base layer image; Encode a second encoded dependency layer image, wherein the second encoded dependency layer image resides in the dependency layer, and the second access unit includes the second encoded dependency layer image; In or along the second encoded dependency layer image, the second encoded dependency layer image is instructed to use the first encoded dependency layer image and the second encoded base layer image as references for prediction; Extract feature parameters from the second input image; The feature parameters are encoded into encoded feature parameters; as well as The encoded feature parameters are indicated to be used for decoding the second encoded dependency layer image.

81. The method of claim 80, further comprising: The first dependent layer image is encoded into an encoded copy image, wherein the encoded copy image is predicted according to the independent layer, and wherein the encoded copy image has no prediction residual.

82. The method according to any one of claims 80 or 81, further comprising: The indication is encoded in or along the second encoded base layer image to omit the output of the reconstructed image decoded from the second encoded base layer image.

83. The method according to any one of claims 80 or 81, further comprising: The instruction is that the first reconstructed base layer image, decoded from the first encoded base layer image, is output; The instruction is that the first reconstructed dependency layer image, decoded from the first encoded dependency layer image, is output; The instruction is that the second reconstructed dependency layer image, decoded from the second encoded dependency layer image, is output; The independent layer and the dependent layer form an output layer set, and the dependent layer is the output layer of the output layer set; or A profile is indicated to the output layer set, wherein the profile includes generative neural network capabilities.

84. The method according to any one of claims 80 to 82, further comprising: A feature input tensor is created based on the encoded feature parameters.

85. The method according to any one of claims 80 to 84, further comprising: Include the encoded feature parameters or the feature input tensor in an adaptive parameter set that indicates the type of generative video; or The encoded feature parameters or the feature input tensor are included in the image header of the second encoded dependency layer image.

86. A method comprising: Reconstruct the first decoded dependency layer image from the first encoded dependency layer image; as well as The second decoded dependency layer image is reconstructed by invoking generative neural network inference to generate an image using the first decoded dependency layer image and feature input tensor as input.

87. The method of claim 86, wherein the grammatical exclusion of the second encoded dependency layer image is irrelevant to the grammatical elements of the second decoded dependency layer image reconstructed using the generative neural network.

88. The method of claim 86, wherein the grammatical exclusion of one or more grammatical structures of the second dependency layer is irrelevant to the reconstructing of the second decoded dependency layer image using the generative neural network.

89. A method comprising: Reconstruct the first decoded dependency layer image from the first encoded dependency layer image; By invoking generative neural network inference with the first decoded dependency layer image and feature input tensor as input, one or more prediction units of the second decoded dependency layer image are reconstructed. as well as The difference between the one or more prediction units and the co-located units of the second input image is encoded as the prediction residual of the second encoded dependency layer image.

90. A method comprising: Reconstruct the first decoded dependency layer image from the first encoded dependency layer image; The generative neural network inference is invoked using the first decoded dependency layer image and the feature input tensor as input to generate inter-layer reference images; as well as Referring to the interlayer reference image, the second encoded dependent layer image is encoded.

91. The method of claim 90, further comprising: The difference between the second input image and the interlayer reference image is encoded as the prediction residual of the second encoded dependent layer image.

92. A method comprising: Receive a first encoded base layer image, wherein the first encoded base layer image is an encoded base image, the first encoded base layer image resides in an independent layer, and the first access unit includes the first encoded base layer image; Decode the first encoded base layer image into a first decoded base layer image; Receive a first encoded dependency layer image, wherein the first encoded dependency layer image resides in a dependency layer, and the first access unit includes the first encoded dependency layer image; Decode the first encoded dependency layer image into a first decoded dependency layer image; Receive a second encoded base layer image, wherein the second encoded base layer image may be an encoded virtual image or an encoded driving image based on the second input image, the second encoded base layer image resides in the independent layer, and the second access unit includes the second encoded base layer image; In response to the fact that the second encoded base layer image is the encoded driving image, the second encoded base layer image is decoded into a second decoded base layer image; Receive a second encoded dependency layer image, wherein the second encoded dependency layer image resides in the dependency layer, and the second access unit includes the second encoded dependency layer image; Decoded from or along the second encoded dependency layer image: The second encoded dependency layer image can use the first encoded dependency layer image and the second encoded base layer image as references for prediction; Obtain the feature input tensor for the decoding of the second encoded dependency layer image; as well as The second decoded dependency layer image is reconstructed by invoking generative neural network inference to generate an image using the first decoded dependency layer image and the feature input tensor as input.

93. A method comprising: Receive the first input image; The first input image is encoded into a first encoded image, wherein the first encoded image is an encoded base image or an encoded driving image, and wherein the first encoded image resides in an independent layer, and wherein the first access unit includes the first encoded image; Receive the second input image; Extract feature parameters from the second input image; The feature parameters are encoded into encoded feature parameters; Encode a second encoded image, wherein the second encoded image resides in a dependency layer, and wherein the second access unit includes the second encoded image; and The encoded feature parameters are used to decode the second encoded dependency layer image.

94. The method according to claim 93, further comprising: Reconstruct the first decoded image from the first encoded image; Create a feature input tensor based on the encoded feature parameters; Using the first decoded image and the feature input tensor as input, a generative neural network inference is invoked to generate a first inter-layer reference image; as well as The second encoded image is encoded by referring to the first interlayer reference image.

95. The method according to any one of claims 93 or 94, further comprising: The encoding of the prediction residual for the second encoded image is omitted.

96. The method according to any one of claims 93 to 95, further comprising: The difference between the second input image and the first inter-layer reference image is encoded as the prediction residual of the second encoded image.

97. The method according to any one of claims 93 to 95, further comprising: In response to the first encoded image being the encoded base image, the first encoded image is indicated to be the output; In response to the first encoded image being the encoded driving image, it is indicated that the first encoded image is not the output; The independent layer and the dependent layer are instructed to form an output layer set, and the dependent layer is the output layer of the output layer set; or This indicates that the second encoded image is output.

98. The method according to any one of claims 93 to 97, further comprising: The first encoded image is indicated in or along the first encoded image, the indication indicating that: the first encoded image is the encoded base image or the first encoded image is the encoded driving image; or The extra bits in the image header indicate whether the first encoded image is the encoded base image or the first encoded image is the encoded driving image.

99. The method according to any one of claims 93 to 98, further comprising: The second encoded image is instructed to use inter-layer predictions from the independent layer.

100. The method according to any one of claims 93 to 98, further comprising: The interlayer prediction from the independent layer is inferred from the first interlayer reference image.

101. The method according to any one of claims 93 to 98, further comprising: Receive a third input image; as well as The third input image is encoded into a third encoded image, wherein the third encoded image is the encoded driving image, and wherein the third encoded image resides in the independent layer, and the third encoded image is after the first encoded image in the decoding order and before the second encoded image in the decoding order.

102. The method according to claim 101, further comprising: In response to the fact that the third encoded image resides in the second access unit, the indication that the third encoded image is the encoded driver image is omitted, and it is inferred that the third encoded image is the encoded driver image.

103. The method according to claim 101, further comprising: An indication is provided in or along the third encoded image, indicating that the third encoded image is the encoded driving image.

104. The method of claim 101, further comprising: The extra bits in the image header indicate that the third encoded image is the encoded driving image.

105. The method according to any one of claims 101 to 104, further comprising: Reconstruct the third decoded image from the third encoded image; as well as The third decoded image is used as additional input in the invocation of the generative neural network inference.

106. The method according to any one of claims 101 to 105, further comprising: Infer when the interlayer prediction from the independent layer references the first interlayer reference image or the third decoded image.

107. The method according to any one of claims 101 to 105, further comprising: The reference image list indicates syntax structure information, which indicates that the first encoded image is used as input to generate the interlayer reference image.

108. The method of claim 107, wherein the information indicating that the first encoded image is used as input for generating the interlayer reference image includes the image order count difference between the current image and the first encoded image.

109. An apparatus comprising components for performing the method according to any one of claims 55 to 108.

110. A computer-readable medium comprising program instructions that, when executed by a device, cause the device to perform the method according to any one of claims 55 to 108.

111. The computer-readable medium of claim 110, wherein the computer-readable medium includes a non-transitory computer-readable medium.

Images